QA/Test Engineer

Help strengthen next-generation agentic evaluation benchmarks for frontier AI models by ensuring complex, multi-step tasks are unambiguous, accurately graded, and resistant to shortcuts. You will run tasks, investigate edge cases, debug environments, and work closely with researchers and task authors to ensure each benchmark measures what it is intended to measure.

• Design test cases that verify tasks function as intended, including challenging edge cases.
• Review tasks and reference solutions before finalization, identifying ambiguity, gaps, and grading issues early.
• Debug task environments and automated checks using Python.
• Create practical, repeatable quality checklists and provide actionable feedback to task authors.
• Identify shortcuts and grading gaps in AI agent runs to protect the reliability of benchmark results.
• Collaborate in a close feedback loop with researchers and task authors.

• Master''s degree or PhD in a STEM field, or equivalent practical experience in a research-intensive or engineering-intensive domain.
• At least 1 year of experience in test engineering, quality assurance, or a research or software engineering role with strong quality ownership.
• Experience designing test cases and quality-review processes and debugging complex systems end to end.
• Working proficiency with Python and Git, including comfort navigating unfamiliar codebases and environments.
• Exceptional attention to detail and strong written documentation skills.
• Experience with AI training, model evaluation, or quality review of AI-generated work is preferred.
• Creative, independent problem-solving ability for ambiguous and open-ended work.

• Remote role available only to candidates located in the United States.
• Full-time W-2 employment, with an expected commitment of approximately 35 hours per week.
• Work as part of an extended workforce supporting a leading AI lab''s internal team and enterprise workflows.
• The employer of record administers employment, onboarding, payroll, benefits, and compliance.

$60 to $90 per hour.

Equal employment opportunity is provided without regard to legally protected characteristics. Reasonable accommodations are available for qualified individuals with disabilities and disabled veterans throughout the application process.