About this role
Role Overview
Validate browser-based evaluation workflows for AI-generated web applications to ensure tests are technically correct, deterministic, and able to produce clear pass or fail outcomes before inclusion in benchmark datasets. You will analyze automation logic, test isolation, fixtures, and execution flow, identify sources of flakiness or ambiguity, and deliver structured technical feedback that helps maintain evaluation quality.
Key Responsibilities- Review browser-based testing workflows for AI-generated web applications, focusing on technical soundness and determinism.
- Assess whether browser interactions and automation steps are feasible, reliable, and reproducible.
- Validate test isolation, fixtures, setup, teardown, and overall execution flow.
- Confirm assertions produce unambiguous pass or fail results and flag ambiguous validation logic.
- Identify flaky tests, hidden dependencies, timing issues, or other sources of nondeterminism.
- Provide structured, rubric-based technical feedback to project owners and reviewers.
- 3+ years of experience in Software Engineering, QA Engineering, SDET, or Test Automation.
- Strong experience testing modern web applications, including automated UI and end-to-end tests.
- Familiarity with browser automation frameworks such as Playwright, Cypress, Selenium, or similar tools.
- Deep understanding of test isolation, fixtures, reproducibility, and reliable browser automation.
- Excellent debugging skills, attention to detail, and ability to communicate technical findings clearly.
- Location: Remote
- Employment Type: hourly
30 - 60 hourly