About this role
Role Overview
Create challenging codebase exploration tasks that help train AI agents to investigate and reason about unfamiliar production Python repositories. These tasks require runtime evidence and cross repository investigation, reflecting the work a senior engineer performs to understand a real system.
Projects draw from widely deployed Python codebases including Django, Twisted, Scrapy, SymPy, mypy, pip, coverage.py, mitmproxy, Certbot, Tornado, Werkzeug, sqlfluff, and Meson.
Key Responsibilities
- Select assignments from a prevalidated pool of Python repositories pinned to specific Git commits. Each assignment includes an engineering question, positive and negative scoring criteria, foils, and a reference solution.
- Investigate repositories, relevant subsystems, and applicable pull request or issue history until the system behavior is understood.
- Rewrite questions so they remain well defined and fair while challenging advanced coding agents.
- Refine scoring criteria and foils to reward sound reasoning and reject credible but incorrect responses.
- Complete the full validation workflow, Check, Golden, Run, and Analyze, iterating until every stage passes.
- Document work in the task README, submit it for expert review, and address feedback on returned tasks.
Qualifications
- At least 3 years of professional software engineering experience, including substantial production Python work.
- Ability to navigate large, unfamiliar Python codebases and explain runtime behavior, not only source code.
- Working knowledge of Python internals used in real systems, including import machinery, descriptors and metaclasses, asynchronous programming and event loops, packaging, and C extension boundaries.
- Experience with web frameworks, developer tooling, type checkers, or scientific and numeric libraries is highly relevant. The task mix emphasizes architecture and system design, 41%, and code onboarding, 31%.
- Precise written English, since the questions and scoring criteria are core deliverables.
- Comfort working with Git at the commit level, containerized environments, and command line tools.
Work Terms
- Remote, asynchronous, per task contract work with expert team leads across US, India, and Nigeria time zones.
- Work is completed in a designated task workspace. Time is tracked through Insightful and access is provisioned through Okta.
- Before receiving task access, candidates must complete identity verification, a background check, required work and confidentiality agreements, a tax form, and a short calibration quiz.
- Each task is reviewed by a second engineer and a senior reviewer before acceptance.
Compensation
$130 for each approved task.
Application Process
- Submit a resume or relevant technical background.
- Qualified applicants may be asked to complete a brief technical assessment or provide additional information.