About this role
Role Overview
Evaluate how large language models perform on real software engineering work by helping build verifiable training and evaluation tasks from public repository histories. This role combines hands-on repository analysis with AI research, expanding coverage across programming languages and task difficulty levels to improve AI-assisted software development.
Key Responsibilities
- Analyze and triage GitHub issues in trending open-source libraries.
- Set up and configure code repositories, including Dockerization and development environments.
- Assess unit test coverage and test quality.
- Modify and run codebases locally to evaluate LLM performance on bug-fixing scenarios.
- Partner with researchers to identify repositories and issues that present meaningful challenges for LLMs.
- Help create verifiable software engineering tasks based on public repository histories using a human-in-the-loop synthetic approach.
- Lead junior engineers collaborating on project work when opportunities arise.
Qualifications
- Tech lead level software engineering experience and familiarity with high-quality public GitHub repositories.
- Experience with well-maintained, widely used repositories with 500 or more stars.
- Strong experience in at least one of the following: Python, JavaScript, Java, Go, Rust, C/C++, C#, or Ruby.
- Proficiency with Git, Docker, and basic software pipeline setup.
- Ability to understand and navigate complex codebases.
- Comfort running, modifying, and testing real-world projects locally.
- Open-source contribution or evaluation experience is a plus.
- Previous LLM research or evaluation work is a plus.
- Experience building or testing developer tools or automation agents is a plus.
Work Terms
- Fully remote contractor assignment.
- Commitment of 20 hours per week, with some overlap with Pacific Time required.
- Three-month contract expected to begin next week.
- This contractor assignment does not include medical benefits or paid leave.