Senior Software Engineer, LLM Evaluation & Repository Validation

Role Overview

Help evaluate how large language models work with real-world codebases, shaping the future of AI-assisted software development. This remote contract role combines hands-on software engineering with AI evaluation work, creating verifiable software engineering tasks from public repository histories and expanding coverage across programming languages and difficulty levels.

Key Responsibilities

• Analyze and triage GitHub issues from widely used open-source libraries.
• Set up and configure repositories, including Dockerization and development-environment automation.
• Assess unit-test coverage and test quality.
• Modify and run codebases locally to evaluate LLM performance on bug-fixing scenarios.
• Partner with researchers to identify repositories and issues that present meaningful challenges for LLMs.
• Contribute to the development of LLM evaluation and training datasets based on realistic software engineering problems.
• Lead junior engineers on project work when opportunities arise.

Qualifications

• Tech lead-level software engineering experience and familiarity with well-maintained public GitHub repositories with 500 or more stars.
• Strong experience in at least one of the following: Python, JavaScript, Java, Go, Rust, C/C++, C#, or Ruby.
• Proficiency with Git, Docker, and basic software pipeline setup.
• Ability to understand and navigate complex codebases.
• Comfort running, modifying, and testing real-world projects locally.
• Open-source contribution or evaluation experience is a plus.
• Experience with LLM research or evaluation projects is preferred.
• Experience building or testing developer tools or automation agents is preferred.

Work Terms

• Fully remote contractor assignment.
• Commitment of 20 hours per week, with some overlap with Pacific Time required.
• Three-month contract, expected to begin next week.
• This contractor assignment does not include medical benefits or paid leave.

Opportunity

Work on advanced AI projects involving leading LLM companies while applying practical engineering judgment to realistic code and repository-validation challenges.