About this role
Role Overview
Design and run practical, language-spanning evaluations and test suites that measure system behavior on real-world coding tasks, surface strengths and weaknesses, and deliver actionable findings to research and engineering teams. Work remotely as part of a distributed, asynchronous talent network contributing to high-impact open source projects.
Key Responsibilities- Design and oversee creation of evaluations for coding tasks across JavaScript, TypeScript, Python, Java, and C.
- Develop test cases that accurately assess system performance in diverse engineering scenarios.
- Analyze system behavior on real-world user use cases to identify strengths and areas for improvement.
- Communicate evaluation results clearly and promptly to the research team to inform development and optimization efforts.
- Demonstrated, recent open source activity, with a strong GitHub or equivalent presence and frequent, high-quality contributions to top projects within the last 12 months.
- Deep expertise in one or more of the following languages: Python, Java, C, JavaScript, TypeScript.
- Well versed in widely used libraries, frameworks, and tools relevant to your language or languages of focus.
- Strong understanding of software architecture, performance tuning, and scalable code patterns.
- Experience collaborating in distributed, asynchronous teams, and ability to work independently to identify and execute improvements with minimal oversight.
- Comfortable using Git, CI/CD systems, and participating in open source governance workflows.
- Location: Remote.
- Employment type: Hourly engagement.
- Rate: 100 to 150 per hour.
- Active open source contributor status is required, demonstrated by frequent, high-quality contributions in the past 12 months.
- Proven ability to assess and improve code across the listed languages, and to participate effectively in open source project workflows.