About this role
Role Overview
Design and deliver evaluations that measure coding systems across multiple languages, develop realistic test cases, analyze system behavior on real-world user scenarios, and report findings to research teams to drive product improvements. This role is centered on open source expertise and consistent, high-quality contributions to public repositories.
Key Responsibilities- Design and manage creation of evaluations for coding tasks in JavaScript, TypeScript, Python, Java, and C.
- Develop test cases that accurately assess system performance across diverse engineering scenarios.
- Analyze system behavior using real-world user use cases to identify strengths and areas for improvement.
- Communicate evaluation results and actionable insights to the research team to support development and optimization.
- Established open source contributor with a strong GitHub or similar presence and frequent, high-quality contributions to leading open-source projects within the last 12 months.
- Expertise in one or more of the following languages: Python, Java, C, JavaScript, TypeScript.
- Deep familiarity with commonly used libraries, frameworks, and tooling in your language or languages of choice.
- Solid understanding of software architecture, performance tuning, and scalable code patterns.
- Experience collaborating in distributed, asynchronous teams, and strong written and verbal communication skills.
- Ability to identify contribution opportunities independently and execute improvements with minimal oversight.
- Comfortable using Git, CI/CD systems, and participating in open-source governance workflows.
- Location: Remote
- Employment type: hourly
- Pay range: 100 - 150 hourly
Remote work is required. No specific work authorization or sponsorship details are provided in this listing; applicants should ensure they can lawfully work remotely from their location.