Senior Software Engineer, Rust for LLM Evaluation

This role focuses on developing and evaluating large language model (LLM) training datasets to address real-world software engineering challenges. You will engage in creating verifiable software engineering tasks derived from public repository histories, utilizing a synthetic approach with human-in-the-loop methodologies. Your contributions will expand dataset coverage across various programming languages and difficulty levels.

Key Responsibilities:

• Analyze and triage GitHub issues across trending open-source libraries.
• Set up and configure code repositories, including Dockerization and environment setup.
• Evaluate unit test coverage and quality.
• Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
• Collaborate with researchers to design and identify repositories and issues that challenge LLMs.
• Lead a team of junior engineers on collaborative projects.

Qualifications:

• Minimum of 3 years of overall experience.
• Strong experience with Rust.
• Proficiency in Git, Docker, and basic software pipeline setup.
• Ability to understand and navigate complex codebases.
• Comfortable running, modifying, and testing real-world projects locally.
• Experience contributing to or evaluating open-source projects is a plus.

Preferred Qualifications:

• Previous participation in LLM research or evaluation projects.
• Experience building or testing developer tools or automation agents.

Work Terms:

• Commitment of at least 4 hours per day and a minimum of 20 hours per week, with 4 hours overlap with PST. Options for 20, 30, or 40 hours per week are available.
• Contractor assignment (no medical/paid leave).
• Contract duration of 3 months, with an expected start date next week.
• Eligible locations include India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and Mexico.

Compensation:

• Competitive compensation based on experience and project scope.

Eligibility:

• Open to candidates from specified countries.

Evaluation Process:

• Two rounds of interviews: 60 minutes technical and 30 minutes technical & cultural discussion.

Why Join Us? You will be part of a rapidly growing AI company, working at the forefront of evaluating LLM interactions with real code, and influencing the future of AI-assisted software development. This role offers a unique opportunity to blend practical software engineering with AI research in a fully remote environment, while working on cutting-edge AI projects with leading LLM companies.