Skip to content
SaidGig
Sign up.

Give me your email, I promise I won't do anything weird with it.

Senior Software Engineer, Python, LLM Evaluation and Repository Validation

from $40/hour

Remote — Mexico, India or NigeriaNon-remote: unknownContracttechnology
Apply Now

About this role

Help evaluate how large language models perform on real-world software engineering work by building and validating challenging tasks from public repository histories. This hands-on contract role combines practical Python engineering, open-source analysis, and AI research to improve AI-assisted software development.

Role Overview

You will support the creation of LLM evaluation and training datasets based on verifiable software engineering tasks. The work expands coverage across programming languages, task difficulty levels, and other task types using public repository histories and a human-in-the-loop approach.

Key Responsibilities
  • Analyze and triage GitHub issues across trending open-source libraries.
  • Set up and configure code repositories, including Dockerization and development environments.
  • Assess unit test coverage and test quality.
  • Run, modify, and test codebases locally to evaluate LLM performance on bug-fixing scenarios.
  • Partner with researchers to identify repositories and issues that are challenging for LLMs.
  • Evaluate how LLMs interact with real code and contribute to the future of AI-assisted software development.
  • Potentially lead junior engineers collaborating on project work.
Qualifications
  • At least 3 years of overall software engineering experience.
  • Strong Python experience.
  • Proficiency with Git, Docker, and basic software pipeline setup.
  • Ability to understand and navigate complex codebases.
  • Comfort running, modifying, and testing real-world projects locally.
  • Open-source contribution or evaluation experience is a plus.
Nice to Have
  • Previous participation in LLM research or evaluation projects.
  • Experience building or testing developer tools or automation agents.
Work Terms
  • Fully remote contractor assignment with no medical benefits or paid leave.
  • Choose a commitment of 20, 30, or 40 hours per week.
  • Commit at least 4 hours per day and at least 20 hours per week, including 4 hours of overlap with Pacific Time.
  • Three-month contract, with an expected start date next week.
  • Applicants must be located in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, or Mexico.
Application Process
  • Complete two interview rounds: a 60-minute technical interview and a 30-minute technical and cultural discussion. The evaluation process is listed as approximately 75 minutes.

Related Jobs

More like this