Senior Software Engineer for LLM Evaluation

This role develops high-quality datasets and evaluation workflows to train and benchmark large language models on software engineering tasks. You will curate and author code examples, correct and improve AI-generated solutions across multiple languages, design automatic verification mechanisms, and work closely with researchers and cross-functional teams to raise model reliability on real-world engineering workflows.

• Create datasets and ground-truth solutions for training and benchmarking LLMs on software engineering problems.
• Curate, implement, and correct code examples in Python, JavaScript including ReactJS, C and C++, Java, Rust, and Go.
• Evaluate AI-generated code for correctness, efficiency, scalability, and reliability, and provide precise, structured evaluation rationales.
• Collaborate with researchers and cross-functional teams to align model outputs with industry performance benchmarks.
• Design and build agents or tooling to verify code quality and detect common error patterns.
• Hypothesize, test, and evaluate model capabilities across software engineering lifecycle steps including prototyping, architecture design, API design, production implementation, launch, experiments, monitoring, and operational maintenance.
• Design verification mechanisms that can automatically confirm a solution to a software engineering task.

• Minimum 3 years of professional software engineering experience.
• Proven experience building full-stack applications and deploying scalable, production-grade software using modern languages and tools.
• Deep understanding of software architecture, design, development, debugging, and code review and quality assessment.
• Strong proficiency in one or more of the target languages: Python, JavaScript (including ReactJS), C/C++, Java, Rust, Go.
• Excellent oral and written communication skills, able to produce clear, structured evaluation rationales.

• Engagement type: Contractor. This is a contractor role without medical or paid leave benefits.
• Commitment: Flexible, minimum 10 hours per week up to 40 hours per week.
• Duration: Initial 1 month engagement, with possible extensions based on performance and fit.
• Work arrangement: Remote.

• Candidates must be based in the United States, Canada, or Western European countries, for example Austria, Belgium, France, Germany.

• The application takes approximately 15 to 30 minutes to complete.
• Applicants are required to complete an AI video interview as part of the evaluation process.