Data Analyst for Machine Learning Evaluation

Role Overview

This position offers an exciting opportunity for experienced Data Analysts to engage in benchmark-driven evaluation projects focused on real-world machine learning systems. The role involves hands-on analytical work with production-like datasets, metrics, and ML outputs, aimed at evaluating, diagnosing, and enhancing the performance of advanced AI systems. The ideal candidate will thrive at the intersection of data analysis and machine learning, demonstrating strong analytical rigor and proficiency in handling real datasets and ML evaluation workflows.

Key Responsibilities

• Analyze structured and unstructured datasets generated from ML training, inference, and evaluation pipelines.
• Define, compute, and validate metrics used to evaluate model performance and behavior.
• Investigate data distributions, model outputs, failure modes, and edge cases relevant to benchmark tasks.
• Write and run Python and SQL code to analyze data, create reports, and support evaluation workflows.
• Validate data quality, consistency, and correctness across datasets and experiments.
• Create clear, well-documented analytical artifacts and reproducible analysis workflows.
• Collaborate with ML engineers and researchers to design challenging, real-world evaluation scenarios for MLE Bench.

Qualifications

• Minimum 3+ years of experience as a Data Analyst or Analytics-focused Engineer.
• Strong proficiency in Python for data analysis.
• Solid experience with SQL and relational datasets.
• Experience analyzing ML outputs and evaluation metrics.
• Strong understanding of statistics and analytical reasoning.
• Ability to work with large, complex datasets and draw reliable insights.
• Experience writing clean, readable, and well-documented analytical code.
• Excellent spoken and written English communication skills.

Work Terms

• Commitments Required: At least 4 hours per day and a minimum of 20 hours per week with 4 hours overlap with PST.
• Engagement Type: Contractor assignment (no medical/paid leave).
• Duration of Contract: 3 months (adjustable based on engagement).
• Location: Open to candidates from India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Brazil, and Mexico.

Compensation

Compensation details will be discussed during the interview process.

Eligibility

This role is open to candidates who meet the specified qualifications and are located in the eligible countries listed above.

Evaluation Process

• Technical Interview with a live coding challenge (60 mins).