Bilingual Marathi Language Evaluator for AI Responses

Assess Marathi AI-generated responses for factual accuracy, reasoning, clarity, tone, and completeness, and produce clear English-language analysis that identifies strengths and specific areas for improvement. Your evaluations will be used to help create the "perfect AI-generated response" in a later phase of the project.

• Fact-check model outputs using trusted public sources and appropriate external tools.
• Produce high-quality human evaluation data that documents response strengths, areas for improvement, and factual inaccuracies.
• Assess reasoning quality, logical consistency, clarity of explanation, tone, and completeness of responses.
• Verify that model responses align with expected conversational behavior and applicable system guidelines.
• Write concise, reproducible evaluation artifacts in English that enable consistent downstream improvements.

• Bachelor''s degree.
• Native speaker of Marathi, with strong written and spoken English skills.
• Significant experience using large language models, and a clear understanding of common LLM uses and failure modes.
• Excellent English writing ability, capable of articulating nuanced feedback clearly.
• Strong attention to detail, able to identify subtle factual or reasoning issues.
• Experience in domains that require structured analytical thinking, such as research, policy, analytics, linguistics, or engineering.

• Prior experience with RLHF, model evaluation, or data annotation.
• Experience writing or editing high-quality written content.
• Experience comparing multiple outputs and making fine-grained qualitative judgments.

• Consistently identifying factual inaccuracies, reasoning errors, and communication gaps in model responses.
• Delivering clear, consistent, and reproducible evaluation artifacts that others can follow.
• Your feedback leads to measurable improvements in model response quality and user experience.

• Remote, global contract position.
• Engagement is hourly, contractor status.
• All evaluation notes and deliverables are expected to be written in English.

• 15 - 20 hourly.

• Fluent language requirement: native Marathi and strong English proficiency.
• To be considered for this role, you must complete the Bilingual Competency interview in Marathi. Candidates who do not take this interview will not be considered.