About this role
Role Overview
Assess Odia AI-generated responses, identify factual errors and areas for improvement, and produce clear English-language analyses that will be used to shape higher-quality AI outputs. This role combines language expertise, analytical thinking, and hands-on experience with large language models.
Key Responsibilities- Evaluate AI-generated responses in Odia for factual accuracy, clarity, tone, completeness, and alignment with expected conversational behavior and system guidelines.
- Conduct fact-checking using trusted public sources and external tools to verify claims in responses.
- Identify strengths, weaknesses, and specific improvement opportunities in model outputs, and document these findings in English.
- Assess reasoning quality and detect logical or factual errors.
- Produce consistent, reproducible evaluation artifacts that can be used to refine model behavior.
- Bachelor''s degree.
- Native fluency in Odia and strong proficiency in English, with excellent English writing skills to articulate nuanced feedback.
- Significant experience using large language models, and a practical understanding of how and why people use them.
- Strong attention to detail and ability to notice subtle issues in text.
- Background or experience in domains requiring structured analytical thinking, such as research, policy, analytics, linguistics, or engineering.
- Experience with RLHF, model evaluation, or data annotation work.
- Experience writing or editing high-quality written content.
- Experience comparing multiple outputs and making fine-grained qualitative judgments.
- Clear identification of factual inaccuracies, reasoning errors, and communication gaps in model responses.
- Delivery of consistent, reproducible evaluation artifacts and actionable feedback.
- Feedback that leads to measurable improvements in response quality and user experience.
- Location: Remote, global.
- Engagement type: Contract work.
- Compensation is paid hourly.
- Hourly rate: 15 to 20 hourly.
- Required language skills: native fluency in Odia and strong proficiency in English.
- To be considered, you must complete a bilingual competency interview in Odia. Applications will only be evaluated after completion of this interview.