Kannada Language AI Response Evaluator

Evaluate Kannada AI-generated responses to identify factual errors, reasoning gaps, clarity or tone issues, and other strengths and weaknesses, then produce clear English-language analyses that will be used to help create an ideal AI response later in the project.

• Conduct fact-checking using trusted public sources and external tools.
• Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
• Assess reasoning quality, clarity, tone, and completeness of model responses.
• Ensure model outputs align with expected conversational behavior and system guidelines.
• Write analysis and evaluation artifacts in English.

• Bachelor''s degree.
• Native fluency in Kannada.
• Strong proficiency in English, including excellent writing skills to articulate nuanced feedback.
• Significant experience using large language models and an understanding of how and why people use them.
• Strong attention to detail, able to notice subtle issues others may miss.
• Background or experience in domains requiring structured analytical thinking, for example research, policy, analytics, linguistics, or engineering.

• Prior experience with RLHF, model evaluation, or data annotation work.
• Experience writing or editing high-quality written content.
• Experience comparing multiple outputs and making fine-grained qualitative judgments.

• You identify factual inaccuracies, reasoning errors, and communication gaps in model responses.
• You produce clear, consistent, and reproducible evaluation artifacts.
• Your feedback leads to measurable improvements in response quality and user experience.

Contract work, paid hourly, fully remote and open to candidates globally.

15 - 20 hourly

• Fluent language requirement, as stated above: Kannada native fluency and strong English proficiency.
• To be considered for this role you must complete a Bilingual Competency interview in Kannada. This interview is required during the application process and is used to assess Kannada bilingual ability.
• Analyses you produce for the role should be written in English.