About this role
Role Overview
You will evaluate AI-generated responses in Malayalam, identify factual errors, reasoning gaps, tone and clarity issues, and produce clear English-language analysis that will be used to create improved model responses. This is a remote, hourly contract role working globally. Note, you must complete a Malayalam bilingual competency interview to be considered.
Key Responsibilities
- Perform fact-checking of model responses using trusted public sources and external tools.
- Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
- Assess reasoning quality, clarity, tone, and completeness of responses.
- Ensure model outputs align with expected conversational behavior and system guidelines.
- Document evaluations consistently so results are reproducible and actionable.
Qualifications
- Bachelor''s degree.
- Native fluency in Malayalam, with strong proficiency in English for written analysis.
- Significant experience using large language models, and an understanding of how and why people use them.
- Excellent English writing skills, able to articulate nuanced feedback clearly.
- Strong attention to detail, able to notice subtle issues others may miss.
- Background or experience in structured analytical fields such as research, policy, analytics, linguistics, or engineering.
Nice-to-Have Specialties
- Experience with reinforcement learning from human feedback, model evaluation, or data annotation.
- Experience writing or editing high-quality written content.
- Experience comparing multiple outputs and making fine-grained qualitative judgments.
What Success Looks Like
- Consistently identifying factual inaccuracies, reasoning errors, and communication gaps in model responses.
- Producing clear, consistent, and reproducible evaluation artifacts.
- Providing feedback that leads to measurable improvements in response quality and user experience.
Work Terms
- Engagement type: Contract work, independent contractor relationship.
- Payment model: Hourly compensation.
- Location: Remote, global work allowed.
Compensation
- Pay range: 15 to 20 USD per hour.
Eligibility and Application Process
- Language requirement: Native Malayalam fluency is required, with strong English proficiency for written evaluations.
- To be considered, you must complete the required bilingual competency interview in Malayalam. Candidates who do not complete this interview will not be considered.
- Apply remotely, and be prepared to demonstrate your Malayalam language skills and your ability to evaluate model outputs. No specific visa sponsorship or work authorization details are provided beyond the global remote eligibility; applicants should confirm they can work as independent contractors from their location.