About this role
Role Overview
Evaluate AI-generated responses in Bengali, identify factual errors and areas for improvement, and produce clear English-language analyses that will help shape higher-quality model outputs. This role combines fact checking, qualitative judgment, and written feedback to improve conversational AI behavior.
Key Responsibilities- Perform fact checking using trusted public sources and external tools.
- Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies.
- Assess reasoning quality, clarity, tone, and completeness of model responses.
- Ensure model outputs align with expected conversational behavior and system guidelines.
- Write evaluation artifacts and feedback in English, with precise, reproducible comments.
- Bachelor''s degree.
- Native fluency in Bengali, strong proficiency in English, with excellent English writing skills to articulate nuanced feedback.
- Significant experience using large language models, with an understanding of how and why people use them.
- Strong attention to detail, able to notice subtle issues others may miss.
- Background or experience in domains that require structured analytical thinking, for example research, policy, analytics, linguistics, or engineering.
- Prior experience with RLHF, model evaluation, or data annotation work.
- Experience writing or editing high-quality written content.
- Experience comparing multiple outputs and making fine-grained qualitative judgments.
- Consistently identify factual inaccuracies, reasoning errors, and communication gaps in model responses.
- Produce clear, consistent, and reproducible evaluation artifacts.
- Provide feedback that leads to measurable improvements in response quality and user experience.
- Location, remote, global.
- Contract work, hourly engagement.
- Pay rate: 15 - 20 hourly.
- Fluent language skills required: Bengali, native fluency, and English, strong proficiency.
- To be considered for this role you must take the Bilingual Competency interview in Bengali, this is a required step in the selection process.