About this role
Help evaluate personalized AI interactions in Turkish by testing how effectively a model uses information from prior conversations, email, search, and video activity to deliver relevant, useful responses.
Key Responsibilities- Design and run multi-turn conversational prompts, typically 1 to 5 turns, based on your own experiences and personal context.
- Assess whether responses correctly address the intent of the starting prompt and apply personalization appropriately.
- Review grounding, ensuring claims about you are supported by available evidence rather than flawed inferences or hallucinations.
- Evaluate how naturally personal information is integrated, including identifying forced connections and excessive narration.
- Compare and rank two model responses side by side for helpfulness, usability, and overall experience.
- Write clear, evidence-based rationales that reference specific conversation turns.
- Extract and verify model debug information to confirm that chat summaries and data sources were used appropriately.
- Delete evaluation conversations to avoid affecting future chat history.
- High-level ability to read and write Turkish.
- Strong analytical judgment for evaluating nuanced and ambiguous AI responses.
- Experience creating creative, multi-turn prompts using personal context.
- Understanding of personalization quality, including incorrect personalization, weak inferences, and forced connections.
- Careful attention to subtle differences between side-by-side responses.
- Excellent written communication skills for concise, structured evaluation rationales and detailed annotations.
- Strong communication and collaboration skills, with the ability to work independently in a remote environment.
- A BS/BA degree or equivalent relevant experience in Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field.
- Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred.
- Willingness to use your primary personal Google account, rather than a test account, and enable personal data sources for authentic evaluation.
- Access to a desktop or laptop and reliable internet connection.
- Remote contractor engagement for 3 months.
- Full-time availability in your local time zone is required as part of a global 24-hour operations team.
- Choose a 30-hour or 40-hour weekly commitment, with at least 4 hours per day and a minimum of 4 hours overlapping Pacific Time.
- $15 per hour.
- Shortlisted candidates receive a job interest form.
- Following profile review, candidates receive an assessment that must be completed within 24 hours.
- Candidates selected based on assessment results are contacted to discuss pre-onboarding requirements.