About this role
Role Overview
Evaluate a new AI personalization feature by testing how effectively it uses information from your past conversations, email, search, and video activity to provide more relevant, helpful responses. You will create prompts rooted in your own experiences, then apply analytical judgment to assess personalization quality, including grounding, integration, and helpfulness.
Key Responsibilities
- Design and run multi-turn conversational prompts, typically 1 to 5 turns, that require use of your personal information and experiences.
- Evaluate responses against the intent of the opening prompt and determine whether personalization was applied appropriately.
- Review grounding to confirm that claims about you are evidence-based and do not rely on flawed inferences or hallucinations.
- Assess whether personal data is integrated naturally, without robotic or excessive narration.
- Compare and rank two responses side by side for helpfulness, ease of use, and overall experience.
- Write clear, defensible comparison rationales that identify specific positive or problematic moments in the conversation.
- Extract and verify model debug information to confirm that chat summaries and data sources were used correctly.
- Delete evaluation conversations after review to avoid affecting your future chat history.
Qualifications
- High proficiency in reading and writing Arabic, the focus language for this project.
- Willingness to use your primary personal Google account, rather than a test account, and enable personal data sources for authentic evaluation.
- Strong analytical judgment for nuanced or ambiguous AI responses and personalization quality.
- Experience creating creative, multi-turn prompts based on personal context.
- Ability to identify incorrect personalization, weak inferences, and forced connections.
- Careful attention to subtle differences in side-by-side responses, including naturalness and excessive narration.
- Excellent written communication skills, including the ability to provide structured rationales that reference specific conversation turns.
- Ability to provide constructive feedback and detailed annotations.
- Strong communication and collaboration skills, with the ability to work independently in a remote setting.
- A desktop or laptop and reliable internet connection.
- Bachelor''s degree or equivalent relevant experience in Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or another analytical field.
- Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred.
Work Terms
- Remote independent contractor engagement for 3 months.
- Full-time availability in your local time zone is required to support a global 24-hour operations team.
- Commit at least 4 hours per day, up to 40 hours per week, including 4 hours of overlap with Pacific Time.
Compensation
$15 per hour.
Application Process
- Shortlisted candidates receive a job interest form.
- Following profile review, selected candidates receive an assessment that must be completed within 24 hours.
- Candidates shortlisted based on assessment results are contacted to discuss pre-onboarding requirements.