About this role
Role Overview
Evaluate personalized AI interactions in Portuguese by testing how effectively a model uses information from your past conversations, email, search activity, and video activity to provide relevant, helpful responses. This role combines creative prompt design with careful analysis of personalization quality.
Key Responsibilities
- Create and run multi-turn conversational prompts, typically one to five turns, based on your own personal context and experiences.
- Assess whether responses appropriately apply personalization in line with the intent of the initial prompt.
- Review responses for grounding issues, ensuring claims about you are supported by evidence rather than flawed inferences or hallucinations.
- Evaluate how naturally personal information is integrated, including whether it feels forced or overly narrated.
- Compare and rank two model responses side by side for helpfulness, ease of use, and overall user experience.
- Write clear, defensible comparison rationales that identify positive and negative moments by specific conversation turn.
- Review debug information to confirm that conversation summaries and relevant data sources were used correctly.
- Delete evaluation conversations to keep them from affecting your future chat history.
Qualifications
- High proficiency in reading and writing Portuguese.
- Willingness to use your primary personal Google account, rather than a test account, and enable personal data sources for authentic evaluation.
- Strong analytical judgment for nuanced or ambiguous AI responses, especially personalization quality.
- Experience designing creative, multi-turn prompts using personal context.
- Ability to identify incorrect personalization, weak inferences, and forced connections.
- Meticulous attention to subtle differences in naturalness and overnarration when reviewing side-by-side responses.
- Excellent written communication skills for concise, structured evaluation rationales and detailed annotations.
- Strong feedback, collaboration, and independent remote-work skills.
- Bachelor''s degree or equivalent relevant experience in areas such as policy, law, ethics, linguistics, journalism, computer science, or another analytical field.
- Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred.
- A desktop or laptop and reliable internet connection.
Work Terms
- Remote contractor engagement lasting three months.
- Full-time availability in your local time zone is required to support a global 24-hour operations team.
- Commit at least four hours per day, up to 40 hours per week, including four hours of overlap with Pacific Standard Time.
Compensation
- $15 per hour.
Application Process
- Shortlisted applicants will receive a job-interest form.
- Following profile review, selected candidates will receive an assessment that must be completed within 24 hours.
- Candidates selected based on assessment results will be contacted to discuss pre-onboarding requirements.