About this role
As an AI Quality Analyst, you will play a crucial role in evaluating a new personalization feature for Gemini. This position involves assessing how effectively the model utilizes information from your past Gemini conversations, Gmail, Google Search, and YouTube activity to enhance the relevance and helpfulness of its responses. The role requires a unique combination of creativity and analytical rigor, where you will design prompts based on your personal experiences and use your analytical skills to evaluate the quality of the model''s personalized responses across various dimensions.
Key Responsibilities:
- Design and execute multi-turn conversational prompts that require the AI to leverage your personal information and experiences.
- Evaluate model responses based on your intent from the starting prompt, ensuring appropriate application of personalization.
- Analyze responses for Grounding issues, verifying that claims about you are supported by evidence and not flawed inferences.
- Assess Integration quality to ensure personal data is naturally woven into the response without robotic overnarrating.
- Rigorously evaluate and stack-rank two model responses side-by-side to determine which is more helpful and user-friendly.
Qualifications:
- Proficiency in Portuguese, with the ability to read and write at a high level.
- Willingness to use your primary personal Google account and enable personal data sources for genuine assessments.
- Full-time availability in your local time zone to support a global, 24-hour operations team.
- Exceptional analytical thinking skills to evaluate nuanced AI responses and assess personalization quality.
- Experience in designing creative, multi-turn prompts based on personal context.
- Strong understanding of personalization concepts, with the ability to identify incorrect personalization and poor inferences.
- Meticulous attention to detail, capable of reviewing side-by-side model responses to spot subtle differences.
- Excellent written communication skills for crafting clear, concise rationales for model rankings.
- Ability to provide constructive feedback and detailed annotations.
- Strong communication and collaboration skills.
- Self-motivated and able to work independently in a remote setting.
- Technical setup with a desktop/laptop and a reliable internet connection.
Work Terms:
Contract position with full-time availability required.
Compensation:
$15 per hour.
Eligibility:
Must be located in Portugal and possess the necessary work authorization.