About this role
As an AI Quality Analyst, you will play a crucial role in evaluating a new personalization feature for Gemini. Your primary responsibility will be to assess how effectively the model utilizes information from your past interactions with Gemini, Gmail, Google Search, and YouTube to enhance the relevance and helpfulness of its responses. This position requires a unique combination of creativity and analytical skills, as you will design prompts based on your personal experiences and rigorously analyze the quality of the model''s personalized responses across various dimensions.
Key Responsibilities:
- Design and execute multi-turn conversational prompts (typically 1-5 turns) that leverage your personal information and experiences.
- Evaluate model responses based on your intent from the starting prompt, ensuring appropriate application of personalization.
- Analyze responses for grounding issues, verifying that claims about you are supported by evidence and free from flawed inferences or hallucinations.
- Assess integration quality to ensure personal data is seamlessly woven into responses without robotic overnarrating.
- Conduct side-by-side (SxS) evaluations of model responses to determine which is more helpful, user-friendly, and engaging.
Qualifications:
- Proficiency in Russian, with the ability to read and write fluently.
- Willingness to use your primary personal Google account and enable personal data sources for authentic assessments.
- Full-time availability in your local time zone, as part of a global, 24-hour operations team.
- Exceptional analytical thinking skills, particularly in evaluating nuanced AI responses and personalization quality.
- Experience in creative prompt engineering, designing multi-turn prompts based on personal context.
- Strong understanding of personalization concepts, with the ability to identify incorrect personalization and poor inferences.
- Meticulous attention to detail, capable of reviewing SxS model responses for subtle differences in naturalness.
- Excellent written communication skills, with the ability to provide clear and structured rationales for model rankings.
- Ability to provide constructive feedback and detailed annotations.
- Strong communication and collaboration skills.
- Self-motivated and able to work independently in a remote setting.
- Technical setup with a desktop/laptop and reliable internet connection.
This is an exciting opportunity to contribute to the development of advanced AI systems while working remotely in a dynamic team environment.