Gmail & Google Calendar AI Assistant Evaluator

Role Overview

Help improve AI assistants by evaluating how they handle everyday digital tasks, including email, calendar management, bookings, and coordination. You will use practical experience managing your own digital life to assess assistant performance and provide feedback that supports stronger real-world task execution.

Key Responsibilities

• Complete realistic digital tasks, including emailing, scheduling, booking, and coordinating, using AI assistant tools in standardized, repeatable conditions.
• Give AI assistants clear instructions for real-world tasks, then review their work for accuracy and completeness.
• Apply detailed grading guidelines to evaluate, rank, and annotate assistant responses with structured feedback.
• Test email threads, calendar invitations, online bookings, and multi-person coordination across varied scenarios and edge cases.
• Use AI training tools to document findings and submit evaluation data according to project standards and milestones.
• Work with trainers and other contributors to clarify guidelines and maintain consistent evaluations.
• Participate in quality reviews and incorporate feedback to uphold rigorous evaluation standards.

Qualifications

• Daily hands-on experience using Gmail and Google Calendar for emails, invitations, scheduling, and changing plans.
• Regular experience coordinating with others by email or text, such as arranging meetups, following up on conversations, and confirming shared plans or payments.
• Ability to independently book restaurants, appointments, travel, deliveries, or similar services online.
• Practical experience using AI tools, including giving an AI assistant instructions and checking its work.
• Exceptional attention to detail and accuracy when reviewing AI-generated content.
• Strong written communication skills for clear reporting and timely progress updates.
• Strong time management and self-organization for independent remote work.

Preferred Qualifications

• Experience with AI evaluation, data labeling, user testing, or similar feedback-focused work.
• Familiarity with prompt writing and an applied understanding of AI assistant behavior and limitations.
• Experience reviewing AI outputs for accuracy, helpfulness, and safety.

Work Terms

• Remote, independent contractor engagement.
• Available only to candidates based in the United States.

Compensation

• $15 to $30 per hour.

Eligibility

• Must have an iPhone with iMessage and an active Facebook or Instagram account.

Application Process

Submit an application by email or through an available application option. Continuing with the application confirms acknowledgment of the candidate Terms and Privacy Policy.