AI Evaluation Specialist

Role Overview

Help improve enterprise AI assistants by evaluating their outputs, identifying weaknesses, and delivering feedback that strengthens how models learn, reason, and perform. Your subject matter expertise and careful judgment are central to this remote contract role. Prior AI experience is not required.

Key Responsibilities

• Assess AI-generated responses using detailed rubrics and quality standards for accuracy, relevance, and guideline adherence.
• Apply impartial, consistent judgment across a high volume of examples.
• Identify reasoning gaps, logic errors, and tool-use failures, then provide actionable feedback for improvement.
• Write concise feedback that highlights both strengths and opportunities to improve.
• Participate in discussions about rubric interpretation, evolving quality standards, process improvements, and best practices.
• Maintain thorough evaluation records and recommendations to support transparent, traceable assessment workflows.

Qualifications

• Experience in grading, quality assurance, editorial review, assessment, annotation, or a comparable field requiring detailed analysis and feedback.
• Advanced daily use of AI assistants, such as ChatGPT or Claude, as a work or productivity tool.
• Ability to synthesize complex information and communicate findings clearly in writing.
• Experience with process improvement, rubric development, or operational quality assessment in an enterprise or educational setting.
• Strong critical thinking, consistency, integrity, fairness, and attention to detail.
• Comfort working independently through large volumes of similar examples.
• A collaborative approach to sharing insights, resolving ambiguous cases, and refining evaluation criteria.

Work Terms

• Independent contractor engagement.
• Remote work is available to candidates located in the United States, Canada, United Kingdom, Ireland, Australia, or New Zealand.

Compensation

$30 to $90 per hour.