AI Rater Guidelines Writer

Write precise, unambiguous rater guidelines and rubrics that enable consistent human evaluation of large language model outputs. You will translate ambiguous program requirements into rater-ready instructions across multiple subject-matter domains, helping a GenAI team build foundational AI models and improve data quality for model training.

• Work with research and program teams to close the gap between ambiguous program requirements and rater-ready instructions across a range of subject-matter domains.
• Design clear, non-contradictory rating guidelines and rubrics that raters can apply consistently, including guidance for edge cases.
• Evaluate draft guidelines for ambiguity, internal contradiction, and coverage gaps, and revise until raters can apply them without escalation.
• Translate program specifications from domains such as finance, retail, insurance, legal, and sports into precise, discipline-specific rater instructions.
• Collaborate with other subject matter experts and program leads to ensure consistency and accuracy across guideline sets.

• 3+ years of professional experience in linguistics, instructional design, technical writing, or a closely related field, with direct experience writing or refining guidelines or rubrics for human raters in a GenAI or RLHF context.
• Demonstrated ability to work across multiple subject-matter domains, for example finance, retail, insurance, legal, and sports, and to translate domain-specific nuance into clear instructions.
• Proven track record of resolving ambiguity and contradiction in written specifications, with concrete before and after examples.
• Evidence of career progression in relevant roles.
• Strong written communication skills, and the ability to explain complex or nuanced guidance clearly and precisely.
• Ability to engage reliably for at least 35 hours per week during weekdays.

This is a W-2 employment position with Cincinnatus LLC, acting as employer of record and placing employees within a leading AI lab as part of their extended workforce. Employment includes payroll and benefits administered by Cincinnatus LLC. Role is based in the United States, and requires availability for a minimum of 35 hours per week on weekdays.

$45 to $65 per hour.

Candidates must be able to be employed on a W-2 basis in the United States. Cincinnatus LLC is an equal employment opportunity employer and does not discriminate on the basis of race, religion, color, national origin, sex including pregnancy and related medical conditions, sexual orientation, gender identity or expression, age, protected veteran status, disability, genetic information, political views, or any other legally protected characteristic.