About this role
Use your administrative expertise to create realistic evaluation tasks that help improve next-generation AI systems. This remote contract role focuses on translating the complexity, judgment, and practical workflows of experienced administrative professionals into high-quality benchmark scenarios. No prior AI experience is required.
Key Responsibilities
- Design realistic, process-driven administrative evaluation tasks involving multi-tool coordination, such as calendar reconciliation, travel and expense workflows, records management, and professional correspondence.
- Create supporting scenario materials, including emails, calendar entries, receipts, meeting notes, and contact records.
- Define the correct outcome for each task and document the step-by-step decisions an experienced professional would make.
- Develop detailed evaluation rubrics with 35 or more criteria that assess process adherence and nuanced judgment in AI agent responses.
- Document administrative workflows and recommend improvements based on current professional best practices.
- Work with project leads to calibrate task complexity against real-world administrative challenges.
- Apply your domain expertise to help train and evaluate AI systems through authentic, expert-authored input.
Qualifications
- At least 5 years of professional administrative experience supporting teams or departments in mid-sized to large organizations.
- Demonstrated ownership of ongoing, recurring processes rather than only ad hoc assignments.
- Experience documenting, assessing, and improving inherited administrative procedures.
- Strong written and verbal communication skills, including professional correspondence standards.
- Excellent attention to detail, written judgment, process ownership, and multi-tool coordination skills.
- Comfort working across digital tools, platforms, and data formats including CSVs, PDFs, spreadsheets, and technical documents.
- Interest in applying administrative knowledge and best practices to intelligent systems.
Work Terms
- Remote independent-contractor engagement.
- Work is output-based, with payment tied to tasks that meet project specifications. Time required may vary based on experience and workflow.
- Minimum submission requirements apply, including a required minimum number of task submissions each week.
- Selected candidates are expected to begin their first tasks within 24 to 48 hours after completing onboarding.
- Openings are generally filled within 48 hours, so immediate availability is preferred.
Compensation
- Advertised compensation is $20 to $40 per hour.
- Payments are made per completed task that satisfies project specifications.
Application Process
- Apply to be considered for the project. Candidates who are selected will complete onboarding before beginning task work.
- By continuing an application, candidates acknowledge the applicable terms and privacy policy.