Key details
- Role type
- Contract
- Compensation
- $50–$70/hr
- Work arrangement
- Remote
- Category
- technology
- Openings
- 4
- Confirmed requirements
- 7
About this role
Apply your software engineering expertise to realistic developer workflows that help improve next-generation AI systems. This remote contract role focuses on evaluating AI-assisted development work, building reliable test contexts, and bringing hands-on technical judgment to a customer-focused automation project. Prior AI experience is not required.
Key Responsibilities- Complete structured evaluations that mirror developer work, including version control, code review, issue tracking, and CI/CD execution with AI-powered tools.
- Assess and score technical outputs, validating results directly in relevant development environments and tools.
- Design, seed, and maintain reproducible test environments, including repositories with authentic commit histories and CI workflows.
- Configure, document, and troubleshoot service integrations and authentication flows across multiple software tools.
- Explore products systematically to identify and document undocumented behaviors.
- Participate in calibration sessions to align evaluation methods and grading consistency across contributors.
- At least 3 years of professional software engineering experience.
- Strong Git and GitHub workflow knowledge, including pull requests, code review, branching, diff analysis, and CI log diagnostics.
- Proficiency with CI/CD systems, ideally GitHub Actions, including writing custom workflows and supporting reproducible builds.
- Experience using connectors with Claude and ChatGPT.
- Advanced Python or Bash scripting skills, including automating environment setup and reset tasks.
- Solid knowledge of REST APIs, OAuth, and webhook integrations.
- Fluency with enterprise SaaS platforms such as Slack, Google Workspace, or Microsoft 365. Cloud workspace administration experience is advantageous.
- Experience using AI assistants or agent tools in day-to-day development work. Familiarity with agent tool calling or Model Context Protocol, MCP, is a plus.
- Technical QA, evaluation, or data annotation experience, with the ability to apply rubrics consistently and identify nuanced edge cases.
- Excellent attention to detail, process discipline, and written and verbal English communication skills.
- Remote independent contractor engagement.
- $50 to $70 per hour.