About this role
Role Overview
Evaluate how well large language models handle real-world Japanese legal questions by running identical prompts through two LLM systems, comparing their outputs, and scoring each answer against a provided rubric. This role focuses on producing concise, high-quality written assessments of AI-generated responses on Japanese commercial and contract law topics.
Key Responsibilities- Run the same set of prompts across two AI LLM platforms.
- Compare the outputs from both platforms side by side.
- Score each response using a standardized rubric provided by the project team.
- Provide concise written feedback for every evaluation completed.
- At least 5 years experience as in-house legal counsel practising in Japan.
- Strong command of Japanese law, especially commercial and contract matters.
- Sharp attention to detail and precise written communication skills.
- Ability to work independently following a rubric and to meet deadlines.
- Location: Remote, must be located in Japan.
- Engagement type: hourly.
- Initial commitment: approximately 10 hours, with strong potential for additional assignments.
- All work is covered by a nondisclosure agreement.
- 140 - 150 hourly.
- Must have the required in-house counsel experience practising in Japan as stated above.