About this role
Role Overview
Design original, graduate-level computational challenges that evaluate whether advanced AI systems can complete research-level scientific and engineering work using real statistical and mathematical software. This role combines domain expertise with problem design, including simulations, result interpretation, experimental design, and data-driven investigation.
Key Responsibilities
- Create challenging problems requiring skilled use of specialized statistical, mathematical, or scientific software.
- Develop fully specified tasks that produce reproducible numerical answers through complex, multi-step workflows.
- Design investigative tasks in which an AI must plan queries or experiments, interpret partial findings, select useful measurements, and efficiently narrow possible explanations.
- Test problems against advanced AI models and refine them until they meet the intended difficulty level.
- Write problem setups, oracle functions, and solution validators in Python.
- Design challenges where careful reasoning, rather than raw computation or superficial pattern matching, is necessary to reach the correct answer.
Qualifications
- MS or PhD in statistics, applied mathematics, a related quantitative STEM field, or equivalent research experience. A PhD is preferred; candidates with an MS should have 10 or more years of relevant experience.
- Hands-on experience using specialized computational packages to solve real research or professional problems, including an understanding of their limitations and edge cases.
- Demonstrated proficiency with at least one specialized statistical, mathematical, or scientific package through research publications, open-source contributions, or professional work.
- Strong Python programming skills.
- Ability to work independently and improve problem designs in response to feedback.
- Comfort working in Linux or terminal environments with remote compute sandboxes.
Relevant Tools and Domains
Deep expertise in one or more specialized R, Python, Matlab, or Scilab packages is sufficient. Relevant areas include Bayesian statistics; item response theory and psychometrics; structural equation and latent variable modeling; topological data analysis; differential equations and dynamical systems; state-space and time-series modeling; survival and event-history analysis; mixed, additive, and advanced regression; spatial statistics and geostatistics; statistical learning; optimization; numerical linear algebra; high-precision computation; and computational geometry.
Examples include rstan, cmdstanr, rjags, brms, lavaan, OpenMx, deSolve, KFAS, forecast, lme4, mgcv, spatstat, gstat, sf, terra, lpSolve, nloptr, RSpectra, Rmpfr, statsmodels, and PyMC. Comparable specialized statistical, mathematical, scientific, or domain-specific packages are also welcome, as is numerical computing and scientific modeling in Matlab or Scilab.
Preferred Experience
- Experience across multiple computational domains or specialized software packages.
- Familiarity with benchmark or evaluation design.
- Scientific teaching, exam design, or problem-set design experience.
- Experience with computational reproducibility and containerized environments.
Work Terms
- Remote, hourly engagement.
- Availability of at least 15 to 20 hours per week is required.
Compensation
Hourly rate of $70 to $90.