We build tasks whose outcomes can be checked by a reference answer, an executable test suite, a graded work product, or held-out re-execution. The library spans Frontier STEM, coding, professional work, and recursive self-improvement.
The unit of evaluation changes with the work: an answer, a code artifact, a professional deliverable, or an improved method.
Closed-ended research problems with determinate reference answers, from graduate-level material to questions that remain difficult for frontier models.
Repository-level tasks with a runnable environment and a concrete artifact contract. The submitted code is evaluated by an executable test suite.
Multi-step assignments modeled on complete professional engagements, with realistic evidence, tools, intermediate state, and graded deliverables.
Runnable research environments in which an AI system iterates on a working method. The submitted method is then re-executed on a sealed split.
Each release is assembled around the requirements of its task rather than forced into one universal schema.
Task instructions, inputs, structured metadata, and—where the task permits—reference answers, completed artifacts, or research trajectories.
Container definitions, dependencies, fixtures, tools, starter artifacts, and run instructions needed to reproduce the task state.
Task-specific tests, graders, or held-out re-execution procedures that turn a submitted answer or artifact into a reproducible result.