Each task combines a work-ticket instruction, a pinned repository or seeded workspace, an executable test suite, and a reference solution. The submitted artifact receives a binary result under the task's verifier.
Difficulty runs from Terminal-Bench-style self-contained work, where fixtures are seeded and the agent builds the tool an in-environment verifier then checks, to DeepSWE-style repository maintenance, where the agent patches a pinned revision of a real upstream project and the submitted patch is replayed against hidden tests in a separate image.
Instruction. Agent-facing, written as a ticket: required interface, the edge cases that matter, explicit definition of done.
Environment. A Docker image with the pinned repo or seeded fixtures, the toolchain, and declared CPU, memory and timeout limits.
Hidden tests. Never shown to the agent, so a solution cannot be written to the tests. They cover variants the seeded fixture does not.
Reference solution. A patch a maintainer would accept. It must score 1, and an empty submission must score 0, before the task ships.
A sample, not the catalog: systems-level work in Rust, Go and Python covering parser state machines, protocol recovery, binary reverse engineering, and vision-to-structure extraction. Each download contains the runnable task, its verifier, reference answer, and instructions for running it. Prior runs, provenance records, and identifying metadata are omitted.
| Task | Area | Runtime | Verification | |
|---|---|---|---|---|
| serde-json-sequence | Recoverable RFC 7464 JSON sequence parsing | Rust | Patch-replay, separate image | Download |
| chi-route-resolution | HTTP radix-tree route resolution | Go | Patch-replay + race checks | Download |
| wsproto-extension-transaction | WebSocket extension parsing, transactional negotiation | Python | Patch-replay, separate image | Download |
| elf-note-calibration-repair | ELF note reverse engineering and binary repair | Python | In-environment verifier | Download |
| greenhouse-frame-roundtrip | Undocumented binary frame decoding and encoding | Python 3.11 | In-env + black-box probe | Download |
| warmboot-config-fragment-recovery | Corrupted key/value page recovery | Python | In-environment verifier | Download |
| plotter-preview-motion-report | PPM parsing, HUD recognition, motion reconstruction | Python | In-environment verifier | Download |
| route-sheet-graph-recovery | Image consensus and semantic graph extraction | Python | In-environment verifier | Download |
Requires Docker and Harbor 0.21.0 or a compatible release. One task runs the same way, by pointing at its directory.
190 KB · 8 tasks · environments, verifiers and reference answers
harbor run \ -p ./tasks \ -a YOUR_AGENT \ -m YOUR_MODEL \ -o ./out -y
Closed-ended research problems with determinate reference answers, from graduate-level material to questions that remain difficult for frontier models.
Multi-step assignments modeled on complete professional engagements, with realistic evidence, tools, intermediate state, and graded deliverables.
Runnable research environments in which an AI system iterates on a working method. The submitted method is then re-executed on a sealed split.