Each RSI environment starts with an executable method, a quantitative objective, and room for improvement. AI systems propose changes, run experiments, interpret results, and roll back failures over a long research horizon. The full trajectory becomes training data; the final method is re-executed on a sealed split to determine whether the improvement generalizes.
RSI data contains more than the successful endpoint. It connects each research decision to the experiment that followed, records whether the change survived, and finally measures whether the resulting method generalizes.
The environment is reusable infrastructure rather than a static prompt. Running it produces complete research trajectories; keeping the verifier sealed turns the same task into an evaluation; reading the research process alongside the held-out result locates the next capability bottleneck.
The working image and the complete research trajectory it produces: every version, the idea behind it, the code, the visible metric, and whether it was kept or reverted. Failed ideas stay in the record instead of disappearing behind the final answer.
The verifier image, hidden split, trusted scoring code and at least one reference solution measured end to end. Stronger references can be added where they exist. Training data and scoring data never cross the seal.
The visible trajectory, final held-out result and reference runs can be read together. They show whether the run stopped iterating, overfit the visible set, ranked candidates worse despite a better model of the problem, or spent its budget in the wrong place.
Each download contains a runnable public research environment, starter method, visible self-check, and instructions for running it. It also includes the corresponding completed rollout and aggregate run result: a Grok trajectory for Gate Sizing, and Codex trajectories for Inventory Policy, Protein–Ligand Co-folding, and Mathematical Post-training. Hidden inputs and labels, private verifier assets, per-hidden-case outputs, provenance records, and identifying metadata are omitted.
Improve a deliberately weak co-folding pipeline using protein sequences, ligand SMILES and MSAs. The final submitted method is re-run on sealed complexes and scored by physical-validity checks and a 2 Å ligand-RMSD threshold.
Improve an algorithm that assigns a library cell to every gate in a placed netlist. The verifier re-runs the solver on hidden designs and scores leakage, timing, slew and capacitance penalties.
Improve a query-efficient optimizer for four controls of an (s,S) replenishment policy under noisy demand and lead times. The final optimizer is re-run on sealed instances with the same 48-query budget.
Improve the mathematical reasoning of a fixed 1.7B student model with LoRA post-training, a local 8B teacher and a sanitized corpus, starting from a weak answer-only baseline. The submitted merged model is re-sampled on sealed integer-answer problems and scored on exact accuracy.
Standard-cell gate sizing: pick a library cell for every instance of a placed chip netlist so no signal arrives late and as little power as possible is wasted. The same gate ships in several drive strengths that do identical work, a bigger one switches faster but burns more power and loads whatever drives it, and enlarging one slows the gate in front of it.
Visible designs. Placed netlists as CircuitOps IR tables, the design files, and the shared cell library. Hidden cases are different designs in the same format.
A crude working baseline. A shipped solver that uniformly upsizes to the strongest drive. It defines the zero point; the agent may delete all of it.
A free self-check. The same legality gate and the same timing-engine score the sealed grader uses, on the visible cases only. No submit step, no per-attempt feedback.
The contract. One function signature and one output format. What the method looks at, what it decides per instance and how it searches are the agent's to design.
The rules. An algorithm, not precomputed answers. Only combinational cells may be resized, and only to a logically equivalent cell. An illegal sizing scores nothing.
The objective. Leakage spent plus weighted timing and design-rule violations, lower is better, with each penalty switching off entirely at zero.
The environments above produce research trajectories and reusable training signal. RSI-Exam uses the same sealed re-execution pattern to compare agents: each agent iterates in the visible research environment, then its final artifact is run once in a fresh offline verifier. This snapshot reports the mean hidden-set score across all 88 active tasks.
| Model | Harness | Full score | ||
|---|---|---|---|---|
| 1 | Opus 5 | claude code · max | 0.464 | |
| 2 | GPT-5.6-sol | codex · max | 0.433 | |
| 3 | GLM 5.3 | claude code · max | 0.403 | |
| 4 | Qwen3.8 Max-0902 | qwen coder · xhigh | 0.392 | |
| 5 | Kimi K3 | kimi cli · max | 0.382 | |
| 6 | Grok 4.6 | grok · xhigh | 0.367 | |
| 7 | GPT-5.5 | codex · xhigh | 0.331 | |
| 8 | DeepSeek V4 Pro | claude code · max | 0.322 | |
| 9 | Gemini 3.7 Flash | antigravity · high | 0.309 |
Closed-ended research problems with determinate reference answers, from graduate-level material to questions that remain difficult for frontier models.
Repository-level tasks with a runnable environment and a concrete artifact contract. The submitted code is evaluated by an executable test suite.
Multi-step assignments modeled on complete professional engagements, with realistic evidence, tools, intermediate state, and graded deliverables.