Each RSI environment starts with an executable method, a quantitative objective, and room for improvement. AI systems propose changes, run experiments, interpret results, and roll back failures over a long research horizon. The full trajectory becomes training data; the final method is re-executed on a sealed split to determine whether the improvement generalizes.
RSI data contains more than the successful endpoint. It connects each research decision to the experiment that followed, records whether the change survived, and finally measures whether the resulting method generalizes.
The environment is reusable infrastructure rather than a static prompt. Running it produces complete research trajectories; keeping the verifier sealed turns the same task into an evaluation; reading the research process alongside the held-out result locates the next capability bottleneck.
The working image and the complete research trajectory it produces: every version, the idea behind it, the code, the visible metric, and whether it was kept or reverted. Failed ideas stay in the record instead of disappearing behind the final answer.
The verifier image, hidden split, trusted scoring code and at least one reference solution measured end to end. Stronger references can be added where they exist. Training data and scoring data never cross the seal.
The visible trajectory, final held-out result and reference runs can be read together. They show whether the run stopped iterating, overfit the visible set, ranked candidates worse despite a better model of the problem, or spent its budget in the wrong place.
Each download contains a runnable public research environment, starter method, visible self-check, and instructions for running it. It also includes the corresponding completed rollout and aggregate run result: a Grok trajectory for Gate Sizing, and Codex trajectories for Inventory Policy and Protein–Ligand Co-folding. Hidden inputs and labels, private verifier assets, per-hidden-case outputs, provenance records, and identifying metadata are omitted.
Improve a deliberately weak co-folding pipeline using protein sequences, ligand SMILES and MSAs. The final submitted method is re-run on sealed complexes and scored by physical-validity checks and a 2 Å ligand-RMSD threshold.
Improve an algorithm that assigns a library cell to every gate in a placed netlist. The verifier re-runs the solver on hidden designs and scores leakage, timing, slew and capacitance penalties.
Improve a query-efficient optimizer for four controls of an (s,S) replenishment policy under noisy demand and lead times. The final optimizer is re-run on sealed instances with the same 48-query budget.
Standard-cell gate sizing: pick a library cell for every instance of a placed chip netlist so no signal arrives late and as little power as possible is wasted. The same gate ships in several drive strengths that do identical work, a bigger one switches faster but burns more power and loads whatever drives it, and enlarging one slows the gate in front of it.
Visible designs. Placed netlists as CircuitOps IR tables, the design files, and the shared cell library. Hidden cases are different designs in the same format.
A crude working baseline. A shipped solver that uniformly upsizes to the strongest drive. It defines the zero point; the agent may delete all of it.
A free self-check. The same legality gate and the same timing-engine score the sealed grader uses, on the visible cases only. No submit step, no per-attempt feedback.
The contract. One function signature and one output format. What the method looks at, what it decides per instance and how it searches are the agent's to design.
The rules. An algorithm, not precomputed answers. Only combinational cells may be resized, and only to a logically equivalent cell. An illegal sizing scores nothing.
The objective. Leakage spent plus weighted timing and design-rule violations, lower is better, with each penalty switching off entirely at zero.
RSI-Exam is the held-out evaluation suite built from this environment library. Each score is computed by re-executing the final method on sealed data, then aggregating performance across the evaluated RSI-Exam environments.
| Model | Harness | Score | ||
|---|---|---|---|---|
| 1 | Opus 5 | claude code · max | 0.514 | |
| 2 | GPT-5.6-sol | codex · max | 0.505 | |
| 3 | Kimi K3 | kimi cli | 0.443 | |
| 4 | Grok 4.6 | grok · xhigh | 0.430 | |
| 5 | Gemini 3.7 Flash | antigravity · high | 0.356 |
Scores map onto each task's own anchors rather than a universal percentage: the shipped baseline scores 0, a reference solution 0.30, the strongest reference the task ships 0.60, and the task's theoretical ceiling 1.00. Tasks without published rollouts carry no score.
Closed-ended research problems with determinate reference answers, from graduate-level material to questions that remain difficult for frontier models.
Repository-level tasks with a runnable environment and a concrete artifact contract. The submitted code is evaluated by an executable test suite.
Multi-step assignments modeled on complete professional engagements, with realistic evidence, tools, intermediate state, and graded deliverables.