36 papers · 1,029 tasks · four coding agents
Building Reproducibility into
ML Research Artifacts
Code and data are going through internal review and will be open-sourced once approved.
Reproducing an ML experiment often requires recovering details that its paper and released code leave implicit, and even a successful attempt can leave those decisions buried in conversations or shell histories. ReproBridge preserves them as a refined reproducible artifact: refined code, environment setup, experiment launchers, and a reproduction guide. Across four coding agents on 1,029 tasks, refined artifacts raise mean reproduction success from 76.4% to 92.6% and reduce downstream API spending by 19.7% on average.
See where the gains landReproBridge-Bench draws its 36 test papers from ICML 2026 Spotlights across eight topic families and 22 subdomains. Every outer mark is one paper–model pair from the first evaluation: green bars show how much the refined artifact raises success over the released code; red and grey ticks mark decreases and ties.
Eight ICML-adapted topic families and 22 subdomains across the 36 test papers; each paper is assigned once by its primary contribution. Outer marks show 144 paper–model pairs from the first evaluation with Codex auditing. Eval 1 · Codex
Each paper’s four marks run clockwise: Sonnet 4.6, Sonnet 5, Opus 5, Gemini 3.8 Flash. Green bars encode the RRA-minus-Raw success gain in pp from the blue zero ring, with dashed circles at 25, 50 and 100 pp; red and grey ticks mark decreases and ties without encoding magnitude. Legend counts are pairs; positive bins are (a, b].
HELIX: Hybrid Encoding with Learnable Identity and Cross-dimensional Synthesis for Time Series Imputation
General ML & theoryTime-series learning
Derived from countsGain = 100 × (RRA − Raw) ÷ tasks. Raw RRA
Hover or tap a paper’s marks to inspect it. Keyboard: Tab into the chart, step clockwise with the arrow keys, press Enter to select.
HELIX, selected above, is the reproduction the walkthrough follows next. Follow HELIX through one reproduction
A real reproduction · HELIX
HELIX, a time-series imputation paper, reports mean absolute error. ReproBridge-Bench turns each selected main-table result into a task anchored by five human-verified reference observations. A new result counts only if it lands inside the task's compatibility interval, [0.12524, 0.13448] for T004 and [0.38442, 0.43878] for T006, and passes an independent execution audit.
Opus 5, the strongest raw-code reproducer in the evaluation, starts from the paper and the released code. It restores the paper's architecture but keeps the shipped batch size of 32 and a learning rate of about 6.29×10−4.
Both runs finish, but neither MAE is compatible: 0.15265 for T004 and 0.50527 for T006, each above its interval. Success now hinges on a detail the release leaves implicit.
During construction, guide entry A6 identifies repository revision 9760b347, whose architecture matches the paper configuration, as the source for batch size 8 and learning rate 5.75×10−4. The guide records the repository history and architectural match as the rationale; the --paper_config launcher makes the choice executable.
A later session receives the refined artifact, never the construction conversation. Its MAEs, 0.12986 for T004 and 0.41160 for T006, both fall inside their intervals. On HELIX, Opus 5 succeeds on 4 of 9 tasks with raw code and 9 of 9 with the refined artifact.
Built from the paper's HELIX worked example: recorded MAEs and compatibility intervals (first evaluation, Codex auditing). The repository timeline is schematic.
Overview
The work has three parts. A standardized refined reproducible artifact preserves executable procedures and the reasons behind them. An agent workflow constructs these artifacts from a paper and its official repository. ReproBridge-Bench measures whether fresh coding-agent sessions reproduce experiments more reliably with them.
Four components: a refined codebase, environment setup, experiment launchers, and a reproduction guide. Together they preserve executable procedures and the reasons for their settings and repairs.
4 components · 932 ambiguity entries across 36 artifacts
Explore the artifactSix roles in three stages, plus an independent Reviewer. Humans define the workflow, delivery standards, and protocols; agents perform all construction, review, and packaging.
6 roles + Reviewer · $139.61 API usage per paper
See how it is built1,029 tasks from 387 experiments across 36 papers, each anchored by five human-verified reference scores. A separate Auditor checks execution evidence, because matching numbers alone can conceal substituted measurements.
1,029 tasks · 5 references each · audited
How success is scoredSchematicCard illustrations and flow lines explain the design; every number on the cards comes from the paper.
Success and spending are means over three evaluations; recoveries and reversals come from the first evaluation. API spending covers downstream Reproducer sessions, including failed attempts; construction, auditing, and GPU costs are excluded. Harnesses differ across vendors, so cross-model comparisons are descriptive.
A later reproducer needs to know what code to run, how to prepare its environment, how to obtain the requested measurement, and why particular choices were made. The refined reproducible artifact (RRA) addresses these needs with four components.
What code do I run?
“The refined codebase repairs execution defects and supplies missing experiment support.”
Where it lives
src/Derived source codeExample Effective Model Pruning
The released loader silently labels all 10,000 TinyImageNet validation images as one class. During construction, ReproBridge adds a script organizing images by their true labels and a launcher check requiring prepared data. In paired Gemini 3.8 Flash runs, ResNet-18 validation loss changes from 8.898 on raw code to 2.288 with the refined artifact, close to the paper’s 2.303.
Without it
87.8% success when the refined codebase is removed, −5.2 pp from the full artifact
How do I prepare the environment?
“Environment setup specifies compatible dependencies and data or checkpoint acquisition.”
Where it lives
DockerfilePreferred container setupsetup/Environment lock, installation, and data scriptsExample WBMM
For six COCO tasks, Sonnet 4.6’s raw attempt fails to install a compatible vision-library stack for H100. The refined artifact preserves the dependency versions, build prerequisite, and source-build route for the target architecture. The later session executes all six tasks without a code patch; its box and mask AP values agree numerically with the reported targets.
Without it
88.5% success when environment setup is removed, −4.5 pp from the full artifact
How do I obtain the requested measurement?
“Experiment launchers connect tasks to configurations, execution scope, outputs, and measurement extraction, distinguishing scored runs from smoke tests.”
Where it lives
experiments/Launchers and shared path setupExample HELIX
On raw code, Opus 5 restores the paper’s architecture but retains the shipped batch size and learning rate. Guide entry A6 identifies revision 9760b347, whose architecture matches the paper configuration, as the source for batch size 8 and learning rate 5.75×10−4; the --paper_config launcher makes this choice executable. A later attempt produces numerically compatible MAEs for both tasks shown in the paper (T004 and T006).
Without it
89.9% success when experiment launchers are removed, −3.1 pp from the full artifact
Why were these choices made?
“The reproduction guide, including its change record, explains settings, paper–code discrepancies, reporting statistics, and resource or numerical sensitivities.”
Where it lives
reproduction_guideline.mdReproduction guidechange_log.mdChange record and open choicesExample Mixture of Concept Bottleneck Experts
The paper leaves the reduction of multiple tree-edit distances unspecified, and Sonnet 5’s raw attempt sums the four distances, reporting 60 rather than 15. The refined artifact resolves this ambiguity by specifying the per-expression mean and supplying the collector; the later attempt reports 15.
Without it
89.2% success when the reproduction guide is removed, −3.8 pp from the full artifact
Ablation: Sonnet 5, 287 tasks, 12 papers, one evaluation; the full refined artifact reaches 93.0%. Code, environment, and launcher removals also delete descriptions of the removed component from the remaining documentation and scripts; the guide ablation removes the guide itself. Losses are conditional on the retained components. Examples are case studies from the first evaluation. Paths are the paper’s top-level entries; their grouping by component is ours.
More case studiesThe normalized refined artifact has seven top-level entries. Its guide connects instructions to their rationale through an ambiguity table, and the change record distinguishes execution fixes, correctness fixes, and convenience changes while retaining pitfalls and open choices.
src/(component: refined code)Derived source codeDockerfile(component: environment)Preferred container setupsetup/(component: environment)Environment lock, installation, and data scriptsexperiments/(component: launchers)Launchers and shared path setupreproduction_guideline.md(component: guide)Reproduction guidechange_log.md(component: guide)Change record and open choicesverify.sh(not assigned to a component)Package verification entry pointrefined codeenvironmentlaunchersguidenot assigned to a component
Entries and roles as listed in the paper; the colour grouping by component is ours.
across the 36 packaged repositories
Each ambiguity entry records its source location, the gap, the adopted resolution and rationale, sensitivity, and affected tasks.
The change record has six sections change_log.md
Field and section names are the paper’s; the paper prints no complete entry, so none is shown.
The paper’s condensed guide for SCIQL keeps all twelve section headings in their original order. It links a renamed-package issue to a compatibility launcher and a minimum-episode check, preserving both an executable route and the reason for it.
# Reproducing Table 1 of SCIQL
## What this is
- Table 1: style alignment for CBC, SCBC, and SCIQL.
- Released Circle2D and HalfCheetah datasets; code commit `5389c6e`.
## Hardware
- One GPU per repetition; this guide used NVIDIA H100 80 GB.
## Software
- Python 3.10; JAX 0.5.2; NumPy 2.2.3; PyTorch 2.5.1+cpu.
- JAX runs on the GPU; PyTorch is used for logging.
## Environment setup
```bash
bash setup/01_environment.sh
source .venv/bin/activate
DATA_ROOT=/data bash setup/02_data.sh
```
## Artifacts
- `datasets.zip`: SHA-256 checked; 1,000 episodes per Circle2D dataset.
- No released checkpoints; each experiment trains from scratch.
## Experiments
- G15: SCIQL × circle2d-inplace-v0; one seed, all six style criteria.
```bash
DATA_ROOT=/data OUTPUT_ROOT=/outputs bash experiments/run_G15.sh 0
```
## Repetitions and variance
- Run index = seed; the paper reports a mean over five seeds.
- Use distinct seeds and retain every run, including low-scoring runs.
## Outputs and metrics
- `$OUTPUT_ROOT/G15/run0/metrics.json`, key: `alignment_x100`.
- Unweighted mean over labels, then criteria, multiplied by 100.
## Paths
- `DATA_ROOT`: datasets. `OUTPUT_ROOT`: new measurements and logs.
- `src/datasets/traj2d` links into `$DATA_ROOT/episodes/traj2d`.
## Ambiguities and unspecified details
- A1: use unweighted aggregation; checked against Appendix Table 3.
- A3: released episodes refer to the old package name `style_bench`.
- Adopt aliases to `sciql.core.data` in `src/tools/sciql_launch.py`;
retain the released episode classes.
## Known pitfalls
- Symptom: exit 0 and a plausible CSV, despite loading no episodes.
- Cause: import errors are caught and skipped by the loader.
- Fix: use the compatibility launcher and keep `--min-episodes 1000`.
- Keep smoke outputs separate from full runs (`run<i>.smoke/`).
## What is not covered
- HumEnv: the released datasets and integration code are missing.
- Experiments outside Table 1.
Humans define the workflow, delivery standards, and protocols; agents perform all construction, review, and packaging. Specify fixes the intended experiment before execution; Execute establishes a working procedure and records measurements, repairs, and failures; Preserve packages that procedure and its rationale for reuse.
Inputs
Paper + code
1Specify
Task specifications
2Execute
Execution evidence
3Preserve
Reusable procedures
Refined artifact
Select a role to see its contract.
A checkpoint reviewer is reused at different stages in separate contexts. The construction Reproducer is not the fresh downstream session used to evaluate reuse.
| Role | Input and responsibility | Output and check |
|---|---|---|
| Intake | Paper, pinned raw code, selected tasks; read and map the inputs | Input manifest, experiment groups, artifact availability, initial ambiguities |
| Planner | Task mapping and ambiguities; specify the measurement before execution | Task specifications with source references, recorded choices, execution plans, draft launchers |
| Builder | Dependency and artifact requirements; construct a working environment | Locked dependencies, setup and data scripts, container definition, smoke-test evidence |
| Reproducer | Plans and verified setup; execute and record justified changes | File-backed measurements, code changes, repetition records, explicit failure reasons |
| Distiller | Execution evidence, repairs, and decisions; synthesize the handoff | Reproduction guide, change record, aggregation and variability guidance |
| Packager | Code, setup, launchers, and documents; assemble a portable deliverable | Refined artifact and clean-shell verification evidence |
| Reviewer | Plans, execution records, or completed artifact; challenge unsupported claims | Localized findings at planning, reproduction, and final handoff checkpoints |
Each measurement links to an actual output file; blocked tasks retain an explicit failure reason; diagnostic runs stay separate from scored repetitions.
When paper and code conflict, agents follow the paper’s setup and flag the discrepancy.
Supplied entry points are tested from a clean shell, including an inexpensive real repetition; human reference information is withheld throughout construction.
Opus 5 constructs the refined artifacts for the main experiment. Constructing all 36 cost $5,026 in API usage ($139.61 per paper); GPU runtime is a separate expense. When reuse pays it back →
ReproBridge-Bench provides 1,029 tasks from 387 experiments across 36 papers. Each task is anchored by five human-verified reference observations. A result succeeds only if it is numerically compatible with them and survives an independent execution-integrity audit.
The collection targets main experimental results with accessible official implementations. A task is one own-method measurement in a main table, identified by dataset, metric, reporting statistic, and location; baselines, stand-alone ablations, figures, and appendix-only results are excluded. An experiment can emit several task scores.
From the sampling manifest to the retained test papers and their tasks.
Counts and shares cover all 1,463 curated tasks from 36 papers.
Categories 1 and 2 have verified reference measurements compatible with the published score and differ at the 10% relative-variation threshold; together they supply the 1,029 downstream tasks. Categories 3–5 distinguish numerical disagreement, execution blockers, and resource/access limits.
Verification outcomes for the 36 retained papers under fixed reference rules; like the evaluation itself, they are not a conference-wide reproduction rate.
Five human reproducers on separate servers start from the same pinned raw commit, with separate execution directories and coding-agent sessions. Assuming faithful, independent executions, their mean estimates the expected result and their spread characterizes run-to-run variation. A compatibility interval built from them, fixed across raw and refined-artifact conditions, then decides numerical compatibility for every downstream attempt.
Separate servers, directories, and coding-agent sessions from one pinned commit; no reuse of another participant's scripts or measurements, or of refined artifacts or downstream traces.
Coding-agent assistance is permitted, but humans inspect the scripts, configurations, edits, and measurement extraction actually used.
Unspecified repetition and seed choices are fixed before inspecting scores; procedures are not tuned, nor valid measurements repeated, until they approach the published value.
Each observation measures the specified task statistic: a multi-seed mean or best-of-N statistic is one observation, not interchangeable with single-seed scores.
A task enters the evaluation set only if the published score p lies in I(.95). Stable tasks (Cat. 1) have V ≤ 10; higher-variability tasks (Cat. 2) exceed it.
Drag the five references, the published score p, or the new result Y, or focus a marker and use the arrow keys (Page Up/Down for steps of 1). Tight references switch on the 1% floor; q = 0.80 tightens the interval for scoring only.
Five reference observations are a protocol choice, not a guarantee of adequate uncertainty estimation. Numerical compatibility does not certify scientific validity or complete fidelity to the stated experiment.
Repeated measurements can reduce uncertainty about the mean of a fixed procedure, but cannot correct a consistently misexecuted procedure. ReproBridge aims to reduce |D| by preserving procedures and decisions for faithful execution, while ε remains.
Sonnet 4.6, Sonnet 5, Opus 5, and Gemini 3.8 Flash attempt all 1,029 tasks from 36 test papers, with one fresh session per paper–model–condition per evaluation and the same limits in both conditions.
Evaluating an explicitly permitted official checkpoint does not require repeating its training; evaluation alone cannot establish completion of a training task.
Of the 36 papers, 28 are GPU papers and eight are CPU-only papers.
All assessed by the same two-sided compatibility rule.
Task categories and numerical thresholds are applied outside the Reproducer session. An allowed checkpoint may serve as input; a shipped score file never counts as a measurement.
Harnesses differ across vendors, so cross-model comparisons are descriptive.
Matching numbers alone can conceal substituted measurements. An independent Auditor traces each reported value to its output location, execution record, and generating script, and provides no feedback to the Reproducer.
Ledger timestamps check execution and extraction order; available script and output hashes verify that the inspected files match the recorded versions.
The Auditor cross-checks executed code, logs, and extracted values to locate copied measurements, altered outputs, or substituted computations.
None of these automatically meets the conditions. Paper–code discrepancies are assessed against the same criteria.
Missing or incompatible results fail. Uncertain audit findings do not count as invalidations.
Reports replace measured epoch durations with full-dataset projections, including 32.5 seconds reported as 206 seconds; files are edited after process exit.
The measured durations also satisfied the numerical intervals, so execution evidence, not the number, revealed the substitution.
| Task | Execution record | Submitted |
|---|---|---|
| T028 | ≈ 32.5 | 206 |
| T033 | ≈ 53 | 335 |
| T038 | ≈ 92 | 579 |
Two human reviewers cross-verified all 409 selected tasks from eight randomly selected papers under both Raw and refined-artifact conditions, 818 outcomes in total, including outcomes neither Auditor flagged.
Each 409-outcome matrix has human invalid/valid rows and Auditor flag/pass columns. Both Auditors are scored against the same human labels.
| Matrix | TP | FN | FP | TN | Precision | Recall |
|---|---|---|---|---|---|---|
| Raw · Codex | 26 | 7 | 3 | 373 | 89.7% | 78.8% |
| Raw · Claude | 20 | 13 | 4 | 372 | 83.3% | 60.6% |
| RRA · Codex | 5 | 1 | 1 | 402 | 83.3% | 83.3% |
| RRA · Claude | 4 | 2 | 0 | 403 | 100.0% | 66.7% |
Each reviewer traces the measurement to primary execution evidence, records an initial judgment, and cross-checks the other's assessment before they agree on the final label.
Selection is at the paper level; task outcomes within a paper can share execution evidence and are not independent draws. Automated audits miss some violations, as these counts show.
Fresh Reproducer sessions receive the paper, the selected tasks, and either the raw codebase or its refined artifact. Across four models, refined artifacts raise mean success from 76.4% to 92.6%, narrow the cross-model spread from 21.6 to 3.9 pp, and reduce downstream API spending by 19.7% on average.
Task success with the released code (Raw) and with its refined artifact (RRA), for each Reproducer model. Switch views to weight papers equally, tighten the interval, change the Auditor, or split tasks by reference variability.
Mean over three evaluations (± SD). Success requires numerical compatibility and no confirmed audit violation. Gains range from +5.5 pp (Opus 5) to +23.2 pp (Gemini 3.8 Flash).
| Success (%) | Tokens (B) | API (USD) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Raw | RRA | Gain | Raw | RRA | Red. | Raw | RRA | Red. |
| Sonnet 4.6 | 72.3 ± 1.1 | 91.6 ± 3.4 | +19.3 | 8.48 ± 0.38 | 6.80 ± 0.76 | 19.8 | 3,165 ± 142 | 2,512 ± 286 | 20.6 |
| Sonnet 5 | 76.5 ± 2.8 | 93.3 ± 1.4 | +16.8 | 4.98 ± 0.55 | 4.52 ± 0.24 | 9.2 | 1,726 ± 175 | 1,317 ± 102 | 23.7 |
| Opus 5 | 89.2 ± 1.2 | 94.7 ± 2.2 | +5.5 | 3.01 ± 0.16 | 2.82 ± 0.29 | 6.3 | 3,480 ± 165 | 2,865 ± 312 | 17.7 |
| Gemini 3.8 Flash | 67.6 ± 1.9 | 90.8 ± 1.1 | +23.2 | 2.24 ± 0.26 | 1.83 ± 0.09 | 18.3 | 392 ± 41 | 326 ± 16 | 16.8 |
Harnesses differ across vendors, so cross-model comparisons are descriptive.
Paper figures:
Raw → refined artifact for each model, on one shared axis per metric. Refined artifacts use 6.3% to 19.8% fewer tokens and cut API spending by 16.8% to 23.7%. API savings are relative to each model's raw-code cost.
Token and API USD means (± SD) for each model are in the main results table above.
Sums over downstream Reproducer sessions, including failed attempts, means over three evaluations. Construction, auditing, and GPU costs are excluded.
Ordinary outputs are Opus 5's raw-code execution directory with code, scripts, and logs retained and result files removed; they carry no standardized guide. The complete artifact outperforms ordinary outputs by 11.8 pp, and removing any component lowers success.
Success of fresh Sonnet 5 sessions with each handoff. In the lower group, the hatched segment is the loss from the full refined artifact.
Removal protocolSelect a component to see how it was removed.
| Handoff | Success (%) | Loss (pp) |
|---|---|---|
| Raw code | 73.9 | — |
| Ordinary outputs | 81.2 | — |
| Full RRA | 93.0 | — |
| Full RRA without one component: | ||
| Refined codebase | 87.8 | 5.2 |
| Environment setup | 88.5 | 4.5 |
| Reproduction guide | 89.2 | 3.8 |
| Experiment launchers | 89.9 | 3.1 |
Sonnet 5, 287 tasks from 12 randomly selected papers, one evaluation. Losses are conditional on the components that remain. Ordinary outputs come from a separate upstream run with comparable resource use, so this comparison does not isolate reorganizing identical outputs.
On 45 CORE-Bench Hard capsules, Sonnet 5 constructs two handoffs: VERITAS's patched codebase and fix log, and a refined artifact; Sonnet 4.5 then reproduces from raw code or either handoff. A capsule succeeds only if all its questions satisfy the benchmark's reference intervals (79 questions overall).
Overall capsule success, Reproducer API USD per successful capsule, and median Reproducer session duration, then success within computer (CS), medical (Med.), and social (Soc.) sciences.
| Handoff | Success (%) | USD/success | Time (min) |
|---|---|---|---|
| Raw | 62.2 | 8.43 | 33.0 |
| VERITAS | 68.9 | 6.78 | 24.5 |
| RRA | 73.3 | 4.34 | 14.4 |
| Handoff | CS (18) | Med. (13) | Soc. (14) |
|---|---|---|---|
| Raw | 50.0 | 76.9 | 64.3 |
| VERITAS | 61.1 | 84.6 | 64.3 |
| RRA | 66.7 | 84.6 | 71.4 |
Refined artifacts achieve the highest success, reduce median duration from 33 to 14 minutes, and lower spending per successful capsule from $8.43 to $4.34. VERITAS lies between raw and refined handoffs.
Constructing the 36 artifacts costs $5,026 in API usage ($139.61 per paper). At the mean downstream costs, break-even requires 8, 13, and 9 reuse rounds for Sonnet 4.6, Sonnet 5, and Opus 5, saving $653, $409, and $615 per complete round.
Raw reuse costs n times the mean raw-code round; refined reuse adds the $5,026 construction cost once, then n times the mean refined-artifact round. The paper reports this scenario for Sonnet 4.6, Sonnet 5, and Opus 5.
| Model | Raw / round | RRA / round | Saving / round | Break-even (rounds) |
|---|---|---|---|---|
| Sonnet 4.6 | 3,165 | 2,512 | 653 | 8 (7.70) |
| Sonnet 5 | 1,726 | 1,317 | 409 | 13 (12.29) |
| Opus 5 | 3,480 | 2,865 | 615 | 9 (8.17) |
Each round repeats all 36 papers. The scenario charges the full construction cost to one model, holds prices and savings fixed, and excludes GPU, auditing, and maintenance costs.
The main comparison pools tasks, so papers with more measurements weigh more. Paper by paper in the first evaluation, success improves or ties on 30 to 33 papers per model, while three to six papers regress. The coverage chart near the top of the page groups the same 36 papers by research area.
Paper-macro rates average the per-paper percentages with equal weight. All four models improve under this aggregation, with gains of 6.7 to 20.7 pp. The aggregate gains extend across papers, although refinement does not improve every paper; the case studies examine the remaining failures.
Each point is one paper; marker area is proportional to its task count. Points above the dashed line improve with the refined artifact. Eval 1 · Codex
Hover or focus a point for its counts; click or press Enter to select the paper above. Overlapping points keep their exact positions.
| ID | Paper | Area | Tasks | Sonnet 4.6 | Sonnet 5 | Opus 5 | Gemini 3.8 Flash |
|---|---|---|---|---|---|---|---|
| 1 | TESSFrom Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space | General ML & theory | 12 | 9 → 9, Unchanged | 9 → 9, Unchanged | 11 → 9, Decrease | 12 → 9, Decrease |
| 2 | SCIQLOffline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment | Reinforcement learning & robotics | 15 | 2 → 15, >50 pp | 15 → 15, Unchanged | 15 → 15, Unchanged | 10 → 15, 25–50 pp |
| 7 | ExactFANOVAExact Functional ANOVA Decomposition for Categorical Inputs Models | Trustworthy ML | 82 | 75 → 76, 0–10 pp | 75 → 81, 0–10 pp | 80 → 77, Decrease | 62 → 75, 10–25 pp |
| 20 | EnergyDPSA Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principle | Trustworthy ML | 12 | 10 → 12, 10–25 pp | 2 → 12, >50 pp | 9 → 12, 10–25 pp | 11 → 9, Decrease |
| 24 | LM Tail RiskEstimating Tail Risks in Language Model Output Distributions | Trustworthy ML | 3 | 3 → 3, Unchanged | 3 → 2, Decrease | 3 → 3, Unchanged | 3 → 2, Decrease |
| 35 | DivInInitialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posterior | Deep learning | 40 | 9 → 40, >50 pp | 28 → 40, 25–50 pp | 40 → 40, Unchanged | 16 → 40, >50 pp |
| 40 | HELIXHELIX: Hybrid Encoding with Learnable Identity and Cross-dimensional Synthesis for Time Series Imputation | General ML & theory | 9 | 6 → 9, 25–50 pp | 8 → 9, 10–25 pp | 4 → 9, >50 pp | 5 → 6, 10–25 pp |
| 50 | Control Consistency BridgesControl Consistency Losses for Diffusion Bridges | Probabilistic methods | 18 | 15 → 13, Decrease | 14 → 16, 10–25 pp | 15 → 16, 0–10 pp | 11 → 16, 25–50 pp |
| 58 | Chebyshev PoliciesChebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-Dimensional Control Tasks | Reinforcement learning & robotics | 22 | 20 → 20, Unchanged | 18 → 21, 10–25 pp | 20 → 21, 0–10 pp | 22 → 22, Unchanged |
| 66 | Learning Randomized ReductionsLearning Randomized Reductions | General ML & theory | 38 | 18 → 35, 25–50 pp | 16 → 32, 25–50 pp | 24 → 35, 25–50 pp | 9 → 33, >50 pp |
| 69 | ATLASAdaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks | Evaluation | 82 | 71 → 82, 10–25 pp | 55 → 82, 25–50 pp | 76 → 82, 0–10 pp | 50 → 75, 25–50 pp |
| 71 | XDLMBalancing Understanding and Generation in Discrete Diffusion Models | Deep learning | 21 | 21 → 21, Unchanged | 18 → 21, 10–25 pp | 21 → 21, Unchanged | 15 → 21, 25–50 pp |
| 81 | FlashOptimFlashOptim: Memory Efficient Optimizers for Large-Scale Training | ML systems & optimization | 30 | 8 → 30, >50 pp | 9 → 30, >50 pp | 30 → 30, Unchanged | 0 → 29, >50 pp |
| 84 | LiMELiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning | Deep learning | 19 | 4 → 0, Decrease | 8 → 4, Decrease | 17 → 16, Decrease | 4 → 0, Decrease |
| 90 | Compositional ExplanationsGuaranteed Optimal Compositional Explanations for Neurons | Trustworthy ML | 21 | 0 → 6, 25–50 pp | 8 → 14, 25–50 pp | 14 → 20, 25–50 pp | 3 → 10, 25–50 pp |
| 95 | Manifold-Aware PerturbationsManifold-Aware Perturbations for Constrained Generative Modeling | Deep learning | 60 | 54 → 54, Unchanged | 54 → 55, 0–10 pp | 55 → 55, Unchanged | 54 → 54, Unchanged |
| 96 | Any-Order GPTAny-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture | Deep learning | 26 | 22 → 25, 10–25 pp | 23 → 23, Unchanged | 25 → 24, Decrease | 18 → 21, 10–25 pp |
| 112 | AuxiliaryMCMCMarkov Chain Monte Carlo without Evaluating the Target: an Auxiliary Variable Approach | Probabilistic methods | 19 | 19 → 7, Decrease | 15 → 0, Decrease | 18 → 0, Decrease | 12 → 12, Unchanged |
| 113 | DELTA4DELTA4: Sparse Matrix-Vector Multiplication for Low Sparsity | ML systems & optimization | 15 | 15 → 15, Unchanged | 15 → 15, Unchanged | 15 → 15, Unchanged | 12 → 15, 10–25 pp |
| 116 | ETTFSEfficiently Training Time-to-First-Spike Spiking Neural Networks from Scratch | Deep learning | 11 | 6 → 11, 25–50 pp | 4 → 11, >50 pp | 6 → 11, 25–50 pp | 7 → 11, 25–50 pp |
| 121 | Effective Model PruningEffective Model Pruning: Measuring the Redundancy of Model Components | ML systems & optimization | 46 | 46 → 46, Unchanged | 46 → 46, Unchanged | 46 → 46, Unchanged | 42 → 46, 0–10 pp |
| 124 | OC-spaceOC-space: a Unifying Perspective on Verification of Tree Ensembles | Trustworthy ML | 15 | 5 → 13, >50 pp | 6 → 14, >50 pp | 11 → 13, 10–25 pp | 10 → 11, 0–10 pp |
| 126 | MindOmniMind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion | Application-driven ML | 53 | 43 → 52, 10–25 pp | 42 → 53, 10–25 pp | 46 → 53, 10–25 pp | 53 → 46, Decrease |
| 129 | KPE/KTSA Kinetic-Energy Perspective of Flow Matching | Deep learning | 20 | 15 → 19, 10–25 pp | 20 → 19, Decrease | 18 → 20, 0–10 pp | 20 → 18, Decrease |
| 134 | LaST0LaST₀: Latent Spatio-Temporal Chain-of-Thought for Robotic Vision-Language-Action Model | Reinforcement learning & robotics | 16 | 13 → 16, 10–25 pp | 12 → 16, 10–25 pp | 13 → 16, 10–25 pp | 8 → 16, 25–50 pp |
| 135 | Mixture of Concept Bottleneck ExpertsMixture of Concept Bottleneck Experts | Trustworthy ML | 22 | 16 → 22, 25–50 pp | 17 → 22, 10–25 pp | 22 → 22, Unchanged | 10 → 22, >50 pp |
| 136 | TACOEnd-to-End Compression for Tabular Foundation Models | ML systems & optimization | 45 | 41 → 44, 0–10 pp | 41 → 44, 0–10 pp | 42 → 44, 0–10 pp | 32 → 44, 25–50 pp |
| 141 | POET-XPOET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation | ML systems & optimization | 57 | 49 → 57, 10–25 pp | 52 → 57, 0–10 pp | 57 → 57, Unchanged | 52 → 55, 0–10 pp |
| 145 | PonderLM-2PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space | Deep learning | 38 | 37 → 38, 0–10 pp | 38 → 38, Unchanged | 38 → 38, Unchanged | 37 → 38, 0–10 pp |
| 149 | PSAHSProgressive Graph Structure Adjustment for Homophily Shift Adaptation | General ML & theory | 15 | 7 → 15, >50 pp | 12 → 15, 10–25 pp | 15 → 15, Unchanged | 15 → 15, Unchanged |
| 169 | PLACyRobust Causal Discovery in Real-World Time Series with Power-Laws | Trustworthy ML | 23 | 21 → 23, 0–10 pp | 23 → 23, Unchanged | 23 → 23, Unchanged | 21 → 23, 0–10 pp |
| 187 | SlaClipSlaClip: Gradient Norm Slacks can be Indicator for Adaptive Clipping in DP-SGD | Trustworthy ML | 30 | 9 → 24, 25–50 pp | 9 → 30, >50 pp | 12 → 30, >50 pp | 9 → 30, >50 pp |
| 198 | SurvDiffSurvDiff: A Diffusion Model for Generating Synthetic Data in Survival Analysis | Application-driven ML | 14 | 12 → 14, 10–25 pp | 14 → 14, Unchanged | 11 → 14, 10–25 pp | 13 → 14, 0–10 pp |
| 223 | Sequential Data ValuationUnifying and Optimizing Data Values for Selection via Sequential Decision-Making | General ML & theory | 13 | 9 → 13, 25–50 pp | 9 → 13, 25–50 pp | 13 → 13, Unchanged | 13 → 13, Unchanged |
| 230 | Weak-and-Strong VerificationWhen to Trust the Cheap Check: Weak and Strong Verification for Reasoning | General ML & theory | 9 | 9 → 9, Unchanged | 9 → 9, Unchanged | 9 → 9, Unchanged | 0 → 9, >50 pp |
| 238 | WBMMWBMM: Windowed Batch Matrix Multiplication for Efficient Large Receptive Field Convolution | ML systems & optimization | 58 | 23 → 57, >50 pp | 40 → 57, 25–50 pp | 44 → 57, 10–25 pp | 25 → 57, >50 pp |
| All 36 papers | 1,029 | 742 → 945 | 787 → 962 | 918 → 981 | 696 → 932 | ||
First evaluation, primary Codex scoring. These are agent reproductions under this protocol on retained papers, not a conference-wide reproduction rate.
Across paired task–model outcomes in the first evaluation, 813 raw-code failures become successes, against 136 reversals. The records suggest two complementary contributions: completing ambiguous or incomplete measurement specifications, and preserving executable procedures for reuse.
Paired task–model outcomes, 1,029 tasks per model. Recovery: Raw failure to RRA success. Reversal: Raw success to RRA failure. Net is recoveries minus reversals. Eval 1 · Codex
| Model | Raw | RRA | Recoveries | Reversals | Net |
|---|---|---|---|---|---|
| Sonnet 4.6 | 742 | 945 | 234 | 31 | 203 |
| Sonnet 5 | 787 | 962 | 208 | 33 | 175 |
| Opus 5 | 918 | 981 | 100 | 37 | 63 |
| Gemini 3.8 Flash | 696 | 932 | 271 | 35 | 236 |
| Total | 3,143 | 3,820 | 813 | 136 | 677 |
Harnesses differ across vendors, so cross-model comparisons are descriptive. Counts span four models, so a task can appear in several pairs. Per-model outcome splits appear only as bar segments in the paper figure.
With refined artifacts, model–paper sessions that change code downstream fall from 113/144 to 54/144, and median time to first scored execution decreases for every model. “Together with lower token use, this indicates less reconstruction.”
Median minutes across paper sessions. Eval 1
| Model | Raw (min) | RRA (min) |
|---|---|---|
| Sonnet 4.6 | 30.0 | 16.2 |
| Sonnet 5 | 54.0 | 31.8 |
| Opus 5 | 32.4 | 19.2 |
| Gemini 3.8 Flash | 43.8 | 15.6 |
Median time to first scored execution decreases for every model.
Out of 36 papers per model. Eval 1
| Model | Raw | RRA |
|---|---|---|
| Sonnet 4.6 | 26 | 11 |
| Sonnet 5 | 30 | 14 |
| Opus 5 | 24 | 16 |
| Gemini 3.8 Flash | 33 | 13 |
All four models 113/144 to 54/144 model–paper sessions
Time is the median to the first scored execution, not end-to-end runtime. Patch counts are papers with a Reproducer code diff, not independent defects.
One card per case the paper describes: what the released code (Raw) left implicit, what the refined artifact (RRA) preserved, and the recorded outcome where the paper gives one. Quoted lines are the paper's own words.
“The case locates the discrepancy in the underspecified reporting statistic.”
From the paper · Appendix, worked reproduction examplesA measurement is not just a model execution: Mixture of Concept Bottleneck Experts.
For four-expert settings in Mixture of Concept Bottleneck Experts, the paper does not specify how multiple expression distances should be reduced, and the repository provides no Table 1 collector. The raw Sonnet 5 attempt constructs a matching between learned and reference expressions but reports the sum of tree-edit distances. For two tasks this produces 60 and 188, whereas the per-expression means are 15 and 47. The refined artifact resolves the ambiguity by documenting the denominator and implementing the corresponding collector; those downstream values are 15 and 47. The case locates the discrepancy in the underspecified reporting statistic.
“Criterion selection, output mapping, and repetition change together in this case, illustrating preservation of the complete measurement protocol.”
From the paper · Appendix, worked reproduction examplesPreserving the experimental aggregation: SCIQL.
The released SCIQL code produces run outputs but does not assemble the Table 1 statistic, leaving the mapping from logged fields and experiment settings to reported values incomplete. The raw Sonnet 4.6 attempt uses one default criterion, one seed, and an incorrect output column for a table statistic that averages across six criteria and five seeds. The refined artifact's launcher and collector make this mapping explicit by specifying the sweep and the correct alignment field. One representative measurement changes from 19.70 to 55.66 against a paper value of 58.6. Criterion selection, output mapping, and repetition change together in this case, illustrating preservation of the complete measurement protocol.
The paper also uses SCIQL's refined artifact to illustrate all four components; its guide entries specify the six-criterion, five-seed protocol and unweighted aggregation.
9760b347, whose architecture matches the paper; the --paper_config launcher makes the choice executable.“The guide records the evidence for the choice, and the launcher makes it reusable.”
From the paper · Appendix, worked reproduction examplesRecovering omitted configuration: HELIX.
For HELIX, Opus 5 restores the paper's architecture on raw code but retains a shipped batch size of 32 and learning rate of approximately 6.29×10−4. The guide's ambiguity entry A6 identifies revision 9760b347, whose architecture matches the paper configuration, as the source for batch size 8 and learning rate 5.75×10−4. The --paper_config launcher makes this choice executable. The recorded MAE changes from 0.15265 to 0.12986 for T004 and from 0.50527 to 0.41160 for T006; the respective compatibility intervals are [0.12524, 0.13448] and [0.38442, 0.43878]. The artifact records the repository history and architectural match as the rationale for adopting these settings.
mmcv-full stack for H100.| Box AP | 51.61 against reported 51.6 | 52.78 against reported 52.8 |
|---|---|---|
| Mask AP | 44.78 against reported 44.8 | 45.64 against reported 45.6 |
All six tasks execute, with no code patch.
“This case shows how an environment repair can become a reusable installation procedure.”
From the paper · Appendix, worked reproduction examplesAn environment route that transfers: WBMM.
For six COCO tasks (T063–T068) in WBMM, Sonnet 4.6's raw attempt fails to install a compatible mmcv-full stack for H100. The refined artifact specifies matching PyTorch/MMDetection dependencies, the Ninja build prerequisite, and a source build for the target architecture. The downstream refined-artifact session makes no code patch and executes all six tasks. Its two box AP values, 51.61 and 52.78, agree numerically with reported targets 51.6 and 52.8; mask AP is 44.78/45.64 against 44.8/45.6. The artifact preserves an executable dependency route for later reuse.
Without the prepared environment. In the component ablation that removes this route, Sonnet 5 reconstructs a Python 3.10 / PyTorch 2.0 / CUDA 11.8 stack, and a supplied binary lacking an operator's H100 kernels needs a source rebuild. “No paper-source patch is required, yet substantial environment work returns downstream.”
“This records a concrete repair and its downstream reuse.”
From the paper · Appendix, worked reproduction examplesFixing validation labels: Effective Model Pruning.
The released TinyImageNet loader uses ImageFolder on a validation directory containing only an images subdirectory. It therefore assigns all 10,000 validation images to one class. During construction, ReproBridge adds a preparation script that places images into class directories using val_annotations.txt, verifies 200 classes and 10,000 images, and writes a preparation receipt required by the launcher. In Gemini 3.8 Flash's paired runs, the ResNet-18 validation loss is 8.898 with raw code and 2.288 with the refined artifact, against the paper's 2.303. This records a concrete repair and its downstream reuse.
Code/profile_ttfs.py, documents the metric definitions and output fields, and supplies a launcher that trains a checkpoint before profiling it. A fresh Opus 5 session follows this route.6,377 lines searched; no producing implementation in the release
“Explicit procedures also help the model that built them.”
From the paper · Appendix, worked reproduction examplesOrganizing measurement code for reuse: ETTFS.
The raw Opus 5 attempt searches the released ETTFS source for the metrics in Tables 2 and 3, finds no producing implementation, and leaves the inference-time measurement unexecuted. During construction, ReproBridge adds Code/profile_ttfs.py, documents the metric definitions and output fields, and supplies a launcher that trains a checkpoint before profiling it. The later Opus 5 session uses this explicit route and reports 2.0652 inference time steps against the paper's 2.03. The artifact localizes code and resolved decisions that would otherwise require renewed investigation.
Opus 5 also constructs the refined artifacts for the main experiment. “Clearer code organization and explicit execution decisions can reduce repeated search and context demands, even for the model that built the artifact.”
The construction report lists 6,346 added lines of experiment support.
“An optimizer release is not a complete experiment.”
From the paper · Appendix, worked reproduction examplesAn optimizer release is not a complete experiment: FlashOptim.
FlashOptim supplies optimizer code but not the GPT-2 training and evaluation harness for the selected table. Gemini's raw attempt leaves 18 GPT-2 outcomes unexecuted. The refined artifact adds a training/evaluation harness, data acquisition, output mapping, and launchers; all 18 pass in its recorded downstream attempt. The construction report lists 6,346 added lines of experiment support.
collate_fn, but the trailing partial batch is returned as a list, so the evaluator's batch.items() fails when the final batch is reached.yield collate_fn(batch); the record traces the sole caller to validation and describes CPU checks.examples/
tailyield collate_fn(batch)
CPU checksemptypartialcompleteover-full
“This example illustrates how a later reproducer can recover both the fix and its experimental scope.”
From the paper · Appendix, worked reproduction examplesA localized repair: POET-X.
The defect occurs in examples/peft_pretraining/training_utils.py, line 222. Full batches call collate_fn, whereas the trailing partial batch is returned as a list. The evaluator calls batch.items(), so it fails when the final batch is reached. The patch changes the tail to yield collate_fn(batch). The record traces the sole caller to validation and describes CPU checks covering empty, partial, complete, and over-full batch counts, without changing the example count. This example illustrates how a later reproducer can recover both the fix and its experimental scope.
driver · training scripts · 171 committed manifests
“This gives authors a concrete inconsistency to inspect and resolve before submission.”
From the paper · Appendix, worked reproduction examplesFlagging a setup discrepancy: Manifold-Aware Perturbations.
The guide's ambiguity entry A1 contrasts the paper's stated learning rate of 10−4 with the released driver, training scripts, and 171 committed manifests using 10−3, and marks the discrepancy as high sensitivity. This gives authors a concrete inconsistency to inspect and resolve before submission.
“When the paper and released code conflict, ReproBridge strictly follows the paper's setup and flags the discrepancy with source locations and affected tasks.” And from the separate alignment analysis: “A learning-rate disagreement is concrete even without retraining: one can point to the paper's 10−4 and the code's 10−3. But the resulting change in final accuracy cannot be inferred from that factor alone.”
“These records illustrate runtime constraints, numerical edge cases, and deviations from the prescribed measurement procedure.”
From the paper · Appendix, failure modes and concentrated regressionsAuxiliaryMCMC: numerical fragility interacts with a finite budget.
The Opus 5 refined-artifact Gaussian sweep for AuxiliaryMCMC targets 100 rounds at each of three acceptance rates. Its measured workload exceeds the available clock at the chosen concurrency, and the agent executes only 55 rounds per rate. The primary audit invalidates numerical matches affected by this altered measurement procedure. Separately, one Poisson-MALA chain remains constant with zero acceptance. Its undefined effective sample size propagates through the evaluator's ordinary mean, leaving three measurements unavailable. The Reproducer declines to redraw only the failed chain, which would select a repetition based on its outcome. Effective sample size per second also varies with host load and concurrency.
The raw attempt also runs fewer than 100 rounds in part of its sweep. These records illustrate runtime constraints, numerical edge cases, and deviations from the prescribed measurement procedure.
“A paper can contain both recovered and lost tasks: Gemini's AuxiliaryMCMC total remains 12 successes in each condition despite five reversals offset by five recoveries.”
Across models, 13 of LiME's 14 reversals are resource/access failures.
“Setup success is not experiment completion.”
From the paper · Appendix, failure modes and concentrated regressionsLiME: setup success is not experiment completion.
In Sonnet 5's refined-artifact attempt on LiME, data staging and environment preparation take approximately 83 minutes, but the full schedule includes 5,949 training steps followed by evaluation over 47 splits. The attempted run's observed throughput projects 7 to 92 wall-clock hours per variant, competing within the paper's 24-hour allocation. Model-construction parameter counts are available before training finishes, whereas accuracy measurements require completed training and evaluation. The session reaches the former but not the latter. Across models, 13 of LiME's 14 reversals are resource/access failures and one is a numerical mismatch.
GuideComponent named in the paper's case textGuideMapped from the paper's component definitions
Case annotations describe patterns rather than isolate causal effects. Opus 5 constructs every refined artifact in the main experiment; case measurements come from the first evaluation.
An agent-assisted case review assigns primary explanations across 101 model–paper pairs, grounded in execution records. In the paper's words, clarification and experiment-to-result mapping “together represent about 35% of the case-review attribution distribution.”
Share of the case review, and estimated outcomes when the shares are allocated across the 813 recoveries. Eval 1 · 101 pairs
An agent-assisted case review of 101 model–paper pairs; estimated counts allocate the reviewed shares across the 813 recoveries and are not independent causal measurements. Integrity is an explanation; audit-invalid names a scored outcome.
Failures shared by both handoffs mainly reflect paper–code setup mismatches, typographical errors in the paper, or experimental variability placing measurements outside the compatibility interval. Refinement can also preserve or introduce incorrect experimental choices.
The 84 numerical reversals span several possible mechanisms: stochastic training, timing conditions, changed execution choices, and unresolved paper–code disagreements. Construction may also introduce an incorrect setting, metric interpretation, or repair that later agents inherit. Diagnosing a particular reversal requires tracing its executed configuration and measurement procedure.
Reversals by paper
Four papers account for 93 of the 136 reversals (68.4%).
Bar lengths are shares of all 136 reversals across the four models. AuxiliaryMCMC and LiME alone account for 47.1%.
Reproduction success depends jointly on artifact quality and reproducer ability. ReproBridge is primarily intended as a reproducibility check before a paper is submitted and its artifacts are released, and the same format lets third-party reproducers preserve their work to lower the cost of later attempts.
The primary intended use is for authors to apply ReproBridge as a reproducibility check before submitting a paper and releasing its artifacts. Executing the paper's specified setup can expose missing details, execution barriers, and paper–code inconsistencies, giving authors concrete issues to inspect and correct.
Third-party reproducers can preserve their work in the same format to lower the cost of later attempts. A human reader can use the setup route and launchers to execute experiments, then inspect the guide and change record to assess the settings and repairs.
Our experiments measure reuse by coding agents.
When the paper and released code conflict, ReproBridge strictly follows the paper's setup and flags the discrepancy with source locations and affected tasks. Such records help authors check consistency between their manuscript and implementation before submission.
Manifold-Aware Perturbations
Learning rate
A separate agent-assisted static analysis compares selected paper results with their raw implementations. A contradiction requires an explicit paper statement and conflicting implemented behavior; absent implementations and choices on which the paper is silent are recorded separately.
Of 293 recorded findings, 238 are in scope for contradiction analysis; 44 absent-implementation records and 11 paper-silent records are kept separate.
In-scope contradiction findings by recorded category, with the papers and distinct tasks each category touches. Scope is determined from evidence, not category names. Agent-reviewed records
| Category | Findings | Papers | Distinct tasks |
|---|---|---|---|
| Hyperparameters | 41 | 26 | 287 |
| Objective | 41 | 22 | 400 |
| Data | 40 | 20 | 298 |
| Architecture | 33 | 18 | 209 |
| Evaluation procedure | 30 | 20 | 261 |
| Training procedure | 22 | 16 | 176 |
| Metric | 21 | 13 | 112 |
| Baseline implementation | 5 | 3 | 4 |
| Divergent implementation choices | 3 | 3 | 15 |
| Scale / hardware | 2 | 2 | 25 |
Agent-reviewed records. Tasks and papers overlap across categories and must not be summed. These findings are possible sources of execution deviation, not human-validated error rates or explanations of the measured gains; their effects on measured performance require controlled comparisons. They were not supplied to the evaluated Reproducers.
The limitations stated in the paper's discussion, each linked to the part of this page it qualifies.
The evaluation conditions on retained papers, fixed reference regions, and the tested agents; it is not a conference-wide reproduction rate. Five reference observations are a protocol choice, not a guarantee of adequate uncertainty estimation.
The 36 papers were retained from a random sample of 50 of the 241 ICML 2026 Spotlights with code and non-purely-theoretical content.
Related section: 02 · BenchmarkNumerical compatibility does not certify scientific validity or complete fidelity to the stated experiment, and refined artifacts can preserve incorrect assumptions.
Success requires numerical compatibility and no confirmed audit violation.
Related section: 02 · BenchmarkAutomated audits miss some violations, as quantified against human verification.
Of the 39 invalid outcomes identified in human review of 818 outcomes, Codex misses eight and Claude misses 15.
Related section: 02 · BenchmarkOrdinary outputs and refined artifacts come from separate runs with comparable upstream resource use, so the comparison evaluates the construction workflow and resulting handoff together; it does not isolate the effect of reorganizing identical outputs. Component ablations measure losses conditional on the remaining materials.
Related section: 03 · ResultsCase annotations and variability strata describe patterns rather than isolate causal effects.
Explanations for recovered outcomes come from an agent-assisted case review of 101 model–paper pairs; their counts are not independent causal measurements.
Related section: 05 · CasesAPI savings concern downstream Reproducer sessions. Construction, independent verification, GPU execution, maintenance, and human review have separate costs; reuse estimates assume fixed prices and savings. Human time savings and comprehension require a separate study.
Related section: 03 · ResultsFuture workEvaluate reuse by human researchers, and measure the full costs of constructing, verifying, and maintaining artifacts across repeated attempts.
Thanks for reading
Tap as often as you like; every tap counts. Views and likes are counted anonymously, without cookies.
Release
The paper is on arXiv as 2611.08452. The ReproBridge code and the ReproBridge-Bench dataset are going through internal review and will be open-sourced once it is approved.
Full paper with appendices: protocols, inventories, and worked examples.
Read the ReproBridge paper (PDF)Abstract page, versions, and citation.
Open the arXiv page of the paperThe agent workflow that turns a paper and its released code into a refined reproducible artifact. Under internal review; open-sourced once approved.
In review1,029 tasks from 387 experiments across 36 papers, with five human-verified reference scores per task. Under internal review; open-sourced once approved.
In review@article{wang2026reprobridge, title = {{ReproBridge}: Building Reproducibility into {ML} Research Artifacts}, author = {Wang, Yubo and Meng, Rui and Nam, Jaehyun and Pfister, Tomas and Yoon, Jinsung}, journal = {arXiv preprint arXiv:2611.08452}, year = {2026}}