ARC-AGI-3 Comparative Performance Study

Interpretation

What can reasonably be concluded

The study establishes substantial task coverage across the ARC-AGI-3 Public Demo, including complete wins in environments where GPT-5.6 Sol Max had scored at or near zero under the standard ARC harness. OpenAI later showed that the model itself could perform much better when the surrounding context system was changed, raising its aggregate Public Demo RHAE from 13.3% to 38.3% without changing the model weights.

It does not establish a controlled causal effect size for the proprietary configuration. The study was exploratory rather than a frozen official submission, so the result should be treated as developmental evidence rather than a replacement for ARC Prize's verified RHAE evaluation.

Evaluation boundary

The underlying model used in the study was GPT-5.6 Sol. The environments were treated as black boxes, and interaction was limited to rendered observations, permitted actions, and their consequences. Environment source code, external solutions, walkthroughs, and answer keys were excluded from the clean comparison. Cases with an integrity or experimental-boundary issue were separately identified rather than silently included.

The proprietary implementation used in this study is not disclosed publicly. This page is limited to the conditions necessary to interpret the results, the evidence that was preserved, and the boundaries of the claims being made.

Why there is no reconstructed RHAE score

ARC-AGI-3 evaluates completed levels partly according to the number of actions required relative to a human baseline. Actions used while exploring an unfamiliar environment therefore matter to the official score.

The research archive contains solved-state records, transition hashes, checkpoints, and many exact clean-solution traces. It does not contain a complete official scorecard preserving every first-exposure action across the full exploratory run. A later clean replay cannot substitute for that missing history because doing so would remove exploration that ARC-AGI-3 is designed to measure.

The 47.5% weighted completion figure is reported separately rather than replacing missing first-exposure data with estimates.

Preserved solution evidence

The archive contains defensible clean-solution action counts for 31 solved levels. Twenty-six of those levels have exact preserved action sequences or traces. Across those 31 solutions, the median clean-solution length is 26 actions.

Those records answer a different question from RHAE. Their value is reproducibility: once a solution had been established, many successful results could be preserved and verified as deterministic action sequences.

Why GPT-5.6 Sol is the comparison

GPT-5.6 Sol is the relevant reference point because it is the underlying model family evaluated in this study. This page does not claim current state-of-the-art ARC-AGI-3 performance. Later model generations and evaluation harnesses have produced substantially stronger official results.

OpenAI's own 13.3% to 38.3% result is important context for this study because it demonstrates that ARC-AGI-3 performance can change materially when the system surrounding the same model changes. This study examines a separate proprietary configuration and reports a different completion-based metric, so its 47.5% index should not be treated as an RHAE score or as a direct numerical improvement over 38.3%.

Next research step

The current benchmark record is frozen. A future evaluation should use a fixed configuration and an official scoring harness from the first action so that completion, action efficiency, and reproducibility can be evaluated together rather than reconstructed afterward.