Reference-based grading / CHIP-8 / 2026-09-17
Grade an emulator by comparing its frame to a reference implementation's and you need a similarity measure. SSIM is the usual answer — it is what GBA Eval settled on after rejecting four alternatives. On the test below, SSIM scores a correct interpreter at 0.9375 and an emulator that draws nothing at all at 0.9688. Pixel proportion and GMSD rank them the same way. Every number here recomputes from the repository.
That failure has a boundary, and it is in closed form. It needs frames where nearly all the drawn content moves and the display is sparse — and a real GBA frame is in neither regime. The measurement that establishes that is below, along with the prediction of mine it falsified.
SSIM 1.0000
SSIM 0.9375
SSIM 0.9688
The middle screen is not a bug. Two real CHIP-8 conventions disagree about one instruction, so the glyph is drawn four pixels apart — and both are right. The grader ranks that legitimate difference below a submission that renders an empty display.
A grader like this is the reward signal in a reinforcement-learning environment. If it ranks a correct implementation below one that does nothing, then the compute spent on that rollout is not wasted — it is actively teaching the model that correct behaviour is wrong. Mechanize argues that cheap RL tasks waste compute; this is the same argument one level down, about the grader rather than the task.
What makes this measurable rather than hypothetical is that the divergence comes from a real platform disagreement, not a synthetic perturbation. And what makes it a demonstration rather than a discovery is stated plainly below.
8XY6 is a shift. The original COSMAC VIP shifts VY into
VX; CHIP-48 and SUPER-CHIP shift VX in place. Both are
documented, and interpreters in the wild do both. This ROM computes an x position with
that instruction and draws the font glyph 0 there — so a correct interpreter
draws the same 14 pixels at x=4 or x=8, depending only
on which convention it follows.
The ROM's design and the predictions about it were registered before the bytes existed, and it had to pass four acceptance gates before any similarity measure touched it. Two more ROMs, covering different kinds of divergence, are below.
: main
v1 := 0x10
v2 := 0x08
v1 >>= v2 # VIP: v1 = v2>>1 = 4
# CHIP-48: v1 = v1>>1 = 8
v0 := 0
i := hex v0 # the interpreter's OWN font
v3 := 0
sprite v1 v3 5
loop again
| Candidate | Exact | Pixel prop. | Thresholded | GMSD | SSIM |
|---|---|---|---|---|---|
| Reference (VIP) | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| Correct, CHIP-48 convention | 0.0000 | 0.9863 | 1.0000 | 0.8492 | 0.9375 |
| Blank screen | 0.0000 | 0.9932 | 1.0000 | 0.8730 | 0.9688 |
| Inverted palette | 0.0000 | 0.0000 | 0.0000 | 1.0000 | −0.0160 |
Amber marks a score the measure gave a cheat; red marks the correct implementation it was ranked above. No threshold separates correct from cheating under any of the five. Exact match rejects both correct frames. Thresholding at τ=.05 accepts the blank screen. And GMSD scores a wholly inverted frame at exactly 1.0000, because gradient magnitude is invariant under inversion — |∇(1−x)| = |∇x| — so every edge sits where it did and the measure reports perfection for an image wrong in every pixel.
SSIM pools over blocks. Moving the glyph four pixels disturbs two 8×8 blocks — the one it left and the one it entered — while erasing it entirely disturbs one. A measure that averages over blocks therefore penalises relocation more than deletion. That is a property of the measure, not of this ROM.
The prediction that follows is falsifiable, so it was registered and run: a divergence that changes what is drawn without moving it should score better. It does.
vs blank 0.9375
vs blank 0.9943
vs blank 0.9688
| ROM | Quirk | Kind | Blocks | Worst correct (SSIM) | Blank | Separates? |
|---|---|---|---|---|---|---|
| A · shift.ch8 | 8XY6 shift source | relocation | 2 | 0.9375 | 0.9688 | none of the five |
| B · digit.ch8 | FX55 index increment | substitution | 1 | 0.9943 | 0.9688 | pixel prop., SSIM |
| C · wrap.ch8 | sprite wrap vs clip | partial addition | 1 | 0.9688 | 0.9375 | pixel prop., SSIM |
| D · jump.ch8 | BNNN jump offset | relocation | 2 | 0.9375 | 0.9688 | none of the five |
| E · vfreset.ch8 | 8XY1 VF reset | substitution | 1 | 0.9943 | 0.9688 | pixel prop., SSIM |
D and E are replications, registered as such before they existed. ROM D
reaches relocation through a different instruction entirely — BNNN adds
V0 on the VIP and VX on CHIP-48, so the two conventions jump to
different instructions, and the VIP path has to leap over the CHIP-48 path or it
falls through and overwrites its own answer. It reproduces ROM A to four decimal places. ROM
E reaches substitution through a register that cannot be drawn at all, routing the VF flag
into the font index so the two conventions render 0 or 3 in the
same place; it reproduces ROM B. Geometry predicts the outcome; the quirk that
produced it does not. Either one separating differently would have meant the
mechanism was wrong.
This narrows the finding and is the more useful version of it: block-pooled
similarity has a predictable blind spot for divergences that move something. A quirk
that changes where output is drawn scores worse than drawing nothing; a quirk that
changes what is drawn in the same place is scored correctly. ROM C also turned up
a defect in two interpreters that had passed every other ROM available here — they place
wrapped columns one row low, computing the framebuffer index as
(x + i + (y+h)*64) % 2048.
A 24-pixel shift scores identically to a 4-pixel one — six times the distance, 0.9375 either way — because both disturb the same two blocks. The score declines only while the glyph positions still overlap, then goes constant. And SSIM sits below the blank screen at every non-zero displacement, so the failure is not an artefact of the magnitude first chosen. Predicted before the run, on the mechanism above.
The obvious objection to everything above is that a 64×32 display with 14 lit pixels is 0.68% lit, so a blank frame is already 99.3% pixel-correct before any metric runs. Earlier drafts of this page named that objection and left it unanswered. Answering it needs two parameters: the lit fraction d, and φ, the fraction of drawn content the divergence moves.
I registered the boundary as φ > 0.5, independent of density, and called it arithmetic rather than a guess. It was wrong. It dropped the collision term — displaced content does not always land on dark pixels, and near half-lit most of it lands on pixels that were already on. Keeping the term:
blank frame differs in d·N displaced frame differs in 2·φ·d·N·(1−d) ← (1−d) is the free landing sites blank wins ⟺ φ·(1−d) > ½ ⟺ φ* = 1 / (2(1−d))
Density does not cancel. The curve leaves the unit square at d = 0.5: above half-lit, no amount of movement lets a blank frame win. Both of my earlier claims are slices of this one surface, and §4.6 sits in its far corner — d=0.0068, φ=1 — which is precisely where the dropped term is negligible and the wrong derivation looked right.
| Density d | Predicted φ* | Measured | Error | Verdict |
|---|---|---|---|---|
| 0.15 | 0.588 | 0.58 | −0.008 | hit |
| 0.30 | 0.714 | 0.72 | +0.006 | hit |
| 0.35 | 0.769 | 0.76 | −0.009 | hit |
| 0.45 | 0.909 | 0.86 | −0.049 | miss |
Three of four inside ±0.01, and the fourth outside the registered tolerance by 0.009 — so the prediction is recorded as falsified. It errs low at high density, in the direction the registration said to check first: this generator moves a contiguous column band, so arriving content collides only in a narrow leading strip rather than in proportion to φ, and that gap widens with d. A flat φ=0.5 boundary is ruled out outright — at d=0.30 it predicts 0.5 against 0.72 measured.
SSIM pools over blocks, so it never sees global density — it sees block occupancy. Two frames with the same 1200 lit pixels, one packed into 19 blocks and one scattered across 600, hold d exactly constant:
| Layout | Lit pixels | Blocks occupied | Pixel proportion | SSIM |
|---|---|---|---|---|
| Concentrated | 1200 | 19 | 0.9688 | 0.9683 |
| Spread | 1200 | 600 | 0.9688 | 0.0027 |
Pixel proportion cannot tell them apart — identical to twelve decimal places. SSIM scores the blank frame 350× apart on the same content. So the second prediction I registered here was also wrong, and wrong at the root: it compared SSIM's crossover in φ against pixel proportion's, and φ is not SSIM's parameter. No verdict actually flipped between the two layouts, so that prediction is falsified too, even though the mechanism behind it came out this strongly. One degeneracy worth stating before a reviewer finds it: a blank frame has zero variance, so SSIM's luminance and contrast terms are carried by the stabilising constants C₁ and C₂ rather than by any similarity judgement.
BNNN and of VF reset, so those two divergent frames come from my own
quirk-configured harness rather than from any third-party interpreter. D and E therefore
test the mechanism and say nothing about a real disagreement, and the small finding in that
is worth stating: the quirks that divide real implementations are a subset of the
quirks that divide the documentation. Both ROMs cleared all four acceptance gates anyway,
which is its own problem — the gates required a population split in prose and never checked
for one, so a fifth gate now reports it, and both ROMs fail it.
Not new: SSIM's sensitivity to small translations is a documented drawback with a purpose-built fix — CW-SSIM, Wang and Simoncelli, 2005. The pixel-proportion result is a replication; GBA Eval reported it themselves. GMSD's invariance under inversion follows from its construction. A page claiming to have discovered any of that would be wrong.
What the literature did not have: the consequence for a reference-based grader — that a documented metric property means a correct implementation is ranked below an empty screen, reachable through a legitimate platform disagreement rather than an adversarial input, with nothing in the score announcing which regime you are in.
Scope, now measured rather than conceded — and it costs this page its headline. I registered, before running it, that if a single displaced sprite on a realistic 240×160 frame lost to a blank frame under none of the three metrics, then §4.6's practical relevance to GBA Eval is materially reduced and this page would have to say so in those words. That is what happened. At d=0.25 with one sprite-sized region displaced, SSIM scores the correct divergence 0.9870 against the blank frame's 0.3033; at d=0.50 the gap is 0.9809 against 0.0550. Pixel proportion and GMSD agree. The failure demonstrated here is a property of sparse displays where nearly all the drawn content moves. It is not their grader's operating regime, and this is not a claim that their grader mis-ranks their submissions.
What survives is narrower and, I think, more useful than the headline was: the regime is real, it is reachable by a legitimate platform disagreement rather than an adversarial input, and it now has a closed-form boundary you can check your own frames against — measure what fraction of drawn content your divergence moves, measure how much of the display is lit, and see which side of φ(1−d) = ½ you are on. For SSIM, measure block occupancy instead. That is a test someone can run on their own grader in an afternoon, which the original finding was not.
Rushil Jaiswal · built with Claude Code, directed and reviewed by me github.com/rushjais/verifier-audit — ROM sources, the seven adapters, and the pre-registered predictions with dated amendments