Beyond Latent Alignment: auditing world-model stitching
A paper draft that re-implements DINO-WM model stitching. A reported collapse did not replicate, and a per-capability audit found door skills degrading in stitches that plan normally.
- Context
- Independent project
- Year
- 2026
- Stack
- PyTorch, DINO-WM, DINOv2 · SigLIP · MAE, DreamerV1-style RSSM, dm_control
0.54–0.74
planning success of MAE-front stitches over 3 seeds, where a collapse to 0.00–0.02 was reported
MAE model alone: 0.65 (mean of 3 seeds)
21 / 21
stitches that keep position, side-of-wall and blocking predictions
next-position R² ≥ 0.958 in every stitch
13 / 21
stitches where knowing alignment with the door degraded
missed by similarity and planning success; post hoc readout
Overview
Problem
A world model is a network that learns how a scene changes, so an agent can plan by imagining the future. Stitching reuses trained models: run the first half of one, pass its internal state through a small learned adapter, and finish with the second half of another. Stitches are usually judged by one similarity score or by planning success, and a recent paper reported that DINO-WM stitches with an MAE image encoder in front collapse. I asked which specific capabilities of the receiving model, such as predicting position or knowing where the door is, still work after the handoff.
Approach
- Re-implemented DINO-WM stitching from the paper text on the Wall task: three frozen image encoders, three seeds each, 21 stitches.
- Built an audit that scores each capability separately, judges it only where the model is competent on its own, and reports confidence bounds against preset thresholds.
- Validated the audit first where the answer is known: planted information gaps in seven pairs of small Dreamer-style models, plus a dose–response control.
Results
The collapse did not replicate: MAE-front stitches planned at 0.63–0.64, against 0.65 for MAE alone and the 0.02 and 0.00 reported. Coarse capabilities survived all 21 stitches, but door alignment degraded in 13. Neither similarity nor planning success showed this. The door readout was added after the fact, the adapters trained for only three epochs, and door capabilities also degrade in one same-encoder control, so whether stitching causes it is open. The work also documents three evaluation pitfalls. This is an unreviewed draft; experiments are still running.
What I built
- Re-implemented DINO-WM layer stitching from the paper text: nine world models on three frozen encoders and 21 stitches. A reported collapse of MAE-front stitches did not replicate.
- Designed a capability-level audit with competence gates, direct readouts and equivalence tests. Coarse capabilities survived all 21 stitches; door alignment degraded in 13.
- Validated the audit on planted gaps and a dose–response control, and documented three evaluation pitfalls: an action leak, validation leakage and a weak rollout baseline.
Results in detail
Stitched models plan about as well as their front model
Planning success on the Wall task: the share of 50 episodes in which the planner reaches the goal. Our bars are means over three seeds in the data-consistent environment. Li et al.'s values are single runs, presumably in the shipped environment.
With 50 episodes per run the standard error is about 0.06 to 0.07. Same-encoder controls (seed 0 into seed 1) reach 0.78 (DINO), 0.76 (SigLIP) and 0.64 (MAE).
In the shipped environment (seed 0 only) the MAE-front stitches reach 0.52 and 0.54, below the MAE model alone there (0.72). The pre-registered test for a collapse (both MAE-front stitches at most 0.15) fails in all three seeds.
Li et al.'s native MAE value is 0.56 in their table and 0.66 in their text; 0.56 is shown.
Coarse capabilities survive every stitch; door capabilities often do not
Verdict for each capability of the stitched model, across 21 stitches (18 cross-encoder stitches over three seeds, plus 3 same-encoder controls). Post hoc readout: this analysis was added after two pre-registered readouts turned out to be invalid.
HOW A VERDICT IS DECIDED
Each capability is judged against the front model's own score. Preserved: the stitched model's lower 95% confidence bound is at least the threshold minus 0.05. Degraded: the upper bound is below the threshold. Inconclusive: neither, or the front model itself is below the threshold. Thresholds: R² 0.90 for position and door distance, balanced accuracy 0.85 for side and door alignment, AUC 0.80 for blocking.
Door distance: 14 of 21 stitches have a confidence interval wholly below 0.90; six of these count as preserved only through the 0.05 tolerance.
Open question, not a finding about stitching: door alignment is only marginally readable even before stitching (direct readout 0.83 to 0.85 against a 0.85 threshold), door capabilities also degrade in the DINO same-encoder control, and the adapters were trained for only three epochs.
Checking the audit, and three pitfalls it exposed
Before the DINO-WM study, the audit was run where the answer is known in advance. Along the way, three evaluation mistakes turned up that make a handoff look better than it is.
PROTOCOL CHECKS
1Planted gap: the audit does not invent capabilities
7 pairs of small Dreamer-style pixel models; a visual cue is shown to the receiver and hidden from the sender
Overall match (latent R²) 0.71 to 0.90; each receiver reads its own cue at 0.995 to 1.000
The hidden cue is not recovered above chance on any of 14 pair-paths
2Dose–response: the audit finds information that is there
Senders shown the true cue with probability q
q = 1: cue recovered at 1.000 (direct readout) and 0.998 (through the handoff)
q = 0.25 is missed (0.314, p = 0.12): with 30 test episodes the detection floor lies between q = 0.25 and 0.5
PITFALLS
AA leak through the driving policy's actions
When the receiver's own policy drove evaluation episodes, its actions carried the hidden cue to the sender
Cue recovered above chance in 4 of 7 pairs (0.279 to 0.450, chance 0.25)
BValidation leakage
Validating on time steps interleaved with the fitting data raised the chosen translator's R² by about 0.26
With a nearest-neighbour translator removed and tuning on separate episodes, the gain is +0.000 and +0.014
CA weak rollout baseline
A direction-aware error check beats the right magnitude baseline by a median 0.27 in R²
Against a baseline that omits the initial error size, the same check looks like 0.85
Technical details
The full list of measured results behind this project.
Show all 20 measurements
| Measurement | Value |
|---|---|
| Planning success, MAE → DINO stitch, seeds 0 / 1 / 2 (50 episodes each) | 0.62 / 0.68 / 0.58 |
| Planning success, MAE → SigLIP stitch, seeds 0 / 1 / 2 | 0.64 / 0.74 / 0.54 |
| Planning success reported by Li et al. for the same two stitches | 0.02 / 0.00 |
| Mean planning success: MAE-front stitches vs MAE alone | 0.63–0.64 vs 0.65 |
| Mean planning success: DINO-front stitches vs DINO alone | 0.78–0.81 vs 0.78 |
| Shipped environment, seed 0: MAE-front stitches vs MAE alone | 0.52 and 0.54 vs 0.72 |
| Released DINO-WM checkpoint: shipped vs data-consistent environment (published 0.96) | 0.94 vs 0.76 |
| Mutual nearest-neighbour overlap, seed 0: DINO–SigLIP / DINO–MAE / SigLIP–MAE (shuffled ≈ 0.002) | 0.62 / 0.39 / 0.34 |
| Coarse capabilities preserved (position, side of wall, blocked by wall, blocked by border) | 21 / 21 stitches |
| 'Aligned with the door': degraded / inconclusive / preserved | 13 / 4 / 4 of 21 |
| 'Aligned with the door': worst stitched vs native balanced accuracy | 0.58 vs 0.89–0.93 |
| 'Distance to the door': degraded under the rule / interval wholly below 0.90 | 8 / 14 of 21 |
| Direct readout from layer-3 states, seed 0: door alignment vs other capabilities | 0.83–0.85 vs ≥ 0.97 |
| Position after 5 open-loop rollout steps, every stitch (R², post hoc, seed 0) | 0.86–0.97 |
| Planted gap, 7 Dreamer-style pairs: global latent R² while the hidden cue is not recovered | 0.71–0.90 |
| Planted gap: pair-paths recovering the hidden cue above chance (leak-free) | 0 of 14 |
| Naive (leaky) evaluation: pairs where the cue is recovered above chance | 4 of 7 (0.279–0.450) |
| Dose–response, q = 1: cue balanced accuracy, direct / through the handoff | 1.000 / 0.998 |
| Validation leakage: R² gain from interleaved validation, then with the fix | ≈ +0.26, then +0.000 / +0.014 |
| Rollout check advantage: right magnitude baseline vs naive baseline (median R²) | 0.27 vs 0.85 |
Limitations (11)
- The paper is an unpublished draft, not peer reviewed, and experiments are still running.
- AI tools assisted with implementing and running the experiments and with editing the text, as the draft states.
- The capability audit of the stitched models is a post hoc readout: it was added after two pre-registered readouts (probes at the stitching interface, readouts over all output channels) turned out to be invalid.
- The door result is an open question rather than a finding about stitching: door alignment is only marginally readable before stitching (direct readout 0.83 to 0.85 against a 0.85 threshold), door capabilities also degrade in the DINO same-encoder control, and the stitch adapters were trained for only three epochs.
- The door-distance count depends on the 0.05 tolerance: 8 of 21 stitches are degraded under the rule, while 14 of 21 have confidence intervals wholly below the threshold.
- DINO-WM study: one task (Wall) with a fixed wall and door layout, 50 planning episodes per run (standard error about 0.06 to 0.07), 10 training epochs, and a re-implementation from the paper text without Li et al.'s code or checkpoints.
- The draft cannot say why its MAE-front stitches differ from Li et al.'s; candidates include the environment inconsistency, encoder checkpoints, token choices and adapter training details. Li et al. do not state which environment they used.
- In the shipped environment (seed 0) the MAE-front stitches trail MAE alone by 0.18 to 0.20; the main results use the data-consistent environment.
- Power: visual pairs use 30 test episodes, so the audit misses weak cue information (q = 0.25) and the equivalence test fails for 2 of 14 pair-paths. Absence claims hold only above the detection floor.
- The planted-gap results use two synthetic recurrent systems and small DreamerV1-style models on two dm_control tasks; chance-level recovery there is guaranteed by construction and only shows the audit does not invent capabilities.
- Dose–response: one seed per sender, the two highest-dose senders have lower returns than both controls, and the tracking slope (1.18) is outside the pre-registered window.