Most write-ups about improving LLMs show you what worked. This one leads with what didn't — four things I fully expected to help and that came back null — because with a frozen verifier, the nulls turn out to be the more trustworthy half of the map.
Four rules keep it honest: the held-out split is sacred — a lever only touches a test input if its program reproduces every training demo, no peeking. The delta is the product — system minus bare, on held-out; a bigger model's better absolute score is not a lever result. Model-swap or it's a trick — the base LLM is a socket, and any real lever survives swapping it. And one lever at a time.
Why this matters for nulls: when a lever comes back zero here, it's the verifier doing its job, not a bug. Nothing reached the test set unless it provably worked on the demos — so a null is a real, trustworthy statement about the world, not "the code broke."
So the rig isn't just failing at everything — three things moved the needle:
Small deltas — one to three tasks — but real, and they stack on top of whatever the base model gives you.
The pattern across all four: on this task, capacity and verified search are the real levers; every "make the model reason differently" reframing came back null. Which is consistent with a known result — attention routes in-context content linearly; the hard-to-manufacture capacity is the nonlinear synthesis a bigger model brings. Reframing re-routes; it can't synthesize.
Intervention — actually acting on the world — recovers the true causes name-blind (F1 1.00) across both labelings. So the "the LLM understands causation" reading is the real null: it's retrieving "rain causes flooding" from the names, not reasoning about the structure. Grounding comes from acting, not association — and you only see that if you run the rename control.
Epistemic: a field drowning in "our method achieves X" needs the other column. A null under a frozen verifier is a trustworthy statement — it says "this lever doesn't add value here," and you can reproduce it. That's rarer, and more useful, than another win you can't check.
Practical: the deliverable of a research programme isn't any single result — it's a trustworthy map of what fits and what doesn't. Knowing that object-structure and multi-model boards are dead ends on this task is worth as much as knowing best-of-N is a live one. It's what stops the next person (or the next you) from re-running a dead experiment.
The claims here are small and bounded — single GPU, one benchmark, deltas of a few tasks. The point was never that they're large. The point is that they're real, and you can check every one.