BenchPress

BenchPress tests one capability: whether a model can hold a precise state in its head and update it correctly across many sequential steps without drifting.

Each item gives the model the full rules of a small deterministic machine (a cellular automaton), a starting grid and a step count. The model has to carry out every step itself, with tools switched off, and report the exact final state. The systems are Turing-complete or chaotic, so there is no formula that shortcuts the answer. The only way to get it right is to simulate it.

Why it matters

Long-horizon, step-by-step reliability underpins agentic and procedural work: following a multi-step process, executing an algorithm, holding state across a long tool-using trajectory. A model that loses the thread at step 15 of a cellular automaton tends to lose it at step 15 of a real workflow. Most benchmarks hide this failure mode by using short answers, allowing tools or grading with a rubric.

What makes the number trustworthy

• Judge-free. Gold answers come from a reference simulator and are compared byte for byte. There is no LLM judge.
• Tools off. With a code interpreter every task is a five-line program, so removing tools measures the model’s own execution.
• Errors compound. One wrong cell early on corrupts every later step, so a good score needs sustained, error-free work.
• Contamination-resistant. Items are generated from a seed, so fresh variants can be minted to check for memorisation.
• Frozen. Calibrated difficulty is locked in a manifest, and harder versions are new versions, never edits.

Results so far

Across a 19-model panel the scores spread widely without saturating. The frontier tops out near 67% exact, every open-weight model sits below 25%, and Conway’s Life is the hardest task. The leaderboard reports both exact-match and per-row accuracy, plus a reasoning-efficiency figure: correct rows per thousand generated tokens.