← All experiments
LLM EvaluationOn GitHub

Tabula

An arena where two language models play backgammon against each other, explaining every move, with Glicko-2 ratings and per-move performance metrics.

Details

Started
28 Sep 2026
Game
Backgammon, full rules engine
Ratings
Glicko-2 with uncertainty
Metrics
Latency, tokens, notional cost, play quality
Stack
React, Vite, Node, SQLite
Stage
Prototype v0.1
Last commit
29 Sep 2026

ConjectureLanguage models can play real backgammon well if the engine keeps the rules and they only choose the move.

FindingStill open.

The write-up

Pick a model for each side, press start, and watch two models play a real game on a realistic board. Each one gives a one-sentence reason for its move, typed out live in its panel.

The engine owns the rules

The most important design decision is that the model never makes a move directly. On each turn the engine rolls the dice and computes every legal play. The model is only asked to choose an index from that list and explain why. A model can therefore never make an illegal move however it formats its reply, which matters in a game as fiddly as backgammon.

The prompt shows the board from the model’s own side, with pip counts, the opponent’s blots and each legal play pre-scored for hits, resulting pip count and exposure. If a call fails, times out or cannot be parsed, the engine’s heuristic plays that move instead. It never hands the move to a different model, so no model is credited with another’s play.

Ratings, not just win counts

Finished games are stored in SQLite and every model earns a Glicko-2 rating with a ± uncertainty that shrinks as it plays. Beating a strong, well-established model counts for more than beating an uncertain or weaker one. A matchmaker suggests the most informative next pairing, so a growing roster gets rated fairly without every model playing every other.

More than who wins

Every decision records latency, tokens, a notional API cost, whether the model actually answered, and how often its pick matched the engine’s best play. Dice make results noisy, so the per-move quality signal builds up much faster than wins and losses do.