decisionbench

How well do AI decision models solve sudoku?

Every OpenRouter decision model gets the same 20 puzzles and fills them in one digit at a time. Every move is recorded, so you can replay any race.

suite v1 · 12 models · 240 runs · $2.4721 total cost · updated 20 Oct 2026

Leaderboard

1d1 UltraLiquid84.0%20/209.610s$0.002697
2Jev 1.14TypeSafe83.2%20/2010.27.4s$0.005278
3Clef OmniCloudflare82.3%20/2010.89.9s$0.0239
4d1Liquid81.5%20/2011.49.5s$0.001373
5Clef FlashCloudflare80.4%20/2012.320s$0.004933
6ClefCloudflare80.0%20/2012.623s$0.0101
7Decision-1Microsoft79.6%20/2012.915s$0.003344
8Decider v1.2Perplexity77.3%20/2014.840s$0.002168
9Mercury DecideInception75.7%20/2016.118s$0.003594
10Solar Decide FlashUpstage74.1%20/2017.651s$0.0433
11GPT-6 Luna DecisionsOpenAI74.0%19/2017.114s$0.0188
12Decider Flash v1.2Perplexity79.1%16/2011.237s$0.004203
Running the full benchmark cost $2.4721 across 240 runs

Triage & Robustness →

Sudoku tests one question type. The second benchmark gives every model 60 support tickets and a written policy, using choice, yes/no and score questions. It checks accuracy, whether stated confidence can be trusted, and whether answers survive reordering, renaming, long inputs and prompt injection.

  1. 1d1 Ultra100.0%
  2. 2Decision-1100.0%
  3. 3Mercury Decide98.3%

Races

Each puzzle is one race. Open one to replay every model solving it.

Easy

5
9
2
4
6
3
4
8
1
5
9
7
5
4
2
1
1
2
5
9
4
7
4
9
3
2
3
7
8
7
1
2
1
8
6
9
5
2
4
7
#0140 clues
Jev 1.14✕2
5
7
6
1
3
7
6
3
9
9
4
4
2
3
8
2
6
5
9
1
4
7
8
1
4
6
2
3
5
2
1
6
1
2
3
6
1
8
5
9
#0240 clues
d1✕2
1
5
2
8
7
6
6
3
4
7
9
7
2
8
5
4
1
9
8
4
5
1
8
6
2
9
9
7
2
3
8
7
2
1
9
9
7
6
3
8
#0340 clues
Decision-1✕3
4
2
3
8
1
1
7
4
5
3
3
1
1
8
6
6
9
4
7
5
4
2
8
6
5
3
2
7
4
9
8
2
7
4
3
1
8
2
9
6
#0440 clues
Jev 1.14✕3
5
6
9
3
7
8
5
3
9
1
3
8
2
6
7
5
4
2
7
8
9
9
3
4
1
8
6
2
4
7
1
6
2
5
5
1
4
6
7
3
#0540 clues
Decider v1.2✕1

Medium

5
4
5
6
3
9
2
5
8
6
3
1
7
8
9
2
7
6
8
6
8
9
4
5
4
3
7
2
9
4
7
3
#0132 clues
Clef Flash✕5
9
7
3
8
1
4
9
2
6
7
6
3
9
3
6
4
5
1
7
4
5
2
5
7
1
8
1
4
3
1
9
7
#0232 clues
Jev 1.14✕6
5
9
1
8
5
2
3
6
8
4
7
4
1
7
6
2
8
3
6
2
1
4
7
2
5
4
8
7
1
1
9
5
#0332 clues
Clef✕5
7
1
9
8
4
3
2
4
6
4
1
2
3
6
3
3
9
2
5
5
9
2
6
1
7
5
4
5
6
2
3
4
#0432 clues
d1 Ultra✕6
3
4
5
2
7
1
7
1
4
9
4
8
5
2
3
4
2
9
3
9
7
3
1
5
2
4
8
6
6
3
7
9
#0532 clues
Clef✕4

Hard

6
4
2
8
8
5
9
7
9
2
8
1
4
1
6
7
4
3
6
7
5
9
1
5
8
5
3
#0127 clues
Decider Flash v1.2✕9
5
3
4
1
8
7
2
4
5
9
8
2
4
1
6
9
5
3
6
4
3
4
1
5
7
8
1
#0227 clues
Jev 1.14✕9
3
4
6
8
2
9
1
5
9
5
6
4
1
8
3
7
6
1
2
9
6
7
5
4
3
6
5
#0327 clues
d1 Ultra✕5
2
5
6
7
4
6
1
5
6
2
2
3
9
7
5
9
8
3
3
7
2
9
3
9
7
2
5
#0427 clues
d1 Ultra✕12
2
3
9
8
8
2
4
6
1
3
2
2
1
7
5
4
9
6
8
9
5
1
6
2
1
8
5
#0527 clues
Jev 1.14✕7

Expert

7
4
5
3
2
6
6
5
9
5
1
2
4
2
9
6
1
3
6
1
8
5
6
#0123 clues
d1 Ultra✕10
9
2
6
5
5
7
8
7
2
4
5
1
8
6
5
6
9
2
7
7
4
1
2
8
#0224 clues
Clef Omni✕11
6
3
4
9
5
8
9
7
2
7
8
4
7
8
9
3
5
1
9
1
2
4
2
8
3
#0325 clues
Clef Omni✕12
1
4
9
6
5
3
7
1
2
4
7
3
3
6
8
2
1
4
2
5
6
7
8
2
#0424 clues
Decision-1✕12
2
7
5
4
6
1
3
7
4
5
6
7
2
7
1
8
5
2
3
5
1
4
2
#0523 clues
d1 Ultra✕12

Method

Puzzles
20 frozen puzzles, 5 per difficulty, each with exactly one solution. Every model gets the same boards.
Each turn
The model sees the grid and gets one choice question per empty cell (up to 40, most constrained first) with options 1–9. Its 3 most confident answers are placed.
Mistakes
Each placed digit is checked against the solution. A wrong digit ends the turn and is crossed off for that cell, so every model eventually finishes. Accuracy = correct digits ÷ all digits placed.
Rules help
Answers that break sudoku rules (a digit already in the row, column or box) are skipped, not counted as mistakes.
Time
Solve time is the sum of the model's API latency, so queueing and rate limits don't count against it. Replays use the same clock.
Ranking
Solve rate, then accuracy, then cost. A run stops early only on errors or a $0.25 cost cap per puzzle.