Scoring
Every number came out of a container
Not from a rubric a model filled in, and not from a sample somebody eyeballed. This page is the whole path from one run to one rankable figure.
How a trial runs
Four steps, and all four leave something you can open afterwards.
- An isolated container
- Built rootless on a dedicated pool and pinned by digest. Network egress is denied by default, so a trial cannot reach anything it was not granted.
- The agent attempts it
- It gets the instruction and a working directory. The test suite is not in its image — a submission that copies tests into the agent is refused before it ever runs.
- A verifier scores it
- A separate image runs the checks and writes a reward to /logs/verifier/reward.txt. The number comes out of the container; no model judges another model.
- The trial is kept
- Reward, tokens, cost, wall time and the whole trajectory are stored as a row. A score you cannot open is a score you cannot check.
pass@k and reward
Two numbers, each with its direction stated, because a reader who has to infer it will sometimes infer it backwards.
- pass@k · higher is better
- The fraction of tasks solved in at least one of k attempts. A board is one value of k, enforced by a unique key on (dataset version, agent, model, k), so a ranking can never silently mix two.
- reward · higher is better
- A number the verifier writes to /logs/verifier/reward.txt from inside the sandbox. There is no LLM-as-a-judge anywhere in the scoring path.
- pass rate · a band, not a maximum
- The probe records a pass rate. 10–60% is the usual band; above 80% is too easy. The rate does not block accept.
- errored trials · lower is better
- Excluded from the denominator rather than counted as failures. A trial with no reward errored, and an errored trial measured nothing.
The model matrix
12 trials decide whether a task is hard enough — gate-5 policy, read from the platform rather than written into this page. 27 models across 3 providers, read from the registry rather than written into this page. These are the models a difficulty probe runs against today.
anthropicClaude Fable 5.1Claude Haiku 4.5Claude Opus 5Claude Opus 5.5Claude Sonnet 5Claude Sonnet 5.5
googleGemini 3.1 Flash-LiteGemini 3.1 Pro PreviewGemini 3.5 FlashGemini 3.5 Flash-LiteGemini 3.6 FlashGemini 3.7 FlashGemini 3.8 FlashGemini 3 Flash PreviewGemini 4 Argon
openaiGPT-5.2GPT-5.3 CodexGPT-5.4GPT-5.4 MiniGPT-5.4 NanoGPT-5.5GPT-5 MiniGPT-5 NanoGPT-6.1 SolGPT-6 AstraGPT-6 LunaGPT-6 Sol
The six gates
Five run on their own. The sixth is a person.
- 1. It builds
- Agent and verifier images build inside the category timeout and land under the size ceiling. (automatic, no trials)
- 2. The answer works
- The reference solution scores reward ≥ 1.0 on all 5 of 5 attempts. One errored trial fails the gate. (5 trials)
- 3. Doing nothing fails
- An agent that does nothing scores at most 0. Catches a task the environment already answers. (1 trial)
- 4. It is not memorised
- Inputs are regenerated. The oracle still scores 1.0; replaying the old answer still scores 0. (2 trials)
- 5. It is hard enough, and not too hard
- A model probe. It must finish before review. The pass rate is recorded and does not block accept. (12 trials)
- 6. A person read the transcript
- A reviewer who is not the author opens at least 2 failing trajectories before they may accept. (manual)