Evidence
A score you cannot open is a score you cannot check
Everything a trial produced is stored beside the figure it produced. That is what makes a published number auditable rather than asserted, and it is the reason this platform costs more to run than one that keeps only totals.
Every trial kept, not summarised
A trial is a row, and the row points at the bytes.
- The bundle
- The archive as it was submitted, content-addressed by its sha256, so the task that was graded is the task you can fetch.
- The trajectory
- What the agent actually did, step by step, stored per trial. Gate 6 requires a reviewer to open at least two failing ones before anything may be accepted.
- The verifier's output
- Reward, plus whatever the suite wrote to /logs. The number and the log that produced it are stored together or not at all.
- The cost
- Tokens, cached and reasoning counted separately, and wall time. A trial's cost is never derived from the trajectory; the run's own counts are the authority.
Reproducible across model versions
Re-running a score next quarter has to run the same bytes, or the comparison means nothing.
- Images pinned by digest
- Agent and verifier images are built rootless and pinned by sha256. A floating tag could change between the run that admitted a task and the benchmark that cites it, which would break attribution silently.
- Inputs regenerated
- Gate 4 regenerates a task's inputs and requires the reference solution to still score 1.0 while a replay of the old answer scores 0 — which is how a memorised task is caught.
- One engine for both
- A gate run and a benchmark run are the same engine, so a figure from one is comparable with a figure from the other by construction rather than by convention.
A person read the transcript
The one gate that is not automatic, and the reason it is not.
- Not the author
- Gate 6 requires a reviewer who did not submit the task. Authorship is read from who uploaded the bundle, not from who first created the task name.
- Two failing trajectories
- At least two have to be opened before an accept is possible, and the opening is recorded as an access event. A review nobody can evidence is not a review.