Reconcile a month's input tax credit
Measured at 33% over 9 counted trials, which is inside the band gate 5 admits. The breakdown below says which model that figure came from.
- 33%
- measured pass rate
- 3 of 9 counted trials met the reward threshold.
- 3
- models measured
- Each cell is one agent against one model, run for the same number of attempts.
- 0
- errored trials
- Trials that produced no reward. Counted separately and never in the rate above.
- 9
- trials dispatched
- Every attempt this round started, including the ones that errored before a reward.
What each model did
One row per agent-and-model cell of the round that produced the figure above. The bar is passes over counted trials.
The table scrolls sideways.
| Agent | Model | Passed | Rate | Errored |
|---|---|---|---|---|
| Terminus 2 | GPT-6 Astraopenai | 3 of 3 | 100% | none |
| Terminus 2 | Claude Fable 5.1anthropic | 0 of 3 | 0% | none |
| Terminus 2 | Gemini 4 Argongoogle | 0 of 3 | 0% | none |
A cell with no passes is not a broken task: it is the point of one. Gate 5 admits a task only when the full matrix lands between 10 and 60 percent, so a set that every model cleared would have been rejected for being too easy before it reached this page.
The task
A finance team closes a credit run by comparing what suppliers filed against what the company itself recorded, giving each invoice a status and an amount. The rules are the company's own and they are written down across several files, not all of which are current.
- Category
- Data processing
- Declared tier
- medium. The tier is the author's declaration and it decides which band passes: a hard-tier task may legitimately sit under 10 percent, where a standard one may not.
- Languages
- python, bash
- Catalogue reference
- gst-itc-reconcile. The name a licensed copy carries, so a run of it is traceable back to this row.
- What a licence includes
- The graded task, its environment and verifier images pinned by digest, and the trial record behind the figures on this page.
- What stays with us
- The instruction, the test suite and the reference solution. Publishing those would retire the task, because a measurement you can read the answer to is not a measurement.