Plan which MCP tools a turn can offer
Measured at 50% over 6 counted trials, which is inside the band gate 5 admits. The breakdown below says which model that figure came from.
- 50%
- measured pass rate
- 3 of 6 counted trials met the reward threshold.
- 2
- models measured
- Each cell is one agent against one model, run for the same number of attempts.
- 0
- errored trials
- Trials that produced no reward. Counted separately and never in the rate above.
- 6
- trials dispatched
- Every attempt this round started, including the ones that errored before a reward.
What each model did
One row per agent-and-model cell of the round that produced the figure above. The bar is passes over counted trials.
The table scrolls sideways.
| Agent | Model | Passed | Rate | Errored |
|---|---|---|---|---|
| Terminus 2 | Claude Fable 5.1anthropic | 3 of 3 | 100% | none |
| Codex CLI | GPT-5.3 Codexopenai | 0 of 3 | 0% | none |
A cell with no passes is not a broken task: it is the point of one. Gate 5 admits a task only when the full matrix lands between 10 and 60 percent, so a set that every model cleared would have been rejected for being too easy before it reached this page.
The task
A turn can hand a model only so many tools. Built-in servers take part of a fixed budget and the configured servers share what is left. Decide which servers fit, given that a server which does not fit should not cost the turn the ones that would have.
- Category
- Software engineering
- Declared tier
- medium. The tier is the author's declaration and it decides which band passes: a hard-tier task may legitimately sit under 10 percent, where a standard one may not.
- Languages
- python
- Catalogue reference
- mcp-tool-budget. The name a licensed copy carries, so a run of it is traceable back to this row.
- What a licence includes
- The graded task, its environment and verifier images pinned by digest, and the trial record behind the figures on this page.
- What stays with us
- The instruction, the test suite and the reference solution. Publishing those would retire the task, because a measurement you can read the answer to is not a measurement.