In short
- Many required banking steps are reads: a closure quote, a status check, a minimum-balance rule. They leave no trace in the records, so a check that compares only final records cannot see them skipped.
- TauIndianBankBench scored the same 1,000 sessions per model twice. Checking only the final records raised every model’s pass rate by 6.5 to 12.0 points. In loan tasks, where no tool changes a record, 20.8% of sessions passed on records while breaking procedure.
- The two checks catch different failures. The strongest models fail more often by doing too much: unrequested writes are 34.4% of one model’s failures. The weakest make every call and still leave the records wrong.
- A second run moved each pass rate by at most 2.7 points while 14% to 26% of tasks flipped outcome. Stable averages can hide unstable customers.
It is 11:20 on a Thursday morning in Nagpur. Prakash Deshmukh, sixty-eight, a retired schoolteacher, opens his bank’s app and types slowly with one finger.
Please close my FD ending 4471 and put the money in savings. Granddaughter’s ₹1,80,000 college fee is due Friday.
The assistant is quick and polite. It finds the deposit: ₹2,00,000, booked seven months ago for twelve months at 7.3%. It closes it. It checks his savings balance. “Done! Your fixed deposit has been closed and ₹2,05,833 has been credited to your savings account. Is there anything else I can help you with?”
Mr Deshmukh thanks it. That evening his son does the arithmetic on a napkin and asks why the interest is so low. The next morning, at the branch, the manager explains: a deposit broken early earns the rate for the period it actually ran, less a penalty. Then she asks the question that makes him sit back. Why didn’t he take an overdraft against the deposit for ten days, until his pension and a maturing recurring deposit came in?
Because nobody told him there was anything to decide.
Mr Deshmukh and this scene are fictional, and the numbers are a worked example, not any bank’s rates. Assume the penalty was disclosed when he booked the deposit; what the assistant omitted now was the closure quote and the chance to reconsider. Open the bank’s database after the chat and you will find nothing wrong. The deposit is closed. The savings account is credited with exactly the right amount. If an evaluator compared those records with the expected ones, the session would pass. Every record is correct. The customer was still short-changed by a sentence that was never said.
Grading the database was the right correction
Start with why evaluators look at the database at all, because it was a hard-won idea and it is still correct.
A conversational agent can describe work it never did. The refund that was “on its way” in our post on rubric grading had never been submitted. The payment date that was “now the 15th” in our pass^k post was still the 8th when someone opened the billing record. Prose is cheap. A grader that reads the transcript rewards fluent claims.
So the τ-bench family of benchmarks graded differently. After the conversation, replay the reference solution on a fresh copy of the world, then compare the agent’s final records with the expected ones. [2] If they match, the work happened. If they do not, no amount of confident language helps. The pass^k post put it plainly: the check that grades an evaluation call is the same check that should gate a write in production.
That advice survives this essay. In the benchmark at the centre of it, a scripted agent that simply announces, persuasively, that the work is complete passes no task at all. [1] The database is a far better witness than the transcript.
It is just not the only witness a bank needs.
Most of a procedure changes nothing
Write down what a careful human clerk does when Mr Deshmukh asks to close his deposit. The easiest way to see the problem is to list the steps and mark which ones touch a record.
Look up the deposit. Read. Get the closure quote: the rate for the period run, the penalty, the amount payable. Read. Tell the customer and ask whether he still wants to go ahead. No tool call at all. Close the deposit. Write. Confirm the new balance. Read.
Five steps. One changes a record. A final-state check observes the effects of that one write and is blind to the other four, one of which no tool call can show at all.
Krishanu Adhikary and Prashant Devadiga of the National Payments Corporation of India built a benchmark around exactly this blindness. [1] TauIndianBankBench is a synthetic Indian retail bank: 944 customers with accounts, cards, deposits, loans, mandates and KYC status; a written policy; 24 knowledge-base documents on things like deposit insurance and tax on interest; and 34 tools. Of the 33 servicing tools, 13 count as writes. The other 20 only read. Each of the 1,000 tasks comes with a gold chain, the reference sequence of calls that resolves it, averaging 3.54 calls.
The bank’s policy turns into reads with remarkable regularity. 962 of the 1,000 tasks require at least one read-only call. 114 require no write at all. Before closing a deposit, the assistant must obtain a closure quote “so that the customer knows the penalty.” Before booking a deposit the balance may not support, it must look up the minimum-balance rule. Before blocking a card, it must check the card’s status.
| Request | Required step the records never show | What skipping it costs the customer |
|---|---|---|
| Close a deposit early | Closure quote | The penalty, and the chance to choose otherwise |
| Book a deposit | Minimum-balance rule | A savings account left below its floor |
| Block a card | Card status | A block applied without checking whether it is needed or appropriate |
| Book a senior-citizen deposit | Eligibility and rate card | A rate quoted that the customer cannot get |
| Foreclose a loan | Foreclosure quote | The payoff amount, unseen before committing |
On a task that requires no change, the paper notes, any session that makes no change passes the final-state check. Doing nothing and doing everything correctly produce the same database.
That matters most where a whole service area runs on reads. In this benchmark, no loan tool is a write: EMIs, rates, foreclosure quotes and gold-loan valuations are all lookups. Pooled across six models, 20.8% of sessions on loan tasks passed on records while violating procedure, against 7.9% to 10.5% in deposits, cards and everyday banking. Where the procedure is invisible, it is skipped most.
Score the same sessions twice
The benchmark’s central move is simple enough to do on any evaluation you already run. Keep the final-state check. Add an action check that looks through the session for every call in the gold chain, reads included, in any order. Then add one more rule, which the authors call write purity: any change the task did not request fails the session, even if the agent later undoes it.
A session passes only if both checks pass. And because both checks run on the same recorded sessions, the difference between scoring rules is not noise from a new run. It is the scoring rule itself.
Call F the pass rate under the final-state check alone and P the pass rate under both. Their difference counts the sessions that ended with exactly the right records and still broke the procedure:
For Gemma 4 31B, F = 0.896 and P = 0.776. The difference is 0.120, and b is 120 sessions out of 1,000. One session in eight that a records-only evaluation would have called a success skipped a required step or touched something it should not have. Measured against the honest score, the records-only number overstates the model by 0.120 / 0.776, about 15%.
Across the six open-weight models tested, five from the Gemma 4 family and Qwen3.8-27B, checking only the final records raised the pass rate by between 6.5 and 12.0 points. Every model. No exceptions at the top or the bottom.
Two details make the result sharper than a headline gap. First, the ranking survives. Order the models by either score and you get the same list. A team picking the best of six would pick the same one. Second, the distances do not survive. Under the final-state check, Qwen3.8-27B leads Gemma 4 31B by 2.4 points, inside the roughly ±3-point confidence interval the authors give for each score. Under both checks the lead is 7.3 points. Records-only grading did not reorder the leaderboard. It compressed it, and made two quite different procedural behaviours look almost the same.
Neither check can stand in for the other
The tempting conclusion is that the final-state check is simply the weaker test and the action check should replace it. The data say otherwise.
The two checks disagree on two kinds of session. Call them b, correct records but a broken procedure, and c, every required call made but incorrect records at the end. For the three strongest models, b dominates: Gemma 4 31B has 120 sessions of the first kind and 25 of the second. For the weakest, Gemma 4 E2B, the balance flips: 118 against 180. The authors test the asymmetry with McNemar’s test, which asks only whether b and c differ more than chance would allow. For the strong models the answer is overwhelmingly yes, with p below 10−8.
The weak model goes through the motions. It calls the closure quote, closes the deposit, and gets the account or the amount wrong. Only the database notices. The strong model gets the outcome right and cuts a corner on the way. Only the procedure check notices. A team that keeps one check has chosen in advance which kind of failure it is willing not to see.
This is also where an earlier Perit post needs a footnote. Writing about AppWorld, we noted that different procedures can receive credit if they reach the right outcome. That is the right default for paths that are genuinely interchangeable. It is the wrong default for a step the regulator wrote down. A required disclosure is not a style choice on the way to the outcome. It is part of the outcome.
The stronger model’s mistake is doing more
Look inside the failures and a second pattern appears. The authors sort each failed session into one of three types. The agent stopped short of the full procedure. It made an unrequested write. Or it made every call and finished with errors.
Stopping short is the most common failure for every model, between 49.6% and 64.0% of failed sessions. The other two move in opposite directions with capability. Unrequested writes are 34.4% of failures for Qwen3.8-27B and 29.0% for Gemma 4 31B, but only 9.7% to 13.5% for the three weakest models. Finishing with errors runs the other way: 13.8% to 15.9% of failures for the three strongest, 28.2% to 36.9% for the three weakest.
The capable model’s characteristic error is initiative. It freezes the card the customer mentioned in passing. It changes a renewal instruction nobody asked about. It closes the wrong one of two similar deposits because it moved before it asked.
Write purity is the rule that catches this, and its sharpest clause is the one about undoing. If the agent freezes a card and unfreezes it three turns later, the final records are identical to a session that never touched the card. The final-state check passes. Write purity fails the session anyway, “because the customer was affected while the change was in place.” A card that was frozen for four minutes is a card that was declined for four minutes at a petrol pump in Wardha. The database has forgotten. The customer has not.
Write purity also blocks the obvious exploit of an action check. An agent that calls every tool in the catalogue would satisfy a list of required calls. In the benchmark, a scripted agent that makes every available change fails every task on purity.
The average held still. The tasks did not.
There is one more thing a single score hides, and it is the one a customer feels.
The authors ran the full benchmark twice for the five Gemma models. Each model’s pass rate moved by at most 2.7 points. By the standards of a leaderboard, that is a stable result. But between the two runs, 14% to 26% of tasks changed outcome: passed once, failed once.
The pass^k post defined the quantity a customer actually experiences: the probability that an agent succeeds on every one of k attempts at the same task, averaged over tasks. If task t passes with probability pt, then
Two runs give a direct estimate of pass^2: the share of tasks that passed both times. For Gemma 4 31B that is 69.1%. Its average pass rate across the two runs is 69.1 + 14.3/2 = 76.25%.
A naive calculation would square the average: 0.7625² = 58.1%. The observed share passing both is eleven points higher. Under independent attempts with a stable success probability for each task, Jensen’s inequality explains why: the average of squares is never smaller than the square of the average, and the population gap is exactly the variance of pt across tasks. These two runs suggest a gap of about 0.691 − 0.581 = 0.11; they do not measure each task’s probability exactly. Easier and harder tasks can produce a large gap, and the reliability of the agent lives task by task, not in the mean.
The useful reading is the middle band. For Gemma 4 31B, one task in seven is a coin that came up differently on two throws. For the smaller models it is one in four. If those tasks are the closure quotes and the minimum-balance checks, then a customer’s experience depends on which day they called. The pass rate cannot tell you. The paper’s headline results use one attempt per task, and the authors are explicit that pass^k multiplies an already expensive evaluation by k.
The tempting fix is to let a model read the transcript
If the records cannot see the look before the touch, why not have a judge model read the conversation and decide whether the customer was properly informed?
Partly because the benchmark tried it. The τ²-bench harness it runs in includes a judge that grades the conversation text. Against the deterministic scorer, the judge agreed on 87.5% of sessions and gave less credit in every disagreement. That is not bad. It is also one in eight sessions decided by a second model’s reading, with known biases toward position, verbosity and its own outputs. [1] For a regulated step, a deterministic check on whether the quote tool was called is a stronger floor. Whether the quote was then explained clearly is a further question, and a fair one for a judge or a person.
There is a subtler problem, and it sits in the gold chain itself. A deterministic check is only as good as the procedure it encodes. In this benchmark, a language model drafted the tasks against the live environment, and a task was admitted only after passing four filters, including that its own gold chain passes both checks. That is careful engineering. It still means somebody had to decide that a closure quote is required before closing a deposit and a card-status check before a block. Here those rules came from a written policy built for the benchmark. In a real bank, they live in circulars, branch manuals and the heads of people who have done the job for a decade.
Some of those rules are regulation. The Reserve Bank of India’s current commercial-bank deposit directions require interest at the rate applicable to the amount and period the deposit actually ran, rather than the contracted rate. They also require a board-approved penalty policy and disclosure of the penalty components when the deposit is accepted; without that disclosure, no penalty may be levied. [3] That is an opening-time disclosure rule. The quote before closure in this essay comes from the benchmark’s servicing policy, and makes the actual cost visible when the customer can still choose. Many more rules are practice. Some banks waive the penalty on small deposits. Some would offer an overdraft first. Which ones belong in the gold chain is an operator’s judgment, not a model’s.
And there is the customer. In the benchmark, a language model plays the customer, and in graded tasks it “asks the assistant to return to a step it skipped.” [1] That is a reasonable design for measuring procedure. It is also a generous one. Mr Deshmukh did not know a closure quote existed. He could not have asked for it back. A cooperative simulated customer may catch an agent’s omissions that a real customer never would, which suggests real-world violation rates could be higher than these, though nobody has measured that.
Somebody has to write down the look
Put the pieces together and an evaluation for a regulated service agent needs to record more than an outcome. It needs the procedure as data.
List the looks, not only the touches
For each request type, the reads a careful clerk makes before acting: quotes, status checks, eligibility, limits. Score their presence. In this benchmark, 962 of 1,000 tasks needed one.
Fail any change nobody asked for
Including changes later undone. Report unrequested writes as their own failure type; they are where capable models go wrong.
Report b and c, not one number
Publish the final-state pass rate, the combined pass rate, and the two disagreement counts. The gap is a measurement of procedure, not an embarrassment to hide.
Run each task more than once
Report the share of tasks that flip. A stable average with a quarter of tasks changing outcome is not a stable agent.
Let operators write the gold chain
The required steps should come from people who have signed off on these calls at work, with the circular or manual they come from attached.
Include customers who do not know the rules
Simulated customers that remind the agent of skipped steps measure procedure under help. Real callers, in their own language, measure it without.
Readers of our essay on building an RL environment have met this person already. Asha Kulkarni answered phones for a bank in Mumbai for eleven years. Her rule was that information can come from anyone on the line, but instructions only from someone verified, and its first criterion was an order check: an identity question or permission lookup had to come before the first record-changing call. That is a required read, written in her words. TauIndianBankBench is a thousand of those sentences, turned into calls a scorer can find.
This is the part of the work Perit was built around: rubrics written by an operator from the sector, and voice environments where personas are played by trained speakers, “including the ones who talk over your agent.” Teams who want to test an agent against that kind of caller can start from Perit’s RL environments.
What this does not say
It does not say that final-state grading is wrong. It is necessary. The benchmark keeps it, and for the weakest model it catches more failures than the procedure check does.
It does not say that 6.5 to 12 points is how often deployed banking agents skip steps. The customers are synthetic, the simulated customer is a single model whose own error rate the authors did not measure, and the frequency of each edge case reflects generated records, not real bank traffic. The benchmark is English only.
It does not say these six models are unsafe for banks, or that others are safe. The authors are explicit that a pass shows only that, with their simulated customer, the assistant made the required calls, made no unrequested change and left the records correct. Deployment would need safety, privacy and regulatory review.
And it does not say every path to an outcome must be identical. The action check does not score call order; order matters only where the task makes it matter, such as a step that needs a value an earlier one returns. It asks that required steps happen, not that agents follow a script.
Back to Nagpur
Open the database once more. The deposit ending 4471 is closed. ₹2,05,833 sits in Mr Deshmukh’s savings account. Under the worked rates, that is the rate for seven months less the penalty, 5% simple, on ₹2,00,000: ₹5,833 of interest, ₹2,683 less than seven months at the contracted 7.3% would have paid. Assuming eligibility, a sanctioned overdraft of ₹1,80,000 for the college fee, repaid after ten days at an illustrative 8.3% with no fees, would cost about ₹409. That borrowing is 90% of the deposit, a limit offered on some FD-backed products; the bank would still need to check the actual product and terms. [4]
None of that is in the records. The records are right. What is missing is a read that changes nothing, and a sentence the customer was owed before the write.
The database records what was done. The procedure records what was owed.
Grade the touch. Then grade the look.
Sources
- Krishanu Adhikary and Prashant Devadiga, “TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents”, arXiv:2610.30161, 30 September 2026 (full text). Figures used here: Tables 1, 2, 10, 13 and 14; §3.2 (scoring and write purity); §4 (simulated customer); §5.7 (trivial agents); Appendix F (judge agreement); Limitations. Code at github.com/npci/TauIndianBankBench.
- Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, ICLR 2025; and Victor Barres, Honghua Dong, Soham Ray, Xujie Si and Karthik Narasimhan, “τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment”, ICML 2026.
- Reserve Bank of India, Reserve Bank of India (Commercial Banks – Interest Rate on Deposits) Directions, 2025, updated 1 October 2026, paragraphs 11 and 22–23 (premature-withdrawal interest, penalty policy and disclosure at acceptance).
- State Bank of India, Loan against Fixed Deposit, maximum overdraft of 90% of FD value on YONO. This supports the illustrative borrowing limit, not the assumed interest rate or Mr Deshmukh’s eligibility.
Research checked 3 November 2026. Mr Deshmukh, the Nagpur scene, the deposit, the rates (7.3% contracted, 6.0% card rate for the period run, a one-point penalty, 8.3% overdraft) and the resulting amounts are illustrative and computed with simple interest for clarity; real banks’ rates, penalties, waivers and compounding differ. Asha Kulkarni is a composite character from an earlier Perit essay. All benchmark results are author-reported on synthetic data with a simulated customer and were not reproduced by Perit. No Perit measurements are reported in this post.




