{"id":960,"date":"2026-10-05T12:07:51","date_gmt":"2026-10-05T12:07:51","guid":{"rendered":"https:\/\/perit.ai\/blogs\/?p=960"},"modified":"2026-10-05T12:11:18","modified_gmt":"2026-10-05T12:11:18","slug":"pass-k-ai-agent-reliability","status":"publish","type":"post","link":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/","title":{"rendered":"pass^k: Measuring AI Agent Reliability Across Repeated Calls"},"content":{"rendered":"\n<div class=\"pp-post\" style=\"max-width: 50rem;margin: 0 auto;font-family: Verdana, Geneva, Tahoma, sans-serif;font-size: 1rem;line-height: 1.68;color: #0b0b0b\">\n<div class=\"meta\" style=\"font-size: 0.92rem;color: #52514e;margin-bottom: 1.6rem;padding-bottom: 0.65rem;border-bottom: 1px solid #e6e5e0\">What pass^k measures, why the obvious estimator is biased, why a transcript grader inflates it, and how to spend a fixed budget of human calls to pin it down.<\/div>\n<div class=\"takeaways\" style=\"margin: 0 0 1.8rem;padding: 0.95rem 1.1rem 0.4rem;border: 1px solid #e6e5e0;border-left: 4px solid #2a78d6;border-radius: 0 10px 10px 0;background-color: #fcfcfb\"> <p class=\"takeaways-head\" style=\"margin: 0 0 0.45rem;color: #2a78d6;font-size: 0.78rem;font-weight: 700;letter-spacing: 0.08em;text-transform: uppercase\">In short<\/p> <ul style=\"padding: 0;margin: 0 0 0.6rem 1.2rem\"> <li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem;font-size: 0.95rem;line-height: 1.55\">pass^k is the probability that an AI agent succeeds on all k independent attempts at the same task, averaged over tasks. On ThinkingBox, Kimi-K3 solved 93.89% of 507 workflows at least once in 20 tries but only 13.41% on all 20.<\/li> <li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem;font-size: 0.95rem;line-height: 1.55\">Grading matters as much as repetition. 67.24% of 79,853 failed ThinkingBox attempts ended cleanly, called a state-changing tool and showed no final tool error. In VAmoS, 54.2% of calls the simulated caller called successful failed verification.<\/li> <li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem;font-size: 0.95rem;line-height: 1.55\">The plug-in estimate (c\/n)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup> is biased upward (by up to 8.6 points for pass^3 from five trials). The unbiased form is C(c,k)\/C(n,k). Confidence intervals must resample tasks, not calls: ThinkingBox&#8217;s published intervals are 3.2 to 4.1 times wider than a naive binomial interval.<\/li> <li style=\"margin: 0 0 0.45rem;padding-left: 0.15rem;font-size: 0.95rem;line-height: 1.55\">With 1,200 human calls and cheap scenarios, the narrowest interval for pass^3 comes from 400 scenarios run 3 times (\u00b14.9 points). Expensive scenarios push the optimum towards more repeats, by the rule n* = \u221a(s\u00b7W\/V<sub style=\"font-size: 0.72em;line-height: 0\">B<\/sub>).<\/li> <\/ul> <\/div>\n<p class=\"lede\" style=\"margin: 0 0 1rem;font-size: 1.08rem\">A caller rings a utility&#8217;s voice agent and asks to move a payment from the 8th to the 15th. The agent looks up the account, says &#8220;Done, your payment is now on the 15th,&#8221; and asks if there is anything else. The caller says no, thanks it, and hangs up. Read the transcript and this is a model call. Open the billing record and the payment is still due on the 8th: the update failed on a lock the agent never mentioned, and a second support ticket sits in the queue.<\/p>\n<p style=\"margin: 0 0 1rem\">The call is constructed, but its shape is measured. When Microsoft and Hugging Face graded 507 business workflows by the database state each attempt left behind, 67.24% of the 79,853 failed attempts had ended cleanly, called a tool that changes state, and reported no final tool error.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-1\" style=\"color: #0000ee;text-decoration: none\">[1]<\/a><\/sup> In Veris AI&#8217;s VAmoS voice benchmark, the simulated caller declared its goal met on 2,072 calls, and 1,122 of those calls failed verification.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-3\" style=\"color: #0000ee;text-decoration: none\">[3]<\/a><\/sup><\/p>\n<p style=\"margin: 0 0 1rem\">Run the same call tomorrow and the agent may get it right. A voice agent&#8217;s reliability is therefore a statement about two things: whether the record changed correctly, and whether it changes correctly every time. This post builds the metric for that, pass^k, from the estimator up: how to estimate it without bias, what transcript grading does to it, how wide its interval really is, and how to spend a fixed budget of human calls on it.<\/p>\n<h2 id=\"passk-asks-whether-an-ai-agent-succeeds-on-every-attempt-not-on-one\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">pass^k asks whether an AI agent succeeds on every attempt, not on one<\/h2>\n<p style=\"margin: 0 0 1rem\">pass^k is the probability that an agent succeeds on all k independent attempts at the same task, averaged over the tasks in a benchmark. It was introduced in \u03c4-bench by Yao and colleagues, who defined it as &#8220;the chance that all k i.i.d. task trials are successful, averaged across tasks&#8221; and graded each trial by comparing the final database state with an annotated goal state.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-7\" style=\"color: #0000ee;text-decoration: none\">[7]<\/a><\/sup> It is the mirror image of pass@k from the Codex paper, the chance that at least one of k attempts succeeds.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-6\" style=\"color: #0000ee;text-decoration: none\">[6]<\/a><\/sup> Perit&#8217;s earlier post on <a href=\"https:\/\/perit.ai\/blogs\/rl-environments-voice-agents\/\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: none\" target=\"_blank\">RL environments for voice agents<\/a> introduced the idea with a single compounding example. Here we need the full machinery.<\/p>\n<p style=\"margin: 0 0 1rem\">Index tasks by i and give task i a per-attempt success probability p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>. If attempts at the same task are independent given p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>, the two metrics for that task are simple powers, and the benchmark numbers are their averages over the task distribution:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">pass@k = E<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>[ 1 \u2212 (1 \u2212 p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup> ] \u00a0\u00a0\u00a0 pass^k = E<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>[ p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup> ] \u00a0\u00a0<span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(1)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">Both equal pass@1 = E[p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>] at k = 1. As k grows, pass@k rises towards the share of tasks with any chance of success, and pass^k falls towards the share of tasks the agent never fails.<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">pass@k asks whether a solution is within the agent&#8217;s reach, which matters when a verifier can pick the winner from k samples, as with unit tests. pass^k asks whether the agent can act without a verifier, which is the situation of every deployed voice agent: each caller gets one attempt, and nobody reruns the call.<\/p>\n<p style=\"margin: 0 0 1rem\">ThinkingBox, released on Hugging Face on 3 October, shows how far apart the two can be. Every one of its 507 workflows was run 20 times from an identical clean backend. Kimi-K3 solved 93.89% of the workflows at least once in 20 tries, 476 of 507, the broadest coverage of any model tested. It passed all 20 attempts on 68 workflows, 13.41%. Claude Opus 5 solved fewer workflows at least once, 79.09%, but passed all 20 on 241 of them, 47.53%.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-1\" style=\"color: #0000ee;text-decoration: none\">[1]<\/a><\/sup> Ranked by pass@20, Kimi-K3 wins. Ranked by pass^20, Opus 5 completes more than three times as many workflows dependably.<\/p>\n<p style=\"margin: 0 0 1rem\">The metric has spread since \u03c4-bench. \u03c4\u00b2-bench reports it,<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-8\" style=\"color: #0000ee;text-decoration: none\">[8]<\/a><\/sup> Mercor&#8217;s APEX-Agents 1.1 reports Pass^4, with GPT-6 Astra highest at 56.3%,<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-9\" style=\"color: #0000ee;text-decoration: none\">[9]<\/a><\/sup> and voice benchmarks use three-run versions named Reliable@3 and Pass\u00b3.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-4\" style=\"color: #0000ee;text-decoration: none\">[4]<\/a><a href=\"#passk-ref-5\" style=\"color: #0000ee;text-decoration: none\">[5]<\/a><\/sup><\/p>\n<h2 id=\"two-thirds-of-failed-attempts-looked-like-successes\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">Two-thirds of failed attempts looked like successes<\/h2>\n<p style=\"margin: 0 0 1rem\">Before estimating anything, decide what counts as a success. A voice call produces two records: the audio and transcript, which is what the caller experienced, and the backend state, which is what the business experiences. They can disagree, and Figure 1 shows the case that matters.<\/p>\n<figure class=\"viz\" style=\"margin: 2.2rem 0\"><div class=\"viz-card\" style=\"border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb;padding: 14px 14px 8px;overflow: auto\"><a href=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-1.webp\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" class=\"wp-image-955\" src=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-1.webp\" alt=\"Diagram of one constructed voice-agent call. The caller asks to move a payment date to the 15th. The agent says the change is done. A transcript grader reads the words and passes the call. A state grader compares the terminal backend record with the expected state: the payment date is unchanged and a duplicate ticket was created, so the call fails. pass^k computed on the transcript grader counts this call as a success; pass^k computed on backend state does not.\" width=\"1520\" height=\"940\" loading=\"lazy\" style=\"display:block;width:100%;min-width:560px;height:auto;max-width:none\" srcset=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-1.webp 1520w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-1-300x186.webp 300w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-1-1024x633.webp 1024w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-1-768x475.webp 768w\" sizes=\"auto, (max-width: 1520px) 100vw, 1520px\" \/><\/a><\/div><figcaption class=\"viz-caption\" style=\"margin-top: 0.65rem;font-size: 0.9rem;line-height: 1.55;color: #52514e\"><strong style=\"color: #0b0b0b\">Figure 1.<\/strong> One constructed call graded two ways. The transcript and the caller&#8217;s reaction say the payment moved; the billing record says it did not, and a duplicate ticket was created. pass^k inherits whichever grader it is computed on. ThinkingBox found that 67.24% of failed attempts would pass a grader that checks only for a clean ending, a state-changing tool call and no final tool error [2].<\/figcaption><\/figure>\n<p style=\"margin: 0 0 1rem\">ThinkingBox&#8217;s ablation measures how often that happens. Across 121,680 valid trials from 12 models, 79,853 attempts failed the executable state checks. Of those failures, 84.86% passed a &#8220;clean termination&#8221; proxy, 80.88% also included a state-changing tool call, and 67.24% additionally showed no explicit error in the final tool response. The state checks found wrong field values in 77.61% of the failures, unintended extra effects in 43.30% and missing required effects in 25.36%.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-2\" style=\"color: #0000ee;text-decoration: none\">[2]<\/a><\/sup> In other words, a grader that looks only at how the conversation ended would accept roughly two failed attempts in every three.<\/p>\n<p style=\"margin: 0 0 1rem\">Voice adds a second weak grader: the caller. VAmoS Energy, a 29 September benchmark of 100 utility-billing calls across fourteen voice stacks, observes that the simulated caller &#8220;often accepts an incorrect result because it hears the agent&#8217;s words but cannot inspect its actions.&#8221;<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-3\" style=\"color: #0000ee;text-decoration: none\">[3]<\/a><\/sup> Its numbers make the point exact. Of the 2,072 calls where the caller declared its goal met, 1,122 failed verification, 54.2%. That is a false discovery rate: the probability that a call the caller called successful was in fact a failure. VAmoS&#8217;s own verifier reads the transcript plus recorded tool calls, arguments and results, and the authors note that it infers account changes from tool results rather than inspecting the ledger directly, so even &#8220;state&#8221; grading comes in degrees.<\/p>\n<p style=\"margin: 0 0 1rem\">APEX-Voice, posted on 28 September, applies the strict version to full-duplex voice agents. A workflow succeeds only with the correct terminal state, a valid process, completed actions and a valid final artifact. Across five frontier real-time agents, none exceeded 25% Pass@1, and the best Reliable@3, the share of workflows solved on all three runs, was 10.8%. GPT-realtime-2.1 reached 23.6% Pass@1 despite 91.4% artifact field accuracy.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-4\" style=\"color: #0000ee;text-decoration: none\">[4]<\/a><\/sup><\/p>\n<h2 id=\"the-obvious-passk-estimator-is-biased-upward\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">The obvious pass^k estimator is biased upward<\/h2>\n<p style=\"margin: 0 0 1rem\">Suppose task i is run n times and succeeds c<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> times. The natural move is to estimate p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> by c<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>\/n and raise it to the k-th power. That plug-in estimator is biased, and the direction of the bias is fixed by convexity. Since x<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup> is convex on [0, 1] for k \u2265 2, Jensen&#8217;s inequality gives E[(c\/n)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>] \u2265 (E[c\/n])<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup> = p<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>. Sampling noise in c\/n is symmetric, but raising it to a power is not: lucky draws above p are amplified more than unlucky draws below it are shrunk.<\/p>\n<p style=\"margin: 0 0 1rem\">The size of the bias follows from the factorial moments of the binomial. Write c<sup style=\"font-size: 0.72em;line-height: 0\">(j)<\/sup> = c(c \u2212 1)\u2026(c \u2212 j + 1). For a binomial, E[c<sup style=\"font-size: 0.72em;line-height: 0\">(j)<\/sup>] = n<sup style=\"font-size: 0.72em;line-height: 0\">(j)<\/sup>p<sup style=\"font-size: 0.72em;line-height: 0\">j<\/sup>, and c<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup> = c<sup style=\"font-size: 0.72em;line-height: 0\">(3)<\/sup> + 3c<sup style=\"font-size: 0.72em;line-height: 0\">(2)<\/sup> + c. So for k = 3:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">E[(c\/n)<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup>] = [ n(n\u22121)(n\u22122)p<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup> + 3n(n\u22121)p<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup> + np ] \/ n<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup> <span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(2)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">At n = 5 and p = 0.7 this is 0.428 against a true p<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup> of 0.343: a bias of +8.5 points. The excess terms vanish only as n \u2192 \u221e.<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">The unbiased estimator comes from a counting argument rather than algebra. Given c successes among n trials, pick k of the n trials uniformly at random without replacement. The probability that all k picks are successes is hypergeometric:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">est. pass^k<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> = C(c<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>, k) \/ C(n, k) \u00a0\u00a0\u00a0 est. pass@k<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> = 1 \u2212 C(n \u2212 c<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>, k) \/ C(n, k) \u00a0\u00a0<span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(3)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">Unbiased for every n \u2265 k. \u03c4-bench gives the first form for pass^k [7]; the Codex paper gives the second for pass@k [6].<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">Why is it unbiased? Average the conditional probability over the distribution of c and you get the unconditional probability that a random k-subset of the n trials is all successes. The trials are i.i.d., so every k-subset has the same joint law as k fresh trials, and that probability is exactly p<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>. The same argument, with &#8220;at least one success&#8221; in place of &#8220;all successes&#8221;, proves the pass@k form. The estimator is also the minimum-variance unbiased one, since it is a function of the complete sufficient statistic c. Averaging over tasks preserves unbiasedness, which gives the benchmark estimator in equation (1).<\/p>\n<p style=\"margin: 0 0 1rem\">The implementation is short; the product form for pass@k is the stable version from the Codex paper.<\/p>\n<div aria-label=\"Python implementation of the pass@k and pass^k estimators\" class=\"code\" role=\"region\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-radius: 6px;font-family: Menlo, Consolas, &quot;Courier New&quot;, monospace;font-size: 0.82rem;line-height: 1.55;color: #0b0b0b;white-space: nowrap;overflow: auto\"><code style=\"font-family:inherit;font-size:inherit;white-space:inherit\">from math import comb, prod<br \/>import numpy as np<br \/> <br \/>def pass_at_k(n, c, k):<br \/>\u00a0\u00a0\u00a0\u00a0\"\"\"Unbiased pass@k for one task (Chen et al., 2021).\"\"\"<br \/>\u00a0\u00a0\u00a0\u00a0if n - c &lt; k:<br \/>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0return 1.0<br \/>\u00a0\u00a0\u00a0\u00a0return 1.0 - prod(1 - k \/ j for j in range(n - c + 1, n + 1))<br \/> <br \/>def pass_hat_k(n, c, k):<br \/>\u00a0\u00a0\u00a0\u00a0\"\"\"Unbiased pass^k for one task (Yao et al., 2024).\"\"\"<br \/>\u00a0\u00a0\u00a0\u00a0return comb(c, k) \/ comb(n, k) if c &gt;= k else 0.0<br \/> <br \/>def benchmark_pass_hat_k(counts, n, k, B=10_000, seed=0):<br \/>\u00a0\u00a0\u00a0\u00a0\"\"\"Mean over tasks, with a task-cluster bootstrap 95% interval.\"\"\"<br \/>\u00a0\u00a0\u00a0\u00a0u = np.array([pass_hat_k(n, c, k) for c in counts])<br \/>\u00a0\u00a0\u00a0\u00a0rng = np.random.default_rng(seed)<br \/>\u00a0\u00a0\u00a0\u00a0idx = rng.integers(0, len(u), size=(B, len(u)))<br \/>\u00a0\u00a0\u00a0\u00a0lo, hi = np.quantile(u[idx].mean(axis=1), [0.025, 0.975])<br \/>\u00a0\u00a0\u00a0\u00a0return u.mean(), (lo, hi)<\/code><\/div>\n<p class=\"code-caption\" style=\"margin: -0.9rem 0 1.5rem;font-size: 0.88rem;color: #52514e\">Python, 20 lines: per-task unbiased estimators and a task-cluster bootstrap. Fed the reconstructed APEX-Voice histogram for GPT-realtime-2.1 in Table 1, it returns 10.8% with a 95% interval of 5.8% to 16.7%.<\/p>\n<p style=\"margin: 0 0 1rem\">Figure 2 checks both estimators by simulation. For each true p from 0.05 to 0.95, 200,000 simulated tasks were each run five times. The plug-in estimate of pass^3 sits on the closed form of equation (2), with a maximum bias of 8.6 points near p = 0.65. The unbiased estimator&#8217;s simulated bias never exceeds 0.1 points. A second simulation over a heterogeneous benchmark (200 tasks with Beta-distributed p, mean 0.6, five trials each, 4,000 replicates) gives the same picture at the benchmark level: the true pass^3 is 0.352, the unbiased mean is 0.352, and the plug-in mean is 0.404.<\/p>\n<figure class=\"viz\" style=\"margin: 2.2rem 0\"><div class=\"viz-card\" style=\"border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb;padding: 14px 14px 8px;overflow: auto\"><a href=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-2.webp\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" class=\"wp-image-956\" src=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-2.webp\" alt=\"Computed chart for n = 5 trials per task and k = 3. The plug-in estimator (c\/n) cubed overstates the true p cubed by up to 8.6 percentage points, peaking near p = 0.65; Monte Carlo means from 200,000 simulated tasks per point sit on the closed-form curve. The unbiased estimator C(c,3)\/C(5,3) has Monte Carlo bias within 0.1 points of zero at every p.\" width=\"1520\" height=\"850\" loading=\"lazy\" style=\"display:block;width:100%;min-width:560px;height:auto;max-width:none\" srcset=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-2.webp 1520w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-2-300x168.webp 300w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-2-1024x573.webp 1024w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-2-768x429.webp 768w\" sizes=\"auto, (max-width: 1520px) 100vw, 1520px\" \/><\/a><\/div><figcaption class=\"viz-caption\" style=\"margin-top: 0.65rem;font-size: 0.9rem;line-height: 1.55;color: #52514e\"><strong style=\"color: #0b0b0b\">Figure 2.<\/strong> Bias of two estimators of pass^3 from five trials per task. Orange line: closed form of equation (2) for the plug-in (c\/n)<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup>. Dots: Monte Carlo means over 200,000 simulated tasks at each p (seed 20261005). Blue: the unbiased estimator C(c,3)\/C(5,3), whose simulated bias stays within 0.1 points of zero.<\/figcaption><\/figure>\n<p style=\"margin: 0 0 1rem\">The bias is not academic, and ThinkingBox shows it in published numbers. The arXiv version of the paper reports pass^20 with the plug-in estimator (c\/N)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>, and says so: the unbiased form is &#8220;necessarily zero whenever&#8221; c &lt; k, so the authors preferred an estimator that keeps resolution on hard tasks.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-2\" style=\"color: #0000ee;text-decoration: none\">[2]<\/a><\/sup> When n = k = 20, the unbiased estimator is simply the share of tasks that passed 20 of 20, which is what the Hugging Face post reports as &#8220;observed 20\/20&#8221;. The two columns therefore come from the same trials with different estimators. For Kimi-K3 the plug-in gives 17.60% and the observed count gives 68 of 507, 13.41%. For GPT-5.4 they are 30.62% and 25.25%.<\/p>\n<p style=\"margin: 0 0 1rem\">The gap is the bias the convexity argument predicts. Fitting a Beta distribution to Kimi-K3&#8217;s pass@1 and 20\/20 count (method below) and taking the expectation of (c\/20)<sup style=\"font-size: 0.72em;line-height: 0\">20<\/sup> under the fitted beta-binomial gives 17.2%, within half a point of the paper&#8217;s 17.60%. The same fit predicts Kimi-K3&#8217;s pass@20 at 94.3% against an observed 93.89%.<\/p>\n<h2 id=\"task-heterogeneity-not-the-per-call-rate-decides-how-fast-passk-falls\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">Task heterogeneity, not the per-call rate, decides how fast pass^k falls<\/h2>\n<p style=\"margin: 0 0 1rem\">Equation (1) hides the most important modelling fact: pass^k depends on the whole distribution of p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> across tasks, not on its mean. Jensen&#8217;s inequality again says E[p<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>] \u2265 (E[p])<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>, so the naive model, in which every call succeeds independently at the pass@1 rate, is a lower bound on pass^k. How far above the bound the truth sits depends on how spread out the tasks are.<\/p>\n<p style=\"margin: 0 0 1rem\">A Beta distribution is the natural model for per-task rates. Parameterise it by its mean m = \u03b1\/(\u03b1 + \u03b2) and its concentration \u03bd = \u03b1 + \u03b2. The moments are closed-form products:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">pass^k = E[p<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>] = \u220f<sub style=\"font-size: 0.72em;line-height: 0\">j=0<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">k\u22121<\/sup> (\u03b1 + j)\/(\u03bd + j) \u00a0\u00a0\u00a0 pass@k = 1 \u2212 \u220f<sub style=\"font-size: 0.72em;line-height: 0\">j=0<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">k\u22121<\/sup> (\u03b2 + j)\/(\u03bd + j) \u00a0\u00a0<span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(4)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">The intraclass correlation between two attempts at the same task is \u03c1 = Var(p)\/[m(1 \u2212 m)] = 1\/(1 + \u03bd). Small \u03bd means tasks are mostly always-pass or always-fail; large \u03bd means every task sits near m.<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">Figure 3 takes two agents with the same pass@1 of 60%. Agent A has \u03bd = 0.5, so its tasks split into ones it reliably solves and ones it reliably fails. Agent B has \u03bd = 20, so every task is a 60% coin. At k = 3, A&#8217;s pass^3 is 47.8% and B&#8217;s is 23.6%. At k = 8 the figures are 39.2% and 3.4%. The ordering reverses for pass@k: B solves 92.2% of tasks at least once in three tries, A only 71.8%. A leaderboard reporting only pass@k ranks them backwards for deployment.<\/p>\n<figure class=\"viz\" style=\"margin: 2.2rem 0\"><div class=\"viz-card\" style=\"border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb;padding: 14px 14px 8px;overflow: auto\"><a href=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-3.webp\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" class=\"wp-image-957\" src=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-3.webp\" alt=\"Two computed panels. Both agents succeed on 60% of single calls. Agent A&apos;s per-task success rates follow a Beta distribution with concentration 0.5, so tasks are mostly always-pass or always-fail; Agent B&apos;s follow concentration 20, so every task sits near 60%. pass^k: A falls to 47.8% at k=3, 39.2% at k=8 and 32.6% at k=20; B falls to 23.6%, 3.4% and 0.1%. pass@k goes the other way: B reaches 92.2% at k=3 while A reaches 71.8%.\" width=\"1520\" height=\"910\" loading=\"lazy\" style=\"display:block;width:100%;min-width:560px;height:auto;max-width:none\" srcset=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-3.webp 1520w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-3-300x180.webp 300w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-3-1024x613.webp 1024w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-3-768x460.webp 768w\" sizes=\"auto, (max-width: 1520px) 100vw, 1520px\" \/><\/a><\/div><figcaption class=\"viz-caption\" style=\"margin-top: 0.65rem;font-size: 0.9rem;line-height: 1.55;color: #52514e\"><strong style=\"color: #0b0b0b\">Figure 3.<\/strong> Two simulated agents with the same 60% pass@1. Per-task success follows Beta(0.3, 0.2) for Agent A and Beta(12, 8) for Agent B; curves are the exact moments of equation (4). pass^k and pass@k rank the agents in opposite orders, and pass@1 cannot separate them.<\/figcaption><\/figure>\n<p style=\"margin: 0 0 1rem\">Real agents show the same split. In ThinkingBox, Kimi-K3 behaves like Agent B (broad coverage, little consistency) and Claude Opus 5 like Agent A. Fitting \u03bd to each model&#8217;s pass@1 and observed 20\/20 share gives \u03c1 \u2248 0.43 for Kimi-K3 and 0.76 for Opus 5.<\/p>\n<p style=\"margin: 0 0 1rem\">APEX-Voice gives a voice example, and its three published numbers are enough to reconstruct each agent&#8217;s outcome histogram. With 120 workflows and three runs each, Reliable@3 fixes the number of workflows with three successes, Pass@3 fixes the number with none, and Pass@1 fixes the total number of successes. The remaining two counts follow by subtraction. GPT-realtime-2.1 and Grok-Voice-Think-2.0 have almost the same Pass@1 (23.6% and 23.1%) but different shapes, as Table 1 shows.<\/p>\n<div class=\"table-wrap\" style=\"max-width: 100%;overflow: auto;margin: 1.5rem 0\"> <table style=\"width: 100%;min-width: 640px;border-collapse: collapse;font-size: 0.9rem;line-height: 1.5\"> <caption style=\"text-align: left;font-weight: 700;margin-bottom: 0.7rem;color: #0b0b0b\">Table 1. APEX-Voice outcome histograms reconstructed from published Pass@1, Pass@3 and Reliable@3 (120 workflows, 3 runs)<\/caption> <thead><tr><th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b\">Agent<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">Pass@1<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">Pass@3<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">Reliable@3<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">0 \/ 1 \/ 2 \/ 3 successes<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">ICC \u03c1<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">p\u0304<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup> if calls independent<\/th><\/tr><\/thead> <tbody> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;white-space: nowrap\">GPT-realtime-2.1<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">23.6%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">36.7%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">10.8%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">76 \/ 16 \/ 15 \/ 13<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.52<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">1.3%<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;white-space: nowrap\">Grok-Voice-Think-2.0<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">23.1%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">41.7%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">5.0%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">70 \/ 23 \/ 21 \/ 6<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.31<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">1.2%<\/td><\/tr> <\/tbody> <\/table> <\/div>\n<p style=\"margin: 0 0 1rem\">The independence model would have predicted a Reliable@3 near 1.3% for both agents. The observed 10.8% and 5.0% are four to eight times higher, because failures cluster on particular workflows. The ICC column uses the factorial moment E[c(c \u2212 1)] = n(n \u2212 1)E[p<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>], which gives Var(p) and therefore \u03c1. One caution: the difference between 13 and 6 reliable workflows out of 120 has a Fisher exact p-value of 0.15 on unpaired counts, and the task-bootstrap interval for GPT-realtime-2.1&#8217;s 10.8% runs from 5.8% to 16.7%. A paired test needs per-workflow results, which the paper does not publish.<\/p>\n<h2 id=\"a-lenient-grader-inflates-passk-more-than-pass1\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">A lenient grader inflates pass^k more than pass@1<\/h2>\n<p style=\"margin: 0 0 1rem\">Now let the grader be imperfect. Suppose it accepts a failed call with probability f (its false-positive rate) and never rejects a true success. If grading errors are independent across calls given the task, the graded success probability of task i is p&#8217;<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> = p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> + f(1 \u2212 p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>) = 1 \u2212 (1 \u2212 f)q<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>, where q<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> = 1 \u2212 p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> is the true failure rate. The grader shrinks every failure rate by the factor (1 \u2212 f), and the measured pass^k becomes:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">pass^k<sub style=\"font-size: 0.72em;line-height: 0\">measured<\/sub> = E[ (1 \u2212 (1 \u2212 f)q)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup> ] \u2248 E[ e<sup style=\"font-size: 0.72em;line-height: 0\">\u2212k(1\u2212f)q<\/sup> ] \u2248 pass^<sup style=\"font-size: 0.72em;line-height: 0\">k(1\u2212f)<\/sup> <span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(5)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">For small failure rates, a grader with false-positive rate f makes pass^k look like the true pass^k(1\u2212f). It stretches the apparent reliability horizon by a factor of 1\/(1 \u2212 f).<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">Equation (5) is the reason grading on transcripts is worse for pass^k than for pass@1. For a homogeneous agent with p = 0.6 and a grader with f = 0.5, measured pass@1 is 0.8, a 1.33\u00d7 inflation. Measured pass^3 is 0.512 against a true 0.216, 2.37\u00d7, and measured pass^8 is ten times the truth. The inflation compounds because each of the k calls gets its own chance to be forgiven. With f = 0.6724, ThinkingBox&#8217;s weak-proxy acceptance rate, a measured pass^10 equals the true pass^3.2 for a task with a 5% failure rate. The approximation k(1 \u2212 f) gives 3.28.<\/p>\n<p style=\"margin: 0 0 1rem\">Heterogeneity damps the effect. Figure 4 applies equation (5) to the Beta fit for Kimi-K3. The true curve falls from 57.3% at k = 1 to 13.4% at k = 20. A grader with the weak-proxy rate reports 86.0% and 25.5%: a 1.5\u00d7 inflation at k = 1 and 1.9\u00d7 at k = 20. A Monte Carlo run of 400,000 simulated tasks at f = 0.3 and k = 8 matched the analytic value to 0.06 points.<\/p>\n<figure class=\"viz\" style=\"margin: 2.2rem 0\"><div class=\"viz-card\" style=\"border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb;padding: 14px 14px 8px;overflow: auto\"><a href=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4.webp\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" class=\"wp-image-958\" src=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4.webp\" alt=\"Computed chart. True pass^k for an agent whose per-task success follows a Beta fit to Kimi-K3&apos;s ThinkingBox numbers falls from 57.3% at k=1 to 13.4% at k=20. A grader that accepts 30% of failed calls reports 70.1% and 16.4%. A grader with ThinkingBox&apos;s 67.24% weak-proxy acceptance rate reports 86.0% and 25.5%. Errors are assumed independent across calls.\" width=\"1520\" height=\"870\" loading=\"lazy\" style=\"display:block;width:100%;min-width:560px;height:auto;max-width:none\" srcset=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4.webp 1520w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4-300x172.webp 300w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4-1024x586.webp 1024w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4-768x440.webp 768w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-4-1400x800.webp 1400w\" sizes=\"auto, (max-width: 1520px) 100vw, 1520px\" \/><\/a><\/div><figcaption class=\"viz-caption\" style=\"margin-top: 0.65rem;font-size: 0.9rem;line-height: 1.55;color: #52514e\"><strong style=\"color: #0b0b0b\">Figure 4.<\/strong> Measured pass^k under graders that accept a share f of failed calls, with errors independent across calls (equation 5). The agent&#8217;s per-task success is a Beta distribution (mean 0.5737, concentration 1.32) fitted to Kimi-K3&#8217;s ThinkingBox pass@1 and observed 20\/20 share. f = 0.6724 is the share of ThinkingBox&#8217;s failed attempts accepted by its strictest completion proxy (clean ending, state-changing call, no final tool error).<\/figcaption><\/figure>\n<p style=\"margin: 0 0 1rem\">Grading errors are rarely independent, though. A judge that misses a silent tool failure on one call of a task will probably miss it on the next, because the failure looks the same. Model the extreme case: with probability f per task, the grader accepts every failure on that task. Then the measured pass^k = pass^k + f(1 \u2212 pass^k), a constant pull towards 1 instead of a stretched horizon. For the Kimi-like agent at f = 0.3, measured pass^8 becomes 45.4% against a true 22.0%, worse than the 27.1% under independent errors. Correlated grader errors therefore inflate pass^k the most, and they are the errors a transcript judge makes when a failure leaves no trace in the words.<\/p>\n<p style=\"margin: 0 0 1rem\">The practical conclusion is that a pass^k number is only as good as its grader&#8217;s false-positive rate, and the rate needs measuring. A team can audit a sample of calls the grader passed against backend state; VAmoS&#8217;s 54.2% is the quantity that audit estimates, and f follows from it once the true failure rate is known. Perit&#8217;s post on rubric grading, <a href=\"https:\/\/perit.ai\/blogs\/the-model-got-8-10-for-what\/\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: none\" target=\"_blank\">The Model Got 8\/10. For What?<\/a>, starts from the same failure: an assistant that says a refund is on its way, with no request submitted, still scores eight out of ten.<\/p>\n<h2 id=\"tasks-are-the-unit-of-replication-so-the-confidence-interval-lives-at-\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">Tasks are the unit of replication, so the confidence interval lives at the task level<\/h2>\n<p style=\"margin: 0 0 1rem\">A benchmark with M tasks and n repeats contains Mn graded calls, and it is tempting to put a binomial interval around the pooled success rate as if those calls were independent. They are not. Calls on the same task share p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>. The law of total variance gives the variance of the pooled pass@1 estimate p\u0302:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">Var(p\u0302) = [ \u03c4<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup> + E[p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>(1 \u2212 p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>)]\/n ] \/ M = [ m(1 \u2212 m)\/(Mn) ] \u00b7 [ 1 + (n \u2212 1)\u03c1 ] \u00a0\u00a0<span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(6)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">\u03c4<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup> = Var(p<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>) is the between-task variance and \u03c1 = \u03c4<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>\/[m(1 \u2212 m)] the intraclass correlation. The bracket is the survey statistician&#8217;s design effect: the factor by which clustering inflates the variance of a naive estimate.<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">The design effect grows with the number of repeats. At \u03c1 = 0.5 and n = 3 it is 2, so the honest interval is 1.41 times wider than the naive one. At n = 20 and \u03c1 = 0.48 it is about 10, and a naive 95% interval covers the truth only about 46% of the time.<\/p>\n<p style=\"margin: 0 0 1rem\">ThinkingBox&#8217;s published intervals let us test equation (6) on real data. The paper reports 95% task-cluster bootstrap intervals for pass@1, resampling the 507 tasks and keeping all 20 outcomes of each.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-2\" style=\"color: #0000ee;text-decoration: none\">[2]<\/a><\/sup> Dividing each reported half-width by the naive binomial half-width over 10,140 calls gives the square root of the design effect, and from that an implied \u03c1. Table 2 compares it with the \u03c1 from the Beta fit to each model&#8217;s 20\/20 share, a different route through the data. Half-widths are in percentage points.<\/p>\n<div class=\"table-wrap\" style=\"max-width: 100%;overflow: auto;margin: 1.5rem 0\"> <table style=\"width: 100%;min-width: 640px;border-collapse: collapse;font-size: 0.9rem;line-height: 1.5\"> <caption style=\"text-align: left;font-weight: 700;margin-bottom: 0.7rem;color: #0b0b0b\">Table 2. Two estimates of the same intraclass correlation from ThinkingBox&#8217;s published numbers<\/caption> <thead><tr><th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b\">Model<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">pass@1<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">Reported CI \u00b1<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">Naive CI \u00b1<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">Design effect<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">\u03c1 (from CI)<\/th><th class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b;text-align: right;white-space: nowrap\">\u03c1 (Beta fit)<\/th><\/tr><\/thead> <tbody> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;white-space: nowrap\">Kimi-K3<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">57.37%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b13.06<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b10.96<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">10.1<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.48<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.43<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;white-space: nowrap\">GPT-5.4<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">65.36%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b13.14<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b10.93<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">11.5<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.55<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.50<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;white-space: nowrap\">Claude Opus 5<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">66.50%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b13.71<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b10.92<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">16.3<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.81<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.76<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;white-space: nowrap\">GPT-6 Astra<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">58.31%<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b13.97<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">\u00b10.96<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">17.1<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.85<\/td><td class=\"num\" style=\"vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0;text-align: right;white-space: nowrap\">0.85<\/td><\/tr> <\/tbody> <\/table> <\/div>\n<p style=\"margin: 0 0 1rem\">The two routes agree to within 0.05: the clustered model fits, and naive intervals on this benchmark would be three to four times too narrow.<\/p>\n<p style=\"margin: 0 0 1rem\">A simulation confirms the coverage claim. With 100 tasks, three repeats and Beta-distributed p (mean 0.7), 2,000 replicate benchmarks gave naive binomial coverage of 91.7%, 87.7% and 82.6% at \u03c1 = 0.1, 0.3 and 0.5. Task-cluster bootstrap coverage was 95.0%, 94.9% and 94.4%. The bootstrap interval was 1.08, 1.25 and 1.40 times wider than the naive one, close to the \u221a(1 + 2\u03c1) of equation (6): 1.10, 1.26 and 1.41.<\/p>\n<p style=\"margin: 0 0 1rem\">For pass^k with n = k, each task contributes a single all-pass indicator, so a binomial interval over tasks is valid. The trap is different there: estimating pass^k as (pooled p\u0302)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>, which assumes the independence that Table 1 refutes. Cekura&#8217;s public voice benchmark shows how wide honest intervals are at typical scale. Its leader passed all three retained runs on 62 of 82 scenarios, 75.61% pass\u00b3; the Wilson interval is 65.3% to 83.6%, and the fourth-ranked platform&#8217;s 56 of 82 sits inside it.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-5\" style=\"color: #0000ee;text-decoration: none\">[5]<\/a><\/sup><\/p>\n<h2 id=\"a-different-human-caller-on-each-repeat-is-a-second-random-effect\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">A different human caller on each repeat is a second random effect<\/h2>\n<p style=\"margin: 0 0 1rem\">With human callers, each repeat also draws a person. Two callers given the same scenario phrase it differently, interrupt at different moments and react differently to a wrong answer. Who calls on each repeat changes both what pass^k means and how precisely it is measured.<\/p>\n<p style=\"margin: 0 0 1rem\">A simple random-effects model makes this concrete. Let the outcome of a call on task i by caller j be Bernoulli with logit(p<sub style=\"font-size: 0.72em;line-height: 0\">ij<\/sub>) = \u03bc + u<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> + v<sub style=\"font-size: 0.72em;line-height: 0\">j<\/sub>, with task effects u<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> ~ N(0, \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">T<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>) and caller effects v<sub style=\"font-size: 0.72em;line-height: 0\">j<\/sub> ~ N(0, \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">C<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>). In deployment every call comes from a new member of the caller population, so the quantity a business cares about is the task&#8217;s caller-averaged success rate p\u0303<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub> = E<sub style=\"font-size: 0.72em;line-height: 0\">v<\/sub>[p<sub style=\"font-size: 0.72em;line-height: 0\">ij<\/sub>], and deployment pass^k = E<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>[p\u0303<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>]. If instead one caller plays all k repeats of a task, the measured quantity is E<sub style=\"font-size: 0.72em;line-height: 0\">i<\/sub>E<sub style=\"font-size: 0.72em;line-height: 0\">j<\/sub>[p<sub style=\"font-size: 0.72em;line-height: 0\">ij<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>], which by Jensen is larger. Same-caller repeats measure whether the agent is consistent with one person, which is a weaker property.<\/p>\n<p style=\"margin: 0 0 1rem\">With \u03bc = 1.2, \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">T<\/sub> = 1.5 and \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">C<\/sub> = 0.6 on the logit scale (illustrative values giving a pass@1 of 69.5%), deployment pass^3 is 43.9% and same-caller pass^3 is 45.6%. The level bias is modest here and grows with \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">C<\/sub>. The variance effect is larger. Callers are crossed with tasks: the same person handles calls across many scenarios, so a caller who is unusually clear lifts many tasks at once. To first order the variance of a benchmark estimate becomes:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">Var(estimate) \u2248 \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">T<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>\/M + \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">C<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>\/J + \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">e<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>\/(Mn) \u00a0\u00a0<span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(7)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">J is the number of distinct callers. When J is small the caller term dominates, however many scenarios are run, and a bootstrap that resamples only tasks cannot see it.<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">Simulation confirms equation (7). With 200 tasks, three repeats and callers drawn at random for each call, the true standard deviation of the pass^3 estimate across 1,500 replicate experiments was 5.6 points with 8 distinct callers, 4.1 with 30 and 3.7 with 200. A task-only bootstrap reported 3.5 points every time, understating the spread by a factor of 1.6 with 8 callers. Crossed designs need a bootstrap that resamples tasks and callers independently, such as Owen&#8217;s pigeonhole bootstrap,<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-12\" style=\"color: #0000ee;text-decoration: none\">[12]<\/a><\/sup> or a fitted random-effects model.<\/p>\n<p style=\"margin: 0 0 1rem\">This is the frame for reading Perit&#8217;s <a href=\"https:\/\/perit.ai\/rl-environments\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: none\" target=\"_blank\">RL environments<\/a> page, which runs live speech-to-speech calls with personas &#8220;played by trained speakers&#8221; and lists &#8220;8 runs per persona, bootstrapped CI&#8221;.<sup style=\"font-size: 0.72em;line-height: 0\"><a href=\"#passk-ref-10\" style=\"color: #0000ee;text-decoration: none\">[10]<\/a><\/sup> Eight runs per persona is enough for the unbiased pass^k of equation (3) up to k = 8. It also estimates pass^3 better than a single three-run block, since C(c, 3)\/C(8, 3) averages all 56 triples of runs. The interval&#8217;s width, though, comes from the number of personas, scenarios and distinct speakers, not from the eight. The page does not say whether one speaker or several play a persona&#8217;s eight runs, and the answer decides which pass^k the runs estimate. The page also scores task success against a rubric written by an operator from the sector. A team that wants pass^k graded on terminal state would export its backend state at hang-up for each call and check it against the scenario&#8217;s expected record.<\/p>\n<h2 id=\"worked-design-how-to-spend-1200-human-calls\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">Worked design: how to spend 1,200 human calls<\/h2>\n<p style=\"margin: 0 0 1rem\">Human calls are the expensive input, so the question is how to split a fixed budget between scenarios M and repeats n. Take a budget of 1,200 calls and a target of pass^3 for a voice agent whose per-scenario success follows a Beta distribution with mean 0.70 and \u03c1 = 0.40. That \u03c1 sits between the two APEX-Voice agents and close to Kimi-K3&#8217;s. True pass^3 for this population is 50.0%. Each scenario contributes the unbiased estimate C(c, 3)\/C(n, 3), and its variance splits into a between-scenario part V<sub style=\"font-size: 0.72em;line-height: 0\">B<\/sub> = Var(p<sup style=\"font-size: 0.72em;line-height: 0\">3<\/sup>) = 0.1345 and a within-scenario part that shrinks with n: 0.116 at n = 3, 0.048 at n = 6 and 0.023 at n = 12.<\/p>\n<p style=\"margin: 0 0 1rem\">Each scenario also carries a fixed cost s, in call-equivalents: writing the scenario and its expected end state, seeding the backend, and briefing callers on the persona. The budget constraint is M(s + n) = 1,200. Minimising the variance (V<sub style=\"font-size: 0.72em;line-height: 0\">B<\/sub> + W\/n)\/M subject to that constraint, with W \u2248 0.28 the within-scenario variance per repeat, gives the classical two-stage sampling optimum:<\/p>\n<div class=\"equation-note\" style=\"margin: 1.5rem 0;padding: 0.9rem 1rem;background-color: #fcfcfb;border: 1px solid #e6e5e0;border-left: 3px solid #2a78d6;border-radius: 6px\"> <span class=\"eq\" style=\"display: block;text-align: center;font-family: &quot;Times New Roman&quot;, Georgia, serif;font-size: 1.15rem;margin: 0.4rem 0;color: #0b0b0b\">n* = \u221a( s \u00b7 W \/ V<sub style=\"font-size: 0.72em;line-height: 0\">B<\/sub> ), \u00a0\u00a0 subject to n* \u2265 k \u00a0\u00a0<span class=\"eq-no\" style=\"color: #8a8984;font-size: 0.9rem\">(8)<\/span><\/span> <p class=\"eq-text\" style=\"margin: 0.5rem 0 0;font-size: 0.92rem;color: #52514e\">Repeat more only when scenarios are expensive relative to calls, or when outcomes within a scenario are noisy relative to the spread between scenarios. For s = 10 the formula gives 4.6; for s = 30 it gives 7.9.<\/p> <\/div>\n<p style=\"margin: 0 0 1rem\">Figure 5 computes the exact half-widths from the beta-binomial distribution. With s = 0, the optimum is the smallest legal design: 400 scenarios run three times, for a half-width of \u00b14.9 points. Six repeats on 200 scenarios gives \u00b15.9, and twelve repeats on 100 scenarios \u00b17.8. With s = 10 the optimum moves to five repeats on 80 scenarios (\u00b19.7, against \u00b110.2 at n = 3). With s = 30 it moves to eight repeats on 32 scenarios (\u00b114.4, against \u00b116.3 at n = 3). The exact optima match equation (8).<\/p>\n<figure class=\"viz\" style=\"margin: 2.2rem 0\"><div class=\"viz-card\" style=\"border: 1px solid #e6e5e0;border-radius: 10px;background-color: #fcfcfb;padding: 14px 14px 8px;overflow: auto\"><a href=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5.webp\" target=\"_blank\" rel=\"noopener\"><img decoding=\"async\" class=\"wp-image-959\" src=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5.webp\" alt=\"Computed chart. Population: per-scenario success Beta with mean 0.70 and intraclass correlation 0.40, true pass^3 = 50%. With no per-scenario setup cost the narrowest interval is at 3 repeats and 400 scenarios, plus or minus 4.9 points. With a setup cost worth 10 calls per scenario the optimum moves to 5 repeats and 80 scenarios, plus or minus 9.7 points. With 30 calls of setup it moves to 8 repeats and 32 scenarios, plus or minus 14.4 points.\" width=\"1520\" height=\"870\" loading=\"lazy\" style=\"display:block;width:100%;min-width:560px;height:auto;max-width:none\" srcset=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5.webp 1520w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5-300x172.webp 300w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5-1024x586.webp 1024w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5-768x440.webp 768w, https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-figure-5-1400x800.webp 1400w\" sizes=\"auto, (max-width: 1520px) 100vw, 1520px\" \/><\/a><\/div><figcaption class=\"viz-caption\" style=\"margin-top: 0.65rem;font-size: 0.9rem;line-height: 1.55;color: #52514e\"><strong style=\"color: #0b0b0b\">Figure 5.<\/strong> Exact 95% interval half-width for the unbiased pass^3 estimator, for a budget of 1,200 calls split as M = 1,200\/(s + n) scenarios of n repeats. Scenario success follows a Beta distribution with mean 0.70 and intraclass correlation 0.40 (true pass^3 = 50.0%). Dots mark each optimum. The caller variance term of equation (7) is not included.<\/figcaption><\/figure>\n<p style=\"margin: 0 0 1rem\">The s = 0 design recovers a familiar rule of thumb. With n = k = 3 each scenario&#8217;s estimate is a Bernoulli indicator with variance at most 0.25, and 1.96<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup> \u00d7 0.25 \/ 0.05<sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup> = 384 scenarios give \u00b15 points in the worst case. That is 1,152 calls, and 400 scenarios use the whole budget.<\/p>\n<p style=\"margin: 0 0 1rem\">Two refinements change the numbers but not the logic. The caller term of equation (7) adds \u03c3<sub style=\"font-size: 0.72em;line-height: 0\">C<\/sub><sup style=\"font-size: 0.72em;line-height: 0\">2<\/sup>\/J whatever the split, so 1,200 calls over 8 callers buy much less precision than over 40. And a team that also wants pass^5 or pass^8 needs n to reach that k.<\/p>\n<div class=\"table-wrap\" style=\"max-width: 100%;overflow: auto;margin: 1.5rem 0\"> <table style=\"width: 100%;min-width: 640px;border-collapse: collapse;font-size: 0.9rem;line-height: 1.5\"> <caption style=\"text-align: left;font-weight: 700;margin-bottom: 0.7rem;color: #0b0b0b\">Table 3. What a pass^k result should state, and why<\/caption> <thead><tr><th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b\">Item<\/th><th style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 2px solid #0b0b0b;font-weight: 700;color: #0b0b0b\">Why it changes the number<\/th><\/tr><\/thead> <tbody> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Grader: terminal state, tool calls, transcript or caller report<\/td><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">A false-positive rate f stretches the apparent horizon from k to about k\/(1 \u2212 f); correlated misses inflate it further<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Measured false-positive rate of the grader, from an audit against backend state<\/td><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Without it the reader cannot correct for equation (5)<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Estimator: unbiased C(c,k)\/C(n,k) or plug-in (c\/n)<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup><\/td><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">The plug-in overstates pass^k; at n = k = 20 it turned 13.41% into 17.60% for one model<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">M scenarios, n repeats, J distinct callers, and who called on each repeat<\/td><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Precision depends on M and J, not on Mn; same-caller repeats measure a weaker property<\/td><\/tr> <tr><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">Interval method: task-cluster bootstrap, or two-way when callers are few<\/td><td style=\"text-align: left;vertical-align: top;padding: 0.6rem;border-bottom: 1px solid #e6e5e0\">A call-level binomial interval is too narrow by \u221a(1 + (n \u2212 1)\u03c1)<\/td><\/tr> <\/tbody> <\/table> <\/div>\n<p style=\"margin: 0 0 1rem\">Teams that already instrument agents for production, as Perit&#8217;s post on <a href=\"https:\/\/perit.ai\/blogs\/engineering-reliable-ai-agents-for-production-systems\/\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: none\" target=\"_blank\">engineering reliable AI agents<\/a> describes, have most of this in place. The terminal-state check that grades an evaluation call is the same check that should gate a write in production.<\/p>\n<h2 id=\"questions-teams-ask-about-passk\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">Questions teams ask about pass^k<\/h2>\n<h3 style=\"font-size: 1.05rem;line-height: 1.4;margin: 1.6rem 0 0.45rem;font-weight: 700;color: #0b0b0b\">What is pass^k?<\/h3>\n<p style=\"margin: 0 0 1rem\">pass^k is the probability that an AI agent succeeds on all k independent attempts at the same task, averaged over tasks. It was introduced in \u03c4-bench in 2024 to measure consistency for agents that act on databases. For a task with per-attempt success p it equals p<sup style=\"font-size: 0.72em;line-height: 0\">k<\/sup>. From n trials with c successes, the unbiased estimate is C(c,k)\/C(n,k).<\/p>\n<h3 style=\"font-size: 1.05rem;line-height: 1.4;margin: 1.6rem 0 0.45rem;font-weight: 700;color: #0b0b0b\">pass@k vs pass^k: what is the difference?<\/h3>\n<p style=\"margin: 0 0 1rem\">pass@k is the chance that at least one of k attempts succeeds; pass^k is the chance that all k succeed. pass@k rises with k and suits settings where a verifier can pick the best attempt, such as code with unit tests. pass^k falls with k and suits deployed agents, where every user gets exactly one attempt and nobody reruns it.<\/p>\n<h3 style=\"font-size: 1.05rem;line-height: 1.4;margin: 1.6rem 0 0.45rem;font-weight: 700;color: #0b0b0b\">How many times should you run an AI agent eval?<\/h3>\n<p style=\"margin: 0 0 1rem\">At least k times per task for the largest pass^k you report, and usually no more than that when scenarios are cheap. Precision comes mainly from the number of tasks, because repeats on one task are correlated. When each scenario is expensive to build, the optimal repeat count is about \u221a(s\u00b7W\/V<sub style=\"font-size: 0.72em;line-height: 0\">B<\/sub>), which reached five to eight repeats in a worked example.<\/p>\n<h3 style=\"font-size: 1.05rem;line-height: 1.4;margin: 1.6rem 0 0.45rem;font-weight: 700;color: #0b0b0b\">How do you measure AI agent reliability?<\/h3>\n<p style=\"margin: 0 0 1rem\">Define success on the terminal state the agent leaves in the backend, not on the transcript. Run each task several times from a clean state, with different human callers for voice agents. Report pass@1 and pass^k with the unbiased estimator, give the grader&#8217;s audited false-positive rate, and compute intervals by resampling tasks, and callers when there are few of them.<\/p>\n<h2 id=\"back-to-the-payment-date\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">Back to the payment date<\/h2>\n<p style=\"margin: 0 0 1rem\">The caller who wanted the 15th heard a confident agent and hung up satisfied. A transcript grader would have scored the call a pass, and a pass^k computed on such grades would have counted it towards a reliable agent. Graded on the record, the call is a failure. Repeated with three different callers, it becomes the measurement that matters: whether the agent moves the date correctly every time, for anyone who asks.<\/p>\n<p style=\"margin: 0 0 1rem\">Each step in between has a trap: an estimator that flatters, a grader that forgives, an interval that treats correlated calls as independent, and a caller pool too small to average over. Teams that want to run this protocol with real callers can connect an agent to Perit&#8217;s <a href=\"https:\/\/perit.ai\/rl-environments\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: none\" target=\"_blank\">RL environments<\/a> and read the <a href=\"https:\/\/perit.ai\/research\/method\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: none\" target=\"_blank\">method page<\/a> for how intervals are produced. The questions to settle before the first call are the ones in Table 3.<\/p>\n<div class=\"refs\"> <h2 id=\"references-and-links\" style=\"font-size: 1.28rem;line-height: 1.35;margin: 2.5rem 0 0.7rem;font-weight: 700;color: #0b0b0b\">References and links<\/h2> <ol style=\"margin: 0.35rem 0 1.2rem 1.35rem;padding: 0;font-size: 0.9rem;line-height: 1.55\"> <li id=\"passk-ref-1\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">T. Kundu et al. (Microsoft and Hugging Face), <a href=\"https:\/\/huggingface.co\/blog\/microsoft\/thinkingbox\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;The Agent Said It Was Done. The Database Disagreed.&#8221;<\/a>, Hugging Face blog, 3 October 2026. Source of the 93.89% and 13.41% (observed 20\/20) figures for Kimi-K3, the 79.09% and 47.53% figures for Claude Opus 5, and the 67.24% headline.<\/li> <li id=\"passk-ref-2\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">Z. Li et al., <a href=\"https:\/\/arxiv.org\/abs\/2608.19741\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;One Success Isn&#8217;t Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows&#8221;<\/a>, arXiv:2608.19741, v4, 1 October 2026. Table 15 (pass^20 with the plug-in estimator, task-cluster bootstrap intervals), Table 16 (evaluator ablation over 79,853 failed trials), Appendix C.7 (estimators and variance).<\/li> <li id=\"passk-ref-3\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">J. Meyer et al. (Veris AI), <a href=\"https:\/\/arxiv.org\/abs\/2609.38512\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;VAmoS Part Deux: Harder, More Realistic Voice-Agent Simulation&#8221;<\/a>, arXiv:2609.38512, 29 September 2026. Source of the 2,072 and 1,122 call counts and the quotation on callers who cannot inspect actions.<\/li> <li id=\"passk-ref-4\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">P. Mathur and D. Manocha, <a href=\"https:\/\/arxiv.org\/abs\/2609.34973\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction&#8221;<\/a>, arXiv:2609.34973, 28 September 2026. Table 2 gives Pass@1, Pass@3 and Reliable@3 for each agent over 120 workflows and three runs.<\/li> <li id=\"passk-ref-5\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">Cekura, <a href=\"https:\/\/benchmarks.cekura.ai\/voice-agent-workflows\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">Cekura Bench: Agent Workflow<\/a> (updated 29 September 2026) and <a href=\"https:\/\/benchmarks.cekura.ai\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">benchmark home<\/a> (updated 1 October 2026). Pass\u00b3 is &#8220;the share of scenarios where all three retained test calls passed&#8221;; 8 configurations, 82 scenarios, 3 repeats.<\/li> <li id=\"passk-ref-6\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">M. Chen et al., <a href=\"https:\/\/arxiv.org\/abs\/2107.03374\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;Evaluating Large Language Models Trained on Code&#8221;<\/a>, arXiv:2107.03374, 2021. The unbiased pass@k estimator and its numerically stable product form.<\/li> <li id=\"passk-ref-7\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">S. Yao, N. Shinn, P. Razavi, K. Narasimhan, <a href=\"https:\/\/arxiv.org\/abs\/2406.12045\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;\u03c4-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains&#8221;<\/a>, arXiv:2406.12045, 2024. Definition of pass^k and its unbiased estimator; grading on final database state.<\/li> <li id=\"passk-ref-8\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">V. Barres et al., <a href=\"https:\/\/arxiv.org\/abs\/2506.07982\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;\u03c4\u00b2-Bench: Evaluating Conversational Agents in a Dual-Control Environment&#8221;<\/a>, arXiv:2506.07982, 2025. Reports pass^k for agents and users who both act on shared state.<\/li> <li id=\"passk-ref-9\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">Mercor, <a href=\"https:\/\/mercor.com\/blog\/introducing-apex-agents-1-1\/\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;Introducing APEX-Agents 1.1&#8221;<\/a>, 8 September 2026. Reports Pass@4 and Pass^4; GPT-6 Astra had the highest Pass^4 at 56.3%.<\/li> <li id=\"passk-ref-10\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">Perit, <a href=\"https:\/\/perit.ai\/rl-environments\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">RL Environments<\/a>: live speech-to-speech calls with trained speakers, six scenario families per sector, 8 runs per persona with bootstrapped intervals, rubric-scored task success.<\/li> <li id=\"passk-ref-11\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">Perit, <a href=\"https:\/\/perit.ai\/research\/method\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">Method<\/a>: held-out sets, two-pass references and 95% bootstrapped intervals.<\/li> <li id=\"passk-ref-12\" style=\"font-size: 0.95rem;line-height: 1.55;margin: 0 0 0.45rem;padding-left: 0.15rem\">A. B. Owen, <a href=\"https:\/\/arxiv.org\/abs\/0712.1111\" rel=\"noopener\" style=\"color: #0000ee;text-decoration: underline\" target=\"_blank\">&#8220;The pigeonhole bootstrap&#8221;<\/a>, Annals of Applied Statistics, 2007 (arXiv:0712.1111). Resampling for crossed random effects.<\/li> <\/ol> <\/div>\n<footer aria-label=\"Notes on the figures and sources\" class=\"post-note\" style=\"margin-top: 2rem;padding-top: 1rem;border-top: 1px solid #e6e5e0;font-size: 0.88rem;line-height: 1.5;color: #52514e\"><p style=\"margin: 0\">The opening call and Figure 1 are constructed examples. Figures 2 to 5 and Tables 1 and 2 are computed from the formulas shown or by seeded Monte Carlo simulation; parameters are given in each caption. Beta fits to ThinkingBox match each model&#8217;s pass@1 and observed 20\/20 share and are a model, not the per-task data. ThinkingBox figures differ between sources: the 3 October blog uses observed 20\/20 counts and lists Claude Opus 5.5 as the pass@1 leader (67.16%), while arXiv v4 (1 October) lists Claude Opus 5 first and reports pass^20 with the plug-in estimator, which is why Kimi-K3 appears as 17.60% there and 13.41% in the blog. This post uses the blog for observed counts and the paper for intervals and the evaluator ablation. Table 1&#8217;s histograms assume the published percentages are rounded from whole counts. The caller-effect parameters in section seven and the population in the worked design are illustrative. arXiv papers are preprints dated by submission; Cekura&#8217;s figures are from a live leaderboard and may change. No Perit measurements are reported in this post.<\/p><\/footer>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"<p>pass^k measures whether an AI agent succeeds on every repeated call: how to estimate it without bias, grade on backend state, and budget human calls.<\/p>\n","protected":false},"author":2,"featured_media":954,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,19],"tags":[],"class_list":["post-960","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","category-research"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>pass^k: Measuring AI Agent Reliability Across Repeated Calls - Perit Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"pass^k: Measuring AI Agent Reliability Across Repeated Calls - Perit Blog\" \/>\n<meta property=\"og:description\" content=\"pass^k measures whether an AI agent succeeds on every repeated call: how to estimate it without bias, grade on backend state, and budget human calls.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/\" \/>\n<meta property=\"og:site_name\" content=\"Perit Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-05T12:07:51+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-10-05T12:11:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-thumbnail.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"2400\" \/>\n\t<meta property=\"og:image:height\" content=\"1260\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Pratyaksh@Perit\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@baldwin_IVth\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pratyaksh@Perit\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"27 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/\"},\"author\":{\"name\":\"Pratyaksh@Perit\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/#\\\/schema\\\/person\\\/6b951f8ca802e00473088655d8a2a1bc\"},\"headline\":\"pass^k: Measuring AI Agent Reliability Across Repeated Calls\",\"datePublished\":\"2026-10-05T12:07:51+00:00\",\"dateModified\":\"2026-10-05T12:11:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/\"},\"wordCount\":5354,\"commentCount\":0,\"image\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/pass-k-ai-agent-reliability-thumbnail.webp\",\"articleSection\":[\"Artificial Intelligence\",\"Research\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/\",\"url\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/\",\"name\":\"pass^k: Measuring AI Agent Reliability Across Repeated Calls - Perit Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/pass-k-ai-agent-reliability-thumbnail.webp\",\"datePublished\":\"2026-10-05T12:07:51+00:00\",\"dateModified\":\"2026-10-05T12:11:18+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/#\\\/schema\\\/person\\\/6b951f8ca802e00473088655d8a2a1bc\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#primaryimage\",\"url\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/pass-k-ai-agent-reliability-thumbnail.webp\",\"contentUrl\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/pass-k-ai-agent-reliability-thumbnail.webp\",\"width\":2400,\"height\":1260,\"caption\":\"Passed once. Every time? Repeated success and failure checks illustrate pass^k reliability for voice agents.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/pass-k-ai-agent-reliability\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"pass^k: Measuring AI Agent Reliability Across Repeated Calls\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/#website\",\"url\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/\",\"name\":\"Perit Blog\",\"description\":\"Frontier training data, collected and graded by people who do the work.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/#\\\/schema\\\/person\\\/6b951f8ca802e00473088655d8a2a1bc\",\"name\":\"Pratyaksh@Perit\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/pfp-150x150.webp\",\"url\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/pfp-150x150.webp\",\"contentUrl\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/pfp-150x150.webp\",\"caption\":\"Pratyaksh@Perit\"},\"sameAs\":[\"http:\\\/\\\/perit.ai\\\/blogs\",\"https:\\\/\\\/x.com\\\/baldwin_IVth\"],\"url\":\"https:\\\/\\\/perit.ai\\\/blogs\\\/author\\\/pratyakshpatel\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"pass^k: Measuring AI Agent Reliability Across Repeated Calls - Perit Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/","og_locale":"en_US","og_type":"article","og_title":"pass^k: Measuring AI Agent Reliability Across Repeated Calls - Perit Blog","og_description":"pass^k measures whether an AI agent succeeds on every repeated call: how to estimate it without bias, grade on backend state, and budget human calls.","og_url":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/","og_site_name":"Perit Blog","article_published_time":"2026-10-05T12:07:51+00:00","article_modified_time":"2026-10-05T12:11:18+00:00","og_image":[{"width":2400,"height":1260,"url":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-thumbnail.webp","type":"image\/webp"}],"author":"Pratyaksh@Perit","twitter_card":"summary_large_image","twitter_creator":"@baldwin_IVth","twitter_misc":{"Written by":"Pratyaksh@Perit","Est. reading time":"27 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#article","isPartOf":{"@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/"},"author":{"name":"Pratyaksh@Perit","@id":"https:\/\/perit.ai\/blogs\/#\/schema\/person\/6b951f8ca802e00473088655d8a2a1bc"},"headline":"pass^k: Measuring AI Agent Reliability Across Repeated Calls","datePublished":"2026-10-05T12:07:51+00:00","dateModified":"2026-10-05T12:11:18+00:00","mainEntityOfPage":{"@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/"},"wordCount":5354,"commentCount":0,"image":{"@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#primaryimage"},"thumbnailUrl":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-thumbnail.webp","articleSection":["Artificial Intelligence","Research"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/","url":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/","name":"pass^k: Measuring AI Agent Reliability Across Repeated Calls - Perit Blog","isPartOf":{"@id":"https:\/\/perit.ai\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#primaryimage"},"image":{"@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#primaryimage"},"thumbnailUrl":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-thumbnail.webp","datePublished":"2026-10-05T12:07:51+00:00","dateModified":"2026-10-05T12:11:18+00:00","author":{"@id":"https:\/\/perit.ai\/blogs\/#\/schema\/person\/6b951f8ca802e00473088655d8a2a1bc"},"breadcrumb":{"@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#primaryimage","url":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-thumbnail.webp","contentUrl":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/10\/pass-k-ai-agent-reliability-thumbnail.webp","width":2400,"height":1260,"caption":"Passed once. Every time? Repeated success and failure checks illustrate pass^k reliability for voice agents."},{"@type":"BreadcrumbList","@id":"https:\/\/perit.ai\/blogs\/pass-k-ai-agent-reliability\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/perit.ai\/blogs\/"},{"@type":"ListItem","position":2,"name":"pass^k: Measuring AI Agent Reliability Across Repeated Calls"}]},{"@type":"WebSite","@id":"https:\/\/perit.ai\/blogs\/#website","url":"https:\/\/perit.ai\/blogs\/","name":"Perit Blog","description":"Frontier training data, collected and graded by people who do the work.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/perit.ai\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/perit.ai\/blogs\/#\/schema\/person\/6b951f8ca802e00473088655d8a2a1bc","name":"Pratyaksh@Perit","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/09\/pfp-150x150.webp","url":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/09\/pfp-150x150.webp","contentUrl":"https:\/\/perit.ai\/blogs\/wp-content\/uploads\/2026\/09\/pfp-150x150.webp","caption":"Pratyaksh@Perit"},"sameAs":["http:\/\/perit.ai\/blogs","https:\/\/x.com\/baldwin_IVth"],"url":"https:\/\/perit.ai\/blogs\/author\/pratyakshpatel\/"}]}},"_links":{"self":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/960","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/comments?post=960"}],"version-history":[{"count":3,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/960\/revisions"}],"predecessor-version":[{"id":970,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/posts\/960\/revisions\/970"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/media\/954"}],"wp:attachment":[{"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/media?parent=960"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/categories?post=960"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perit.ai\/blogs\/wp-json\/wp\/v2\/tags?post=960"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}