In short
- MKA evaluates finished knowledge work, not isolated answers. Its 200 private tasks require agents to inspect mixed files, use tools and deliver reports, workbooks, slides or annotated drawings.
- Vision is load-bearing. Removing direct image viewing from the same agent reduced its Climb score by 5.94 percentage points, while input tokens rose 1.85×.
- In one road-inspection case, the vision-disabled agent turned a photograph into ASCII, spent more compute and repeatedly missed defects that the visual agent found.
- The enterprise lesson is practical: evaluate the complete artifact, preserve the original modality, and measure whether extra tokens buy better work rather than more elaborate failure.
The assignment looked ordinary for a civil-engineering team: inspect a road-condition report, compare the photographs against a distress-classification manual, find defects the written log had omitted, and turn the result into a priced workbook.
The evidence was already there. A field photograph showed a depression and a pothole. The contractor’s rates were available. The output format was clear. Nothing required a speculative answer about the world.
One AI agent opened the photograph, zoomed into targeted regions and found the defects. Another version of the same agent had its image-viewing tool removed. It still had the files, code execution, web access and computer-vision libraries. It could extract the image. It could read its pixels. It simply could not look.
So it improvised. The agent reduced the road into rows of ASCII characters—a field of punctuation intended to stand in for light and shade. The pothole became a dark cluster among other dark clusters. Across the reported runs, the defect disappeared from the answer.
Visual agent · targeted image crops → defects found
Vision-disabled agent · pixels → ASCII luminance map → defects missed
This is not a hypothetical vignette. It is a case study in the technical report for the new Multimodal Knowledge Agent benchmark, or MKA, released by Google DeepMind and Artificial Analysis on 10 November 2026. [1] [2]
The photograph had not vanished. The agent had transformed it into a representation that preserved some data while destroying the meaning the task required.
A benchmark made of work, not questions
Most benchmarks make a model’s job easy to recognise. A prompt arrives. The model returns text. A short reference answer decides whether it was right. That format has been enormously useful, but it removes the part of knowledge work where many systems now fail: opening a disorderly bundle, deciding which evidence matters, preserving visual structure, building a deliverable and checking whether the finished object is usable.
MKA was designed around that missing middle. It contains 200 private tasks from 10 sectors and 83 occupations. Each task was authored by a professional with at least five years of relevant experience and reviewed by a second expert. The authors spent between three and eight hours assembling the inputs, producing a gold-standard deliverable, and writing a detailed rubric with justifications. [2]
The inputs resemble an inbox after a real project has been running for a week. Across the benchmark are 1,459 files—an average of 7.3 per task—with bundles ranging from two files to 45. PDFs make up 43% of them; images and scans another 22.9%. Spreadsheets, Word documents, geospatial files, text, JSON, slide decks and CSVs supply the rest.
The required outputs matter just as much. Eighty-eight per cent of tasks demand a standalone file: a spreadsheet, deck, report, HTML document, image or marked-up drawing. The agent is not merely asked what it thinks. It has to leave behind work that another person can open.
That produces tasks such as auditing a construction package, analysing medical images alongside documentation, preparing a market-research deck, reconciling financial records or annotating an engineering drawing. The private 200-task set limits direct benchmark leakage. Ten disjoint showcase tasks—one per sector—are publicly available under Apache 2.0 for inspection and experimentation. [3]
The answer is an artifact
An enterprise deliverable has at least three layers of correctness. Its facts must be right. Its visual structure must survive. And its file must function.
A workbook can contain the correct total and still fail because every value was pasted into one column. A deck can quote the source material accurately while shrinking its charts into unreadable thumbnails. A drawing can identify the right component in prose and place the annotation on the wrong component. A PDF can look polished while omitting the one exception the executive needed.
MKA turns these layers into roughly 70 or more binary rubric criteria per task, weighted from one to five. Google describes the overall rubric mix as approximately 75% factual and 25% visual design. An agentic judge inspects each produced artifact and marks the criteria that pass. Artificial Analysis runs the benchmark independently in a Linux sandbox with internet access, code execution, image viewing and a limit of 250 agent turns. [1] [4]
The task score is a weighted satisfaction rate:
Consider an illustrative four-criterion task—not an actual MKA item—with weights 5, 4, 3 and 1. If the agent passes the first, second and fourth criteria but misses the third, its score is:
This is more informative than declaring the entire task a failure because one check was missed. With 70 independent criteria and a hypothetical 98% chance of passing each, the probability of passing every one would be only 0.9870 ≈ 24.3%. The independence assumption is unrealistic and the calculation is only a toy, but it shows why an all-or-nothing task score becomes brittle as rubrics grow.
The paper also divides criteria into Regression and Climb. Regression criteria are capabilities older systems routinely pass; Climb criteria remain difficult and unsaturated. There are 7,888 Climb criteria across all 200 tasks, while 6,396 Regression criteria cover 180 tasks. The distinction stops a model from appearing frontier-ready merely because it formats the date and copies the title while missing the expert judgement at the centre of the assignment.
The leaderboard is less important than the ceiling
In the paper’s fixed four-model comparison, Claude Fable 5.2 produced the highest Climb score at 56.56%. Gemini 4 Carbon followed at 54.34%, Claude Opus 5.5 at 53.01%, and GPT-6 Astra at 50.24%. Overall scores ranged from 69.34% to 73.26%; Regression scores clustered above 95%. [2]
| Model in paper | Climb | Overall | Regression |
|---|---|---|---|
| Claude Fable 5.2 | 56.56 | 73.26 | 95.71 |
| Gemini 4 Carbon | 54.34 | 71.89 | 96.36 |
| Claude Opus 5.5 | 53.01 | 70.98 | 95.81 |
| GPT-6 Astra | 50.24 | 69.34 | 95.17 |
Scores are percentages from the technical paper’s four-model evaluation, not a claim about the live leaderboard. Confidence intervals omitted here for readability.
The small ordering at the top should not become a dramatic horse race. The reported Climb difference between Fable and Carbon was 2.22 percentage points, with a 95% confidence interval from −0.94 to +5.49. In other words, those results did not establish that the apparent leader was better. The top three were statistically indistinguishable on Climb.
The robust story is the unsolved half. Every evaluated system passed most of the basic requirements. None crossed 57% on the criteria specifically chosen because they represent the remaining climb. Frontier agents can produce recognisable professional artifacts while leaving a large part of the difficult work unfinished.
The judge deserves the same caution. The paper reports 96.2% criterion-level agreement with human reviewers on a 50-task study, Cohen’s κ of 0.91 and task-level Pearson correlation of 0.966. Those are strong reported results, but they come from a selected study of high-weight criteria with balanced pass and fail examples. They do not prove that automated judging will notice every small formatting defect or domain-specific risk.
When blindness becomes expensive
The most revealing MKA experiment does not compare two companies’ models. It compares Gemini 4 Carbon with itself.
In one condition the agent could call view_image. In the other, that tool was removed. Everything else remained: the same 250-turn harness, code execution, web search, OCR and computer-vision libraries. This is important because the ablation did not make images inaccessible. It made them indirect.
Across MKA, removing direct vision reduced Climb by 5.94 percentage points, Overall by 4.58 and Regression by 1.96. At the same time, the agent consumed 1.85 times as many input tokens, 1.40 times as many output tokens and 1.28 times as many turns. On GDPval, a different professional-work benchmark, the same removal changed the overall score by only 0.08 percentage points, within uncertainty, and barely changed compute. [2]
MKA’s images were not decorative attachments that OCR could bypass. They carried the task.
The road case makes the aggregate result tangible. With vision, the agent made 68 targeted image-view calls and scored 86.7% on Climb and 96.0% Overall. Without vision, it scored 14.3% on Climb and 64.7% Overall. It used 3.7 times the input tokens, 2.2 times the output tokens and 1.8 times the turns. The agent was not merely worse. It was more expensive while being worse.
A cleaner CAD task produced a subtler result. Code, geometry and OpenCV could recover much of the drawing’s structure, so the vision-disabled score fell by 13 points on Climb rather than 72.4. Yet it still used 4.6 times the input tokens and 2.4 times the turns. In one run the workaround nearly matched direct vision; in others it missed fine clearances.
The difference is a useful design principle. Structured geometry can sometimes be reconstructed from structured operations. Natural photographs and medical imagery contain meaning that collapses when reduced to an improvised text proxy.
More reasoning can be evidence of a broken interface
AI systems are often evaluated as though more tokens imply more diligence. MKA exposes the opposite possibility. A long trajectory can be the exhaust plume of a bad representation.
The vision-disabled road agent extracted images, calculated luminance, experimented with character maps, inspected the output and tried again. Each step was locally rational. Together they formed an expensive detour around a missing sensory interface. The failure did not begin in the final conclusion. It began when the system converted evidence into a form its reasoning loop could not use faithfully.
The paper’s four-model cost table tells a related story. Fable used 40.48 million total tokens per task on average and cost $35.28 at the provider prices used by the authors. Carbon used 7.05 million tokens and cost $5.63. Fable’s apparent 2.22-point Climb advantage was statistically inconclusive, yet it required 5.74 times the tokens and 6.26 times the cost. Prices will change, and these figures include the paper’s caching assumptions, but the systems lesson survives the tariff.
Efficiency should therefore be measured against useful rubric progress:
That is not a single official MKA metric; it is a deployment lens. An enterprise may rationally pay more for a high-value criterion—finding a safety defect is not equivalent to aligning a heading. But it should know which additional passes the additional compute purchased.
What an enterprise MKA should look like
A company does not need to reproduce a 200-task benchmark to learn from MKA. It needs a small set of work samples where the modalities and the deliverables are genuine.
Choose a workbook, deck, report or annotation that a professional already produces and another person actually uses.
Supply the scans, charts, photographs and layouts. Do not silently replace them all with OCR text.
Separate factual findings, calculations, traceability, file integrity and visual usability.
Give safety, compliance and decision-critical findings more weight than cosmetic preferences.
Remove vision, code or retrieval one at a time. Observe whether quality falls and compute rises.
Open the deliverable in its real application and let a domain expert inspect it at normal working size.
The ablation is especially valuable. If removing image access changes nothing on a supposedly visual task, either the images are irrelevant, the benchmark leaks their content through text, or the agent never used them. If removing a tool increases tokens without increasing quality, the remaining interface is creating compensatory work.
The rubric should also record provenance. A passed criterion is more trustworthy when the reviewer can trace it to page 17, cell D42 or a bounding box on the third photograph. This makes evaluation useful beyond ranking: it reveals whether the system reached a correct conclusion through the right evidence or simply produced a plausible match.
Finally, keep a human-authored gold artifact—but do not confuse similarity with correctness. Two good decks can look different. A spreadsheet may use different formulas and reach the same auditable result. MKA’s binary criteria are valuable because they describe obligations rather than demand pixel-for-pixel imitation.
What MKA still cannot tell a buyer
MKA is intentionally broad, but it is not a universal estimate of workplace automation. The tasks are bounded assignments, not month-long projects. The agent starts with an assembled file bundle. It does not negotiate access with a colleague, discover that a missing attachment exists, or live with the consequences of an error. Its private data helps control leakage but limits external reproduction of the full score.
The vision ablation was run on one model, Gemini 4 Carbon. It shows that direct image access mattered strongly for that system on these tasks; it does not prove that every architecture would fail in the same way. The road and CAD examples are case studies, not distributions over all enterprise imagery.
Nor does a high MKA score establish security, privacy or operational reliability. A system can build an excellent workbook and still send it to the wrong destination. It can cite every source and expose confidential data in a temporary file. Artifact quality is one layer of readiness, not the entire stack.
These limits are reasons to adapt the method, not discard it. The benchmark asks a better question than “Can the model answer?” It asks whether the complete agent, with its tools and interfaces, can turn mixed evidence into work that survives inspection.
The pixels were present
Return to the road photograph. The vision-disabled agent was not starved of intelligence. It could write code, call libraries, reason across documents and spend millions of tokens. It had the pixels.
What it lacked was a faithful route from those pixels to the decision. Its workaround compressed a pothole into punctuation, then asked language reasoning to recover what the transformation had erased. More effort followed. Better work did not.
That is why MKA matters beyond its leaderboard. As agents move from chat boxes into accounting, engineering, healthcare, construction and operations, their mistakes will increasingly live between modalities: a chart flattened into text, a note detached from its location, a formula replaced by its displayed value, a photograph reduced to an OCR caption.
The enterprise task is not to feed a model every possible byte. It is to preserve the form in which evidence remains meaningful, give the agent the right instruments, and judge the object it leaves behind.
The photograph was available. The pixels were present. The meaning disappeared in the interface between them.
References
- Google DeepMind, “Introducing the Multimodal Knowledge Agent benchmark”, 10 November 2026.
- Google DeepMind and Artificial Analysis, “Multimodal Knowledge Agent: A Benchmark for Professional Knowledge Work”, technical report, November 2026.
- Google, Multimodal Knowledge Agent public showcase tasks, accessed 13 November 2026.
- Artificial Analysis, MKA evaluation and live leaderboard, accessed 13 November 2026.