Claude Fable 5.2 edges Gemini 4 Carbon on Google's benchmark at 6x the cost

Google DeepMind on November 10, 2026 released a benchmark built to test whether AI agents can do office work that depends on reading blueprints, schematics, radiographs and maps. The highest-scoring model in its first set of results is not one of Google's.

The Multimodal Knowledge Agent benchmark, shortened to MKA, was developed with Artificial Analysis, according to a post on Google's blog by Rohan Doshi, Senior Product Manager at Google DeepMind, and Ben Caine, Principal Engineer at Google DeepMind. The post states that "Gemini 4 Carbon achieves #2 in quality among the models we tested, and completes tasks at 6x lower cost on average compared to the top scoring model." It does not say which model came first. The 25-page technical report linked from the post does. According to that report, Anthropic's Claude Fable 5.2 reached an overall weighted satisfaction rate of 73.26%, against 71.89% for Gemini 4 Carbon, 70.98% for Claude Opus 5.5 and 69.34% for OpenAI's GPT-6 Astra.

The gap between first and second is small, and Google's researchers describe the top three models as statistically indistinguishable on their headline measure. The gap in cost is not small. According to the report, Claude Fable 5.2 spent an average of $35.28 per task at list prices. Gemini 4 Carbon spent $5.63.

In Short

Google DeepMind and Artificial Analysis released MKA on November 10, 2026, a 200-task test of whether AI agents can read visual files such as floor plans, scans and charts and turn them into finished reports, slide decks and spreadsheets. Google's own technical report puts Anthropic's Claude Fable 5.2 first and Gemini 4 Carbon second by a margin it treats as statistically insignificant, while Carbon's runs cost about one sixth as much per task. The results show large differences in cost and effort between models that score almost the same, on a test designed by one of the vendors being ranked and graded by an OpenAI model.

What the benchmark asks agents to do

Google presents MKA as a correction to the way agents are currently measured. "Most knowledge work benchmarks can be solved with text alone, without ever needing to look at an image or page layout," the post states. "Across many industries, day-to-day work is deeply visual," the authors add, pointing to architectural blueprints, engineering schematics and the decks, reports and spreadsheets that professionals produce from them.

To build the test, Google worked with domain experts across 10 verticals and 83 occupations "where day-to-day work is inherently visual," according to the post. Those specialists, who range from architects and biologists to underwriters and data scientists, wrote the 200 tasks in the evaluation set. Each task reflects a multi-step project that takes a seasoned professional three to eight hours. Each pairs a prompt and a bundle of input files with an expert-written rubric of roughly 70 yes-or-no checks, which the post says are weighted 75% on factual accuracy and 25% on visual design.

The technical report, dated November 10, 2026 and credited to 31 Google DeepMind researchers, fills in the detail. The tasks are split evenly, 20 per sector, across business, construction, education, engineering, finance, healthcare, law and government, logistics and mapping, science, and software and data. Every task author has at least five years of professional experience, and a second expert with the same job profile reviewed each task for realism and accuracy.

The 200 tasks carry 1,459 input files between them, from 2 to 45 per task with a mean of 7.3. PDF documents account for 43.0% of those files (628), images and scans for 22.9% (334), spreadsheets for 10.9% (159) and Word documents for 10.2% (149). Geospatial and text files make up 6.0% (88), slide decks 3.5% (51) and CSV or TSV files 3.4% (50). The inputs include CAD drawings, CT scans and radiographs, SEC Form 10-K filings, commercial leases, municipal boundary layers and LiDAR elevation data.

Output matters as much as input. According to the report, 176 of the 200 tasks (88%) require the agent to create or modify a standalone file. PDF reports are the primary deliverable in 45 tasks, Word documents in 42, presentations in 34 and spreadsheets in 32. Thirty tasks call for text reports, nine for images or charts and eight for interactive HTML dashboards.

The rubrics contain 14,284 criteria, a mean of 71.4 per task, ranging from 20 to 150. Each criterion carries an integer weight from one to five, set by the expert who wrote it. The report does not describe the 75/25 split between factual accuracy and visual design that appears in the blog post.

Three published examples

The report walks through three tasks from a public sample. A workforce planning analyst task gives the agent six files, including alluvial charts of quarterly leave flows between 15 departments, and asks for a branded two-sheet workbook ranking each department's main feeders. It is graded on 90 criteria. A GIS analyst task requires the agent to identify the City of Nottingham from an unlabeled reference map, combine building-footprint and heritage layers with 1-meter LiDAR elevation and solar-irradiance data, and produce a Word report ranking the top five council buildings for rooftop solar, with 62 criteria. A mechanical engineering instructor task asks for a six-slide lab deck with phase-portrait charts, built from a mechanical schematic, a flowchart and a reference paper, and graded on 54 criteria.

The business sector sits closest to advertising. Its reference job profiles, according to the report, include management consultants, marketing and media analysts, customer support specialists, revenue management analysts and procurement specialists.

The harness, the sandbox and the grader

Every model runs in Stirrup, an open-source agent harness maintained by Artificial Analysis, with the same tools, sandbox and turn budget, according to the post. The report specifies version 0.2.1 and sets out the limits. The agent works without human intervention until it declares itself finished or reaches 250 turns, with a context window of up to 1 million tokens where the model supports it and an 8-hour wall-clock limit per task. When a conversation reaches 70% of the model's context length, the harness compacts it.

Each task runs in a fresh Linux container with 4 virtual CPUs and memory that can burst to 16 GB, preloaded with LibreOffice, the Tesseract OCR engine and Python libraries for Office files and PDFs. The agent gets four tools: code execution, image viewing, web search and a finish command. The sandbox has internet access.

The image-viewing tool carries much of the weight. The harness hands the agent only the task prompt and the file paths, as text. To look at a PDF page or a slide, the agent has to render it to an image with code and then load that image into its visual context.

Grading is also handled by an agentic system, one that works through files with tools rather than reading extracted text. "An agentic autorater built on a non-Google model grades each deliverable," the post states. "It opens every file, renders pages, inspects spreadsheet formulas, and checks the work against the expert rubric." The report names the model: OpenAI's GPT-6 Luna. Its authors write that a judge from outside the Google model family avoids favoring Gemini, and that they see no sign of same-family preference because GPT-6 Astra, the OpenAI model under test, ranked last. The autorater runs as its own Stirrup agent, without web access, and sees only the candidate's deliverables and final message, not the task's input files. Rubric criteria state the expected value or tolerance directly, so the grader checks the work rather than solving the task again.

How reliable is that grader? "When domain experts graded model outputs themselves, they agreed with the autorater 96% of the time," according to the post. The report's study behind that figure covered 50 tasks across nine verticals. Three domain experts graded each deliverable independently, without knowing which model produced it. Pairwise agreement between the autorater and the experts was 96.2%, with a 95% confidence interval of 94.7% to 97.6%, against 99.8% agreement among the experts themselves. Cohen's kappa, which adjusts for agreement expected by chance, was 0.911.

The report breaks the 3.8% error rate down further. False passes, where the autorater accepted work the experts rejected, accounted for 2.4% of criteria, mainly through leniency such as crediting an alternative value or accepting equivalent content when a required label was missing. False fails accounted for 1.4%, mostly from overly strict string matching or from inspecting interactive deliverables without running them. The study was limited to high-weight criteria, those weighted three to five, and the report acknowledges that the 96.2% figure may not transfer exactly to low-weight formatting checks.

The ranking the blog post did not name

The report evaluates four frontier models: Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5 and GPT-6 Astra. Each ran once per task.

Its headline metric needs some unpacking. A weighted satisfaction rate is the weighted share of rubric criteria a deliverable meets. Google then split the criteria in two, using a fixed panel of four earlier models: Gemini 3.8 Flash, Gemini 3.6 Flash, Claude Sonnet 5.5 and GPT-6.1 Sol. Criteria that every panel model passed in at least two of three runs were labeled "regression" checks, 6,396 of them across 180 tasks, covering skills that older models already have. The remaining 7,888 criteria, spread across all 200 tasks, are "climb" checks. The score on those, called Climb-WSR, is what the report treats as the measure of frontier capability.

Source: Google DeepMind MKA technical report, task-level means, November 10, 2026.

All four models clear 95% on the regression checks, so the differences sit almost entirely in the climb criteria. There, Claude Fable 5.2 leads Gemini 4 Carbon by 2.22 percentage points, with a 95% confidence interval running from minus 0.94 to plus 5.49. Carbon's lead over Claude Opus 5.5 is 1.33 points, with an interval from minus 1.70 to plus 4.41. Both intervals cross zero, which is why the report treats the top three as statistically indistinguishable. Carbon's 4.10-point lead over GPT-6 Astra, with an interval from plus 1.00 to plus 7.11, is significant, and so is Fable's.

Google's "#2" claim is therefore accurate as a ranking, but the report's own statistics allow the order of the top three to change on a different sample of tasks. The absolute level matters too. Even the best model met 56.56% of the weighted climb criteria, which leaves about 43% unmet, by PPC Land's calculation.

Six times the cost for two points

"On the same tasks, the models we tested have extreme differences in efficiency, using between 30 and 130 turns on average to complete each task," the post states. The report's efficiency table shows where those figures come from.

Source: Google DeepMind MKA technical report, mean per task, at provider list prices with prompt caching.

Claude Fable 5.2 consumed 5.74 times as many total tokens as Gemini 4 Carbon, took 2.76 times as many turns and ran for 2.85 times as long. At list prices that comes to 6.26 times the API cost, for a lead the report calls statistically non-significant. Claude Opus 5.5 cost 1.83 times as much as Carbon while scoring 1.33 points lower. Across the four models, total token use per task varied by more than 16 times, from 2.43 million for GPT-6 Astra to 40.48 million for Fable.

GPT-6 Astra complicates the picture. It used the fewest tokens and turns of any model, yet cost $6.03 per task because of higher per-token list pricing. That is about 7% more than Gemini 4 Carbon, by PPC Land's calculation. The "6x" figure in Google's post holds only against the top scorer; against OpenAI's model, the cost gap is narrow and the difference that stands out is Carbon's statistically significant quality lead. The blog post's chart caption says Carbon "is on the pareto frontier of quality and price for the MKA eval." The report's own chart plots quality against tokens, and on that measure GPT-6 Astra, Gemini 4 Carbon and Claude Fable 5.2 all sit on the frontier.

Caching shapes these numbers heavily. According to the report, between 93.7% and 97.7% of cumulative input tokens were served from cache, and the cost figures account for each provider's cache pricing. The report also notes that list prices will change. When Google priced Gemini 4 Argon at $2 per million input tokens and $10 per million output tokens in October, cached input came at a 95% discount. Google's separate Carbon launch post, published the same day as MKA, gives Carbon the same introductory price as Argon.

Agent workloads make those discounts matter. OpenRouter data published in August showed agents using nearly five times more tokens per task than human users, with more than 85% of agentic token consumption in cached prompts. Anthropic, for its part, said in its IPO filing that Claude Opus 5.5 cut customer cost per task by about 40% compared with Opus 5, without publishing a method.

What happens when the agent cannot see

The post says the report shows "how strong perception abilities are necessary to solve each task," and that "we also show how models try and fail to compensate when vision is removed."

The test behind that claim ran Gemini 4 Carbon with and without its image-viewing tool, on MKA and on GDPval, the workplace benchmark OpenAI published in 2025, which the report describes as spanning 44 occupations. Without the tool, the agent kept code execution, web search and every preinstalled parsing, OCR and computer vision library. On GDPval, removing image viewing changed nothing: the overall score was 88.1% with the tool and 88.2% without, and the agent used 0.95 times as many turns. Google's researchers attribute that to GDPval's task-critical inputs sitting in spreadsheets and machine-readable text.

On MKA, the same change cut Climb-WSR by 5.94 points and the overall score by 4.58 points, while the agent took 1.28 times as many turns and produced 1.40 times as many output tokens. The report describes Carbon falling back on workarounds that go beyond text extraction, including rendering image crops as ASCII art or Unicode Braille blocks in its terminal and writing bespoke OpenCV pipelines.

Two case studies show how far those workarounds go. In an engineering task, the agent had to judge whether a machined part shown only in a raster CAD drawing could be made with a shop's equipment. Without image viewing, 30 of its 49 code calls ran pixel scans and Braille renders. One run matched the visual baseline, scoring 86.2% overall against 88.7%. The other missed tight clearance violations and scored 78.7%, using 2.4 times the turns and 4.6 times the input tokens. In a construction task, the agent audited road-inspection photos for pavement defects missing from a written log. With image viewing, it found the undocumented depressions in every run. Without it, 67 of 84 tool calls went to image-processing pipelines, every run concluded there were no undocumented defects, and Climb-WSR fell from 86.7% to 14.3%, despite 3.7 times the input tokens.

The report notes that this experiment ran on Gemini 4 Carbon only, and that other models may rely on vision to different degrees.

Who built the test, and who grades it

Several design choices sit beside the results.

The first is authorship. Google DeepMind designed MKA, commissioned its tasks and wrote the report, and one of its models is among the four ranked. Artificial Analysis runs the 200-task set and keeps it private, "so the tasks can't leak into training data or be found by the model via search," according to the post, and hosts a public leaderboard where the post says anyone can evaluate their own models. A separate set of 10 public tasks, one per sector, is published on Hugging Face so the format can be inspected.

The second concerns that search claim. The report lists among its limitations that the sandbox has internet access, which it says mirrors real deployments but may let agents retrieve some information online instead of extracting it visually.

The third concerns the headline metric. The four-model panel that decides which criteria count as "climb" includes two earlier Gemini models, Gemini 3.8 Flash and Gemini 3.6 Flash. The report acknowledges that the split therefore reflects what that panel already solves.

The fourth is sample size. Each model ran once per task and was graded in a single autorater pass, so the report's confidence intervals capture variation across tasks but not run-to-run variation in either the agents or the grader. Only four models were tested. None of them is an open-weight model, and none comes from Meta or xAI.

The fifth is less conventional. The report's acknowledgments credit Gemini Argon and Gemini Carbon as AI co-authors, coding collaborators and editors during the design of the benchmark, the analysis of results and the writing of the manuscript.

The report discloses each of these points itself. Together they mean that MKA's first results are a Google-designed measurement, with a Google model's predecessors helping to define the hardest criteria and an OpenAI model doing the grading. Artificial Analysis has also served as a reference point for Google's earlier model claims. When Alphabet reported third-quarter results in October, Google said Carbon ranked first on the Artificial Analysis Intelligence Index with a score of 59. In July, Google cited the same index when it said Gemini 3.6 Flash used 17% fewer output tokens than 3.5 Flash.

MKA also arrived as part of a launch. Google's Gemini 4 Carbon announcement, published on November 10, 2026 by Koray Kavukcuoglu, cites the benchmark in a single paragraph, and the MKA post closes with a section on Carbon and a note that the model is available through the Gemini API in Google AI Studio.

Why this matters for marketing teams

The deliverables MKA grades are the routine output of agencies and in-house marketing teams: slide decks, spreadsheets, PDF reports and Word documents, often built from charts, screenshots and scanned material rather than clean text. The benchmark's business sector names marketing and media analysts among its reference job profiles. Google has been moving its consumer products in the same direction; in late October it gave the Gemini app an in-chat editor for generated PDFs, Docs and Sheets. In July, OpenAI folded its Atlas browser into a new ChatGPT Work agent.

What MKA adds is a price per finished deliverable. Model pricing is usually quoted per million tokens, and the report's efficiency table shows how little that rate says on its own. GPT-6 Astra used about a third of Carbon's tokens and still cost more per task. By PPC Land's calculation from the same table, Astra also had the highest effective cost per token, about $2.48 per million against roughly $0.87 for Claude Fable 5.2, yet Fable cost almost six times as much per task because it used nearly 17 times as many tokens. For agentic work, token volume per task varies by an order of magnitude between models that reach similar quality.

Capacity is part of the same story. After Gemini 4 Argon processed almost 25 trillion tokens in its first 72 hours, Sundar Pichai said Google remained short of compute. A model that finishes a task in 7 million tokens instead of 40 million places a different load on that capacity, whoever is paying.

The scores themselves also deserve a sober reading. Agent productivity in agency work has so far been modest: Omnicom's PHD panellists in Asia-Pacific put the share of work handled by agents at 10% to 20%, and Google's own ATLAS study found that AI appears in 68% of occupations but covers a median of only 21% of their tasks. Benchmarks have a mixed record as predictors. In 2025, LiveCodeBench Pro found frontier models scored 0% on hard coding problems without external tools, Salesforce's CRMArena-Pro found enterprise agents succeeded on only 35% of multi-turn tasks, and a Stanford review of 84 papers found that technical metrics dominated 83% of AI productivity evaluations.

MKA is more demanding than most of those tests, and more transparent than many vendor benchmarks: it publishes its grader, its agreement study, its cost table and its limitations. Its top result is still a model that leaves about two in five of the hardest weighted checks unmet, at a cost of more than $35 per task. Whether a deliverable that meets 73% of an expert's weighted checklist is usable depends on which checks it misses. The report deliberately leaves out a task-level all-pass rate, arguing that with 70 or more criteria per task, a single missed minor check would zero out the result.

Timeline

• 2025 - OpenAI publishes GDPval, a workplace benchmark the MKA report describes as spanning 44 occupations; Artificial Analysis releases Stirrup, the open-source agent framework later used for MKA
• June 1, 2025 - Stanford review of 84 papers finds technical metrics dominate 83% of AI productivity evaluations
• June 10, 2025 - Salesforce's CRMArena-Pro finds enterprise agents succeed on 35% of multi-turn tasks
• June 13, 2025 - LiveCodeBench Pro shows frontier models scoring 0% on hard coding problems
• July 9, 2026 - OpenAI folds Atlas into a new ChatGPT Work agent
• July 21, 2026 - Google cites the Artificial Analysis Index for Gemini 3.6 Flash's 17% output token reduction
• July 23, 2026 - Google's ATLAS study finds AI in 68% of occupations but a median 21% of tasks
• August 6, 2026 - Omnicom's PHD panellists put agent-handled work at 10% to 20% in Asia-Pacific
• August 25, 2026 - OpenRouter data shows agents using nearly five times more tokens per task than humans
• October 13, 2026 - Google prices Gemini 4 Argon at $2 per million input tokens
• October 21, 2026 - Pichai says Argon processed almost 25 trillion tokens in 72 hours and Google remains short of compute
• October 26, 2026 - Anthropic's IPO filing says Claude Opus 5.5 cut cost per task about 40%
• October 28, 2026 - Alphabet reports third-quarter results; Google says Carbon ranks first on the Artificial Analysis Intelligence Index
• October 29, 2026 - Google gives the Gemini app an in-chat editor for PDFs, Docs and Sheets
• November 10, 2026 - Google DeepMind and Artificial Analysis release the MKA benchmark, its technical report and 10 public tasks; Gemini 4 Carbon becomes available through the Gemini API in Google AI Studio, and Google's Carbon launch post cites its MKA ranking

Related PPC Land coverage

• Google prices Gemini 4 Argon at $2 per million tokens, paid users only - Argon's introductory pricing, including the 95% discount on cached input.
• Gemini 4 Argon's 25 trillion tokens in 72 hours leave Google short of compute - Pichai on Argon's early token volume and Google's capacity constraints.
• Gemini app gains 350m users in a year to pass 1bn, Alphabet says - Alphabet's third-quarter figures, including Google's claim that Carbon tops the Artificial Analysis Intelligence Index.
• Claude Opus 5.5 cuts cost per task 40%, Anthropic IPO filing says - Anthropic's cost-per-task claim and customer figures from its S-1.
• AI agents use five times more tokens than humans, OpenRouter data shows - How agent workloads change token consumption and the role of cached prompts.
• Google cuts Gemini Flash prices as 3.6 uses 17% fewer output tokens - Gemini 3.6 Flash pricing and token efficiency measured on the Artificial Analysis Index.
• Gemini gains an in-chat editor for PDFs, Docs and Sheets - Google's move to let Gemini users preview and edit generated files.
• OpenAI kills Atlas browser, folds it into new ChatGPT Work agent - OpenAI's shift toward a work agent for knowledge tasks.
• Omnicom's PHD gains just 10-20% productivity from AI in APAC - Agency panellists on how much work agents actually handle.
• Google finds AI touches 68% of jobs but only 21% of their tasks - Google's ATLAS study of how AI use spreads across occupations and tasks.
• Salesforce study reveals enterprise AI agents fail 65% of multiturn tasks - CRMArena-Pro results on agent performance in business workflows.
• New benchmark reveals AI coding limitations despite industry claims - LiveCodeBench Pro and the gap between coding benchmarks and vendor claims.
• Research paper exposes flaw in AI productivity claims - A Stanford review of how AI productivity is measured.
• AI models fake understanding while failing basic tasks - Research on models that define concepts correctly but fail to apply them.

Summary

Who: Google DeepMind, which designed the Multimodal Knowledge Agent (MKA) benchmark, with Artificial Analysis, which runs its private 200-task set and maintains the Stirrup harness. The blog post is by Rohan Doshi and Ben Caine of Google DeepMind. The ranked models are Anthropic's Claude Fable 5.2 and Claude Opus 5.5, Google's Gemini 4 Carbon and OpenAI's GPT-6 Astra, with OpenAI's GPT-6 Luna as the grader.

What: A benchmark of 200 expert-written tasks across 10 sectors and 83 occupations, in which AI agents must read visually complex files and produce finished reports, decks, spreadsheets and other deliverables. Google's post says Gemini 4 Carbon ranks second at 6x lower average cost than the top model; the technical report shows the top model is Claude Fable 5.2, at 73.26% against 71.89%, a lead the report treats as statistically indistinguishable, at $35.28 per task against $5.63.

When: The blog post and technical report are both dated November 10, 2026, the same day Google made Gemini 4 Carbon available through the Gemini API.

Where: On Google's blog, in a technical report hosted by Google DeepMind, on a leaderboard hosted by Artificial Analysis, and in a 10-task public sample on Hugging Face.

Why: Google argues that most knowledge work benchmarks can be solved from text alone and fail to measure whether agents actually use vision. For marketers and agencies, the results put a price on finished deliverables and show more than a 16-fold spread in tokens per task between models of similar quality, on a test designed by one of the vendors being measured.