Google today made Pathways, the distributed system it built to train its own large models including Gemini, generally available on Google Cloud Platform. The release came in a Google Developers Blog post written by ten Google staff, the first in a series titled "Pathways on Google Cloud", which argues that the standard way of spreading a training job across thousands of chips breaks down once the job stops being one uniform program, and sets out how a single controller is meant to take its place.

In Short

Google has made Pathways, the software it uses to coordinate thousands of its own AI chips when training Gemini, a generally available product on Google Cloud. It matters to anyone paying for large-scale AI training on Google's TPU chips, because the system is designed to keep a run going when a machine fails and to stop expensive chips from sitting idle while other parts of the job catch up. What changes is the programming model: one Python program on an ordinary computer describes the whole job, and Google's runtime decides where each piece runs and moves the data between chips.

What Google released today

The post's summary opens with one line: "Today, we announce the General Availability of Pathways on Google Cloud Platform." Its authors are eight software engineers - Sujeeth Jinesh, Shaurya Gupta, Luke Baumann, Akanksha Gupta, Anthony Owusu, Jing Yang, Sadi Kneipp and Yu Li - together with product manager Monica Song and Ivan Nardini of Google's AI Developer Relations team. Google calls Pathways "the distributed system Google built to train its own large models, including Gemini." It "runs JAX programs on TPUs," according to the post, which means it serves code written in JAX, Google's numerical computing library, on the company's own tensor processing units.

Pathways is not new inside Google. The design was described in a paper titled "Pathways: Asynchronous Distributed Dataflow for ML," submitted to arXiv on March 23, 2022, with Paul Barham as first author and 16 authors in all. That paper reported that the system matched state-of-the-art systems, at close to 100% accelerator utilization, when running standard workloads across 2,048 TPUs, and that it could pipeline Transformer models across 16 stages. Google now uses it to train its largest frontier models, Gemini among them, according to the post.

Outside access came three years later. On April 9, 2025, the opening day of Google Cloud Next 2025, Google said Pathways, described by ITPro as the internal machine learning runtime developed by Google DeepMind, would be made available to Google Cloud customers. Neither that report nor a Google Cloud blog post on the subject the following month described the offering as a preview or as generally available. Today's general availability arrives 19 months after that announcement.

The post does not explain what general availability changes in practice. It gives no pricing, no service-level terms, no list of regions and no customer names. What it provides instead is a technical argument for why large training jobs need a different execution model, and five links for further reading: the Pathways on Google Cloud documentation, the 2022 paper, documentation on resilient training, a GitHub repository called pathways-utils, and the repository for MaxText, the training codebase Google used in an earlier post on elastic training.

Where the standard approach runs short

Most large-model training runs as Single Program, Multiple Data, or SPMD, programs, according to Google. Every accelerator runs the same program in lockstep and coordinates through collective operations such as AllReduce, which averages gradients across devices. The two methods PyTorch users most commonly rely on to spread training across chips follow this model. Distributed Data Parallel runs a copy of the training script on every accelerator. Fully Sharded Data Parallel splits parameters, gradients and optimizer states across workers so that a model too large for one chip can fit at all.

Inside one tightly connected group of accelerators, collectives are cheap, because they run over a fast interconnect. Stretch a job across several groups and the same operations cross the datacenter network, which is far slower. Google's example is its TPU v5p: "each chip gets 1,200 GB/s of bidirectional chip-to-chip bandwidth inside its group, but only 50 Gb/s (gigabits, about 6 GB/s) to the datacenter network which is roughly two orders of magnitude less." Converted precisely, 50 gigabits per second is 6.25 gigabytes per second, which puts the ratio at about 192 to one, by PPC Land's calculation.

Because every training step blocks until its collectives finish, that communication lands directly in step time. The more groups a job spans, the post argues, the larger the share of each step spent moving data, so throughput grows more slowly than the number of accelerators added. Fitting a model into memory is only part of the problem. "Keeping every accelerator busy is the harder part, and this is where the SPMD model starts to show its limits."

Two jobs that break the pattern

The post uses two examples. The first is pipeline parallelism, in which a model is split into blocks across devices and each device processes smaller micro-batches. Activations flow forward and gradients flow back. While the first device works on the first micro-batch, the second and third wait. Google's diagram shows three devices handling four micro-batches, with hatched slots marking the idle time known as the pipeline bubble. Those idle accelerators still have to be paid for, and utilization and throughput drop even though every device has work once the pipeline is full.

The second is reinforcement learning used in post-training, the stage in which an LLM is refined after its initial training. Three separate programs are involved: a sampler that generates rollouts, a reward model that scores them, and a trainer that updates the policy and sends the new weights back to the sampler. Run one after another, the sampler sits idle while the trainer works and the trainer sits idle while the sampler generates. Run in overlap, as Google's third diagram shows, the sampler is already producing the next batch while the trainer updates.

"In both scenarios, the job is now several different programs running on different devices, and their dependencies can no longer be managed through a single collective," the post states. Someone has to decide which devices share which tensors, which processes run while others finish, and how early to queue the next buffer so that no chip waits for input. Since each worker sees only its own devices and data, no single process is in a position to manage all of it.

Three consequences

Google lists three. The first is fragility. Each process holds its accelerators for the entire run and every collective waits for all devices, so one failed machine stops the whole job and forces a restart from the last checkpoint. Devices cannot be added during a run, and they cannot be released.

The second is stranded capacity. A pipeline stage that needs fewer chips than another keeps its surplus anyway. "They sit idle, you keep paying for them, and no other part of the job can use them until the run ends," according to the post. Across a team, the same logic applies to entire clusters, which have to be carved into fixed per-job allocations, each job bringing up its own processes on its own hosts.

The third is the development cycle. On a single machine, an engineer can run a script, open a Python shell, set a breakpoint and debug. Across many hosts, a launcher must push the script to each one, coordination bugs appear that never showed up on one machine, and there is no single process to break into. "When something hangs, each worker is stuck at a collective waiting for the others, with no one place that shows the whole job."

Failures at Gemini scale

Hardware failure is the most concrete of these problems. According to Google, large runs "have to survive hardware failures, which interrupted Gemini 2.5 training multiple times per hour." Under SPMD, one lost machine halts every process and sends the run back to its last checkpoint.

The post links that claim to Google's Gemini 2.5 technical report, submitted to arXiv on July 7, 2025. The report describes interruptions from hardware failures multiple times per hour at the scale of that training run. Its response, according to the report, was a system that continues training with fewer slices of TPU chips when a localized failure occurs, losing tens of seconds of training time per interruption and running at around 97% throughput while the failed slice recovers. Over the whole run, 93.4% of the time went on TPU computation. The report credits the single-controller design of Pathways with making these techniques relatively simple to implement.

A smaller public demonstration followed a year later. On July 6, 2026, Luke Baumann, one of today's authors, published a Google Developers Blog post with Abhinav Singh and Ivan Nardini describing a test in which they deliberately terminated a TPU during a MaxText training run. The setup was modest: three TPU v5e slices of 16 chips each, 48 chips in total, training the small qwen3-0.6b model. According to that post, the run took about one minute and 50 seconds to get from the kill to the next training step. Roughly 13 seconds passed before the training loop noticed anything was wrong, about 50 seconds went on waiting for Kubernetes to schedule a replacement pod, and the job lost 88 steps of progress, rewinding to a checkpoint at step 3,300. Today's post adds no new resilience figures.

How Pathways works

Google's answer to the coordination problem is to take it away from individual workers. "With Pathways, you define the end-to-end computation from a single Python client, without writing custom coordination code, and a distributed runtime executes it," the post states.

One client and virtual devices

That client "is decoupled from the hardware." It runs on an ordinary CPU machine, and programs are written against virtual devices rather than physical chips. A resource manager maps those virtual devices onto real chips, and a Pathways worker on each accelerator host runs the compiled programs it receives.

Most of the features Google describes follow from that indirection. In a reinforcement learning loop, the trainer and the samplers rarely need the same number of chips, so the sampler stage can be given more by asking for more virtual devices. If a host dies, the resource manager remaps its work onto healthy chips and the run continues instead of relaunching, which Google describes as "the basis for elastic training." And because a program targets virtual devices, compilation can start before the physical chips are provisioned, so the two overlap instead of running in sequence.

A cluster shared between researchers

Because the resource manager owns the chips, one Pathways cluster can serve several clients at once. Through what Google calls the Shared Pathways Service, the cluster stays up as a long-running service on TPUs hosted on GKE, Google's managed Kubernetes platform. Each client connects on demand from a laptop or virtual machine, receives its own slices - groups of tightly connected TPU chips - from a shared pool, and returns them when it disconnects. "When every chip is busy, the next workload waits until capacity frees up," the post states. Its link for the service points to a folder marked experimental in the pathways-utils repository on GitHub. The post does not say whether the shared service is covered by today's general availability.

A graph instead of hand-offs

Pathways treats a training job as a distributed dataflow graph. Taking the three-block pipeline as its example, the post describes declaring what each block computes, which devices it targets and where its outputs go. Each block becomes a compiled SPMD program, and each edge becomes a tensor that the runtime is responsible for delivering.

Within one group of devices, nothing changes: the framework handles the collectives as it would without Pathways, and code stays the same. The difference shows up when a job runs several programs on different groups, a pattern the post calls Multiple Programs, Multiple Data, or MPMD. Then no single process sees both sides of a transfer, so moving a tensor from block A to block B is a hand-off engineers have had to coordinate themselves. With Pathways, according to Google, the dependency is declared once, and the runtime moves and reshards the tensor - over the fast interconnect within a group and over the datacenter network between groups.

Dispatch before the data exists

To keep chips busy, Pathways prepares launches in advance using futures and parallel asynchronous dispatch. Block B cannot run while block A is still computing, but its input shapes and data types are already known. So the runtime allocates B's buffers and schedules the transfer "before A's output even exists." Host-side setup for every block happens up front, and each block launches as soon as its inputs arrive.

The same mechanism, the post argues, keeps the single client from becoming a bottleneck. Because dispatch is asynchronous, coordination overlaps with computation already running on the chips, and the client stays "off the critical path instead of stalling the accelerators between steps." This is presented as a design argument. The diagram that accompanies it is labeled "illustrative durations," and no measured overhead is given.

One order for every device

When a group of devices has more than one operation ready, the runtime decides which runs first. In pipelined training, the devices holding block B often have both a forward pass for an upcoming micro-batch and a backward pass for an earlier one queued. Google's diagram shows what happens if each device chooses on its own: one starts the forward pass for micro-batch 3, the other the backward pass for micro-batch 1, their collectives fail to match, and the group stalls. Pathways uses "a central scheduler that puts every program's operations into one consistent order, so operations that must communicate start together instead of deadlocking."

Debugging from one process

A single client also brings back the development loop that multi-host jobs take away. "It's the same single Python process whether you drive one host or thousands of hosts," the post states, so an engineer can open a shell, set a breakpoint and inspect the run as it goes. When something stalls, one place knows what is queued on every device and which dependency has not been met.

Google sums up the design as a separation of model building from execution. The client decides how the model is partitioned across hardware, while the runtime handles the transfers, ordering and scheduling "across the whole cluster rather than just your slice of it."

Not a replacement for Ray

The post anticipates a comparison with Ray, the open-source framework for orchestrating Python-based AI workloads across large clusters. According to Google, Ray provisions the cluster, schedules Python code as tasks and actors, and launches workloads across hosts, holding data loading, training, serving and tuning under one control plane. "To Ray, an accelerator is a resource to schedule."

What Ray does not change, in Google's account, is how training runs on the chips. The job inside each Ray worker still runs as SPMD, with each process seeing only its own devices. Pathways works one layer below. "So concretely, Ray's job is about orchestration while Pathways' job is about execution," the post states.

Google describes the two as complementary, with Ray orchestrating the cluster and the pipeline while Pathways handles how the training step executes. It qualifies that claim with "in principle," and it does not show the two running together. The post links to Google's own documentation for running Ray on TPUs and to an earlier post in a series titled "Run Ray on TPU", so Google supports both routes on its hardware.

What the post leaves out

Several gaps matter for anyone weighing the release.

Performance data. The post contains no throughput, utilization, step-time or cost comparison between Pathways and SPMD on Google's hardware. Its diagrams are toy schedules and illustrations. The utilization figure in the 2022 paper concerned standard SPMD workloads, where Pathways was shown to match existing systems rather than beat them.

Commercial terms. No price, region list, support commitment or customer appears, and the post does not say what general availability adds to the access Google announced in April 2025.

Framework and chip scope. The post opens by addressing PyTorch users, yet states only that Pathways runs JAX programs on TPUs. Whether PyTorch code can run on Pathways is not addressed, and GPUs are not mentioned at all. Virtual devices abstract over Google's own chips. They do not make a job portable to another vendor's hardware.

Resilience at scale. The public numbers on elastic recovery come from the July 2025 Gemini 2.5 report and the 48-chip test published in July 2026. Today's post adds none.

There is also a small editorial slip. The diagram showing the single client above the distributed runtime carries the caption of Figure 3, the reinforcement learning schedule, and the post has no Figure 4.

Why this matters for the marketing industry

Pathways is training software, and today's release changes no advertising product. Its relevance lies in the economics underneath the Gemini models that sit behind Google's AI search features, the Gemini app and Google's ad tools, and in Google's expanding business of selling its chips to other AI companies.

Compute is the constraint Google keeps naming. In an interview published on October 21, Alphabet chief executive Sundar Pichai said Gemini 4 Argon processed almost 25 trillion tokens in its first 72 hours and that Google remained capacity constrained, with consumer access limited to paying subscribers while smaller versions are prepared. Pre-training of Gemini 4 had begun by July. Training and inference compete for the same fleets of accelerators, so chips left idle by pipeline bubbles or by restarts after hardware failures represent capacity that is not available to serve models to users and advertisers. Google has not put a figure on how much capacity Pathways recovers, and today's post does not either.

The spending behind that capacity is large. Alphabet announced an equity raise of about $85 billion in June to fund AI infrastructure, and its third-quarter results showed capital expenditure of $51.4 billion and negative free cash flow of $9.1 billion. At that level of spending, small changes in how much of each chip's time goes on useful work translate into large sums.

The chips are also a product in their own right. Google Cloud revenue rose 103% to $30,709 million in the third quarter, and Alphabet said TPU systems sales were a meaningful contributor to that growth, describing a stack supported by JAX, PyTorch and the XLA compiler. Pathways adds Google's own training orchestration to what it offers around that hardware, and because it runs JAX on TPUs only, it ties the workloads that adopt it more closely to Google's chips.

The largest outside buyer of those chips is Anthropic. Its IPO filing on October 26 showed at least $114.5 billion of future spending with Google Cloud, and its exhibits described TPU8i pods that Google sells to Broadcom, which leases them to an Anthropic subsidiary. The amended filing on November 2 estimated additional TPU compute from Google at $125.0 billion to $150.0 billion, for delivery between 2028 and 2031. Anthropic's own approach differs from the one Google describes today. Its filing lists a multi-chip architecture spanning AWS Trainium, Google TPUs, NVIDIA GPUs and AMD GPUs, managed as one pool, whereas Pathways pools Google's chips only.

That dependence runs in both directions. UBS estimates cited in August put OpenAI and Anthropic at more than 48% of Google Cloud revenue in 2027. Rivals are building their own capacity: Amazon switched on Project Rainier with nearly half a million Trainium2 chips in 2025, and Meta's infrastructure chief put Meta's 2026 AI spending at well over $100 billion. Each hyperscaler competes in part on how much useful work it can extract from each chip, and software of this kind sits alongside the silicon in that contest.

For advertisers and publishers, the effects are indirect. How quickly Google can train new Gemini versions, and how widely it can afford to serve them in Search and its ad products, depends in part on how efficiently it turns capital spending into finished models. Today's post explains how Google says it does that. It does not supply the measurements that would show how much difference the system makes.

Timeline

Summary

Who: Google, through a Google Developers Blog post by software engineers Sujeeth Jinesh, Shaurya Gupta, Luke Baumann, Akanksha Gupta, Anthony Owusu, Jing Yang, Sadi Kneipp and Yu Li, product manager Monica Song and Ivan Nardini of AI Developer Relations. The release affects Google Cloud customers training large models in JAX on TPUs, and indirectly companies such as Anthropic that buy TPU capacity, as well as advertisers and publishers whose tools run on Gemini models.

What: Google made Pathways, the single-controller distributed system it built to train Gemini, generally available on Google Cloud Platform. One Python client on a CPU machine describes the whole job as a graph against virtual devices, while a runtime maps the work onto TPUs, moves and reshards tensors between groups of chips, dispatches work ahead of time, orders operations through a central scheduler and remaps work around failed hosts. The post includes no pricing, customer names or performance measurements.

When: Today, November 9, 2026. The underlying design was published in March 2022, and Google first said it would offer Pathways to cloud customers on April 9, 2025.

Where: On Google Cloud Platform, running JAX programs on Google's TPUs, with a shared cluster option on GKE that clients can reach from a laptop or virtual machine.

Why: According to Google, the SPMD model behind most large-model training keeps every chip in lockstep, so pipelines and reinforcement learning loops leave accelerators idle, one failed machine stops the entire job, and multi-host debugging becomes hard. Hardware failures interrupted Gemini 2.5 training multiple times per hour. Google is capacity constrained, is spending heavily on infrastructure and is selling TPU systems to outside buyers, which makes the share of each chip's time spent on useful work a commercial question as well as a technical one.