Google reworks TPU silicon into specialized systems for agents and security

Google's eighth-generation TPU family has now expanded further. Back in April, the company announced TPU 8t for training AI models and TPU 8i for general-purpose inference workloads. Google has now introduced TPU 8tx, built on the same silicon as the previously announced 8t, for confidential, hardware-isolated inference aimed at regulated enterprises and governments, and TPU 8i Max, which combines two 8i chips on one package for long-context agentic work.

TPU 8tx swaps the Arm-based Axion hosts of the 8t and 8i for a 5th Gen AMD EPYC CPU, adopting a Boardfly network, and the 8i Max doubles the compute, SRAM, and HBM bandwidth of the original 8i. The original split, the TPU 8t and 8i, made natural sense. The 8t targets large-scale training, while the 8i leans toward low-latency inference and reasoning. TPUs have become a fast-growing business, with the company claiming nearly 100 customers, including Meta’s recent move to TPUs. The more modular approach is flexible and can deliver specialized systems “in a matter of months rather than years,” helping that popularity along.

For the new parts, the TPU 8i Max isn't innovative on the surface, as it’s just two TPU 8i chips combined into a single package. The result is twice the theoretical compute, but more importantly, it fits twice the silicon within the “existing rack footprint,” and Google claims more compute per square foot and per watt.

Doubling both SRAM and HBM bandwidth helps keep things scaling, which would put the 8i Max at about 20.2 FP4 PFLOPS, 768 MB SRAM, and about 17.2 TB/s HBM bandwidth. This compares to Nvidia’s Rubin R200 GPU at a rated 50 FP4 PFLOPS for inference and 22 TB/s of HBM4 per socket, or AMD’s MI455X with 40 FP4 PFLOPS. The compute gap is closing.

The memory bandwidth doubling is also potentially one way around moving to HBM4. The memory wall, as Google calls it, or the bandwidth bottleneck, is a real concern “when dozens of AI agents work together across an entire software codebase.” Larger context windows — with larger models targeting 1 million tokens — mean larger KV caches, partly held in SRAM, and each new token has to read that cache back from memory, which is where the extra HBM bandwidth helps. It’s a different route from Nvidia’s LPX: more bandwidth in one package, rather than a second chip type for decoding and a storage tier for context.

Google also provides useful software for agents to get the most out of its hardware. TPU Sync is an open-source “model-state layer for TPU clusters” that moves KV caches off the chip’s HBM, “automatically tiering massive KV caches off the chip in milliseconds” down to host DRAM and, experimentally, shared storage. Google says this can cut KV transfer latency by “more than 90%,” with time-to-first-token gains that vary by workload. The company has also announced the Agent Compute Engine, a Google Cloud service that pauses idle agent sessions and moves their working memory aside until they’re needed again, another memory-saving effort.

Google credits its vLLM-TorchTPU stack with “state-of-the-art price-performance marks” on InferenceX and AgentX, which test realistic agentic coding and long-context work. However, the only published third-party numbers so far are for the last-generation Ironwood TPU. In SemiAnalysis’s InferenceX tests in September, running Qwen3.5 397B in FP8, Ironwood delivered 50.4% more tokens per dollar than Nvidia’s B200 and 96.0% more than the B300 at 20 tokens per second per user. One issue with that, however, is that Ironwood lacks native FP4, although newer November results show its FP8 beating Nvidia’s FP4 GB300 NVL72 on the same model, at 1.67 times the performance per dollar. There is also no eighth-gen result yet to compare how the new generation differs.

In contrast to the 8i Max, which is designed for long-context work, the new TPU 8tx, “built on the same silicon as TPU 8t,” is “optimized for secure and confidential inference” as well as small-scale training. The security is built into the hardware with isolation to prevent snooping on sensitive data, including model weights. Normally, data has to be unlocked in memory during processing, which is the step this isolation is meant to cover.

This chip is intended for areas where security and confidentiality are high priorities, which means regulated enterprises and governments, particularly those who are using frontier or proprietary models. In that case, the operators may want to “fine-tune models using their private data, distill large models into smaller ones, and train their agents,” without having to move sensitive data between clusters. Agents and security meet here for “secure agent execution.” Back in April, the Pentagon and Google reportedly held discussions for TPU usage in classified settings, although the 8tx chip is not specifically named.

Google’s Private AI Compute, its “hardened TPU platform” with protections “comparable to a typical TEE,” per its September technical brief, already runs confidential serving on TPUs. The 8tx appears to be that capability turned into a product for customers. Alternatives include Nvidia’s NVL72, with one security domain across 72 Rubin GPUs, 36 Vera CPUs, and the interconnects, and Intel’s TDX, which is adding trusted I/O to cover data moving to accelerators.

The TPU 8tx hosts are based on 5th Gen AMD EPYC boards, instead of the Axion hosts on the 8t, which replaced x86 in April. There's one Axion for every two TPUs on the 8i, compared with one Xeon per four on Ironwood. Google gives no reason for the change, but its Confidential VM configuration list has both AMD and Intel series chips, with no Arm-based ones listed. Private AI Compute uses “an AMD-based hardware Trusted Execution Environment” for its CPU workloads, suggesting Google swapped something it already runs on its confidential stack.

The 8tx also adopts the 8i’s Boardfly network over the 8t’s torus, which Google claims can deliver “up to 2x lower latency compared to TPU 8t” with almost no performance penalty, although it’s not clear what measure Google is using. The alternative torus network suits training, since chips mostly swap data with neighbors in a predictable manner, but inference on large mixture-of-experts and reasoning models can send smaller messages between distant chips, with every hop adding delay. With Boardfly, it’s at most seven hops against 16 at 1,024 chips.

What is lacking is a full specifications table, the amount of HBM capacity, pod sizes, an availability date, pricing, and customers for either system. Details on attestation, that is, the ability for the customer to verify confidentiality, are also missing for 8tx. The 8tx and 8i Max are currently listed, along with the 8i and 8t, as “coming soon” on Google’s Cloud TPU page.

Delivering two new TPU systems nearly seven months after April’s pair shows Google wants to turn out specialized systems much more quickly than has been the case in the past. Now, it just needs to prove them, through benchmarks and an availability date when the hardware is actually in customers’ hands.