👋🏻 Hello, Rafael here - every week I cover interesting challenges and developments that I’ve come across recently through the lens of an engineer building AI systems.
For most of the modern LLM era, text generation has followed a familiar pattern: predict the next token, append it to the context, then predict the one after that.
That basic loop has scaled remarkably well. It sits underneath most frontier language models, coding models, and agent systems in production today.
It also creates a structural constraint. Generation is sequential. Even when models become better at predicting several tokens ahead, the decoding process still depends on a left-to-right chain of commitments.
Google DeepMind’s DiffusionGemma explores a different design.
The model starts from Gemma 4 26B A4B, but changes how output is produced. Instead of generating one token at a time, DiffusionGemma operates on blocks of 256 tokens and refines them iteratively. The paper reports roughly 1,500 output tokens per second on a single H100, along with about 20 tokens per forward pass on average.
Those numbers make more sense once you look at the architecture behind them.
MTP still lives inside the autoregressive model
Gemma 4 can already be served with multi-token prediction (MTP), a speculative-decoding technique designed to accelerate autoregressive generation.
The goal of MTP is straightforward. Instead of relying on one next-token prediction per forward pass, a drafter proposes multiple tokens, which the main model verifies.
Speculative-decoding approaches can produce several tokens per forward pass, although their efficiency depends on how many proposed tokens are accepted. In DiffusionGemma’s own benchmark comparison, Gemma 4 with MTP averages about 1.4 tokens per forward pass
The underlying sequence remains autoregressive, though. Generation still moves from left to right. Once a token has been accepted and added to the history, later tokens condition on it. The model cannot revise that part of the sequence during the same decoding process.
This is the important distinction: MTP improves how quickly an autoregressive model moves through a sequence. DiffusionGemma changes the structure of the generation step itself.
A 256-token canvas
DiffusionGemma generates text in blocks.
The model first encodes the user prompt and the existing conversation using causal attention and stores the result in a KV cache.
It then creates a 256-token canvas filled with noisy tokens. A decoder processes that canvas with bidirectional attention while also attending to the KV cache.
Over a sequence of denoising steps, the model progressively refines the entire block. Once the canvas Once the canvas is fully denoised or meets the adaptive stopping criterion, the generated block is committed to the history and encoded back into the KV cache. The process then repeats for the next block.
The architecture therefore has two different forms of temporal structure: across blocks, generation is causal. Inside each block, generation is bidirectional. That creates a useful middle ground between fully autoregressive generation and fully parallel generation.
The model can preserve a long conversation history through a standard KV cache while giving itself much more flexibility inside the part of the response it is currently writing.
What bidirectional generation changes
The most immediate consequence is that tokens inside the active canvas are provisional.
If the model produces an awkward or incorrect token early in the denoising process, that token can still change on a later step.
The paper describes this as a form of self-correction. Because every token in the canvas can attend to the rest of the canvas, later emerging context can influence earlier positions before the block is finalised. This is easy to miss if DiffusionGemma is viewed only as an inference optimisation.
A speculative decoder can predict several tokens ahead, but it still works within the logic of left-to-right generation. Once those tokens are verified, they are part of the sequence. DiffusionGemma keeps the whole block open for revision during generation.
That difference becomes visible in some of the examples in the paper. On reasoning tasks, the model can revise an initial answer as the reasoning develops elsewhere in the canvas. On structured outputs, predictable portions of the response can settle quickly while uncertain regions continue to change.
For tasks such as JSON generation, code editing, extraction, or constrained formatting, this structure is particularly interesting.
A large part of the desired output is often already implied by the prompt. An autoregressive model still has to emit every token sequentially. A diffusion model, on the other hand, can resolve many of those predictable positions in parallel. The paper shows structured generation examples converging in only a few denoising steps.
Turning an autoregressive Gemma into a diffusion model
Another important part of the paper is the training recipe. DeepMind did not pretrain DiffusionGemma from scratch. The model starts from the publicly released Gemma 4 checkpoint and is converted through a two-stage process that uses less than 10% of the original model’s training-token budget.
The first stage is supervised fine-tuning. This is the stage where an autoregressive model begins learning the mechanics of discrete diffusion. The model learns to reconstruct clean blocks of text from corrupted versions of those blocks. It uses bidirectional attention inside the current canvas while conditioning on the clean prompt and previous blocks through the causal encoder.
The second stage is more unusual. DeepMind combines reinforcement learning with sampler distillation in a single online training phase. The reinforcement-learning objective pushes generation quality and reward upward, while sampler distillation compresses that quality into fewer denoising steps.
The sampler distillation component pushes the model toward producing good outputs in fewer denoising steps. Both objectives are optimised together.
That means training is shaping more than the final output distribution. It is also shaping how efficiently the model reaches that output. This matters, because diffusion models have an inference trajectory. They may need many refinement steps, or they may converge quickly. The training process explicitly tries to shorten that trajectory.
Training the model to finish sooner
Another trick used by DiffusionGemma is adaptive stopping. At every denoising step, the model measures its own uncertainty. If average predictive entropy falls below a threshold and the deterministic predictions have stopped changing, generation can terminate early.
The model is allowed up to 48 denoising steps, but the paper reports an average of around 12 effective steps across its evaluation suite. The reinforcement learning and sampler distillation stage gradually reduces predictive entropy as training progresses.
That creates an interesting feedback loop. As the model becomes more confident, adaptive stopping activates earlier. The training distribution then contains shorter denoising trajectories. The model becomes increasingly specialised at solving tasks in fewer steps.
This gives DiffusionGemma a different relationship with test-time compute. An autoregressive model has a relatively simple cost model. A longer response requires more generated tokens and therefore more sequential decoding steps.
DiffusionGemma introduces another variable: the number of refinement steps required for each block. An easy, highly constrained task may converge quickly. A harder reasoning problem may use more denoising steps before the output stabilizes.
The gap between MTP and diffusion shows up in the numbers
The benchmark results make the architectural difference easier to quantify.
In the paper’s aggregate measurements, Gemma 4 with MTP reaches around 303 output tokens per second and about 1.4 tokens per forward pass.
DiffusionGemma reaches roughly 1,479 output tokens per second and around 19.7 tokens per forward pass in its thinking configuration. These measurements are specific to the hardware, serving configuration, and benchmark suite used in the paper, so they should not be treated as universal performance ratios.
Still, they show why DiffusionGemma belongs in a different category from MTP. MTP squeezes more useful tokens out of an autoregressive forward pass. Diffusion decoding allows many token positions to be refined during the same forward pass.
The hardware consequences are also different. Traditional autoregressive serving is often constrained by memory bandwidth because each decoding step repeatedly moves model weights and KV cache data. DiffusionGemma performs heavier individual forward passes, but requires far fewer of them. The paper argues that this shifts more of the workload toward compute and away from repeated memory transfers.
That trade appears especially favourable at low batch sizes, where per-user latency matters. At higher concurrency levels, the advantage decreases. The paper reports that autoregressive models begin to regain a throughput advantage at moderate batch sizes, around 32 concurrent requests in their tests.
One set of weights, two decoding modes
DiffusionGemma retains another useful property from its Gemma origins.
The final model can still generate autoregressively. The same weights can be loaded into the original causal architecture and used for standard left-to-right decoding. The paper reports some quality recovery when the model runs in AR mode, although throughput is lower than in diffusion mode.
That opens up several practical possibilities. A serving system could use diffusion decoding for latency-sensitive tasks and switch to autoregressive decoding when the task benefits from it. A system could potentially use diffusion for drafting or planning and AR decoding for final generation. Different requests could be routed according to latency, quality requirements, context length, or output structure.
The impact for agents
The implications for agents are worth considering because agent workloads tend to amplify inference latency.
An agent may make a model call to plan, another to select a tool, another after observing the tool result, and several more while refining the final output. Each iteration adds latency. Faster generation changes how many iterations can fit inside a responsive interaction.
Blockwise generation may also fit naturally with several common agent tasks. Tool calls often require structured arguments. Code-editing agents frequently modify an existing artifact where most of the output is already known. Data-extraction agents produce outputs constrained by schemas. Planning agents may benefit from a generation process that can revise earlier parts of a local plan while later parts are still taking shape.
There is still a large gap between those possibilities and a demonstrated production advantage. The paper evaluates agentic behavior with Tau-bench, but it does not provide a full study of end-to-end agent architectures built around diffusion decoding. That makes this an interesting research direction rather than a settled conclusion.
The trade-offs are still substantial
DiffusionGemma does not dominate its autoregressive parent model on capability.
The paper explicitly reports a performance gap relative to the original Gemma 4 checkpoint. The authors attribute this to the cost of converting an autoregressive model rather than training a diffusion-native model from scratch, the relatively short fine-tuning budget, and the decision to optimise aggressively for low-latency generation.
The model can also produce occasional token repetition and stuttering, an artifact the authors connect to operating with very short denoising trajectories.
And as mentioned previously, the serving economics vary with workload. The latency advantage is strongest in low-batch settings. At higher batch sizes, the higher compute cost of diffusion becomes more important, and autoregressive models eventually regain a throughput advantage.
The open question is where these trade-offs add up to a better overall system. Interactive agents, coding environments, structured generation and low-concurrency workloads are obvious candidates, but the paper does not settle that question.
A wider design space for language generation
Autoregressive decoding has accumulated an enormous amount of infrastructure around it.
Serving systems, KV cache optimisations, speculative decoding, quantisation techniques, agent frameworks, and API conventions have all been built around next-token generation.
DiffusionGemma is interesting because it preserves enough of that infrastructure to remain practical, while changing an important part of the generation process. The model still uses a transformer. It still uses a KV cache. It still has causal sequence history. It still supports ordinary autoregressive decoding.
Inside the active generation block, however, it behaves differently. A 256-token region becomes something the model can refine in parallel, revisit, and resolve using bidirectional context.
That gives language generation a different set of engineering trade-offs. The result may end up as a specialized decoding mode rather than a replacement for autoregression. It may prove most useful for particular workloads. Future models may combine AR and diffusion more tightly than DiffusionGemma does today.
References
Note: the weights for DiffusionGemma had already been released in July 2026; the technical report was released by Google DeepMind on 31/07/2026.