Subscribe
Transcript
MayaImagine trying to write a paragraph while also proofreading the one you just wrote. Not line by line, not in a neat left-to-right way, but by revisiting chunks, fixing them, and only then committing them to the page.
LeoThat already sounds more like editing than writing.
MayaExactly. And that is the core tension in this paper. NVIDIA's Nemotron-Labs-TwoTower asks: what if the model that represents clean context and the model that denoises noisy tokens were not the same set of weights?
LeoSo one tower keeps the story straight, and the other tower does the cleanup.
MayaThat is the intuition. And the result is interesting: they keep most of the quality of the autoregressive baseline, while getting a big wall-clock speedup.
LeoWhich is the kind of sentence that makes every infrastructure person sit up a little straighter.
MayaLet's start with the bigger picture. Autoregressive language models are the default because they are simple to reason about: one token, then the next token, then the next. But that simplicity comes with a throughput bottleneck.
LeoOne token at a time is elegant, but it is not exactly aggressive.
MayaRight. Diffusion language models try to break that bottleneck by generating blocks in parallel and refining them iteratively. The promise is obvious: if you can fill in several tokens at once, you can potentially get more throughput and maybe more controllability.
LeoBut the paper says the standard setup puts too much burden on one network.
MayaYes. Existing approaches often use a single network to do two different jobs. One job is context representation: keep track of the clean prefix, the committed text, the state of the sequence. The other job is iterative denoising: look at a noisy block and decide which tokens should survive the next refinement step.
LeoThose are related jobs, but they are not identical.
MayaThat is the whole point. If the same weights have to be excellent at both, they can get pulled in different directions. The paper argues that this entanglement limits how good the model can get at either role.
LeoSo TwoTower is really a specialization story.
MayaExactly. Specialize the roles. Keep one tower frozen as the causal autoregressive context tower. Train the other tower as the diffusion denoiser.
LeoWalk me through the mechanics in plain English.
MayaYou start with the prompt. The context tower reads the prompt causally and produces the usual machinery: key-value cache and Mamba state. That tower is frozen, so it preserves the pretrained autoregressive backbone. Then generation happens block by block. Each new block starts as a set of mask tokens. The denoiser tower takes that noisy block and iteratively refines it over several denoising steps.
LeoSo the block begins as "I know nothing," and then it gets cleaned up step by step.
MayaYes. And while it is doing that cleanup, the denoiser does not work in isolation. It cross-attends to the corresponding layers of the context tower. That is a subtle but important design choice.
LeoWhy layer-aligned cross-attention instead of just passing one final hidden state?
MayaBecause the same-index layers in the two towers are operating at comparable representation levels. So the denoiser can get multi-scale access to the backbone's internal states, rather than depending on one compressed summary.
LeoThat sounds more like a real conversation between the towers instead of a handoff through a single bottleneck.
MayaExactly. The paper also modulates the denoiser with the diffusion timestep using adaptive layer normalization. In practice, that means the denoiser knows how far along the refinement process it is.
LeoAnd the context tower stays causal the whole time.
MayaYes. It keeps the clean prefix and committed blocks coherent while the denoiser works on the noisy block in front of it.
LeoThe paper builds on Nemotron-3-Nano-30B-A3B, right?
MayaRight. That is a 30B hybrid model with a mix of Mamba-2 layers, attention layers, and mixture-of-experts layers. The point is not just that the backbone is large. It is that the backbone already mixes different computation styles, and TwoTower reuses that backbone in a more specialized way.
LeoSo the context tower is basically the pretrained model doing what it already knows how to do.
MayaYes. Frozen, causal, preserving the original strengths. The denoiser tower is where the diffusion-specific training happens. The paper also tests a bidirectional Mamba variant, but the gains were marginal, so the final design keeps Mamba causal.
LeoThat is a nice example of an experiment that sounds clever but does not earn its keep.
MayaAnd that is why I like the paper. It is not just "we made a novel tower-shaped thing." It is a fairly disciplined engineering tradeoff. They ask which parts actually improve quality and throughput, and which parts are just expensive flourishes.
LeoLet's talk results.
MayaThe headline claim is that Nemotron-Labs-TwoTower retains 98.7 percent of the autoregressive baseline's quality while delivering 2.42 times higher wall-clock generation throughput.
LeoThat is the part everyone cares about. If you are going to change the generation regime, you need the payoff.
MayaExactly. The paper also says the model was trained on about 2.1 trillion tokens, which is a fraction of the 25 trillion tokens used to pretrain the backbone.
LeoSo they are not rebuilding the whole foundation from scratch. They are leveraging a pretrained backbone and reshaping the inference behavior.
MayaYes. That is the more interesting production story. They are trying to preserve the expensive general-purpose competence that already exists, while changing the generation loop so the system can commit more than one token per refinement step early in decoding.
LeoThat helps explain why the wall-clock gain can be real even if the model is still doing iterative refinement.
MayaRight. Throughput is not just a theoretical property. It is how fast the system actually pushes text out of the door.
LeoThis is where I want to make the bridge to coding agents.
MayaThe bridge is pretty direct. TwoTower is a design about splitting stable context from iterative correction. That is a pattern we see everywhere in agent systems.
LeoFor example, a coding agent might keep a durable repo summary or task state in one place, then run a separate repair loop against a draft patch.
MayaExactly. The lesson is not "use diffusion everywhere." The lesson is: do not force one component to be equally good at memory, representation, and local correction if those jobs want different inductive biases.
LeoSo the analog for software is not just model architecture. It is harness design.
MayaYes. A clean context tower is like the part of a system that preserves the truth: the task brief, the repo state, the committed changes, the running caches. The denoiser tower is like the part that explores candidate fixes, revises them, and converges.
LeoThat sounds a lot like the difference between a planner and a patcher.
MayaOr between a repository index and a repair loop. Same general idea: keep the source of truth stable, and let the noisy iterative work happen in a separate lane.
LeoI want to return to the layer-aligned cross-attention, because that feels like one of the most reusable ideas.
MayaIt is. A shallow summary would be "the denoiser looks at the context tower." But the paper is more precise than that. Each denoiser layer attends to the corresponding context layer.
LeoSo the relationship is structurally matched.
MayaRight. That matters because same-index layers in the two towers are already operating at comparable abstraction levels. It is a little like giving each specialist agent the exact stage of the project state it needs, instead of dumping the whole notebook on it.
LeoThat would make the denoiser less like a noisy intern and more like a specialist with the right references on the desk.
MayaExactly. And that is why this paper is a nice example of architecture doing real work. The network wiring is not just aesthetic. It encodes who should look at what, when.
LeoWhat are the cautions here?
MayaFirst, this is not a blanket argument that diffusion beats autoregression. The paper is careful about the operating point. The claim is about preserving most of the baseline quality while improving throughput on a particular hybrid architecture and training recipe.
LeoSo you should not read this as "throw away AR models."
MayaNo. You should read it as "there is room to decouple roles inside the generation stack." That distinction matters. Second, the throughput number is wall-clock throughput, which is what users feel, but it depends on implementation details, denoising budgets, and how the towers are connected.
LeoIn other words, the architecture is only half the story. The system implementation still matters.
MayaExactly. And third, the bidirectional Mamba experiment is a reminder that not every plausible extension pays off. The paper tested it, found only marginal gains, and did not force it into the final design.
LeoThat restraint is a good sign.
MayaIt is. Good engineering papers usually tell you what they tried and why they did not ship the flashiest version.
LeoIf you had to compress the whole paper into one sentence, what would it be?
MayaSomething like: if one model has to both preserve context and denoise noisy blocks, it may do both jobs acceptably, but you may get better throughput and nearly the same quality if you split those responsibilities into specialized towers.
LeoThat is the essence.
MayaAnd for builders, the practical lesson is broader than text generation. Whenever a system is mixing stable memory, causal context, and iterative repair, ask whether those roles should really share the same mechanism.
LeoBecause the wrong answer is usually expensive in two directions: slower generation and muddier internal state.
MayaExactly. TwoTower is a reminder that architecture can be a form of organizational design. Who keeps state, who revises, who commits, and who just stays frozen because they already know enough.
LeoThat is a pretty good description of how half of engineering actually works.
MayaAnd maybe that is why this paper feels so satisfying. It is not just about a smarter model. It is about separating responsibilities so the system can move faster without forgetting what it is doing.
LeoBefore we wrap, here is the question I want to leave people with: if your agent pipeline is slow, which part is genuinely supposed to be doing the thinking, and which part is just carrying context that should have been frozen all along?
MayaThat is the right question. And it is a good one to sit with the next time you are tempted to make one component do every job.