Subscribe
Transcript
MayaWhat do you change when training ever more capable agents on roughly the same size of open model starts looking like a ceiling?
LeoThe obvious answer is, make the model bigger. The expensive answer. Kimi's report says that's only half of it: K3 scales the pre-trained foundation to 2.8 trillion total parameters, then scales reinforcement learning, reasoning effort, and long-horizon interaction on top. Bigger base and harder practice, together.
MayaSet that against the oversight problem we just examined.
LeoThere, the question was how to catch an agent pursuing a hidden objective. Here, the report is deliberately building more capable long-horizon execution: capability construction rather than capability detection.
MayaAnd the systems claim?
LeoMake information flow farther in three directions: across a very long sequence, back through model depth, and out across a huge expert pool.
MayaHmm. Three architecture slides stapled together unless the connection earns its keep.
LeoStart with sequence length. K3 repeats three Kimi Delta Attention layers, then one Gated M-L-A layer. Delta attention carries a fixed-size recurrent state; the periodic global-attention layer can still connect distant content directly.
MayaSo, efficient running memory with an occasional full-room conversation.
LeoThat's it. Technically, K-D-A updates a recurrent state with a learned write strength and channel-wise forgetting. K3 bounds that decay so the chunk computation stays inside B-F-sixteen range and can use dense Tensor Core operations, including the diagonal tiles that were a bottleneck before.
MayaWait — a change in the range of a forget gate becomes a hardware win?
LeoYes. The report fixes the lower log-decay bound at minus five. That keeps the reciprocal scaling over a sixteen-token tile below the B-F-sixteen limit, so the kernel avoids a special position-by-position path.
MayaThat's the detail I trust more than “efficient attention.” It says what stopped overflowing and what computation disappeared.
LeoNow, quick check for you listening: does a one-million-token window mean every layer performs global attention over every token? No. Most layers use the recurrent K-D-A path, and each repeating block ends with Gated M-L-A for unrestricted global content interaction.
MayaAnd no R-o-P-E retuning?
LeoThe report says no explicit positional encoding. Position is carried implicitly by K-D-A's recurrence and decay, so context extension doesn't require rescaling a positional frequency. But they still train the ability: eight thousand to sixty-four thousand tokens in pre-training, then two hundred fifty-six thousand to one million during cooldown.
MayaGood. A big address space isn't the same thing as knowing how to use the far end of it.
LeoWhich is why they also synthesize tasks whose needed clues are scattered across the full context. Length alone, the report says, can collapse back into local pattern matching. All right. That's movement across time. The route through depth—
Maya—is Attention Residuals. A normal residual stream keeps compressing everything from earlier layers into the current state. AttnRes lets a layer selectively retrieve representations from earlier blocks, more like opening the relevant folder than inheriting one repeatedly photocopied stack.
LeoThe layer gets to ask which earlier representation it needs, not just accept the accumulated blend.
MayaYes. K3 groups the network to control memory and communication. The report uses twelve-layer blocks; the partial final block and embedding source bring the accounting to nine blocks.
LeoAnd width is the expert system.
MayaStable LatentMoE. There are eight hundred ninety-six routed experts, but each token activates sixteen, plus two shared experts. The routed experts work in a smaller latent width, so expanding the expert pool doesn't multiply full-width communication sixteen times.
LeoTry this check: when the headline says 2.8 trillion parameters, are all 2.8 trillion active for each token? No. The architecture table reports about 104.2 billion activated parameters.
MayaStill enormous, just a different kind of enormous. And that sparsity creates nasty training problems. More experts means more chances for a few to overheat while others barely train; the latent routed branch can also produce exploding internal activations.
LeoSo “Stable” is doing work in the name.
MayaThree pieces of work. An R-M-S normalization before the up-projection, a capped activation called SiTU-G-L-U, and Quantile Balancing for routing. The activation caps the product at one hundred with the report's settings, instead of letting both SwiGLU branches grow without bound.
LeoAnd Quantile Balancing?
MayaIt adjusts each expert's routing bias from the score quantile that corresponds to its target load. It finds the cutoff that would send the right share of tokens to that expert, then applies that bias on the next step.
LeoRather than nudging an overloaded expert down by some fixed little step and hoping the system settles.
MayaRight. At scale they estimate those quantiles with pooled histograms. The counts add across ranks, so they get the global-batch distribution without gathering millions of raw margins.
LeoHere's my first real objection. The report bundles new attention, new residuals, a new expert layout, data changes, and training changes, then says scaling efficiency improved by about two and a half times over K2. Where's the evidence that the architecture is the cause?
MayaThe honest answer is that the headline number is collective. Their held-out scaling-law curves compare the K2 and K3 families, and the report attributes the gain to architecture, data, and training recipes together. It gives component ablations in places — the from-scratch vision encoder, for example — but it doesn't isolate a clean percentage for each part of the two-and-a-half-times claim.
LeoSo builders should read it as a system result, not a coupon they can redeem by copying one layer.
MayaAnd after pre-training?
LeoThe pipeline starts with supervised fine-tuning, trains reinforcement-learning experts, then folds them back into one model.
MayaHow many experts are we talking about here? Not MoE experts — behavioral versions.
LeoNine teacher policies. Three domains — general tasks, general agents, and coding agents — crossed with low, high, and maximum reasoning effort. Multi-Teacher On-Policy Distillation consolidates those policies into one effort-conditioned model.
MayaOne instructor teaches coding at maximum effort, another teaches general agent work at low effort, and the final model absorbs all nine lessons without becoming nine separate products.
LeoThat's the post-training move. And the tasks aren't just answer-and-grade. The report trains a loop of reasoning, acting, observing, verifying, and adapting in white-box environments with varied harnesses, tools, memories, skills, and context-management strategies.
MayaVaried harnesses matters. If every training run uses one tool schema—
Leo—the model may learn the costume instead of the job. They generate configurations resembling Kimi Code, Claude Code, Codex, OpenClaw, and Hermes. For autonomous execution tasks, an independent verifier inspects the final environment state; the agent doesn't get rewarded just for saying, “done.”
MayaTheir own personal-assistant tasks can run for thousands of tool calls and millions of accumulated context tokens.
LeoWhat happens when a rollout takes forever? They don't make the whole training batch wait. Partial rollouts pause once a chosen fraction finishes, optimize those completed trajectories, and queue the unfinished ones to resume in the next iteration.
MayaBut resuming isn't just restoring text. The tool environment may have changed files, opened processes, built half a project—
Leo—so they preserve the world too. AgentENV uses resumable Firecracker micro-V-M sandboxes. The report gives checkpoint and resume latencies as low as one hundred thirty-three and forty-nine milliseconds, and says a paused sandbox uses no memory or C-P-U.
MayaThat's a remarkable claim. Also a reminder that the training mechanism is an infrastructure mechanism.
LeoCompletely. They report creating 51,219,741 sandboxes across 1,505,678 images during training and evaluation. This isn't “run R-L on a few prompts.” It's a factory for persistent experience.
MayaGive me the evidence without turning the next minute into a leaderboard recital.
LeoLook for a pattern across different kinds of work. In the report's public suite, K3 scores 77.8 percent on ProgramBench, the best listed result; 88.3 on Terminal-Bench 2.1, just under GPT-5.6 Sol at 88.8; and 81.2 on the long-horizon FrontierSWE score, second to Claude Fable 5 at 86.6.
MayaAgent work?
LeoBrowseComp at 91.2 percent, DeepSearchQA at 95.0 F-one, and AutomationBench at 30.8, all highest in their comparison table. The BrowseComp footnote matters: with the full million-token window and no context management, K3 gets 90.4; the 91.2 result uses compaction triggered at three hundred thousand tokens.
MayaThat footnote changes the story from “the window solved it” to “the window plus context management solved it a little better.”
LeoAnd the overall claim stays bounded. The report says K3 still trails Claude Fable 5 and GPT-5.6 Sol overall, even while leading many individual open and proprietary baselines in its suite.
MayaResearch-level reasoning is one place the gap shows. The report gives K3 23.4 percent on CritPt and 43.5 on Humanity's Last Exam without tools, 56.0 with them. Those aren't failure numbers, but they're not “problem solved.”
LeoNor are benchmark tables deployment guarantees. Harnesses differ, several results come from internal evaluations, and all K3 main evaluations use maximum reasoning effort. The report is unusually detailed about configurations, but a builder still has to reproduce the workload that matters.
MayaThen here's the systems objection. “Open frontier intelligence” sounds accessible. But 104.2 billion active parameters, custom K-D-A kernels, expert-parallel routing, prefix-cache machinery, and fleet scheduling don't.
LeoThat's the limitation I'd put in red ink. Open weights remove a permission barrier; they don't remove the systems bill. The paper says its million-token R-L experiments stay within a few hundred G-P-Us, which is presented as efficient for this scale, not small in ordinary builder terms.
MayaSo downloading the weights is not reproducing K3.
LeoNot remotely. The released model gives builders a frontier artifact. Recreating the training efficiency requires MoonEP's balanced expert execution, memory offload, K-D-A context parallelism, persistent sandboxes, and serving logic that jointly manages recurrent states and M-L-A caches.
MayaThere is a practical upside, though. Sparse activation means you're not paying dense 2.8-trillion-parameter compute for every token, and the report's cost study puts BrowseComp at two dollars and three cents per task, about half its stated GPT-5.6 Sol cost.
LeoYes, but that's their production system doing the work. The architecture and the infrastructure are a matched pair. Separate them and the headline efficiency may not travel.
MayaI think that's where the report is strongest, actually. It doesn't pretend the model is a magic file.
LeoThe mechanism runs from a bounded forget gate all the way out to which cluster holds a session's cache. Which makes the capability gain easier to believe, and harder to copy.
MayaAnd that closes the thesis. K3 pushes both scaling axes — a much larger sparse foundation and much richer test-time training — by engineering information flow through tokens, layers, experts, and long-lived environments as one system.
LeoSo here's the question to sit with: if you wanted to build on K3, which part of its claimed frontier could you actually reproduce and evaluate — the weights, the long-horizon behavior, or the infrastructure that makes the two meet?