williamliu.ai
EN中文

All podcasts · Series

Mastering Language Models: From Architecture to Optimization

Maya and Leo open the series with the map: seven stops from the Transformer blueprint to the machinery under massive models, anchored by a three-person startup building an insurance-claims assistant on eight GPUs. They lay out the mental models every LLM expert shares — trust curves, find the bottleneck, separate capability from behavior — then stage the field's cleanest fight on air: bigger models versus more data, from OpenAI's 2020 scaling curves to Chinchilla's flip to the serving-cost era that ran past both camps. Plus trailers for the live attention debate and the alignment fight to come.

45 episodes· 45 with audio · Updated 2026-07-19
Apple Podcasts Spotify Overcast Pocket Casts Android RSS

Episodes

  1. SE0 — Series Overview — Mastering Language Models: From Architecture to Optimization
  2. T1E0 — Foundations of Sequence Modeling: The Transformer Revolution
  3. T1E1 — Attention Is All You Need
  4. T1E2 — Kimi Linear: An Expressive, Efficient Attention Architecture
  5. T1E3 — Attention Residuals
  6. T2E0 — Scaling and Training Large Models Efficiently
  7. T2E1 — Scaling Laws for Neural Language Models
  8. T2E2 — Training Compute-Optimal Large Language Models
  9. T2E3 — Scaling Data-Constrained Language Models
  10. T3E0 — Advanced Distributed Training: Overcoming Bottlenecks
  11. T3E1 — GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
  12. T3E2 — Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  13. T3E3 — ZeRO: Memory Optimization Towards Training Trillion Parameter Models
  14. T3E4 — Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs
  15. T3E5 — Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing
  16. T3E6 — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  17. T3E7 — FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  18. T3E8 — Embarrassingly Simple Self-Distillation Improves Code Generation (SSD)
  19. T3E9 — Test-Time Scaling Makes Overtraining Compute-Optimal
  20. T4E0 — Fine-Tuning and Specialization: LoRA and Beyond
  21. T4E1 — LoRA: Low-Rank Adaptation of Large Language Models
  22. T4E2 — QLoRA: Efficient Finetuning of Quantized LLMs
  23. T4E3 — LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
  24. T4E4 — Continual Learning of Large Language Models: A Comprehensive Survey
  25. T5E0 — Reinforcement Learning from Human Feedback (RLHF)
  26. T5E1 — Proximal Policy Optimization Algorithms
  27. T5E2 — Learning to Summarize with Human Feedback
  28. T5E3 — Training Language Models to Follow Instructions with Human Feedback
  29. T5E4 — Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  30. T5E5 — Constitutional AI: Harmlessness from AI Feedback
  31. T5E6 — Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  32. T5E7 — RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  33. T5E8 — Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  34. T5E9 — Reinforcement Learning from Human Feedback: Progress and Challenges
  35. T6E0 — The Latest in Fine-Tuned and Open Models: From LLaMA to DeepSeek
  36. T6E1 — Llama 2: Open Foundation and Fine-Tuned Chat Models
  37. T6E2 — The Llama 3 Herd of Models
  38. T6E3 — DeepSeek-V3 Technical Report
  39. T6E4 — DeepSeek-V4-Flash
  40. T7E0 — Mixture-of-Experts Models and Handling Massive Models: Sparsity, Data, and Optimization
  41. T7E1 — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
  42. T7E2 — Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
  43. T7E3 — The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  44. T7E4 — AutoML from Basics to State-of-the-Art Techniques
  45. T7E5 — Recent Advances in Optimization Methods for Machine Learning: A Systematic Review