William Liu · Podcasts

All podcasts · Series

Mastering Language Models: From Architecture to Optimization

Maya and Leo open the series with the map: seven stops from the Transformer blueprint to the machinery under massive models, anchored by a three-person startup building an insurance-claims assistant on eight GPUs. They lay out the mental models every LLM expert shares — trust curves, find the bottleneck, separate capability from behavior — then stage the field's cleanest fight on air: bigger models versus more data, from OpenAI's 2020 scaling curves to Chinchilla's flip to the serving-cost era that ran past both camps. Plus trailers for the live attention debate and the alignment fight to come.

45 episodes 45 with audio
Apple Podcasts Spotify Overcast Pocket Casts Android RSS
  1. SE0 — Series Overview — Mastering Language Models: From Architecture to Optimization
  2. T1E0 — T1E0 · Foundations of Sequence Modeling: The Transformer Revolution
  3. T1E1 — T1E1 · Attention Is All You Need
  4. T1E2 — T1E2 · Kimi Linear: An Expressive, Efficient Attention Architecture
  5. T1E3 — T1E3 · Attention Residuals
  6. T2E0 — T2E0 · Scaling and Training Large Models Efficiently
  7. T2E1 — T2E1 · Scaling Laws for Neural Language Models
  8. T2E2 — T2E2 · Training Compute-Optimal Large Language Models
  9. T2E3 — T2E3 · Scaling Data-Constrained Language Models
  10. T3E0 — T3E0 · Advanced Distributed Training: Overcoming Bottlenecks
  11. T3E1 — T3E1 · GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism
  12. T3E2 — T3E2 · Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  13. T3E3 — T3E3 · ZeRO: Memory Optimization Towards Training Trillion Parameter Models
  14. T3E4 — T3E4 · Fully Sharded Data Parallel: Faster AI Training with Fewer GPUs
  15. T3E5 — T3E5 · Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing
  16. T3E6 — T3E6 · FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  17. T3E7 — T3E7 · FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
  18. T3E8 — T3E8 · Embarrassingly Simple Self-Distillation Improves Code Generation (SSD)
  19. T3E9 — T3E9 · Test-Time Scaling Makes Overtraining Compute-Optimal
  20. T4E0 — T4E0 · Fine-Tuning and Specialization: LoRA and Beyond
  21. T4E1 — T4E1 · LoRA: Low-Rank Adaptation of Large Language Models
  22. T4E2 — T4E2 · QLoRA: Efficient Finetuning of Quantized LLMs
  23. T4E3 — T4E3 · LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
  24. T4E4 — T4E4 · Continual Learning of Large Language Models: A Comprehensive Survey
  25. T5E0 — T5E0 · Reinforcement Learning from Human Feedback (RLHF)
  26. T5E1 — T5E1 · Proximal Policy Optimization Algorithms
  27. T5E2 — T5E2 · Learning to Summarize with Human Feedback
  28. T5E3 — T5E3 · Training Language Models to Follow Instructions with Human Feedback
  29. T5E4 — T5E4 · Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  30. T5E5 — T5E5 · Constitutional AI: Harmlessness from AI Feedback
  31. T5E6 — T5E6 · Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  32. T5E7 — T5E7 · RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  33. T5E8 — T5E8 · Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  34. T5E9 — T5E9 · Reinforcement Learning from Human Feedback: Progress and Challenges
  35. T6E0 — T6E0 · The Latest in Fine-Tuned and Open Models: From LLaMA to DeepSeek
  36. T6E1 — T6E1 · Llama 2: Open Foundation and Fine-Tuned Chat Models
  37. T6E2 — T6E2 · The Llama 3 Herd of Models
  38. T6E3 — T6E3 · DeepSeek-V3 Technical Report
  39. T6E4 — T6E4 · DeepSeek-V4-Flash
  40. T7E0 — T7E0 · Mixture-of-Experts Models and Handling Massive Models: Sparsity, Data, and Optimization
  41. T7E1 — T7E1 · Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
  42. T7E2 — T7E2 · Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training
  43. T7E3 — T7E3 · The Pile: An 800GB Dataset of Diverse Text for Language Modeling
  44. T7E4 — T7E4 · AutoML from Basics to State-of-the-Art Techniques
  45. T7E5 — T7E5 · Recent Advances in Optimization Methods for Machine Learning: A Systematic Review