ML 八股速查

• Both: cut what a decode step reads, in opposite ways.
• Sparse attention (NSA, DSA): keeps token-level K/V, so selected old tokens come back in detail. Gives up: the cache keeps every token, reads still grow with length, and bad selection can miss information.
• Fixed state (SSM, linear attention): memory and per-step cost stay constant. Gives up: a finite state can't keep every token as context grows.

• Why mix: no single layer gives both cheap long context and direct lookup.
• Recurrent layers: a fixed-size state keeps decode cheap, but history gets compressed.
• Attention layers: retrieve earlier tokens directly, but need a growing KV cache.
• Kimi K3: mostly KDA layers (efficient, position-aware mixing), with occasional Gated MLA layers (global content attention).
• Nemotron 3 Ultra: mostly Mamba-2, with a few global attention layers as anchors.

• Memory: all experts must be held, so total memory follows total parameters.
• Communication: under expert parallelism, tokens are exchanged all-to-all between GPUs holding different experts.
• Weight traffic: large-batch decode touches nearly all experts.
• Load imbalance: with static tensor shapes, buffers waste compute or overflowing tokens are dropped.
• Training: hard routing adds instability, and balancing needs extra machinery.

中文讲解、推导和追问见 速查页

• MRoPE: splits position into temporal, height and width IDs.
• Text: uses identical IDs, so it reduces to 1D RoPE.
• Image: one temporal ID; height and width IDs come from spatial position.
• Video: the temporal ID increases across frames.

• What: extra heads predict tokens beyond the next one.
• Training: predicting tokens further ahead gives a richer training signal.
• Inference: heads draft tokens for speculative decoding. The backbone verifies the drafts in one pass, and several can be accepted.
• Limit: helps mainly at small batches; at large batches verification overhead can erase the gain.

• Late fusion (Qwen2.5-VL): a separate ViT feeds an LLM initialized from a text model, then they train jointly.
• Its tradeoff: gains native resolution, fine detail and retained text strength, but covers understanding, not image generation.
• Early fusion (Chameleon): images become discrete tokens in one Transformer trained from scratch, so reasoning and generation can happen in any order.
• Its tradeoff: training is hard to stabilize, and the tokenizer limits detail.

• Kaplan: as compute grows, spend mostly on model size.
• Chinchilla: scale model and data about equally, so existing models were undertrained. It blames Kaplan's fixed token horizon and schedule.
• Why train past compute-optimal: that point counts only training compute. Llama 3 trains smaller models far beyond it, spending more training compute for cheaper inference.

• Fair evaluation (DCLM): fix everything except data: same raw data pool, model, training recipe, compute and evaluation tasks. Only curation changes.
• Small vs. large: small models largely predict large ones, so iterate cheaply small, then scale up.
• Limit: small runs separate large differences but don't settle close calls.

• Purpose: mid-training (older name: annealing) prepares the base model for better post-training.
• Goal 1: raise reasoning capability by switching to higher-quality math, code and science data.
• Goal 2: extend long-context ability by raising context length with longer documents.
• Learning rate: ends near zero, which the lecture explains as avoiding disrupting what was learned.

• Sources: softmax logits and activations drifting upward, low precision losing small values, hard MoE routing, and weights growing over long runs.
• Normalization: bounds values (QK-Norm, reordering); z-loss penalizes drift in the final softmax.
• Precision: selective high precision protects sensitive parts.
• Optimizer: weight decay limits growth, and update scaling evens update size.

• QK-Norm: layer norm on queries and keys, which controls norm growth going into the attention softmax.
• Norm reordering (Swin-style): normalizes each sublayer's output, which bounds feed-forward growth that SwiGLU can amplify.
• Z-loss: a small penalty on the softmax normalizer, which restrains logit drift in the final softmax, where QK-Norm doesn't reach.

• Selective precision: only the router runs in float32, giving float32's stability at bfloat16's speed.
• Smaller initialization: improved quality and cut variance across runs.
• Expert dropout: higher inside experts during fine-tuning, which counters overfitting from many more parameters on small tasks.

• PreNorm: normalize before each sublayer: x + F(Norm(x)). Stable, because gradients bypass sublayers through residuals, but this reduces effective depth.
• PostNorm: normalize after adding the residual: Norm(x + F(x)). Keeps depth, but risks vanishing and exploding gradients, so MiniMax adds DeepNorm.
• Why PostNorm: its hybrid architecture depends on effective depth, and PostNorm scored higher on most benchmarks.

• Weight decay: without it, weights and layer outputs keep growing past bf16's high-precision range, limiting long runs. So AdamW-style decay was added.
• Update RMS: Muon's update size depends on matrix shape, so large matrices update too little and small per-head ones too much, causing instability.
• Fix: scale updates by matrix shape. Updates become consistent and match AdamW's size, so hyperparameters transfer without tuning.

• Problem: AdamW is the default. Muon beat it on small models, but its gains diminished when scaled up.
• Main idea: compute gradient momentum, then approximately orthogonalize that matrix, so weights don't learn along only a few dominant directions.
• Scaling fixes: weight decay and shape-aware update scaling make it work without tuning.
• Result: scaling laws show about 2× compute efficiency over AdamW.

• Main idea: treat chain-of-thought as an action sampled before predicting the next token.
• Reward: information gain, meaning how much the thought raises the observed token's log-likelihood over a no-think EMA baseline.
• Difference from RLVR: no verifier or curated problems, runs on ordinary pretraining text in pretraining's last phase, and rewards predictive usefulness rather than correct answers.
• Afterwards: SFT and RLVR still follow.

• Preference optimization needs: comparisons from humans or AI judges. InstructGPT also trains a reward model and samples during RL; DPO uses a static pair dataset.
• It optimizes: a learned proxy for preference under a KL constraint, so it handles non-verifiable tasks.
• RLVR needs: hard questions, a reliable verifier, RL compute and a capable base, but no human reasoning traces (R1-Zero).
• It optimizes: final-answer correctness.

中文讲解、推导和追问见 速查页

• Preference RM: sparse, one learned score per response. Over-optimized, so scores rise while true quality falls.
• Verifier: sparse, outcome only. Hard to hack where rules are reliable, yet agents may cheat.
• Rubric: one weighted score per response. Harder to game, but still over-optimizes and can reward verbosity.
• Teacher: dense, per token. Stylistic tokens can dominate KL, and off-support prefixes give unreliable guidance.

• Off-policy (fixed data): SFT, word- and sequence-level KD, offline DPO.
• On-policy (model's own samples): PPO, GRPO, GKD at λ=1, OPSD.
• Problem 1, exposure bias: prefixes at inference differ from prefixes in training, so errors compound.
• Problem 2, staleness: samples from old policies weaken the signal, and performance falls as they get staler.

• RLHF: a KL penalty to the SFT model limits drift away from where the reward model is accurate, reducing over-optimization.
• DPO: solves that same KL-constrained objective implicitly, with no KL term in the loss; β sets its strength.
• Distillation: KL to the teacher is the objective itself, pulling the student's next-token distribution toward the teacher's.
• Summary: a leash in the first two, the training target in the third.

• Forward KL(T‖S): weighted by the teacher. Wherever the teacher puts mass, the student must too. It covers modes, but a weak student may spread across them and hallucinate.
• Reverse KL(S‖T): weighted by the student. It heavily penalizes tokens the teacher rejects, but the student can drop teacher modes, so it seeks one mode.
• Which is better: GKD finds it depends on the task.

• SFT and off-policy KD: train on fixed prefixes, so errors compound at inference (exposure bias).
• RL: uses the model's own rollouts, but spreads one score across all tokens, and gives zero gradient when a group ties.
• On-policy distillation: dense teacher feedback on the student's own rollouts (token-level DAgger).
• Costs: needs a teacher, adds generation and teacher passes to the loop, and weakens when rollouts leave the teacher's support.

• Where compute goes: mostly rollout generation. Inference uses several times more compute than training, the learner sits idle much of the time, and a few slow generations dominate rollout time.
• Why speed matters: RL is a large and growing share of frontier compute, and more RL steps still help, so wasted generation limits how far training can go.

• How: generation and learning run in parallel on separate GPUs, using fast inference engines. Training uses samples from slightly older policies.
• Cost: off-policy data gives a worse signal, and performance falls as staleness grows.
• Findings: Online DPO is most robust; robustness rises with policy size, not reward-model size; one-step async matches sync while running faster.

• Cut idle compute: fully asynchronous actors and learner; continuous batching (a finished slot immediately takes a new prompt); inflight weight updates that keep the KV cache (the largest measured gain).
• Cut wasted rollouts: filter zero-gradient groups and easy problems; active sampling, which stabilizes batches.
• Result: training time fell substantially with similar performance.

• Why wasteful: GRPO advantage is a response's reward minus the group mean. If all answers are right or all wrong, every advantage is zero, so the expensive generation yields no gradient.
• Fix 1: drop such groups from training.
• Fix 2: filter offline, mainly removing easy problems.
• Fix 3: active sampling keeps collecting nonzero groups until the batch fills, instead of oversampling in advance.

• Matthew effect: RL improves problems roughly in proportion to initial ability, so easy problems gain most and hard ones barely improve.
• Why: fixed samples per prompt spend compute on easy prompts, and raising that number doesn't fix it.
• NGU: sample a few completions per prompt, skip all-correct prompts, train on mixed groups and retry all-wrong prompts. Async training shifts compute toward hard problems.

• Problem: RL needs just-right problems. Easy ones give no signal, and overly hard ones give consistently poor rewards.
• Environments: verifiable environments generate unlimited problems with programmatic verifiers.
• Difficulty: each keeps a sliding difficulty window, raised once the model is accurate enough.
• Effect: many environments keep problems appropriately hard, speed learning and transfer to unseen tasks.
• Limits: expert engineering, verifiable tasks only.

• Observation: token relationships look hierarchical: low hyperbolicity means tree-like structure, and very few tokens are very frequent.
• Problem: trees grow exponentially, but Euclidean space grows only polynomially, so leaves on different branches crowd together.
• Solution: hyperbolic space has constant negative curvature and exponentially growing room, so trees fit with low distortion.
• Cost: basic operations like addition must be redesigned, so models are harder to build.