# Weeks 7–8 (Generative, EM) — Notes Catalog & Rewrite Plan

**Status (2026-10-15): EXECUTED.** All five notes are written, compile clean (0 errors, 0 overfull
boxes), and are linked from the syllabus and HW4. The plan below is kept as the record of *why* each
note is shaped the way it is; §8 records what was actually built and where the execution departed from
the plan.
**Scope:** everything in `3_generative_models/` + the weeks 7–8 items in `5_features_dimensions/`,
plus the already-written `bayesian_ridge_regression.pdf`.
**Target:** 4 consolidated notes (2 basic carrying HW4, 2 advanced), in the course-wide `*_claude_*` system.

> Weeks 7–8 are the only weeks of the course without consolidated notes. Every other module has them
> (`REGRESSION`, `DT`, `BOOST`, `LOCAL_CLASSIFIERS`, `GD_LOGREG`, `LOGREG_LOSS`, `PERCEPTRON`, `PCA`,
> `KERNELS`, `SVM`, `OPTIMIZATION`, `NN_BASICS`, `NN_ADVANCED`, `VAE`, `GAN`).

---

## 1. Decisions (settled)

| # | Decision | Consequence |
|---|---|---|
| 1 | **KDE stays in HW4 P2** ("histogram or KDE") and gets taught in the lecture note | KDE is *already* taught — `LOCAL_CLASSIFIERS_claude_0908 §“Weighted/Soft k-NN: Kernel Density Estimation”` (Parzen windows, bandwidth, worked example, and the per-class Bayes rule). Basic-1 **cross-references it** and adds only the new wrinkle: HW1 used KDE over the whole feature vector, HW4 P2 uses it **per feature (1-D)**. |
| 2 | **Notation follows the course, not the board** | Adopt `π_k` (mixture weight) and `r_ik` (responsibility), matching the VAE note **and HW4 P3(C), which already uses exactly these symbols**. The board's `w_k` / `⟨Π_ik⟩` gets a bridge box. No VAE-note edit needed. Full standard in §4. |
| 3 | **HW4 P4's symbols realign** to the course standard | `K` = number of coins (= components, as everywhere else), `n` = flips per sequence, `M` = sequences. HW4 currently has `K`=flips, `n_components`=coins — the two most important symbols swapped. Cosmetic rename in HW4 text + starter + solution. |
| 4 | **Retire only after the content is safely inside a new note** | And the handwritten lectures + slide decks **stay linked regardless** — they are primary material, not redundancy. The retire list is duplicates and third-party redundancy only. |
| 5 | **`bayesian_ridge_regression` → cross-reference, don't re-derive** | Rename to house convention, link to `REGRESSION_ADVANCED §“A Bayesian (MAP) View of Ridge”` (week 1) and recap only the algorithm/result. |
| 6 | **Lecture plans: yes**, for both basic notes | One page each, the GAN/VAE format (100-min table, `xr-hyper` into the main note). Planning aid, not a script. |

---

## 2. House style the new notes must match

From `PCA_claude_1001`, `NN_BASICS/ADVANCED_claude_0929`, `VAE/GAN_claude_0929`:

- **Naming/location:** `TOPIC[_ADVANCED]_claude_MMDD.{tex,pdf}` in `3_generative_models/lecture_notes/<subfolder>/`,
  with `images_claude/` and a `build_*_figs.py` that regenerates every figure.
- **Preamble:** `extarticle` 12pt, `margin=0.5in`, `\normalsize`→13/16pt, TikZ; environments
  `remark`, `keypoint` (boxed), `pseudo` (algorithm box), `example`.
- **Title:** `<Topic>: <subtitle>` + `\large Lecture Notes Consolidated with Claude AI --- CS6140 Machine Learning`.
- **Arc:** motivating opener → narrated derivations → **worked numeric example usable as a unit test for the
  student's code** → algorithm box → **HW map + common bugs** → optional/forward-pointer section.
- **Length:** basic ≈ 350–620 `.tex` lines (14–20 pp); advanced ≈ 200–280 lines (10–14 pp).
- Advanced notes are labelled in the syllabus `(optional — …; not required for HWx)`.

---

## 3. Catalog

### 3.1 Tier 1 — the instructor's own material (absorb; **keep linked**)

| File | Len | What's in it | Verdict |
|---|---|---|---|
| `Lecture_060625.pdf` “L9: GDA” | 8 pp | “Fit a curve = fit a **density**”; 1-D→D-dim Gaussian, **row-vector** quadratic form; MLE via `∂LL/∂μ` and **`∂LL/∂Σ⁻¹`**; Bayes rule (`P(x)` “ignore”); GDA, shared vs per-class Σ | absorb + link |
| `Lecture_061125.pdf` “L10: Naive Bayes” | 13 pp | Poisson (buses/tweets) + Taylor normalizer check; multinomial; mixtures previewed (3-hump figure); NB with height/weight/shoe-size; **parametric vs histogram** per feature; log-space; bag-of-words; **Laplace / ε as prior strength** | absorb + link |
| `Lecture_GMM_061325.pdf` “L11+12: GMM” | 11 pp | **“Not-so-naive Bayes → belief networks”**; membership `Π_ik` as a 0/1 RV; **K-means → soft K-means** on-ramp; E-step *is* Bayes; the `^Π_ik` filter → “too hard” → ELL; M-step μ, Σ, w (Lagrangian, λ=N) | absorb + link |
| `Lecture_mixt_binom_poisson_062025.pdf` “L13” | 11 pp | **Mixture of binomials = HW4 P4** (generative recipe, `C(n,x)` cancels, `p_k=S_k/(nR_k)`); Poisson as the `n→∞` limit; mixture of Poissons; forward pointers (HMM, topic models, sampling, VAE) | absorb + link |
| `slides/3_generative_likelihoods.pdf` | 15 pp | The **10¹⁰⁰-cell impossibility argument**; the **three options** (independence→NB / parametric-or-mixture→GDA, EM / grouped factors→Bayes nets); MLE setup | absorb + link |
| `slides/3_GDA.pdf` | 12 pp | Gaussian fit by ML; three Σ contour pictures (I, diag, full); GDA with common Σ; “closed form, no GD”; equidistant-line picture | absorb + link |
| `slides/3_naive_bayes.pdf` | 15 pp | Independence factorization; parametric vs histogram; **constant feature → variance floor ε** (source of HW4 P2's regularizer); zero probability → Laplace; foreground/background interpolation | absorb + link |
| `bayes_net.pdf` | 10 pp | His own 2008 Bayes-net notes — almost entirely figures | **rebuild** → Advanced-2 |
| `slides/3_EM.pdf` | 100 pp | Course EM deck | keep linked |

### 3.2 Tier 2 — course-adapted (absorb; best ones stay linked)

| File | Len | What | Verdict |
|---|---|---|---|
| `mix_gauss/mix_gauss_w_mu_piik_sigma.pdf` | 12 pp | Week-8 anchor: all three M-step updates incl. the Lagrange step, E-step, algorithm box, a Jensen lower-bound section, K=2,3,4 examples | absorb + link |
| `mix_binomials/EM_Mixture_Binomial_Derivation.tex` | 87 ln | Mixture of K binomials = HW4 P4. Correct; Lagrange and `∂/∂p_k` steps stated, not shown | absorb (expand the terse steps) |
| `mix_binomials/EM_Mixture_Poisson_Derivation.tex` | — | Poisson limit + mixture-of-Poissons EM | absorb |
| `Gen_models_overview/Generative_Models_Summary.pdf` | 4 pp | Bullet catalog NB/GDA/EM/mixtures/HMM/topics/VAE + table | absorb as the module “map” |
| `HMM/HMM_Detailed_Overview.pdf` | 3 pp | Components, joint, 3 tasks, forward, Viterbi, Baum–Welch | absorb → Advanced-2 (backs HW4 P16) |
| `EM algorithm for coin flipping problem.pdf` | 1 p | Cheng Li's E/M formulas; shipped with HW4 as `EM_coins.pdf` | absorb (notation bridge) |
| `5_features…/LDA/LDA.tex` | 6 pp | Fisher LDA, full derivation → `θ* ∝ S_W⁻¹(μ₂−μ₁)`, multiclass | absorb → Advanced-1 (PCA note has a condensed version — cross-link) |
| `5_features…/feature_analysis/features_analysis.pdf` | 2 pp | **Missing values**: NB marginalizes them out since `∑ P(x¹|y)=1` | absorb (1 subsection of Basic-1) |
| `smoothing.pdf` | 17 pp | IR smoothing; Laplace + Lindstone are what HW4 needs, rest is IR-specific. ⚠ Laplace slide extracts as mojibake | absorb part + link |
| `bayesian_ridge_regression.tex` | 7 pp | Already written for HW4 P5 | keep (per decision 5) |

### 3.3 Tier 3 — keep as links

**EM theory/mixtures:** `UWEETR-2010-0002` (**best general-EM source** — the only one with both a monotonicity
proof and the tight-lower-bound picture) · `em_berkeley` (Baum–Welch) · `EM_cmu` (identifiability, pLSA) ·
`em_tutorial_bioinfo` (**the two-coin worked example — absorb the example**) · `tomasiEM` (EM vs Newton) ·
`em_hastie_tibshirani` · `EMslides_Sontag_{1,2}` · `cours_m_nadif_mm` · `em_details` · `Murphy_mixtureModels`

**Why HW4 P4 behaves as it does:** `Carriera-Perpinan` (identifiability of Bernoulli mixtures ⇒ overlapping
coins recover weakly) · `10.1.1.104.4104` (initialisation schemes ⇒ restarts)

**GDA/NB/probability:** `NBayesLogReg` (Mitchell — GNB→sigmoid + Ng–Jordan = all of HW4 P9) ·
`Andrew_Ng_generative` · `Murphy_gaussians` · `Murphy_NB` · `Murphy_bernoulli_multinom` (Beta/Dirichlet
conjugacy) · `em_naivebayes` (Collins) · `icml03-nb` · `Marginal and conditional…MVN` · `recap_probability` ·
`Aslam_prob_smoothing_MaxLike_notes` (⚠ scan, no text layer)

**Graphical models:** `slides-8-graphical_models_modified` · `Murphy_DirectedGraphModels` · `Murphy_GraphModel_Infer`

### 3.4 Tier 4 — retire **after** the new notes are written and verified (decision 4)

Four copies of the D'Souza GMM note (`mix_gauss.{tex,pdf}`, `mix_gauss_EM_dsouza.pdf` ×2,
`materials/mix_gauss*`) · three md5-identical pairs (`3_graphical_models_Bishop` ≡ `slides-8-graphical_models_modified`,
`materials/8-em` ≡ `slides/EM_Bottou`, `materials/EM` ≡ `materials/EM_Ajit_Singh`) · two Wikipedia dumps
(`Maximum_likelihood_Wikipedia`, `Multivariate normal…Wikipedia`) the syllabus links live ·
`EMTalk2` · `materials/EMnotes` · `materials/mixture_EM` (image-only, broken refs) ·
`Murphy-bayesianConcepts` (tangential).

---

## 4. Notation standard

Surveyed across `REGRESSION`, `GD_LOGREG`, `LOCAL_CLASSIFIERS`, `PCA`, `KERNELS`, `SVM`, `VAE`.
**The good news: the course standard and the instructor's board notation already agree on the data layer.**

| Role | Symbol | Source of truth |
|---|---|---|
| data matrix | `X`, **N × D** | REGRESSION, PCA, KERNELS all state it |
| datapoint, index | `x_i`, `i = 1..N` (**row** vector, 1×D) | dominant in every note (PCA uses `k` only because it needed `j` for directions) |
| feature, index | `x_i^j`, `j = 1..D` — **superscript** | REGRESSION, GD_LOGREG, LOCAL, and the board |
| quadratic form | `(x−μ) Σ⁻¹ (x−μ)ᵀ`; `Σ = (1/N)Σ_i (x_i−μ)ᵀ(x_i−μ)` | PCA note + both GDA lectures |
| label / classes | `y_i`; classes `c = 1..C` | PCA §LDA, LOCAL_CLASSIFIERS (board writes `ℓ = 1..L`) |
| **mixture component** | `k = 1..K` | universal |
| **mixture weight** | **`π_k`** | VAE note **and HW4 P3(C)** (board writes `w_k`) |
| **latent label** | `z_i ∈ {1..K}`; indicator `z_ik` | VAE note's `z` — flows straight into week 10 |
| **responsibility** | **`r_ik` = E[z_ik] = p(z_i=k \| x_i)** | VAE note **and HW4 P3(C)** (board writes `⟨Π_ik⟩`) |
| coins (HW4 P4) | `K` coins, `n` flips/sequence, `M` sequences | board; HW4 realigns to it (decision 3) |

Keep the board's pedagogy — the indicator is a random variable, the responsibility is its expectation —
but land on the course symbols: introduce `z_ik`, then `r_ik = E[z_ik]`. One **bridge box** per basic note:

> `w_k ≡ π_k` · `Π_ik ≡ z_ik` · `⟨Π_ik⟩ ≡ r_ik` · `ℓ ≡ c` · and in the coin problem the board's `n/M/K` are flips/sequences/coins.

---

## 5. The four notes

### 5.1 `GDA_NB_claude_MMDD` — “Generative Classifiers: Fitting Densities, GDA, and Naive Bayes”
**Basic · carries HW4 P1, P2, P7** · ≈16–20 pp

1. Fitting a curve = fitting a density; `p(x)` as “proportion of patients like this”
2. Why not the joint: the 10¹⁰⁰-cell argument; **the three options** (sets up the whole module)
3. The Gaussian: 1-D → D-dim, row-vector form, dimension bookkeeping, the three Σ contour pictures
4. MLE for `μ` and `Σ` (differentiating w.r.t. `Σ⁻¹`), with the matrix-derivative rules boxed
5. Bayes rule for classification; prior × class-conditional; why `p(x)` drops out
6. **GDA**: per-class Gaussians; **LDA vs QDA**; why the boundary is linear vs quadratic
7. Practical: rank-deficient Σ, regularization, **the sklearn `reg_param` crash HW4 hit**, standardization
8. **Naive Bayes**: the factorization; **what it assumes vs. what it says nothing about** → *HW4 P7*
9. The three per-feature likelihoods: Bernoulli · Gaussian · non-parametric (**histogram, and 1-D KDE —
   cross-ref `LOCAL_CLASSIFIERS §KDE`**); the constant-feature **variance floor**; *why a Gaussian on binary
   data is the wrong family* → *HW4 P7*
10. Log-space scoring; zero probabilities; **Laplace / ε-smoothing as prior strength**; text/bag-of-words NB
11. Missing values marginalize out
12. Scores → threshold `τ` → ROC/AUC (HW4 P2.2)
13. **Worked numeric example** (hand-checkable)
14. **HW4 map** (P1, P2, P7) + common bugs
15. Forward: Fisher LDA → Advanced-1; mixtures → Note 2

### 5.2 `EM_MIXTURES_claude_MMDD` — “Mixture Models and the EM Algorithm”
**Basic · carries HW4 P3 (incl. **P3(C) ELBO**), P4, P6** · ≈16–20 pp

1. Why mixtures: one bump isn't enough (the 3-hump figure)
2. The generative process; `z_i`, the indicator `z_ik`, and the two sum identities
3. **K-means as hard EM → soft K-means** (the on-ramp); distance → probability
4. GMM: the E-step **is** Bayes' rule; prior `π_k` vs posterior `r_ik`
5. The complete-data likelihood with the indicator exponent; why it's intractable; the expected log-likelihood
6. M-step ① `μ_k` ② `Σ_k` (via `Σ⁻¹`) ③ `π_k` (Lagrangian, λ=N)
7. **The ELBO**: `log p(x) ≥ E_q[log p(x|z)] − KL(q‖p(z))`; what plays `q(z|x)` and `p(z)` here → ***HW4 P3(C)***
8. **Monotonicity — stated and explained** (the likelihood must not decrease; what it means when it does)
   → ***HW4 P6*** (the *proof* lives in Advanced-2)
9. Algorithm box; initialization; convergence test; **numerics** (log-sum-exp, singular Σ, empty components)
10. **Worked numeric example** + what to expect on `2gaussian.txt` / `3gaussian.txt`
11. **Mixture of binomials = the coins problem** (HW4 P4): recipe, E-step (`C(n,x)` cancels), `p_k = S_k/(n R_k)`
12. Mixture of Poissons: same recipe, new likelihood
13. Label switching, identifiability, restarts — **why overlapping coins recover weakly**
14. **Notation bridge box** (§4)
15. **HW4 map** (P3, P4, P6, optional P11) + common bugs
16. Forward: HMM → Advanced-2; **VAE (HW5 P3)** → week 10

### 5.3 `GDA_NB_ADVANCED_claude_MMDD` — “Generative Classifiers: Advanced Topics”
**Advanced · optional · covers HW4 optional P8, P9, P10, P13–P15** · ≈10–12 pp

1. Decision-boundary geometry: LDA linear, QDA quadratic, derived
2. LDA ⟺ Fisher's discriminant ⟺ least squares on ±1 labels
3. NB as diagonal-covariance GDA; the binary/ternary discriminant → **P13, P14, P15**
4. Generative vs discriminative: GNB → sigmoid; Ng–Jordan → **P9**
5. Regularized / shrinkage discriminant analysis (where `reg_param` comes from)
6. Kernel Fisher Discriminant → **P8**
7. MLE vs MAP; Beta-Bernoulli, Dirichlet-multinomial; **“Laplace smoothing is a Dirichlet prior”**;
   predictive distribution → **P10**; cross-ref `bayesian_ridge_regression`
8. Further reading

### 5.4 `EM_ADVANCED_claude_MMDD` — “EM Theory and Latent-Variable Models”
**Advanced · covers HW4 optional P16 · prerequisite depth for HW5 P3** · ≈12–14 pp

1. Why EM works: Jensen's inequality and the ELBO, properly
2. **The monotonicity proof**; the KL decomposition; why the E-step makes the bound tight
3. EM as coordinate ascent on the ELBO — and how that maps onto the board's ELL route
4. Local optima, MAP-EM, regularized covariance
5. K-means as a limit; identifiability; model selection (BIC, held-out likelihood)
6. Missing data: EM's original motivation
7. “Not-so-naive Bayes” → Bayes nets → **HMM: forward/backward, Viterbi, Baum–Welch as EM** → **P16**
8. Topic models as mixtures with Dirichlet priors
9. Variational inference → **the VAE**: the EM ↔ VAE table, feeding HW5 P3(C)
10. Further reading

### 5.5 Fifth note (already written)
`bayesian_ridge_regression` → rename `BAYES_REGRESSION_claude_MMDD`, house preamble, cross-reference
week 1's MAP-view-of-Ridge instead of re-deriving (decision 5).

---

## 6. HW4 / HW5 accounting

### 6.1 HW4 — 310 required points, 7 required + 9 optional

| # | Problem | Pts | Covered by |
|---|---|---|---|
| 1 | GDA (LDA vs QDA), k-fold | 40 | Basic-1 §3–7 |
| 2 | Naive Bayes ×3 likelihoods, smoothing, ROC/τ | 50 | Basic-1 §8–12 (+ KDE cross-ref to `LOCAL_CLASSIFIERS`) |
| 3 | EM for a Gaussian mixture — (A,B) 50 + **(C) ELBO 10** | 60 | Basic-2 §1–10, **§7 for (C)** |
| 4 | EM for a mixture of Binomials (coins) | 50 | Basic-2 §11 |
| 5 | Bayesian linear regression (MAP=Ridge, posterior, evidence) | 60 | `bayesian_ridge_regression` (+ week-1 cross-ref) |
| 6 | **(THEORY)** may the EM log-likelihood decrease? | 25 | **Basic-2 §8** (statement + failure modes); proof in Advanced-2 |
| 7 | **(THEORY)** choosing the right likelihood family | 25 | **Basic-1 §8–9** |
| 8 | KFDA | opt | Advanced-1 §6 |
| 9 | NB ↔ Logistic Regression | opt | Advanced-1 §4 |
| 10 | Bayes math / double-headed coin | opt | Advanced-1 §7 |
| 11 | NB with a mixture of K Gaussians per feature | opt | Basic-2 §15 (map) + Advanced-1 |
| 12 | St.-Petersburg games | opt | — (self-contained probability) |
| 13–15 | DHS Ch.2 Pb.43/44/45 | opt | Advanced-1 §3 |
| 16 | HMM forward–backward (DHS 3.50) | opt | Advanced-2 §7 |

### 6.2 HW5 — 260 required points; what it needs from weeks 7–8

| # | Problem | Pts | Dependency on weeks 7–8 |
|---|---|---|---|
| 1 | Backprop from scratch, 8-3-8 autoencoder | 50 | none |
| 2 | Classifier network, Fashion-MNIST | 100 | none |
| 3 | **VAE (PyTorch)** | 50 | **Heavy.** P3(C) is written as a direct continuation of **HW4 P3(C)**: same ELBO, `q(z|x)` exact (responsibilities) vs learned (encoder). Needs `z`, `π_k`, `r_ik`, KL, ELBO to mean the same thing in both HWs |
| 4 | **GAN** | 10 | Back-references **HW4's EM** twice: “wrote down a likelihood and maximized it” vs implicit density; and “compare with HW4's EM fit of the same data” — **P4 reuses `2gaussian.txt`/`3gaussian.txt`** |
| 5 | (THEORY) dropout | 25 | none |
| 6 | (THEORY) learning rate from a loss curve | 25 | none |
| 7–8 | DCGAN · matrix backprop | opt | none |

**The bridge is already built and consistent** — HW4 P3(C) states the ELBO and names `r_ik`/`π_k`;
HW5 P3(C) picks up exactly there; HW5 P4 reuses HW4 P3's datasets and contrasts EM against GANs.
So the notes must deliver: the ELBO at HW4 level (Basic-2 §7), the vocabulary (`z`, `π_k`, `r_ik`, KL),
and the EM↔VAE table (Advanced-2 §9, mirroring the one already in the VAE note).

**Pedagogical flow across the two HWs:**
density → class-conditional density (GDA) → factorized density (NB) → *mixture* density fit by EM
(exact posterior) → **ELBO** → neural decoder with an approximate learned posterior (VAE) →
no density at all (GAN).

### 6.3 Cosmetic HW edits implied by this plan
- **HW4 P4** symbol rename: `K`=coins, `n`=flips, `M`=sequences (decision 3) — text, starter, solution.
- **HW4 P2**: say explicitly that the KDE option is the 1-D per-feature version of
  `LOCAL_CLASSIFIERS §KDE`, and link that note.
- **HW4 P3(C) / P6**: add the "see `EM_MIXTURES` §7/§8" pointers once the note exists.
- Add `Lecture notes: …` links at the top of HW4 P1–P4 (HW5 already does this for VAE/GAN).

*(No substantive HW change is required by this plan. §7 lists the one candidate.)*

### 6.4 Stale elsewhere
`claude_renovation_0826_code/HW_general_format.md` still says **HW4 = 250 points / 5 problems**;
it is now **310 / 7 required (+9 optional)**, with P3 at 60 and two 25-point theory problems.

---

## 7. Still open

1. **Substantive (needs debate):** HW4 P6 asks whether the EM log-likelihood may decrease, but weeks 7–8
   currently never state the monotonicity property — the board's route is “LL is too hard ⇒ take E[LL]”.
   Plan is to state and explain it in Basic-2 §8 (proof in Advanced-2). Alternative: keep Basic-2 purely
   procedural and make Advanced-2 required reading for P6.
2. Which of the two basic notes carries the **“three options” taxonomy** figure — it introduces NB, GDA *and*
   mixtures, so it could open either; currently Basic-1 §2.
3. Whether Advanced-2 should also absorb **sampling** (the board's “next: sampling” pointer), or leave it out.
4. Subfolder names: `GDA_NB/` and `EM/` under `lecture_notes/`, matching `GAN/`, `VAE/`, `HMM/`?

---

## 8. Execution record (2026-10-15)

### 8.1 What was built

| Note | Path | Pages | Carries |
|---|---|---|---|
| Basic-1 | `GDA_NB/GDA_NB_claude_1015.tex` | 14 | HW4 P1, P2, P7 |
| Basic-2 | `EM/EM_MIXTURES_claude_1015.tex` | 16 | HW4 P3 (incl. 3C), P4, P6 |
| Advanced-1 | `GDA_NB/GDA_NB_ADVANCED_claude_1015.tex` | 10 | HW4 opt. P8, P9, P10, P13–P15 |
| Advanced-2 | `EM/EM_ADVANCED_claude_1015.tex` | 11 | HW4 opt. P16; depth for HW5 P3 |
| Fifth | `BAYES_REGRESSION/BAYES_REGRESSION_claude_1015.tex` | 7 | HW4 P5 |

Lecture plans: `GDA_NB/GDA_NB_claude_lecturePlan.tex`, `EM/EM_MIXTURES_claude_lecturePlan.tex`
(3 pp each, `xr-hyper` cross-references all resolve).

Figure scripts (every figure regenerable, no checked-in-only images):
`GDA_NB/build_gda_nb_figs.py` (3 figures), `GDA_NB/build_gda_nb_adv_figs.py` (2),
`EM/build_em_figs.py` (4), `EM/build_em_adv_figs.py` (2). All use real data where real data makes the
point — Spambase, `2gaussian.txt`, `3gaussian.txt` — rather than illustrative synthetics.

### 8.2 Resolutions to §7's open items

1. **Monotonicity (the one substantive candidate).** Resolved as planned, no HW change needed:
   stated with its failure modes in `EM_MIXTURES` §8, proved in `EM_ADVANCED` §3. `EM_MIXTURES` is
   therefore sufficient for HW4 P6 on its own; `EM_ADVANCED` stays optional.
2. **The "three options" taxonomy** opens Basic-1 (`GDA_NB` §2) as planned. Basic-2 §1 and §16 both
   point back at it, and the Basic-1 lecture plan tells the instructor to leave it on the board for
   both weeks.
3. **Sampling** was *not* absorbed into Advanced-2 — it has no HW4/HW5 hook, and Advanced-2 was
   already carrying Jensen, the proofs, HMMs, topic models and the VAE bridge. Left for a future note.
4. **Subfolders** `GDA_NB/`, `EM/`, `BAYES_REGRESSION/` under `lecture_notes/`, each with
   `images_claude/` — matching `GAN/`, `VAE/`.

### 8.3 Departures from the plan, and one correction

- **HW4 P4 symbols (decision 3) landed as `K` coins / `n` flips / `N` sequences**, not `M` sequences.
  Using `N` for the number of sequences makes each sequence a datapoint `i=1..N`, matching every other
  note in the course; `M` would have been a third convention for "number of datapoints".
  Also `p`/`r` → `p_1`/`p_2`, which additionally removed a real collision: `r` was the second coin's
  head-probability *and* the responsibility matrix in the same function.
- **A claim in Basic-2 was wrong and got corrected by measurement.** The first draft said a single
  random EM start merges two humps of the three-hump example "about a third of the time". Measured
  over 300 restarts (`EM/build_em_adv_figs.py`): **11%**. Text fixed, and the measurement is now the
  `local_optima.png` figure in `EM_ADVANCED` §5. The HW's own `2gaussian`/`3gaussian` files turned out
  to be too well separated to show local optima at all — every restart finds the same optimum — which
  is itself worth saying in lecture.
- **The starter notebook's problem list was stale** (P3 shown as 50 points, no mention of the theory
  problems P6/P7). Updated along with the symbol rename and regenerated.

### 8.4 Link-and-reference pass (2026-10-16, instructor review)

Four style rules applied across the notes and the syllabus:

1. **Older links: remove if fully absorbed, otherwise move to the last column.** Actioned —
   see §8.4b below for what was removed vs. moved.
2. **Handwritten `Lecture ..` notes stay in the lecture column.** They now sit directly under the
   consolidated notes in both weeks, ahead of the slide decks.
3. **Optional notes are annotated just `(optional)`** — no content list. Applied to all seven
   advanced/optional note links in the syllabus, not only weeks 7–8, so the file is internally
   consistent (`REGRESSION_ADVANCED` and `NN_ADVANCED` also had content lists).
4. **No bare `supports HW-x Problem-y` tags**, in the notes or in the syllabus links. Removed
   everywhere. What was *kept* is the opposite case the instructor called out: references that
   address a specific issue — the per-note **HW Map sections** (the `BAYES_REGRESSION` §8 pattern),
   and inline sentences that tell the student what to do, e.g. `EM_MIXTURES` §7's "identify
   $q(z|x)$ and $p(z)$… the answer is…", §13's "try at least one well-separated setting and one
   overlapping setting", and `GDA_NB_ADVANCED` §4's "derive Eq. 16 and verify numerically".
   Zero `supports HW` strings remain in `schedulen_mod.html`.

Page counts after the pass: 14 / 16 / 9 / 11 / 7 (GDA_NB_ADVANCED lost a page to the shorter
header). All five still compile with 0 errors and 0 overfull boxes.

**Week 7's slide decks moved to the last column (2026-10-16).** Read all three
(`3_generative_likelihoods.pdf`, `3_GDA.pdf`, `3_naive_bayes.pdf`, 15/12/15 pp) before moving them.
Every technical point in them is inside `GDA_NB` at greater depth and in the course's notation: the
$10^{100}$ argument, Bayes' rule, the independence-vs-restrict-the-shape fork, the Gaussian fit by
maximum likelihood, and the per-feature model-vs-non-parametric choice. They now sit inline with the
Wikipedia-style links, as `Slides: Generative Intro | GDA | Naive Bayes`.

**One real gap the decks exposed, now closed.** The Naive Bayes deck's last slides cover
*interpolation* smoothing --- $\hat p = \lambda P + (1-\lambda)Q$ against a background estimate
pooled over all experiments --- which is a genuinely different idea from the additive/Laplace
smoothing `GDA_NB` §10 had. Added as a new subsection, "A second kind of smoothing: back off to a
background estimate" (Eq. 20), framing additive smoothing as the special case where the background is
uniform, and naming Jelinek--Mercer. Note still 14 pp, still clean.

**Not touched:** the HW4 page's own "All lecture notes for this homework" index, which maps problems
to notes. That is the HW telling students where to read — the direction the instructor asked for —
not a note advertising which problem it serves. Also not touched: the internals of the Week 9/10/13
notes (`NN_BASICS`, `NN_ADVANCED`, `VAE`, `GAN`), which still carry their own "Supports HW5 …"
headers; only their *syllabus links* were restyled. Worth a sweep if those notes are revisited.

### 8.4c Covariance vs. summation sigma (2026-10-16)

The instructor's board convention is that a covariance $\Sigma$ carries small vertical tips at the
ends of its arms and a summation $\sum$ does not. **LaTeX already does this** --- `\Sigma` (the Greek
letter) and `\sum` (the big operator) are different glyphs, and `\Sigma` has exactly those terminals.
The problem was never shape, it was *weight*: at body size both are triangular and the eye does not
separate them inside a dense formula.

Rendered a comparison sheet of seven treatments at the notes' real 13/16pt
(`scratchpad/sigtest/sig.tex`) and picked **bold upright**, `\boldsymbol{\Sigma}`:

- it keeps the vertical tips and in fact makes them more prominent, so the board convention still reads;
- bold-means-matrix is the standard typographic convention and is already how this course writes
  vectors (`\mathbf{x}` via `\bx`), so this makes the notes *more* conventional, not less;
- rejected: `\mathit{\Sigma}` (slanted) is lighter-weight and reads *closer* to `\sum`;
  `\mathsf{\Sigma}` (sans) loses the tips, which is backwards.

Implemented as a macro so it is a one-line change if a different variant is ever preferred:

```latex
\newcommand{\Sig}{\boldsymbol{\Sigma}}   % covariance matrix
```

in all seven preambles (5 notes + 2 lecture plans), with 213 occurrences of `\Sigma` rewritten to
`\Sig`. Each file now contains exactly one literal `\Sigma` --- the macro definition. `\boldsymbol`
needs no new package (`amsmath` provides it, and `BAYES_REGRESSION` already used it).

Also: `GDA_NB` §3 gained a short "Two different sigmas" remark stating the convention, and
`shrinkage.png`'s axis label now uses `\mathbf{\Sigma}` so the figures match (matplotlib's mathtext
supports `\mathbf` but not `\boldsymbol`). All seven files recompile at 0 errors / 0 overfull, same
page counts.

**Not touched:** `VAE_claude_0929.tex` (4 `\Sigma`) and `GAN_claude_0929.tex` (1) in Week 10, same
reason as the HW-reference sweep --- different batch. Worth doing if those are revisited, since
HW5 P3 reads straight on from HW4 P3.

### 8.4d Latent-variable notation: three symbols down to two (2026-10-16)

Instructor review: the basic note carried `z_i` (categorical), `z_ik` (indicator) and `r_ik`
(responsibility) — one symbol too many, and `r_ik` in particular "costs a lot pedagogically for little
benefit."

**`z_i` cut outright.** The categorical $z_i\in\{1..K\}$ earned nothing the indicator didn't; the
die-roll recipe reads fine as "the die comes up $k$; set $z_{ik}=1$". Bayes' rule and the joint are now
written $p(z_{ik}{=}1\mid\bx_i,\theta)$. A bare $z$ survives for "all of point $i$'s indicators at
once" and as the latent variable in the ELBO, which is what HW4 P3(C) and the VAE need.

**`r_ik` → `⟨z_ik⟩`.** Candidates were rendered on the heaviest formulas before deciding
(`scratchpad/ztest/z.tex`):

- **`⟨z_ik⟩` chosen.** The decisive argument is that it is *already the board's notation* — §14's
  bridge table records the board as $\langle\Pi_{ik}\rangle$, angle brackets around the indicator. So
  this adopts the lecture convention rather than inventing one, and the bridge table shrinks. It also
  makes the key step *visible in the symbols*: Eq. 8's complete-data log-likelihood has $z_{ik}$ and
  Eq. 9's ELL has $\langle z_{ik}\rangle$, so "replace the indicator by its expectation" no longer
  needs to be remembered as a definition. Bonus: the hard-vs-soft contrast in
  `kmeans_vs_em.png` is now notational too ($z_{ik}\in\{0,1\}$ vs $\langle z_{i1}\rangle+\langle z_{i2}\rangle=1$).
- **`z̄_ik` rejected:** a bar conventionally means *sample mean over samples*, but this is an
  expectation over the posterior of $z$ for one point — semantically wrong, and it nearly vanishes
  inside the ELBO fraction.
- **`[z_ik]` rejected.** The instructor rightly pushed back that the Iverson-bracket clash is remote
  for an ML audience. The decisive objection is local instead: square brackets already do heavy
  grouping duty in the same formulas ($\mathbb{E}[\cdot]$, $[\log\pi_k+\log\N(\cdot)]$, $[0,1]$),
  so the ELL would read $\sum\sum[z_{ik}][\log\pi_k+\dots]$. Angle brackets collide with nothing —
  $\langle\cdot\rangle$ appears nowhere else in these notes.
- **Curly rejected:** `{z_ik}` is set notation, and the note does discuss collections of $z$'s.

Implemented as `\newcommand{\ez}[1]{\langle z_{#1}\rangle}` so the bracket style is one line to
change. 74 occurrences rewritten across `EM_MIXTURES` (66), `EM_ADVANCED` (7) and the EM lecture plan
(6). `EM_MIXTURES` is now 17 pp (was 16) — the extra half-page is a fuller treatment of "the posterior
probability *is* the expectation for a 0/1 variable", which is the whole justification for the
notation.

**Three riders kept deliberately:**
1. The **word** "responsibility" stays throughout (17 uses in the basic note) — it is the universal
   term and what students will search for.
2. `r_ik` is named **once**, in the bridge table, alongside Bishop's $\gamma(z_{nk})$, so outside
   reading is unblocked.
3. **Code keeps `r`** as the array name; short names are right in code. The two TODO comments that
   said "responsibilities r_ik" now say "responsibilities (the note's <z_ik>)".

**Downstream:** one HW4 sentence (P3(C)'s "the responsibilities r_ik" → "&lang;z_ik&rang;"); HW5 P3(C)
needed nothing, since it says only "q(z|x) (the responsibilities)" with no symbol.
`kmeans_vs_em.png` regenerated; both notebooks regenerated and the solution re-executed (0 error
cells).

### 8.4e EM_MIXTURES revisions (2026-10-16, second pass)

1. **Opening blurb cut.** The "Week 8 — continues from `GDA_NB`…" paragraph is gone; the note now
   opens on §1. (The other notes keep theirs: those carry *prerequisite* information — "assumes
   maximum likelihood from Week 1" — rather than pure cross-referencing.)
2. **`π_k` vs `⟨z_ik⟩` stated in plain English**, right under the boxed E-step where both symbols sit
   in the same formula: `π_k` is the probability source *k* produced a datapoint **in general** (a
   random point you have not looked at); `⟨z_ik⟩` is the probability source *k* produced **this
   particular, observed** point. One number per source vs. one per source *per point*; looking at the
   point is what turns the first into the second. The sharper keypoint later in §4 stays.
3. **"Membership" recorded as a synonym** for responsibility — it is the instructor's own word for it
   and students hear it in lecture. Named at first use and in the §14 bridge table.
4. **§6 derivations filled in**, roughly doubling each of the three steps:
   - **means:** the matrix-calculus identity ∂/∂μ[(x−μ)A(x−μ)ᵀ] = −2(x−μ)A stated explicitly (with
     the scalar analogue), then the right-multiply-by-Σ_k step shown rather than asserted.
   - **covariances:** the substitution Λ_k = Σ_k⁻¹ written out, the two identities
     ∂log|Λ|/∂Λ = Λ⁻¹ and ∂(aΛaᵀ)/∂Λ = aᵀa quoted, the stationarity condition derived, and an honesty
     note that differentiating as if all D² entries were free puts a factor of 2 on the off-diagonals
     that cancels on both sides.
   - **mixing weights:** the inner sum collapsed to Σ_k N_k log π_k first; *why* the constraint is
     load-bearing (unconstrained, every π_k → ∞); the ∂L/∂λ = 0 equation named as the one that closes
     the K+1 unknowns; and λ = N derived rather than asserted.

Still 17 pp, 0 errors, 0 overfull.

### 8.4f GDA_NB restructured: 15 sections to 8 (2026-10-16)

Instructor: too many top-level sections. Merged as agreed, 15 -> 8, same 14 pp:

| # | Section | absorbed |
|---|---|---|
| 1 | Densities, Classifiers, and Bayes' Rule | old 1 + 2 + 5 |
| 2 | The Gaussian in *D* Dimensions | old 3 + 4 |
| 3 | Gaussian Discriminant Analysis: LDA and QDA | old 6 + 7 |
| 4 | Naive Bayes | old 8 + 9 + 10 + 11 |
| 5 | Turning a Score into a Decision: Thresholds and ROC | old 12 (retitled) |
| 6 | Worked Example | old 13 |
| 7 | HW4 Map and Implementation Notes | old 14 |
| 8 | Where This Goes Next | old 15 |

**§12 and §13 kept separate**, each for a different reason. §12 (ROC) because of *scoping*: it is not a
Naive Bayes topic — it applies equally to GDA's log-odds score — and filing it under NB would plant
that misconception. Retitled to signal it covers either model's score. §13 (worked example) because of
*function*: topically it is GDA, but its job is to be a findable reference artifact ("the numbers I
check my code against"), which a top-level heading serves and burial at the end of a four-page section
does not.

**Two paragraphs added** where the merge created stacked headings (a `\section` immediately followed by
a `\subsection`): an orienting paragraph under §1, and under §2 a bridge that doubles as the
transition the reorder needed — old §5 (density → classifier) now precedes the Gaussian, so the recipe
arrives as a schema before its first instance, and §2 opens by saying so.

**Reference handling.** The lecture plan and all 50 of the note's internal references use *labels*, so
they re-resolved automatically; the plan needed no edit and now reads §1.1, §1.2, §2.3, §3.2, §4.2 etc.
Only hard-coded numbers needed hand-fixing: four in sibling notes (`GDA_NB` §1 → §1.1, §2 → §1.2 (x2),
§7 → §3.2, §10 → §4.3, and "§ Naive Bayes" → §4), two section ranges in `HW4_26F.html`
(§3–7 → §2–3, §13 → §6, §8–12 → §4–5), and one in `build_hw4_starter.py` (§8, §9 → §4).

**One bug caught by checking the `.aux`.** The transform applied heading-demotion *after* retitling, so
the two sections being turned into subsections got demoted a second time, to `\paragraph` — which
silently flattened two levels and left `sec:families` and `sec:smoothing` both resolving to "4.1". Found
it by reading the generated label numbers rather than trusting the structure grep (whose pattern had
been narrow enough to miss the missing headings). Fixed; all 16 labels now resolve uniquely.
"Naive Bayes on text" was promoted to its own subsection at the same time — it is not a smoothing topic.

### 8.4g Opening blurbs (2026-10-16)

Instructor: the opening italic paragraph should carry **prerequisites only** — no week label, no
"the companion note X continues from here", no "deeper material is in Y". Applied across the set:

- `GDA_NB`: now just *"Before starting, make sure to recap maximum likelihood and the Gaussian noise
  model from the **Regression** notes, and kernel density estimation from the **Local Classification**
  note."*
- `BAYES_REGRESSION`: "Week 8 supplement" dropped and the prerequisite reframed in the same recap
  voice. Kept the two things that are instructions about *this* note rather than boilerplate — that
  the ridge-as-MAP derivation is recapped and not repeated, and the row-vector notation convention.
- `GDA_NB_ADVANCED`, `EM_ADVANCED`: "(Week 7)" / "(Week 8)" parentheticals dropped. Their blurbs are
  kept otherwise: "optional companion to X, which answers these questions" is an identity statement,
  not the cross-reference boilerplate being removed.
- `EM_MIXTURES`: no blurb at all (cut earlier at the instructor's request).

The lecture plans keep their "supports the Week 7 lecture" openers — a lecture plan is literally about
scheduling a lecture, so the week label is the point there.

### 8.4h ELBO staircase widened (2026-10-16)

`EM_MIXTURES` §7's staircase diagram was ~11 cm on a 19.0 cm textwidth. Wrapped in
`\resizebox{0.95\linewidth}{!}{...}`, so it now spans ~18 cm. Uniform scaling means the in-figure
labels grew with it (they land at roughly body size, which suits a figure this wide) and the geometry
of the leader lines and the broken dotted verticals is preserved exactly.

Cost: the note went 17 -> 18 pp, because uniform scaling grows the height too (~5.3 cm -> ~8.7 cm).
That is inherent to enlarging it — scaling coordinates only (`scale=1.78`) grows the height
identically and merely leaves the labels small. Drop to `0.85\linewidth` if the page matters more
than the size. Section numbering and all labels unchanged.

### 8.4i EM_MIXTURES restructured: 16 sections to 10 (2026-10-16)

| # | Section | was |
|---|---|---|
| 1 | Mixture Models | 1+2 |
| 2 | Chicken and Egg: *k*-Means as EM | 3, rewritten |
| 3 | The E Step: Bayes' Rule | 4 |
| 4 | The M Step | 5+6 |
| 5 | The Algorithm, Start to Finish | 9, moved up |
| 6 | Why It Works: the ELBO and Monotonicity | 7+8 |
| 7 | Mixtures of Other Things | 11+12 + new |
| 8 | What Goes Wrong: Local Optima, Label Switching, Identifiability | 13 |
| 9 | HW4: Worked Numbers, Map, and Common Bugs | 10+15 |
| 10 | Notation, and Where This Goes Next | 14+16 |

**§2 rewritten around the chicken-and-egg**, with the two steps named **E** and **M** from the first
paragraph instead of "assign"/"update" — the instructor's idea and better than the original. Hard
*k*-means is then presented as *"you have already run this algorithm"*, with a remark conceding that its
"expectation" is degenerate (a 0 or a 1) and that this degeneracy is precisely what makes it the easy
case. Soft *k*-means follows as "softening the E step".

**Order now runs practice-before-theory**: model → chicken/egg → E → M → algorithm → why it works. The
ELBO/monotonicity material moved *after* the algorithm box, which matches the order students actually
work in, and the transition sentence ("the derivation above is a recipe; it does not yet say why")
reads better there than it did before.

**§9 merges the worked numbers with the HW map, per the instructor.** My objection (that the worked
example should stay standalone to be findable, as in `GDA_NB`) did not survive scrutiny: `GDA_NB`'s
worked example is a generic eight-point hand calculation, whereas this one already quotes the HW's own
data files. "What do I check my code against" and "what does the HW want" are the same question, so
they belong in the same place. §5 ends with a pointer to it for anyone mid-implementation.

**New content in §7.3, "Choosing a Component Family"** — the commentary the instructor asked for and
the note genuinely lacked: a six-row table (real vector / real scalar / heads-out-of-n / yes-no /
unbounded count / word counts) giving the component, its parameters and its M-step update; a paragraph
on what changes between families (only the M step) and what does not (the E step, and the cancellation
of every $k$-free factor); and a remark on the three cases where mixtures actually earn their keep,
with the honest counterweight that a discriminative model usually wins if the real goal is a boundary.

**Six orienting paragraphs** added where merging left a section heading stacked on a subsection heading.

20 pp (was 18: +1 for the new family material, +1 for the §2 rewrite and the intros). 0 errors,
0 overfull. **All 23 labels verified unique against the regenerated `.aux`**, and the 16 external
citations updated from it rather than by hand: 4 in `EM_ADVANCED` (§7→§6.1, §11--12→§7, §1→§1.1,
§9→§5), 7 across two HW4 blocks, 5 in the two notebook builders (§8→§6.2, §11→§7.1, §13→§8 ×3). Both
notebooks regenerated, solution re-executed with 0 error cells.

### 8.4j Advanced-notes pass (2026-10-16)

A self-directed pass over both advanced notes to bring them in line with the revised basic notes. Two
real bugs, two consistency fixes, one restructure.

**Bugs found.**
1. **The HMM trellis in `EM_ADVANCED` drew two overlapping nodes.** The `\foreach` ran `1..4` and
   created `z4`/`o4` labelled $z_4$/$o_4$, then separate `z_T`/`o_T` nodes were placed at the *same*
   coordinates — so the last column rendered as "$z_T$" superimposed on "$z_4$". It had never been
   looked at: this figure was written and compiled but never visually checked. Rewritten so the loop
   covers $t=1,2,3$ only, each emission arrow is labelled $b_t(o_t)$, and the parameter legend sits on
   one line below.
2. **A hard-coded `\S2` inside the Jensen figure** pointed at "the KL of §2" — correct before the
   restructure, wrong after it. Now `\S\ref{sec:kldecomp}`, resolving to §1.2. This is exactly the
   failure mode the GDA_NB pass warned about, found by grepping for `\S[0-9]` outside `\ref{}`.

**Consistency.** `GDA_NB_ADVANCED`'s two figures were at `0.84\linewidth` while every other figure in
the set is at `0.98` (bar `roc_tau`, a square plot at `0.74`); raised to `0.98`. And `E-step`/`M-step`
hyphenation was mixed — `EM_MIXTURES` had 39 hyphenated and 10 not, after the §2 rewrite introduced the
unhyphenated form. Normalized to the noun form "E step"/"M step", keeping the hyphen only where the
term modifies a following noun ("M-step update", "E-step quantities"), which is the correct English
distinction: 76 instances changed, 10 correctly left hyphenated.

**`EM_ADVANCED` restructured 11 sections to 7**, matching the basic notes' granularity:

| # | Section | was |
|---|---|---|
| 1 | Why EM Works | 1+2+3+4 |
| 2 | Local Optima, MAP-EM, and Regularized Covariance | 5 |
| 3 | *k*-Means as a Limit, and Choosing *K* | 6 |
| 4 | EM's Original Problem: Missing Data | 7 |
| 5 | Giving the Latent Variable Structure: Bayes Nets and HMMs | 8 |
| 6 | When the Posterior Is Intractable | 9+10 |
| 7 | Further Reading | 11 |

Jensen, the KL decomposition, the monotonicity theorem and the coordinate-ascent reframing are one
argument, so they became §1.1–1.4 under one heading — mirroring the basic note's single "Why It Works".
Topic models merged with variational inference/the VAE under "When the Posterior Is Intractable",
because the topic-model subsection's actual job is to be the bridge from an exact E step to an
intractable one. Five external citations updated from the regenerated `.aux` (`EM_ADVANCED` §5→§2,
§6→§3 ×3, and HW4's "EM Theory §1--3" → "§1").

`GDA_NB_ADVANCED`'s structure was left alone: 8 flat sections over 10 pp is already the basic notes'
granularity, and its sections are genuinely distinct results rather than facets of one argument.
Style density across all five notes is consistent at ~1.1 keypoints-plus-remarks per page.

### 8.4k Weeks 9--10 consistency pass (2026-10-16)

Checked `NN_BASICS`, `NN_ADVANCED`, `VAE` and `GAN` against the revised Week 7--8 notation.

**NN notes: zero overlap, zero changes.** Neither contains a single `\Sigma`, `r_ik`, `E-step`, `ELBO`
or `KL`. Nothing to reconcile, as expected.

**VAE: flow already consistent, notation was not.** The ELBO treatment needed no rework at all --- its
identity ($\log p(x) = \mathrm{ELBO} + \KL(q\|\text{posterior})$), its reconstruction-minus-KL split,
and its "the E step sets $q$ to the posterior, so the gap vanishes" are exactly what `EM_MIXTURES` §6.1
and `EM_ADVANCED` §1.2 now say. Four notational fixes:

1. **`r_ik` -> `⟨z_ik⟩`** (3 places, including the EM↔VAE table and the line quoting HW4 Problem 3(C)).
   This was the one that actually mattered: HW4 P3(C) had already been changed to `⟨z_ik⟩`, so the VAE
   note was contradicting the homework it is the forward reference for.
2. **Bold covariance** --- 5 `\Sigma` in `VAE`, 1 in `GAN`, now `\Sig` via the same macro.
3. **Hyphenation** --- 13 `E-step`/`M-step` in `VAE` normalized to the noun form.
4. **Cross-references added.** The VAE's one-paragraph GMM recap now names `EM_MIXTURES` as the full
   treatment, its "This is EM" subsection points at the staircase picture in `EM_MIXTURES` §6.1, and a
   sentence glosses why the brackets are there (expected indicator; most texts write $r_{ik}$) together
   with the $\pi_k$-vs-$\ez{ik}$ "in general / for this point" distinction, so the reader meets the
   same gloss in both weeks.

**The two EM↔VAE tables were diffed** (`EM_ADVANCED` §6.2 and `VAE` §9) --- same seven rows, no
contradictions: prior learned vs fixed, decoder Gaussian vs network, posterior exact vs learned, gap
$0$ vs $>0$, same ELBO, alternation vs joint SGD. Having it in both places is deliberate: one is the
Week 8 forward pointer, the other the Week 10 recap.

**GAN: one `\Sigma`, no other overlap.** Its many references to HW4's EM are substantive comparisons
(likelihood fit vs implicit density, monotone vs oscillating dynamics), not boilerplate --- left alone.

All four notes plus both lecture plans recompile at 0 errors / 0 overfull; page counts unchanged
(16 / 8 / 13 / 10).

### 8.4l Week 10 column and the VAE/GAN story (2026-10-17)

**Two links removed from Week 10's last column.** (i) The Week 8 EM-notes pointer I had added there —
the right place for that cross-reference is inside the VAE and GAN notes, which now carry it (the VAE's
GMM recap names `EM_MIXTURES`, its "This is EM" subsection points at the staircase in §6.1, and the GAN
note's new opening names both). Removing it also retired a stale citation: that line still said
`EM_MIXTURES §7` and `EM_ADVANCED §10`, both wrong after the restructures. (ii) The "Higher-level
overviews: VAE | GAN" links. Read both PDFs first (`VAE_Detailed_Overview`, 6 pp;
`GAN_Detailed_Overview`, 5 pp): they are condensed earlier drafts of the same notes, strictly subsumed,
with the same title block. Files left on disk, nothing links to them.

**Opening stories added to both notes**, since the overview's job — easing a reader in — belongs in the
note itself. Each now opens with a prerequisite line in the house format, then three short paragraphs
that set the problem before any machinery:

- **VAE:** *you can compress and you can generate, but not with the same object* — the autoencoder needs
  an input so it cannot invent; the mixture invents freely but its generator is three Gaussians. One
  change (a distribution over codes, held near a prior) gives you both, and the price is a posterior you
  can no longer compute. Closes with the arc of the note.
- **GAN:** *every generative model so far maximized a likelihood* — exactly (mixture) or approximately
  (VAE), but always with a formula for $p(x)$. A GAN drops that commitment, keeps only the sampler, and
  replaces "make the data likely" with "can anything tell these apart?". Names the cost up front: no
  likelihood, no monotone curve, new failure modes.

**A "story in N beats" page added to each lecture plan**, after the 100-minute table — the instructor's
narrative thread rather than a script. One paragraph per stage, each saying what to put on the board and
why that beat exists, ending with a single line to hold on to (*"the VAE is EM with a learned E step"*;
*"a GAN replaces a likelihood with an opponent"*). Both plans go 1 pp → 2 pp.

Notes: 13 pp (VAE, unchanged), 10 → 11 pp (GAN). All compile at 0 errors / 0 overfull, no undefined
references.

### 8.4m HW4 solution notebook: P6 and P7 answers added (2026-10-18)

Instructor asked for complete worked answers to the two theory problems, with the observation that P6
probably needs code. P1--P5 untouched; 13 new cells appended (45 -> 58), notebook re-executed with
0 error cells.

**P6 is code-backed, and writing it corrected the lecture note.** The note's bug list claimed
"updating $\Sigma_k$ with the old $\mu_k$ ... is the classic version of this bug". Tested it before
building the demo around it: **zero decreases**, across both data files and six random starts each.
The claim was wrong. The reason is worth more than the original bullet: $\Sigma_k$ about the old mean
equals the correct one plus $(\mu^{old}-\mu)^T(\mu^{old}-\mu)$, so it is merely *inflated* --- still a
valid covariance, still an increase in $Q$, hence a legal **generalized-EM** step. Monotonicity
survives; it just converges slower and worse, and the plot never warns you. `EM_MIXTURES` §6.2's bullet
has been replaced accordingly, and the "what does *not* break it" case is now stated explicitly.

The demo instead uses **hard assignment in the E step** (round the responsibilities, keep reporting the
soft log-likelihood), which dips reliably --- 6 decreases, worst step $-0.82$. Three-panel figure:
full scale (both curves look fine --- the point), zoomed on the dip, and per-iteration step size
against a zero line. The first panel earns its place: at full scale a $-0.8$ dip in a range of $3000$
is invisible, which is the argument for testing monotonicity numerically rather than by eye.

**P7's measurements overturned the answer I had drafted.** The intended story was "wrong family, worse
numbers". Measured on Spambase: under presence/absence encoding the Gaussian is *marginally better*
than the Bernoulli on both accuracy and AUC; under mean-thresholding it loses 5 points of accuracy but
its **AUC is unchanged**. So the section now tells the true story --- a wrong family damages the
*threshold*, not the *ranking*, which is the same lesson as "Naive Bayes survives dependent features"
--- and rests the case against the Gaussian on principle rather than on accuracy: 3 features per class
where the density is literally undefined without an arbitrary floor, ~26% of fitted mass on negative
values at $p=0.3$ (computed), two parameters that are not identifiable as a pair, and uncalibrated
scores. Closing line makes the caution explicit: that it ranks well is a warning about using accuracy
as evidence of a correct model, not a defence of the Gaussian.

**Starter notebook:** one factual correction only. Its header said Problems 6--7 are "theory --- no
code", then immediately asked for a log-likelihood plot. Now reads "written answers; Problem 6 also
wants the log-likelihood plot from your Problem 3 run". No P6/P7 cells added to the starter --- say so
if you want scaffolding there.

### 8.4n P6/P7 scaffolded into the starter (2026-10-18)

Agreed plan, trimmed on the instructor's instruction from six code TODOs to **four**, with the written
side collapsed from three prompts per problem to **one answer cell with three labelled sub-parts** ---
which also matches the HW text, since both problems are posed there as a single question with clauses.
Estimated ~1 h per problem. Cut as plumbing: counting decreases (P6) and calling the two NB fitters
(P7); both are now given.

| | TODO (student fills) | Insight |
|---|---|---|
| P6 | hard-assign the responsibilities | rounding = running *k*-means while plotting EM's objective; dips reliably |
| P6 | centre the covariance on the *previous* means | predict-then-test: this one does **not** dip |
| P7 | mass the fitted Gaussian puts outside $\{0,1\}$ | the density describes impossible outcomes (~32%) |
| P7 | count zero-within-class-variance features | the Gaussian is undefined on 3 features per class |

Both sections are **theory-first**: the question, then the experiments, then an empty `### Your answer`
cell. P6 opens with a blockquote naming its dependency on Problem 3 (`ll_2g`, `ll_3g`, `X_2g`), as
asked.

**The stale-mean experiment corrected the lecture note a second time.** Last pass established it stays
monotone; this pass measured what it *costs*, because the solution now prints the comparison. Answer:
**nothing.** Final log-likelihood identical to the correct run to ten decimal places across both data
files and three seeds, iteration counts within one. The claim I had written --- "converges slower and
to a worse optimum" --- was wrong. The reason is better than the claim: at the fixed point
$\mu^{old}_k=\mu_k$, so the stale and correct updates coincide in the limit and the bug perturbs only
the transient. Both `EM_MIXTURES` §6.2 and the notebook's answer (c) now say so, and the pedagogical
point sharpens: here is genuinely wrong code with a clean curve **and** the right answer, so a monotone
plot is a *necessary* condition for correctness and never a sufficient one.

Solution 58 -> 60 cells (0 errors); starter 42 -> 59 cells, 42 -> 46 TODOs. One stale figure caught on
re-read: the prose cited "26% of mass on negative values", which was P(x<0) alone, while the collapsed
table now reports the combined mass --- corrected to ~32% outside $\{0,1\}$.

**Point split named (instructor approved).** Both P6 and P7 are now **(A) 8 points** for the code
measurements and **(B) 17 points** for the written answer, with part (B) spelling out the same three
sub-parts the starter's answer cell asks for. Required total unchanged at **310**. P6 also gained a
`Lecture notes:` block (`EM_MIXTURES` §6.2, proof in `EM_ADVANCED` §1.3) --- it was the only required
problem without one. Both parts state their dependency (P6 needs Problem 3 working; P7 needs Problem
2's fitting functions). The starter's header carries the same split.

### 8.4b Retire list — now partly actioned

**Removed from the syllabus (2026-10-16)** — both verified fully subsumed by reading them in
full; both third-party, neither instructor-authored. The files themselves are untouched on disk:

| Candidate | Linked from | Subsumed by | Verified |
|---|---|---|---|
| `mix_gauss/mix_gauss_w_mu_piik_sigma.pdf` (D'Souza, USC, 12 pp — EM for GMM) | syllabus Week 8, and Week 10 as the VAE's ELBO prerequisite | `EM_MIXTURES` §2–§6 (same derivation, course notation, plus the notation bridge §14) and `EM_ADVANCED` §1–§3 (the proofs it omits) | read in full; nothing in it is missing from the new notes |
| `5_features_dimensions/.../feature_analysis/features_analysis.pdf` (2 pp — missing values in Naive Bayes) | syllabus Week 7 | `GDA_NB` §11 "Missing Values Come for Free" — the identical marginalization argument | read in full; identical content |

**Moved to the last column rather than removed** (not read in full this session, so not safe to
delete — rule 1's "otherwise"): the **Probability Primer** (cs229, an explicit prerequisite),
**Ng's cs229-notes2** on GDA/Naive Bayes, **CMU's NBayesLogReg**, and the **MIT EM slides**. All four
are plausibly subsumed; verify before deleting.

**Week 10's ELBO-prerequisite pointer** now leads with `EM_MIXTURES` and no longer carries the
D'Souza link, because that prerequisite is taught in-course now rather than outsourced.

**Explicitly NOT retire candidates**, per decision 4: all handwritten lectures (`Lecture_060625`,
`Lecture_061125`, `Lecture_GMM_061325`, `Lecture_mixt_binom_poisson_062025`), all slide decks
(`3_generative_models/slides/*`), `EM_coins.pdf` and the four coin-mixture background notes linked from
HW4 P4, and every external link (Ng, CMU, MIT, Berkeley, Wikipedia).

### 8.5 Downstream edits made

- `3_generative_models/HW3/HW4_26F.html`: `Lecture notes:` blocks on P1–P5; KDE cross-reference in P2;
  P4 symbol rename; `bayesian_ridge_regression.pdf` → `BAYES_REGRESSION/BAYES_REGRESSION_claude_1015.pdf`
  (2 places); an "All lecture notes for this homework" entry in References. All relative links verified
  to resolve.
- `code_HW3/build_hw4_solution.py` + `build_hw4_starter.py`: P4 rewritten with the new symbols; both
  notebooks regenerated and the solution re-executed end to end (0 error cells).
- `html/schedulen_mod.html`: Week 7 gains Basic-1 and Advanced-1; Week 8 gains Basic-2 and Advanced-2;
  Week 7's Bayesian-regression link repointed; Week 10's ELBO prerequisite repointed. All relative
  links verified to resolve.
- `claude_renovation_0826_code/HW_general_format.md`: dated amendment for HW4 250→310 points,
  5→7 required problems, optional bin renumbered P6–P14 → P8–P16, P4 symbols, starter-built status.

### 8.6 Not done / left for the user

- **HW5 needs no change.** P3(C) continues HW4 P3(C) as written, and HW4 P3(C) stayed required.
- The old `bayesian_ridge_regression.{tex,pdf}` are still in `lecture_notes/` alongside the restyled
  copy. Nothing links to them any more; delete when you're satisfied with the replacement.