CS6140 Machine Learning — Fall 2026

HW6 — Convolutional and Recurrent Networks

Make sure you check the syllabus for the due date. Please use the notations adopted in class.

Where this HW sits. HW5's networks were stacks of fully-connected layers that ignore the structure of their input. HW6 adds the two architectures built around structure. Convolutional networks (Problems 1–2) exploit the fact that images are local and that the same pattern can appear anywhere. Recurrent networks (Problems 3–4) exploit the fact that sequences unfold in order, carrying a hidden state from step to step. As in every HW of this course, each key piece is built from scratch and then compared with the PyTorch library version. Problem 4's translator is the baseline that HW7's attention models and Transformers are compared against.

Setup. Same environment as HW5; see the NN setup page. The notebook gets every dataset itself.

Dev and full runs. The starter has one switch, MODE; both settings are given and printed by the notebook, so you never choose sizes.

• MODE = "dev" (the default): small data subsets and the same networks at smaller sizes, about 4 minutes for the whole notebook on a laptop CPU. Write and debug your code here. The checkpoint cells and the "overfit one batch" checks are as trustworthy as in full mode, but every dev result is labelled [DEV — not for answers]: dev accuracies, perplexities and BLEU scores mean nothing.
• MODE = "full": the real data and models. All required answers, and anything you show in office hours, must come from full runs. Each training run saves its model (folder hw6_runs/) and is reloaded instead of retrained when you re-run its cell, so you can finish one problem at a time. Each comparison has an Expected in a full run cell with the range your numbers should land in.
• HW7 needs your own HW6 full results: Problem 4's last cell writes results_hw6.json (including Problem 3's language models) and hw6_seq2seq.pt. Keep both. Dev writes results_hw6_dev.json / hw6_seq2seq_dev.pt, which HW7 does not accept.

Expected full-run time. Measured on an Apple M1 Max (32-core GPU): about 25 minutes for the whole notebook, the longest single training run (Problem 4's translator) about 5 minutes, plus the first-run downloads. Estimates for other machines: base M1/M2/M3 laptops (8–10-core GPU) about 2.5–3.5× longer; Colab's free T4 GPU about the same as the M1 Max; CPU only: use Colab, or run full mode overnight.

Instructions. Submit the completed notebook with its outputs, from full runs. Libraries (torch, torchvision, d2l, numpy, matplotlib) are allowed for data handling, plotting and standard layers. In this HW, "from scratch" means built from torch tensor operations, with autograd allowed, but without the built-in being replaced (no nn.Conv2d/F.conv2d inside your convolution, no nn.BatchNorm2d inside your batch norm, no nn.RNN inside your RNN). You may read the d2l book's code, but inside a TODO_STUDENT blank you may not call the d2l package's version of the piece you are building (nor the version in the shared course files lm_common.py / seq2seq_common.py).

Requirements.

• Fill in every TODO_STUDENT blank in the required problems/parts according to the algorithm steps covered in lecture and the linked materials, and make sure the notebook runs top to bottom without errors.
• Understand the whole notebook — including the provided/given code, not just the blanks you filled in — well enough to explain any part of it during office hours.

Datasets. Fashion-MNIST (Problem 1; the same data as HW5, so HW5's MLP is your baseline). CIFAR-10 (Problem 2; 32×32 colour photos, 10 classes; 170 MB). TinyStories (Problem 3; short children's stories written with a small vocabulary, so a small language model can produce fluent text; we use the official 19 MB validation file, about 22,000 stories and 4.4 million words). Spanish–English sentence pairs (Problem 4; Tatoeba, sentence_pairs_large.tsv, filtered to pairs of at most 12 tokens per side: 100,000 training pairs, and 2,000 test sentences never seen in training, each scored against all its translations), the same data HW7 uses. The notebook downloads all of it on first run into the shared data/ folder (see the setup page). Problems 3–4 load their data through two shared files that come with the starter, lm_common.py and seq2seq_common.py, which HW7 uses too.

Starter code: hw6_26F_starter.ipynb with lm_common.py and seq2seq_common.py (download all three into one folder, e.g. cs6140/hw6/). The starter provides the device cell, data loading, training loops, plotting, and a checkpoint cell after each piece you implement: it copies your weights into the matching PyTorch module and checks that both give the same output, so you know a piece is right before you train with it. Written answers go in the notebook's markdown cells, each with a short How to think about it hint.

PROBLEM 1 — A CNN from Its Building Blocks (Fashion-MNIST)   [50 points]

Prerequisites: HW5 Problem 2 (an MLP classifier in PyTorch, softmax + cross-entropy, training loop); matrix products. Lecture notes: CNN note (Week 11) — cross-correlation, channels, padding and stride, pooling, LeNet, batch normalization.

• (A — 15 points) Convolution and pooling, from scratch. corr2d with loops → summing over input channels → max and average pooling. Convolution as one matrix product (unfold the patches, then multiply) is given: read it and check it matches your corr2d.
• (B — 10 points) LeNet, yours vs PyTorch's. LeNet from your layers (the layer list is given) and from nn layers; check they give the same output; train both. Table against HW5's MLP (retrained in the notebook with the same optimizer and number of epochs): accuracy and number of parameters.
• (C — 10 points) Modernize. ReLU and max pooling; retrain; then test the MLP and LeNet on images shifted by a few pixels (provided).
• (D — 15 points) Batch normalization, from scratch. Training mode (statistics over the batch and the image positions, then γ, β) and evaluation mode (running averages); batch-normalized LeNet at a higher learning rate.

What to see: the 1998 LeNet (sigmoid) trails the MLP; modernized, it matches the MLP with less than a third of its parameters and holds up far better on shifted images, because convolution builds in locality and weight sharing; batch norm makes training faster and more forgiving of the learning rate, and the batch-normalized LeNet beats the MLP outright.

Library calls: torch.nn.functional.conv2d, torch.nn.functional.max_pool2d, torch.nn.Conv2d, torch.nn.BatchNorm2d.

PROBLEM 2 — Deeper CNNs on Natural Images (CIFAR-10)   [50 points]

Prerequisites: Problem 1; HW5 Problem 2(F) (vanishing gradients through many layers). Lecture notes: CNN note (Week 11) — residual networks, data augmentation, transfer learning.

• (A — 20 points) Residual block. The skip connection, plus a 1×1 convolution when the shapes change; a small ResNet (layout given) against the batch-normalized LeNet (given).
• (B — 15 points) Data augmentation. Random crops and flips on the training data only (given); compare the train/test gap with and without.
• (C — 15 points) Transfer learning. On a small subset (5,000 images): your ResNet trained from scratch, against a pretrained ResNet-18 with only its last layer trained, against the same network fully fine-tuned (images resized to 128×128 for ResNet-18).

What to see: the 11-layer ResNet beats the batch-normalized LeNet by a wide margin; the explanation from lecture is that its shortcuts give the signal and the gradient a direct path around every block (this comparison changes depth and shortcuts together; optional Problem 7 separates them). Augmentation shrinks the train/test gap; pretrained features win by a wide margin when labelled data is scarce.

Library calls: torchvision.models.resnet18, torchvision.transforms.v2.

PROBLEM 3 — A Recurrent Language Model (TinyStories)   [60 points]

Prerequisites: HW5 (training loop, softmax + cross-entropy); the starter's opening walkthrough of tokens → IDs → embedding lookup → next-token targets. Lecture notes: RNN note (Week 12) — language models and perplexity, the recurrent cell, backpropagation through time, exploding and vanishing gradients, GRU and LSTM.

• (A — 30 points) An RNN from scratch, step by step. One cell step (checked against a hand-computed value) → the whole sequence → the same output as nn.RNN → gradient clipping. Then the baseline that ignores context (word frequencies only), and your RNN trained against nn.RNN: perplexity and speed.
• (B — 15 points) GRU and LSTM. Swap in nn.GRU and nn.LSTM; perplexity table and generated stories at two temperatures. Side question: TinyStories was built so that about 5,000 words cover 99.6% of it. What would go wrong with whole words as tokens on real text, and what do real language models use instead?
• (C — 15 points) How far back do gradients reach? Run the provided experiment (gradient size against distance in time, RNN vs LSTM) and explain the plot; Problem 5 builds on it.

What to see: perplexity drops from the RNN to the GRU and LSTM, and the RNN's gradient fades with distance much faster than the gated models'. In the generated stories, look for which characters and scenes carry over from sentence to sentence and where a story contradicts itself (one sample per model is noisy; the perplexities are the firmer evidence).

Library calls: torch.nn.RNN, torch.nn.GRU, torch.nn.LSTM, torch.nn.utils.clip_grad_norm_.

PROBLEM 4 — Sequence-to-Sequence Translation, Spanish → English   [50 points]

Prerequisites: Problem 3 (an RNN that predicts the next word); padding a batch of different-length sentences. Lecture notes: RNN note (Week 12) — the encoder–decoder, teacher forcing, padding masks, greedy decoding, BLEU.

• (A — 10 points) Encoder. Embedding + GRU; shape check.
• (B — 15 points) Decoder, trained with teacher forcing, and the one-line masked loss that ignores padding. Check: decoding the whole target at once and one word at a time give the same scores.
• (C — 10 points) Greedy decoding. Translate a new sentence one word at a time, stopping at the end-of-sentence token.
• (D — 15 points) Evaluate and save. BLEU (standard corpus BLEU-4) overall and by sentence length, good and bad examples, the book's d2l.Seq2Seq trained the same way for comparison, and the save cell for HW7.

What to see: short everyday sentences come out right; longer sentences tend to score worse. One suspect is that the whole source sentence must fit into one fixed-size vector; HW7 tests that idea with attention.

Library calls: torch.nn.GRU, torch.nn.CrossEntropyLoss (with ignore_index); the d2l book's d2l.Seq2Seq as the package baseline.

PROBLEM 5 (THEORY) — Why an LSTM Reaches Further Back Than a Plain RNN   [25 points]

Prerequisites: Problem 3(C).

Problem 3(C) described what the curves show; here explain why. Using your Problem 3(C) plot: why does the plain RNN's gradient shrink with distance, and what in the LSTM changes that? Why does gradient clipping fix exploding gradients but not vanishing ones?

PROBLEM 6 (THEORY) — What Convolution Is (and Isn't) Robust To   [25 points]

Prerequisites: Problem 2(B).

A CNN trained on centred, upright objects fails on objects moved to a corner or rotated. Which changes does convolution + pooling handle on its own, which does it not, and which of these did your Problem 2(B) augmentation teach it?

OPTIONAL PROBLEMS [no credit]

Background: the CNN and RNN advanced notes.

• PROBLEM 7 — Plain vs residual at depth (20+ layers): the deeper plain network trains worse.
• PROBLEM 8 — An LSTM cell from scratch, checked against nn.LSTM (set up in the starter).
• PROBLEM 9 — Your Problem 3 RNN inside the Problem 4 translator. More than a one-line swap: your RNN can't read the packed sequences the encoder uses (run it on the padded batch instead) and the translator has 2 layers (stack two of yours).
• PROBLEM 10 — An LSTM translator.
• PROBLEM 11 — Why dev mode looks the way it does: short questions in the notebook (why dev accuracies are meaningless while the checks stay trustworthy, why a dev perplexity is not comparable, why dev can't show the long-sentence BLEU drop, what would make dev too small).

References

• NN setup page.
• Lecture notes for this HW: CNN note (Week 11) and RNN note (Week 12); optional: CNN advanced, RNN advanced. The annotated lectures: convolutional networks, RNN, LSTM, GRU, encoder-decoder, seq2seq.
• d2l book: CNNs, modern CNNs, fine-tuning, RNNs, modern RNNs.
• TinyStories: Eldan & Li (2023).