Make sure you check the syllabus for the due date. Please use the notations adopted in class.
Where this HW sits. HW6 built a language model and a translator out of recurrent networks. Recurrence passes information one step at a time, and the translator squeezes a whole sentence into one fixed-size vector. Attention lets every output position look directly at every input position (Problems 2–3). The Transformer then drops recurrence entirely (Problem 4), and its decoder-only form is the GPT architecture behind today's large language models (Problem 5). Problem 1 covers what these models read: subword tokens and their embeddings. Every model is compared with its HW6 counterpart on the same data.
Setup. Same environment as HW5–HW6, plus Hugging Face transformers and tokenizers, and gensim (see the NN setup page).
Dev and full runs. The notebook has one switch, MODE. Debug in "dev": small models, few steps and small data subsets, every code path exercised and every training cell well under a minute even on a laptop CPU; its numbers are labelled [DEV — not for answers]. Then run "full": the real experiments. Your required answers, and anything you show in office hours, must use full runs. You never choose sizes: both settings are fixed in the notebook. Full runs are saved per problem and reloaded when you re-run the notebook. Expected full-run time: about 30 minutes on an Apple M1 Max (32-core GPU; measured), the longest training cell about 5 minutes; estimates for other machines: base M1/M2/M3 laptops about 2.5–3.5× longer, Colab's free T4 GPU about the same as the M1 Max, CPU only: use Colab, or run full mode overnight. The whole notebook in dev mode takes about 3 minutes on a laptop CPU. The numbers under What to see are from full runs; the notebook gives the expected ranges after each comparison.
Instructions. Submit the completed notebook with its outputs, from full runs. As in HW6, "from scratch" means built from torch tensor operations (autograd allowed), without the built-in it replaces: your attention may not call F.scaled_dot_product_attention, your multi-head attention may not call nn.MultiheadAttention, your blocks may not use nn.Transformer*, your GPT may not call Hugging Face. Those appear only as the reference your code is checked against. You may read the d2l book's code, but inside a TODO_STUDENT blank you may not call the d2l package's version.
Requirements.
Datasets and HW6 results. TinyStories (Problems 1, 5; the HW6 Problem 3 data and split). Spanish–English sentence pairs (Problems 3–4; the HW6 Problem 4 data, split and test set). HW7 needs your own HW6 full-run results: HW6 Problem 4(D)'s saved results_hw6.json (BLEU, test translations, and Problem 3's perplexities and stories) and hw6_seq2seq.pt (the translator), from HW6 in full mode. The notebook looks for them in the HW7 folder, then in your HW6 folder next to it (../hw6/), and in full mode stops with a message if they are missing. Dev mode runs without them (it uses your HW6 dev results, or a tiny stand-in), so you can start HW7 before HW6's full runs are finished.
Starter code: hw7_26F_starter.ipynb with lm_common.py, seq2seq_common.py and gensim_tinystories_course.json (download all four into one folder next to your HW6 folder, e.g. cs6140/hw7/). The starter provides data loading, HW6's encoder, decoder and training loops, plotting, and a checkpoint cell after each piece you implement (your weights copied into the matching PyTorch or Hugging Face module, same output expected, in float64 on the CPU with dropout off), plus an overfit-one-batch check for the Transformer and the GPT. Written answers go in markdown cells, each with a short How to think about it hint.
Prerequisites: HW6 Problem 3 (word tokens → IDs → nn.Embedding), and its side question on whole words. Lecture notes: TOKENS note (Week 13) — words vs characters vs subwords, byte-pair encoding, word vectors.
What to see: your BPE learns the same merges as the library's (in the reference run the ordered lists part ways only at a tie, which the notebook checks); a 5,000-piece BPE vocabulary keeps TinyStories' words whole and still covers typos and names, which the word vocabulary turns into <unk>, and GPT-2's bytes cover everything. Words used in similar contexts get similar vectors: animals, colours, family members and feelings form clusters, and the neighbour lists agree with gensim's on about half the words on average (in the reference run).
Library calls: Hugging Face tokenizers (BPE) and GPT-2's tokenizer, gensim.models.Word2Vec.
Prerequisites: matrix products and softmax (HW5); HW6 Problem 4 (padding). Lecture notes: ATTENTION note (Week 13) — queries, keys and values; scaled dot-product attention; masking; multi-head attention; positional encoding; the causal mask.
What to see: attention is a soft lookup; without the √d the softmax saturates as d grows (the largest of 10 weights goes from about 0.5 to above 0.9 between d = 4 and 256, and the softmax's sensitivity to its scores, the factor every gradient to the queries and keys passes through, shrinks about 4×); positions and the causal mask must be added on purpose.
Library calls: torch.nn.functional.scaled_dot_product_attention, torch.nn.MultiheadAttention.
Prerequisites: HW6 Problem 4 (the encoder–decoder); Problem 2(A). Lecture notes: ATTENTION note (Week 13) — attention in an encoder–decoder (Bahdanau), alignment.
What to see (full runs): attention improves the translations clearly (about 28 → 38 BLEU, in every length bucket). On the reverse task the difference is stark: without attention, accuracy collapses as the sequence gets longer (about 97% at 5 symbols, 30% at 20, 10% at 60); with attention it stays near 100% up to 60 symbols. And the model learns a rough, soft word alignment it was never told (some words look at their translation, others at the end of the sentence).
Prerequisites: Problem 2; HW6 Problem 2 (residual connections) and Problem 1(D) (batch normalization). Lecture notes: TRANSFORMER note (Week 14) — the Transformer block, add & norm, layer normalization, the encoder and decoder stacks, cross-attention.
What to see (full runs): no recurrence at all, yet clearly better translations (about 45 BLEU vs 38 for the attention GRU and 28 for HW6's, better in every length bucket) with fewer parameters and less training time; yours and PyTorch's (the same architecture, dropout included) reach about the same quality, PyTorch's a little faster; how the three kinds of attention fit together.
Library calls: torch.nn.LayerNorm, torch.nn.TransformerEncoderLayer, torch.nn.TransformerDecoderLayer, torch.nn.Transformer.
Prerequisites: Problem 4(B) and Problem 2(D) (the causal mask); HW6 Problem 3 (the same language-model task and data). Lecture notes: TRANSFORMER note (Week 14) — decoder-only models, pre-norm vs post-norm, sampling: greedy, temperature, top-k.
What to see (full runs): the same task as HW6 Problem 3, clearly better results (perplexity about 15 vs 18.6 for HW6's LSTM and 24 for its RNN); your GPT matches the library's within a perplexity point; pre-norm and post-norm are about equal at 4 blocks (the reason for pre-norm is depth); greedy decoding is safe but repetitive, temperature trades variety for mistakes, top-k cuts the worst words.
Library calls: Hugging Face GPT2LMHeadModel, generate, TemperatureLogitsWarper, TopKLogitsWarper.
Prerequisites: Problem 2(C); the attention maps of Problem 3(B) and of your Problem 4 Transformer's four cross-attention heads (the notebook draws them).
Why can several small heads do more than one large head of the same total size? Use your own attention maps.
Prerequisites: Problems 4 and 5. Reading: the TRANSFORMER note, the section “Three Kinds of Transformer”; or d2l 11.9 (large-scale pretraining with Transformers): what BERT and T5 are trained to do is not something your experiments show.
Encoder-only (BERT), decoder-only (GPT) and encoder–decoder (translation, T5): which blocks and masks does each use, and what is each trained to do? What does cross-attention give a translator? Why did decoder-only models become the common choice?
Background: the TOKENS and TRANSFORMER advanced notes.