{ "cells": [ { "cell_type": "markdown", "id": "ffd442b3", "metadata": {}, "source": [ "# Recurrent networks — lecture demos (RNN-A and RNN-B)\n", "\n", "Laptop demos for the RNN note (`RNN_claude_1017.pdf`), one segment per board block of the lecture plan.\n", "The code is the **d2l book's** (chapters 9–10) wherever the book has it; the first line of every code cell\n", "says whether it is the book's code as is, the book's code with one change, or something added for CS6140.\n", "\n", "| Seg. | Lecture | Board topic | Note | Time on an M1 Max |\n", "|---|---|---|---|---|\n", "| 1 | A | Tokens → IDs → embeddings → shifted targets | §2 | 2 s |\n", "| 2 | A | Language models and perplexity | §3 | 0 s |\n", "| 3 | A | The recurrent cell, by hand and in nn.RNN | §4 | 0 s |\n", "| 4 | A | Training the book's RNN language model; temperature | §4 | 47 s |\n", "| 5 | A | A product of factors; gradient clipping | §5 | 0 s |\n", "| 6 | A | The encoder–decoder on Spanish → English | §7 | 48 s |\n", "| 7 | A | Teacher forcing vs generation; the consistency check | §8 | 0 s |\n", "| 8 | B | Gates: the book's GRU and LSTM | §6 | 1.9 min |\n", "| 9 | B | How far back do gradients reach? | §5–6 | 1 s |\n", "| 10 | B | Greedy translations and BLEU | §9 | 1 s |\n", "\n", "**What differs from the book**\n", "\n", "| | Segments | What |\n", "|---|---|---|\n", "| Changed | 1, 4, 8, 9 | **TinyStories, word level** (HW6 Problem 3's data, vocabulary 5,274) instead of the book's *Time Machine* characters; `lm_common.TinyStories` is a `d2l.TimeMachine` subclass, so the book's code runs on it unchanged |\n", "| Changed | 4, 8 | fewer steps (1,000 per model, about a minute) and 128 hidden units instead of 32 |\n", "| Changed | 6, 7, 10 | **Spanish → English** (HW6 Problem 4's Tatoeba data, dev-sized: 10,000 pairs) instead of the book's French → English; smaller model (embeddings 64, hidden 128) and 12 epochs |\n", "| Added | 2 | perplexity of a uniform guess, computed with the library loss |\n", "| Added | 3 | the note's hand-computed cell step, run in the book's `RNNScratch` and in `nn.RNN` (transposed weights, two biases) |\n", "| Added | 4 | samples at two temperatures with HW6's given `lm_common.generate` (the book's `predict` is greedy and joins characters) |\n", "| Added | 5 | the per-step factor $0.9^k$ / $1.1^k$ and `nn.utils.clip_grad_norm_` on the note's $(3,4)$ |\n", "| Added | 7 | teacher-forced guesses next to the model's own generation; whole target = word by word |\n", "| Added | 9 | gradient of the last prediction w.r.t. inputs $d$ words back, for the three trained models |\n", "| Added | 10 | BLEU with NLTK (standard BLEU-4) next to the book's `d2l.bleu` on the note's example; clipping with two references |\n", "\n", "Run the setup cell first; then each segment runs on its own after the ones it says it needs\n", "(segments 4 → 5, 8, 9; segment 6 → 7, 10). Numbers vary a little from run to run." ] }, { "cell_type": "code", "execution_count": 1, "id": "180d87f0", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:25:55.539755Z", "iopub.status.busy": "2026-10-18T03:25:55.539618Z", "iopub.status.idle": "2026-10-18T03:25:57.900312Z", "shell.execute_reply": "2026-10-18T03:25:57.899750Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "device: mps\n" ] } ], "source": [ "# [Book §9.6] imports, plus the device setup the HW notebooks use (d2l's GPU helpers only know CUDA)\n", "import sys, time, math, collections\n", "import torch\n", "from torch import nn\n", "from torch.nn import functional as F\n", "from d2l import torch as d2l\n", "\n", "\n", "def get_device():\n", " if torch.cuda.is_available():\n", " return torch.device(\"cuda\")\n", " if torch.backends.mps.is_available():\n", " return torch.device(\"mps\")\n", " return torch.device(\"cpu\")\n", "\n", "\n", "DEVICE = get_device()\n", "if DEVICE.type == \"mps\": # point d2l's GPU helpers at Apple's GPU\n", " d2l.num_gpus = lambda: 1\n", " d2l.gpu = lambda i=0: DEVICE\n", " d2l.try_gpu = lambda i=0: DEVICE\n", "NUM_GPUS = 1 if DEVICE.type != \"cpu\" else 0\n", "CPU = torch.device(\"cpu\")\n", "print(\"device:\", DEVICE)" ] }, { "cell_type": "code", "execution_count": 2, "id": "47701cf4", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:25:57.901795Z", "iopub.status.busy": "2026-10-18T03:25:57.901611Z", "iopub.status.idle": "2026-10-18T03:25:57.908885Z", "shell.execute_reply": "2026-10-18T03:25:57.908476Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shared modules from /Users/vip/Dropbox/CS6140/2.5_NN/HW6_advNN | data in /Users/vip/Dropbox/CS6140/data\n" ] } ], "source": [ "# [Added for CS6140] HW6's shared data modules and the course data folder (as in the HW starters)\n", "import os\n", "import urllib.request\n", "\n", "# The shared data folder: cs6140/data, next to your HW folders (NN setup page, section 3).\n", "DATA = next((d for d in [\"../data\", \"../../data\", \"../../../data\"] if os.path.isdir(d)), \"../data\")\n", "COURSE_DATA_URL = \"https://www.khoury.northeastern.edu/home/vip/teach/MLcourse/data\"\n", "\n", "\n", "def course_file(name, fallback_url=None):\n", " \"\"\"DATA/name. Downloaded the first time from the course website (or from fallback_url).\"\"\"\n", " path = os.path.join(DATA, name)\n", " if not os.path.exists(path):\n", " os.makedirs(os.path.dirname(path), exist_ok=True)\n", " for url in [f\"{COURSE_DATA_URL}/{name}\"] + ([fallback_url] if fallback_url else []):\n", " try:\n", " print(f\"downloading {name} from {url} ...\")\n", " urllib.request.urlretrieve(url, path + \".part\")\n", " os.replace(path + \".part\", path)\n", " break\n", " except OSError as err:\n", " print(f\" failed: {err}\")\n", " else:\n", " raise FileNotFoundError(f\"could not download {name}: put it into {os.path.abspath(DATA)}\")\n", " return path\n", "\n", "_places = [\".\", \"../hw6\", \"../../../2.5_NN/HW6_advNN\"] # next to this notebook, your HW6 folder, the course tree\n", "COMMON = next((p for p in _places if os.path.exists(f\"{p}/lm_common.py\")), None)\n", "if COMMON is None:\n", " raise FileNotFoundError(\"lm_common.py / seq2seq_common.py (they come with the HW6 starter) not found in: \"\n", " + \", \".join(os.path.abspath(p) for p in _places))\n", "sys.path.insert(0, COMMON)\n", "import lm_common\n", "import seq2seq_common\n", "course_file(\"sentence_pairs_large.tsv\") # Segment 6's Spanish-English pairs (24 MB, downloaded once)\n", "print(\"shared modules from\", os.path.abspath(COMMON), \"| data in\", os.path.abspath(DATA))" ] }, { "cell_type": "markdown", "id": "04d25d9f", "metadata": {}, "source": [ "---\n", "# Lecture RNN-A\n", "\n", "## Segment 1 — Tokens → IDs → embeddings → shifted targets (note §2)\n", "**What to watch:** a story becomes a list of word IDs; the target sequence is the input shifted by one word;\n", "a few thousand words cover almost all of TinyStories, and anything else becomes ``." ] }, { "cell_type": "code", "execution_count": 3, "id": "1f55a55d", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:25:57.910115Z", "iopub.status.busy": "2026-10-18T03:25:57.910014Z", "iopub.status.idle": "2026-10-18T03:26:00.190231Z", "shell.execute_reply": "2026-10-18T03:26:00.189738Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TinyStories: 20,891 training / 1,099 validation stories; 4,229,528 / 221,116 tokens; vocabulary 5,274 words (min frequency 10); 0.45% of training tokens are \n", "(loaded in 2 s)\n" ] } ], "source": [ "# [Changed from book §9.2–9.3] HW6's TinyStories (word level) instead of the book's Time Machine (characters)\n", "t0 = time.time()\n", "data = lm_common.TinyStories(root=f\"{DATA}/tinystories\", batch_size=128, num_steps=32, min_freq=10,\n", " batches_per_epoch=50)\n", "print(data.describe())\n", "print(f\"(loaded in {time.time() - t0:.0f} s)\")" ] }, { "cell_type": "code", "execution_count": 4, "id": "c59d507d", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.191479Z", "iopub.status.busy": "2026-10-18T03:26:00.191369Z", "iopub.status.idle": "2026-10-18T03:26:00.193729Z", "shell.execute_reply": "2026-10-18T03:26:00.193343Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['once', 'upon', 'a', 'time', ',', 'there', 'was', 'a', 'little', 'girl', 'named', 'lily', '.']\n", "[3004, 4917, 9, 4700, 2, 4632, 5018, 9, 2506, 1778, 2861, 2483, 3]\n", "['once', 'upon', 'a', 'time', ',', 'there', 'was', 'a', 'little', 'girl', 'named', 'lily', '.']\n", "vocabulary size: 5274 | most frequent: [('.', 343055), ('the', 194374), ('and', 172689), (',', 165816), ('to', 122829), ('a', 111452), ('was', 93510), ('\"', 89705)]\n" ] } ], "source": [ "# [Book §9.2] tokens -> IDs with the book's Vocab; IDs -> tokens back\n", "sentence = \"once upon a time , there was a little girl named lily .\"\n", "tokens = sentence.split()\n", "ids = data.vocab[tokens]\n", "print(tokens)\n", "print(ids)\n", "print(data.vocab.to_tokens(ids))\n", "print(\"vocabulary size:\", len(data.vocab), \"| most frequent:\", data.vocab.token_freqs[:8])" ] }, { "cell_type": "code", "execution_count": 5, "id": "a0a364d7", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.194875Z", "iopub.status.busy": "2026-10-18T03:26:00.194779Z", "iopub.status.idle": "2026-10-18T03:26:00.197568Z", "shell.execute_reply": "2026-10-18T03:26:00.197207Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "the little dog was happy . -> the little dog was happy .\n", "the archaeologist examined a byzantine manuscript . -> the examined a .\n", "lily sayd helo to her frend . -> lily to her .\n", "validation tokens that are : 0.62% (real LLMs avoid this with subword tokens: Week 13)\n" ] } ], "source": [ "# [Added for CS6140] whole words are fine for TinyStories, not for real text: unknown words become \n", "for text in [\"the little dog was happy .\", \"the archaeologist examined a byzantine manuscript .\",\n", " \"lily sayd helo to her frend .\"]:\n", " toks = lm_common.tokenize(text)\n", " print(f\"{text:52s} -> {' '.join(data.vocab.to_tokens(data.vocab[toks]))}\")\n", "val_unk = (data.val_ids == data.vocab.unk).float().mean().item()\n", "print(f\"validation tokens that are : {val_unk:.2%} (real LLMs avoid this with subword tokens: Week 13)\")" ] }, { "cell_type": "code", "execution_count": 6, "id": "bb698e4a", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.198710Z", "iopub.status.busy": "2026-10-18T03:26:00.198615Z", "iopub.status.idle": "2026-10-18T03:26:00.201239Z", "shell.execute_reply": "2026-10-18T03:26:00.200890Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "X: (128, 32) Y: (128, 32)\n", "input : and feeling ashamed . she asked for help and listened to the\n", "target: feeling ashamed . she asked for help and listened to the advice\n" ] } ], "source": [ "# [Book §9.3] a minibatch: X and Y are the same windows, Y shifted by one word\n", "X, Y = next(iter(data.train_dataloader()))\n", "print(\"X:\", tuple(X.shape), \" Y:\", tuple(Y.shape))\n", "print(\"input :\", \" \".join(data.vocab.to_tokens(X[0, :12].tolist())))\n", "print(\"target:\", \" \".join(data.vocab.to_tokens(Y[0, :12].tolist())))" ] }, { "cell_type": "code", "execution_count": 7, "id": "4f78f63b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.202354Z", "iopub.status.busy": "2026-10-18T03:26:00.202261Z", "iopub.status.idle": "2026-10-18T03:26:00.206250Z", "shell.execute_reply": "2026-10-18T03:26:00.205799Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "one-hot x E : tensor([-0.3894, 0.7702, 0.6750, 0.1561])\n", "row of E for ID : tensor([-0.3894, 0.7702, 0.6750, 0.1561])\n" ] } ], "source": [ "# [Book §9.5] the book feeds one-hot vectors; one-hot times a matrix = picking a row = an embedding lookup\n", "V = len(data.vocab)\n", "E = torch.randn(V, 4) # an embedding matrix with 4-dimensional rows\n", "onehot = F.one_hot(torch.tensor([ids[1]]), V).float() # the word \"upon\"\n", "print(\"one-hot x E :\", (onehot @ E).squeeze())\n", "print(\"row of E for ID :\", E[ids[1]])" ] }, { "cell_type": "markdown", "id": "4bb22825", "metadata": {}, "source": [ "## Segment 2 — Language models and perplexity (note §3)\n", "**What to watch:** perplexity = exp(mean cross-entropy) = \"the number of words the model is choosing\n", "between\". A uniform guess gives exactly the vocabulary size." ] }, { "cell_type": "code", "execution_count": 8, "id": "80bf4a77", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.207418Z", "iopub.status.busy": "2026-10-18T03:26:00.207274Z", "iopub.status.idle": "2026-10-18T03:26:00.225454Z", "shell.execute_reply": "2026-10-18T03:26:00.225061Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cross-entropy 8.571 nats -> perplexity 5,274 (vocabulary 5,274)\n", "the ladder in HW6 (reference run): uniform 5,274 -> word frequencies only about 303.5 -> RNN about 23.8 -> LSTM about 18.6 -> GRU about 17.5\n" ] } ], "source": [ "# [Added for CS6140] a model that knows nothing: equal scores for every word -> perplexity = |V|\n", "Xv, Yv = next(iter(data.val_dataloader()))\n", "uniform_logits = torch.zeros(*Yv.shape, V)\n", "loss = F.cross_entropy(uniform_logits.reshape(-1, V), Yv.reshape(-1))\n", "print(f\"cross-entropy {loss.item():.3f} nats -> perplexity {math.exp(loss.item()):,.0f} (vocabulary {V:,})\")\n", "print(\"the ladder in HW6 (reference run): uniform 5,274 -> word frequencies only about 303.5 -> RNN about 23.8 \"\n", " \"-> LSTM about 18.6 -> GRU about 17.5\")" ] }, { "cell_type": "markdown", "id": "64b476f9", "metadata": {}, "source": [ "## Segment 3 — The recurrent cell, by hand and in `nn.RNN` (note §4)\n", "**What to watch:** the note's hand-computed step, $H_t=\\tanh(X_tW_{xh}+H_{t-1}W_{hh}+b_h)=(0.3364,-0.0500)$,\n", "from the book's from-scratch cell and from PyTorch's `nn.RNN`, which stores the matrices *transposed* and\n", "has *two* biases." ] }, { "cell_type": "code", "execution_count": 9, "id": "50077e37", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.226704Z", "iopub.status.busy": "2026-10-18T03:26:00.226616Z", "iopub.status.idle": "2026-10-18T03:26:00.230073Z", "shell.execute_reply": "2026-10-18T03:26:00.229714Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RNNScratch: tensor([[ 0.3364, -0.0500]])\n" ] } ], "source": [ "# [Changed from book §9.5] the book's RNNScratch, with the note's weights put in instead of random ones\n", "cell = d2l.RNNScratch(num_inputs=2, num_hiddens=2)\n", "with torch.no_grad():\n", " cell.W_xh.copy_(torch.tensor([[0.1, 0.2], [0.3, 0.4]]))\n", " cell.W_hh.copy_(0.5 * torch.eye(2))\n", " cell.b_h.zero_()\n", "X_t = torch.tensor([[[1.0, 0.0]]]) # (steps=1, batch=1, inputs=2): time-major, as the book\n", "H_prev = torch.tensor([[0.5, -0.5]])\n", "outputs, H_t = cell(X_t, (H_prev,)) # the book's cell takes the state as a 1-tuple\n", "print(\"RNNScratch:\", H_t.detach())" ] }, { "cell_type": "code", "execution_count": 10, "id": "b16a8fa8", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.231246Z", "iopub.status.busy": "2026-10-18T03:26:00.231165Z", "iopub.status.idle": "2026-10-18T03:26:00.234164Z", "shell.execute_reply": "2026-10-18T03:26:00.233807Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "nn.RNN : tensor([[ 0.3364, -0.0500]])\n", "shapes: weight_ih (2, 2) weight_hh (2, 2) | the SAME three matrices are used at every time step\n" ] } ], "source": [ "# [Added for CS6140] the same step in nn.RNN: weight_ih = W_xh transposed, weight_hh = W_hh transposed\n", "rnn = nn.RNN(2, 2)\n", "with torch.no_grad():\n", " rnn.weight_ih_l0.copy_(cell.W_xh.T)\n", " rnn.weight_hh_l0.copy_(cell.W_hh.T)\n", " rnn.bias_ih_l0.zero_()\n", " rnn.bias_hh_l0.zero_() # b_h = bias_ih + bias_hh\n", "out, h = rnn(X_t, H_prev.unsqueeze(0))\n", "print(\"nn.RNN :\", h.detach().squeeze(0))\n", "print(\"shapes: weight_ih\", tuple(rnn.weight_ih_l0.shape), \"weight_hh\", tuple(rnn.weight_hh_l0.shape),\n", " \"| the SAME three matrices are used at every time step\")" ] }, { "cell_type": "markdown", "id": "b3034039", "metadata": {}, "source": [ "## Segment 4 — Training the book's RNN language model; temperature (note §4)\n", "**What to watch:** the book's concise RNN, trained for about a minute on TinyStories: perplexity falls from\n", "5,274 to under 100 and the stories start to make local sense. Temperature trades safety for variety." ] }, { "cell_type": "code", "execution_count": 11, "id": "9f38edff", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.235379Z", "iopub.status.busy": "2026-10-18T03:26:00.235298Z", "iopub.status.idle": "2026-10-18T03:26:00.237777Z", "shell.execute_reply": "2026-10-18T03:26:00.237436Z" } }, "outputs": [], "source": [ "# [Book §9.6] the book's concise RNN and RNN language model, as is\n", "class RNN(d2l.Module): #@save\n", " \"\"\"The RNN model implemented with high-level APIs.\"\"\"\n", " def __init__(self, num_inputs, num_hiddens):\n", " super().__init__()\n", " self.save_hyperparameters()\n", " self.rnn = nn.RNN(num_inputs, num_hiddens)\n", "\n", " def forward(self, inputs, H=None):\n", " return self.rnn(inputs, H)\n", "\n", "\n", "class RNNLM(d2l.RNNLMScratch): #@save\n", " \"\"\"The RNN-based language model implemented with high-level APIs.\"\"\"\n", " def init_params(self):\n", " self.linear = nn.LazyLinear(self.vocab_size)\n", "\n", " def output_layer(self, hiddens):\n", " return self.linear(hiddens).swapaxes(0, 1)" ] }, { "cell_type": "code", "execution_count": 12, "id": "a8ec404b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:00.238826Z", "iopub.status.busy": "2026-10-18T03:26:00.238747Z", "iopub.status.idle": "2026-10-18T03:26:45.893475Z", "shell.execute_reply": "2026-10-18T03:26:45.892981Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "46 s | validation perplexity 104.1\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:26:45.870611\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.10.7, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Changed from book §9.6] TinyStories, 128 hidden units, 20 epochs of 50 batches (1,000 steps) instead of 100 epochs\n", "torch.manual_seed(0)\n", "rnn_lm = RNNLM(RNN(num_inputs=V, num_hiddens=128), vocab_size=V, lr=1)\n", "trainer = d2l.Trainer(max_epochs=20, gradient_clip_val=1, num_gpus=NUM_GPUS)\n", "t0 = time.time()\n", "trainer.fit(rnn_lm, data)\n", "PPL = {\"RNN\": lm_common.perplexity(rnn_lm, data, DEVICE)}\n", "print(f\"{time.time() - t0:.0f} s | validation perplexity {PPL['RNN']:.1f}\")" ] }, { "cell_type": "code", "execution_count": 13, "id": "77105f0f", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:45.894944Z", "iopub.status.busy": "2026-10-18T03:26:45.894826Z", "iopub.status.idle": "2026-10-18T03:26:46.789432Z", "shell.execute_reply": "2026-10-18T03:26:46.788883Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "T = 0.5: once upon a time, there was a little girl named lily. she was very happy and he was very to. it was so excited to be all. she was a happy and ran. it was a little girl named \n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "T = 1.0: once upon a time, there was a buckets spicy with to the park and was very heroes and me. they have the overcome train that vehicle. all felt a attic on, suddenly. a grateful with the matt folds out \n", "\n" ] } ], "source": [ "# [Added for CS6140] stories at two temperatures, with HW6's given sampler (softmax(scores / T), then a draw)\n", "for T in [0.5, 1.0]:\n", " print(f\"T = {T}:\", lm_common.generate(rnn_lm, data.vocab, \"once upon a time\", DEVICE, n_words=40,\n", " temperature=T), \"\\n\")" ] }, { "cell_type": "markdown", "id": "ad335836", "metadata": {}, "source": [ "## Segment 5 — A product of factors; gradient clipping (note §5)\n", "**What to watch:** the gradient reaching a word $k$ steps back is a product of $k$ per-step factors, so it\n", "shrinks or blows up geometrically. Clipping rescales a too-long gradient; it cannot bring back a vanished one." ] }, { "cell_type": "code", "execution_count": 14, "id": "c227d793", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:46.791165Z", "iopub.status.busy": "2026-10-18T03:26:46.791042Z", "iopub.status.idle": "2026-10-18T03:26:46.793742Z", "shell.execute_reply": "2026-10-18T03:26:46.793069Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "k = 10: 0.9^k = 0.3487 1.1^k = 2.59 0.95^k = 0.599\n", "k = 20: 0.9^k = 0.1216 1.1^k = 6.73 0.95^k = 0.358\n", "k = 50: 0.9^k = 0.0052 1.1^k = 117.39 0.95^k = 0.077\n" ] } ], "source": [ "# [Added for CS6140] the note's numbers: a per-step factor of 0.9 vs 1.1\n", "for k in [10, 20, 50]:\n", " print(f\"k = {k:2d}: 0.9^k = {0.9 ** k:8.4f} 1.1^k = {1.1 ** k:8.2f} 0.95^k = {0.95 ** k:.3f}\")" ] }, { "cell_type": "code", "execution_count": 15, "id": "0b1a84c1", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:46.795547Z", "iopub.status.busy": "2026-10-18T03:26:46.795413Z", "iopub.status.idle": "2026-10-18T03:26:46.799087Z", "shell.execute_reply": "2026-10-18T03:26:46.798589Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "theta = 1.0: norm before 5.0 -> gradient after [0.6, 0.8]\n", "theta = 10.0: norm before 5.0 -> gradient after [3.0, 4.0]\n" ] } ], "source": [ "# [Added for CS6140] clipping with PyTorch's clip_grad_norm_ (what the book's gradient_clip_val does): g=(3,4)\n", "for theta in [1.0, 10.0]:\n", " p = torch.zeros(2, requires_grad=True)\n", " p.grad = torch.tensor([3.0, 4.0])\n", " norm = nn.utils.clip_grad_norm_([p], max_norm=theta)\n", " print(f\"theta = {theta:4.1f}: norm before {norm.item():.1f} -> gradient after {[round(v, 4) for v in p.grad.tolist()]}\")" ] }, { "cell_type": "markdown", "id": "a07f8a3d", "metadata": {}, "source": [ "## Segment 6 — The encoder–decoder on Spanish → English (note §7)\n", "**What to watch:** two RNNs. The encoder reads the Spanish sentence; its last state is the *context*, fed to\n", "the decoder at every step. The decoder is a language model for English, conditioned on that context." ] }, { "cell_type": "code", "execution_count": 16, "id": "63849b8b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:46.800574Z", "iopub.status.busy": "2026-10-18T03:26:46.800461Z", "iopub.status.idle": "2026-10-18T03:26:49.357018Z", "shell.execute_reply": "2026-10-18T03:26:49.356311Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tatoeba es->en, both sides <= 12 tokens: 243,067 pairs, 222,600 distinct Spanish sentences.\n", "train 10,000 pairs (of 237,612 available) | val 1,093 pairs | test 2,000 Spanish sentences (1.09 references each) | final test 2,000 (kept untouched)\n", "vocabularies: Spanish 3,566, English 2,884 tokens (min frequency 2); num_steps = 13\n", "one pair: tengo casi cuarenta años y aún soy . | i am almost 40 years old and i am still single . \n", "decoder input (starts with ): i am almost 40 years old and i am still single .\n" ] } ], "source": [ "# [Changed from book §10.5] HW6's Spanish-English pairs (dev-sized: 10,000 training pairs) instead of French\n", "mt = seq2seq_common.TatoebaEsEn(root=DATA, batch_size=128, num_train=10_000)\n", "print(mt.describe())\n", "src, tgt_in, src_len, label = next(iter(mt.train_dataloader()))\n", "print(\"one pair:\", \" \".join(mt.src_vocab.to_tokens(src[0].tolist())), \"|\",\n", " \" \".join(mt.tgt_vocab.to_tokens(label[0].tolist())))\n", "print(\"decoder input (starts with ):\", \" \".join(mt.tgt_vocab.to_tokens(tgt_in[0].tolist())))" ] }, { "cell_type": "code", "execution_count": 17, "id": "ec3a11f5", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:26:49.358609Z", "iopub.status.busy": "2026-10-18T03:26:49.358489Z", "iopub.status.idle": "2026-10-18T03:27:34.423512Z", "shell.execute_reply": "2026-10-18T03:27:34.423006Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "45 s\n", "encoder outputs (steps, batch, hidden): (13, 4, 128) | final state (layers, batch, hidden): (2, 4, 128) | context = the last layer's final state\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:27:34.405532\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.10.7, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Changed from book §10.7] the book's encoder, decoder and Seq2Seq (in seq2seq_common.book_seq2seq), smaller\n", "# (embeddings 64, hidden 128, 2 layers) and 12 epochs instead of 30; trained on the CPU (small GRUs are faster there)\n", "torch.manual_seed(0)\n", "s2s = seq2seq_common.book_seq2seq(mt, embed_size=64, num_hiddens=128, num_layers=2, dropout=0.2, lr=0.005)\n", "t0 = time.time()\n", "d2l.Trainer(max_epochs=12, gradient_clip_val=1, num_gpus=0).fit(s2s, mt)\n", "print(f\"{time.time() - t0:.0f} s\")\n", "enc_out, enc_state = s2s.encoder(src[:4], src_len[:4])\n", "print(\"encoder outputs (steps, batch, hidden):\", tuple(enc_out.shape), \"| final state (layers, batch, hidden):\",\n", " tuple(enc_state.shape), \"| context = the last layer's final state\")" ] }, { "cell_type": "markdown", "id": "f7587d71", "metadata": {}, "source": [ "## Segment 7 — Teacher forcing vs generation; the consistency check (note §8)\n", "**What to watch:** in training the decoder always sees the *correct* previous words (teacher forcing); when\n", "it generates, it sees *its own* previous words, so one early mistake changes everything after it. And scoring\n", "the whole target at once gives the same scores as feeding it one word at a time." ] }, { "cell_type": "code", "execution_count": 18, "id": "0d3acfcc", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:27:34.424912Z", "iopub.status.busy": "2026-10-18T03:27:34.424802Z", "iopub.status.idle": "2026-10-18T03:27:34.430877Z", "shell.execute_reply": "2026-10-18T03:27:34.430513Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "true previous words : where is the train station ?\n", "guesses (teacher-forced): where is the nearest to to \n", "own generation : where is the nearest to the station ?\n" ] } ], "source": [ "# [Added for CS6140] the decoder's next-word guesses with the true previous words vs its own generation\n", "s2s.eval()\n", "sent = seq2seq_common.tokenize(\"¿ dónde está la estación de tren ?\")\n", "ref = seq2seq_common.tokenize(\"where is the train station ?\")\n", "s, sl = mt.src_batch([sent])\n", "t_in = torch.tensor([mt.tgt_vocab[[\"\"] + ref]])\n", "with torch.no_grad():\n", " forced = s2s(s, t_in, sl).argmax(-1)[0] # teacher forcing: one pass, true words in\n", " bos = torch.full((1, mt.num_steps), mt.tgt_vocab[\"\"])\n", " own, _ = s2s.predict_step([s, bos, sl, bos], CPU, mt.num_steps) # the book's generation loop\n", "print(\"true previous words :\", \" \".join([\"\"] + ref))\n", "print(\"guesses (teacher-forced):\", \" \".join(mt.tgt_vocab.to_tokens(forced.tolist())))\n", "print(\"own generation :\", \" \".join(seq2seq_common.ids_to_tokens(own[0].tolist(), mt.tgt_vocab)))" ] }, { "cell_type": "code", "execution_count": 19, "id": "d9d44d2c", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:27:34.432382Z", "iopub.status.busy": "2026-10-18T03:27:34.432264Z", "iopub.status.idle": "2026-10-18T03:27:34.437244Z", "shell.execute_reply": "2026-10-18T03:27:34.436814Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "max |whole - step by step| = 6.67572021484375e-06\n" ] } ], "source": [ "# [Added for CS6140] the whole target at once = one word at a time, carrying the decoder state (max difference)\n", "with torch.no_grad():\n", " state = s2s.decoder.init_state(s2s.encoder(s, sl), sl)\n", " whole, _ = s2s.decoder(t_in, state)\n", " state = s2s.decoder.init_state(s2s.encoder(s, sl), sl)\n", " steps = []\n", " for t in range(t_in.shape[1]):\n", " out, state = s2s.decoder(t_in[:, t:t + 1], state)\n", " steps.append(out)\n", "print(\"max |whole - step by step| =\", (whole - torch.cat(steps, 1)).abs().max().item())" ] }, { "cell_type": "markdown", "id": "b560a2af", "metadata": {}, "source": [ "---\n", "# Lecture RNN-B (needs Segment 1's `data` and Segment 4's `rnn_lm`; Segment 10 needs Segment 6)\n", "\n", "## Segment 8 — Gates: the book's GRU and LSTM (note §6)\n", "**What to watch:** the same data, steps and width; only the cell changes. The gated models reach a clearly\n", "lower perplexity than the plain RNN." ] }, { "cell_type": "code", "execution_count": 20, "id": "f55a1044", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:27:34.438658Z", "iopub.status.busy": "2026-10-18T03:27:34.438538Z", "iopub.status.idle": "2026-10-18T03:27:34.441067Z", "shell.execute_reply": "2026-10-18T03:27:34.440621Z" } }, "outputs": [], "source": [ "# [Book §10.2, §10.1] the book's concise GRU and LSTM, as is\n", "class GRU(d2l.RNN):\n", " def __init__(self, num_inputs, num_hiddens):\n", " d2l.Module.__init__(self)\n", " self.save_hyperparameters()\n", " self.rnn = nn.GRU(num_inputs, num_hiddens)\n", "\n", "\n", "class LSTM(d2l.RNN):\n", " def __init__(self, num_inputs, num_hiddens):\n", " d2l.Module.__init__(self)\n", " self.save_hyperparameters()\n", " self.rnn = nn.LSTM(num_inputs, num_hiddens)\n", "\n", " def forward(self, inputs, H_C=None):\n", " return self.rnn(inputs, H_C)" ] }, { "cell_type": "code", "execution_count": 21, "id": "67117aa5", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:27:34.442365Z", "iopub.status.busy": "2026-10-18T03:27:34.442266Z", "iopub.status.idle": "2026-10-18T03:28:37.990257Z", "shell.execute_reply": "2026-10-18T03:28:37.989397Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "63 s | validation perplexity 58.5\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:28:37.964516\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.10.7, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Changed from book §10.2] GRU on TinyStories, 128 hidden units, 1,000 steps (the book's lr = 4)\n", "torch.manual_seed(0)\n", "gru_lm = d2l.RNNLM(GRU(num_inputs=V, num_hiddens=128), vocab_size=V, lr=4)\n", "t0 = time.time()\n", "d2l.Trainer(max_epochs=20, gradient_clip_val=1, num_gpus=NUM_GPUS).fit(gru_lm, data)\n", "PPL[\"GRU\"] = lm_common.perplexity(gru_lm, data, DEVICE)\n", "print(f\"{time.time() - t0:.0f} s | validation perplexity {PPL['GRU']:.1f}\")" ] }, { "cell_type": "code", "execution_count": 22, "id": "33422422", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:28:37.992030Z", "iopub.status.busy": "2026-10-18T03:28:37.991899Z", "iopub.status.idle": "2026-10-18T03:29:26.167625Z", "shell.execute_reply": "2026-10-18T03:29:26.166709Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "48 s | validation perplexity 79.2\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:29:26.141620\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.10.7, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Changed from book §10.1] LSTM on TinyStories, 128 hidden units, 1,000 steps (the book's lr = 4)\n", "torch.manual_seed(0)\n", "lstm_lm = d2l.RNNLM(LSTM(num_inputs=V, num_hiddens=128), vocab_size=V, lr=4)\n", "t0 = time.time()\n", "d2l.Trainer(max_epochs=20, gradient_clip_val=1, num_gpus=NUM_GPUS).fit(lstm_lm, data)\n", "PPL[\"LSTM\"] = lm_common.perplexity(lstm_lm, data, DEVICE)\n", "print(f\"{time.time() - t0:.0f} s | validation perplexity {PPL['LSTM']:.1f}\")" ] }, { "cell_type": "code", "execution_count": 23, "id": "5a32c6f5", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:29:26.169493Z", "iopub.status.busy": "2026-10-18T03:29:26.169356Z", "iopub.status.idle": "2026-10-18T03:29:26.900559Z", "shell.execute_reply": "2026-10-18T03:29:26.899926Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RNN perplexity 104.1 | cell parameters 691,712\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " once upon a time, there was a little girl named lily. she was very happy and he was very to. it was so excited to be all. she was a\n", "GRU perplexity 58.5 | cell parameters 2,075,136\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " once upon a time, there was a little boy named lily. she loved to play outside with her mommy. \" lily was so excited to be all of the park.\n", "LSTM perplexity 79.2 | cell parameters 2,766,848\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " once upon a time, there was a little girl named lily. she loved to play with his mom. she was so happy that she was very happy. she was a\n", "(the RNN's book learning rate is 1, the gated models' 4; HW6 trains all three the same way with Adam)\n" ] } ], "source": [ "# [Added for CS6140] the three side by side: perplexity, cell parameters, one story each (T = 0.5)\n", "for name, m in [(\"RNN\", rnn_lm), (\"GRU\", gru_lm), (\"LSTM\", lstm_lm)]:\n", " n_cell = sum(p.numel() for p in m.rnn.parameters())\n", " print(f\"{name:5s} perplexity {PPL[name]:6.1f} | cell parameters {n_cell:,}\")\n", " print(\" \", lm_common.generate(m, data.vocab, \"once upon a time\", DEVICE, n_words=30, temperature=0.5))\n", "print(\"(the RNN's book learning rate is 1, the gated models' 4; HW6 trains all three the same way with Adam)\")" ] }, { "cell_type": "markdown", "id": "bf2f96de", "metadata": {}, "source": [ "## Segment 9 — How far back do gradients reach? (note §5–6)\n", "**What to watch:** how much the last prediction's loss depends on the input word $d$ positions earlier,\n", "relative to the most recent word. The plain RNN's curve falls like a straight line on the log scale (a product\n", "of factors); the gated models fall more slowly." ] }, { "cell_type": "code", "execution_count": 24, "id": "66723e2a", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:29:26.902573Z", "iopub.status.busy": "2026-10-18T03:29:26.902433Z", "iopub.status.idle": "2026-10-18T03:29:27.539795Z", "shell.execute_reply": "2026-10-18T03:29:27.539057Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "d = 10: RNN 2.7e-02 GRU 1.9e-02 LSTM 3.2e-02\n", "d = 20: RNN 1.2e-03 GRU 1.5e-03 LSTM 6.1e-03\n", "d = 40: RNN 3.5e-06 GRU 1.8e-04 LSTM 2.8e-04\n", "d = 79: RNN 6.3e-11 GRU 3.7e-07 LSTM 1.1e-06\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:29:27.506470\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.10.7, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Added for CS6140] gradient of the last word's loss w.r.t. the one-hot input d steps back (64 validation windows)\n", "def grad_by_distance(m, T=80, n=64):\n", " m.eval()\n", " starts = torch.randint(len(data.val_ids) - T - 1, (n,), generator=torch.Generator().manual_seed(1))\n", " win = torch.stack([data.val_ids[s:s + T + 1] for s in starts.tolist()]).to(DEVICE)\n", " x = F.one_hot(win[:, :T].T, V).float().requires_grad_() # (T, n, V): time-major, as the book\n", " out, _ = m.rnn(x)\n", " logits = m.output_layer(out) # (n, T, V)\n", " F.cross_entropy(logits[:, -1], win[:, T]).backward()\n", " g = x.grad.norm(dim=-1).mean(dim=1).cpu() # one number per position\n", " m.train()\n", " return (g / g[-1]).flip(0) # index d = d steps back\n", "\n", "\n", "curves = {name: grad_by_distance(m) for name, m in [(\"RNN\", rnn_lm), (\"GRU\", gru_lm), (\"LSTM\", lstm_lm)]}\n", "d2l.plot(list(range(80)), [c.tolist() for c in curves.values()], xlabel=\"d (words back)\",\n", " ylabel=\"relative gradient\", legend=list(curves), yscale=\"log\", figsize=(5, 3))\n", "for d in [10, 20, 40, 79]:\n", " print(f\"d = {d:2d}: \" + \" \".join(f\"{k} {v[d]:.1e}\" for k, v in curves.items()))" ] }, { "cell_type": "markdown", "id": "24f80420", "metadata": {}, "source": [ "## Segment 10 — Greedy translations and BLEU (note §9)\n", "**What to watch:** the book's greedy loop on a few sentences (a dev-sized model: expect mistakes). BLEU counts\n", "matching $n$-grams, each credited at most as often as it appears in one reference, times a brevity penalty.\n", "The standard BLEU-4 and the book's `d2l.bleu` are different formulas." ] }, { "cell_type": "code", "execution_count": 25, "id": "db4af8a5", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:29:27.541597Z", "iopub.status.busy": "2026-10-18T03:29:27.541502Z", "iopub.status.idle": "2026-10-18T03:29:27.550626Z", "shell.execute_reply": "2026-10-18T03:29:27.549949Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tengo hambre . => i am hungry .\n", "me gusta leer libros . => i like to go .\n", "¿ dónde está mi perro ? => where's my ?\n", "ella no sabe nadar . => she doesn't know what she knew .\n", "mañana vamos a la playa con mis amigos . => my father will be my friends .\n" ] } ], "source": [ "# [Changed from book §10.7.5] the book's predict_step on Spanish sentences (cut at )\n", "spanish = [\"tengo hambre .\", \"me gusta leer libros .\", \"¿ dónde está mi perro ?\", \"ella no sabe nadar .\",\n", " \"mañana vamos a la playa con mis amigos .\"]\n", "toks = [seq2seq_common.tokenize(x) for x in spanish]\n", "s, sl = mt.src_batch(toks)\n", "bos = torch.full((len(toks), mt.num_steps), mt.tgt_vocab[\"\"])\n", "preds, _ = s2s.predict_step([s, bos, sl, bos], CPU, mt.num_steps)\n", "for es, p in zip(spanish, preds):\n", " print(f\"{es:42s} => {' '.join(seq2seq_common.ids_to_tokens(p.tolist(), mt.tgt_vocab))}\")" ] }, { "cell_type": "code", "execution_count": 26, "id": "4e9f0a81", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:29:27.552528Z", "iopub.status.busy": "2026-10-18T03:29:27.552421Z", "iopub.status.idle": "2026-10-18T03:29:28.607794Z", "shell.execute_reply": "2026-10-18T03:29:28.607072Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "precisions p1..p4: [1.0, 0.75, 0.667, 0.5]\n", "BLEU-4 (standard, NLTK): 0.579\n", "d2l.bleu (book, k=4) : 0.694 <- a different formula\n" ] } ], "source": [ "# [Added for CS6140] the note's example: standard BLEU-4 (NLTK) vs the book's d2l.bleu\n", "from nltk.translate.bleu_score import sentence_bleu, modified_precision\n", "hyp, ref = \"the cat sat on mat\".split(), \"the cat sat on the mat\".split()\n", "print(\"precisions p1..p4:\", [round(float(modified_precision([ref], hyp, n)), 3) for n in range(1, 5)])\n", "print(f\"BLEU-4 (standard, NLTK): {sentence_bleu([ref], hyp):.3f}\")\n", "print(f\"d2l.bleu (book, k=4) : {d2l.bleu(' '.join(hyp), ' '.join(ref), k=4):.3f} <- a different formula\")" ] }, { "cell_type": "code", "execution_count": 27, "id": "b5460ae9", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:29:28.609739Z", "iopub.status.busy": "2026-10-18T03:29:28.609502Z", "iopub.status.idle": "2026-10-18T03:29:28.612519Z", "shell.execute_reply": "2026-10-18T03:29:28.611931Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "unigram precision 3/4 -> 3 of 4 words matched (summing over references would wrongly give 4)\n", "in HW6 (reference run): BLEU about 27.9; by Spanish length 1-4 / 5-6 / 7-8 / 9-12 tokens: about 32.4 / 37.0 / 30.1 / 18.6\n" ] } ], "source": [ "# [Added for CS6140] clipping with two references: \"the\" is credited at most twice (its max in ONE reference)\n", "hyp = \"the the the cat\".split()\n", "refs = [\"the cat is the best\".split(), \"a cat the dog the\".split()]\n", "p1 = modified_precision(refs, hyp, 1)\n", "print(f\"unigram precision {p1} -> {p1.numerator} of {p1.denominator} words matched (summing over references \"\n", " f\"would wrongly give 4)\")\n", "print(\"in HW6 (reference run): BLEU about 27.9; by Spanish length 1-4 / 5-6 / 7-8 / 9-12 tokens: \"\n", " \"about 32.4 / 37.0 / 30.1 / 18.6\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.0" } }, "nbformat": 4, "nbformat_minor": 5 }