{ "cells": [ { "cell_type": "markdown", "id": "cc7ea928", "metadata": {}, "source": [ "# The Transformer and GPT — lecture demos (Parts A and B, plus the 12/14 appendix)\n", "\n", "Laptop demos for the TRANSFORMER note (`TRANSFORMER_claude_1017.pdf`), one segment per board block of the lecture\n", "plan. The code is the **d2l book's** (chapter 11.7) wherever the book has it; the first line of every code cell\n", "says whether it is the book's code as is, the book's code with one change, or something added for CS6140.\n", "Part A = the second half of the extra lecture, Part B = Thu 12/10.\n", "\n", "| Seg. | Lecture | Board topic | Note | Time on an M1 Max |\n", "|---|---|---|---|---|\n", "| 1 | A | The big picture: the book's Transformer on Spanish → English | §1 | 3 s |\n", "| 2 | A | The wiring: three attentions and their masks | §1 | 0 s |\n", "| 3 | A | Layer norm = batch norm over the other axis | §2 | 0 s |\n", "| 4 | A | Add & Norm, the FFN, the encoder block; parameter counts | §3–4 | 0 s |\n", "| 5 | B | The decoder block: the future stays hidden | §5 | 0 s |\n", "| 6 | B | Train in parallel, generate in a loop | §6 | 50 s |\n", "| 7 | B | Delete the encoder: a tiny GPT on TinyStories | §7 | 39 s |\n", "| 8 | B | Pre-norm vs post-norm | §8 | 0 s |\n", "| 9 | B | Sampling: greedy, temperature, top-k | §9 | 1 s |\n", "| 10 | 12/14 | Three kinds of Transformer: attention patterns | §10 | 0 s |\n", "\n", "**What differs from the book**\n", "\n", "| | Segments | What |\n", "|---|---|---|\n", "| Changed | 1, 2, 6 | **Spanish → English** (HW7's Tatoeba data, dev-sized: 10,000 pairs) instead of the book's French → English; `seq2seq_common.TatoebaEsEn` is a `d2l.MTFraEng` subclass, so the book's code runs on it |\n", "| Changed | 1, 6 | width 128 (FFN 256), 10 epochs instead of 30, so training takes about a minute |\n", "| Changed | 3 | the note's toy $X=[[1,2,3],[2,4,6]]$ instead of the book's 2×2 example |\n", "| Changed | 6 | test translations scored with **corpus BLEU-4** (as in HW6/HW7) instead of the book's per-sentence `d2l.bleu` |\n", "| Added | 2 | shapes of the three attentions' weights and their three masks, as heatmaps |\n", "| Added | 4 | parameter count of one block (`nn.TransformerEncoderLayer`): 3,280 at $d=16$, 789,760 at $d=256$ |\n", "| Added | 5 | the \"change the last English word\" test with `nn.TransformerDecoderLayer` and a causal mask |\n", "| Added | 6 | one teacher-forced pass vs the step-by-step generation loop (time); the embedding scale against the positional encoding |\n", "| Added | 7 | GPT = decoder-only: Hugging Face's GPT-2 class, tiny (width 128, 2 blocks, context 64), trained about a minute on HW6/HW7's TinyStories; tied output layer; perplexity (the book has no GPT code) |\n", "| Added | 8 | `nn.TransformerEncoderLayer(norm_first=False / True)`: what comes out of each |\n", "| Added | 9 | the note's sampling table via Hugging Face's logits warpers; stories with `generate` (greedy, temperature 0.5 / 1.0 / 1.5, top-10) |\n", "| Added | 10 | attention-pattern grids for encoder-only, decoder-only and encoder–decoder models |\n", "\n", "Needs `transformers` (HW7's setup). Run the setup cells first; segments 2, 5 and 6 need segment 1, segments 8–9\n", "need segment 7. Numbers vary a little from run to run. The models here are tiny and trained for a minute: the\n", "HW7 numbers quoted along the way come from the full runs." ] }, { "cell_type": "code", "execution_count": 1, "id": "cbe2789c", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:25.701254Z", "iopub.status.busy": "2026-10-18T03:35:25.701080Z", "iopub.status.idle": "2026-10-18T03:35:27.563436Z", "shell.execute_reply": "2026-10-18T03:35:27.562946Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "device: mps\n" ] } ], "source": [ "# [Book §11.7] imports, plus the device setup the HW notebooks use (d2l's GPU helpers only know CUDA)\n", "import sys, time, math, collections\n", "import torch\n", "from torch import nn\n", "from torch.nn import functional as F\n", "import matplotlib.pyplot as plt\n", "from d2l import torch as d2l\n", "\n", "\n", "def get_device():\n", " if torch.cuda.is_available():\n", " return torch.device(\"cuda\")\n", " if torch.backends.mps.is_available():\n", " return torch.device(\"mps\")\n", " return torch.device(\"cpu\")\n", "\n", "\n", "DEVICE = get_device()\n", "if DEVICE.type == \"mps\": # point d2l's GPU helpers at Apple's GPU\n", " d2l.num_gpus = lambda: 1\n", " d2l.gpu = lambda i=0: DEVICE\n", " d2l.try_gpu = lambda i=0: DEVICE\n", "NUM_GPUS = 1 if DEVICE.type != \"cpu\" else 0\n", "CPU = torch.device(\"cpu\")\n", "print(\"device:\", DEVICE)" ] }, { "cell_type": "code", "execution_count": 2, "id": "1e7272de", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:27.565009Z", "iopub.status.busy": "2026-10-18T03:35:27.564864Z", "iopub.status.idle": "2026-10-18T03:35:27.569844Z", "shell.execute_reply": "2026-10-18T03:35:27.569348Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "shared modules from /Users/vip/Dropbox/CS6140/2.5_NN/HW7 | data in /Users/vip/Dropbox/CS6140/data\n" ] } ], "source": [ "# [Added for CS6140] HW7's shared data modules and the course data folder (as in the HW starters)\n", "import os\n", "import urllib.request\n", "\n", "# The shared data folder: cs6140/data, next to your HW folders (NN setup page, section 3).\n", "DATA = next((d for d in [\"../data\", \"../../data\", \"../../../data\"] if os.path.isdir(d)), \"../data\")\n", "COURSE_DATA_URL = \"https://www.khoury.northeastern.edu/home/vip/teach/MLcourse/data\"\n", "\n", "\n", "def course_file(name, fallback_url=None):\n", " \"\"\"DATA/name. Downloaded the first time from the course website (or from fallback_url).\"\"\"\n", " path = os.path.join(DATA, name)\n", " if not os.path.exists(path):\n", " os.makedirs(os.path.dirname(path), exist_ok=True)\n", " for url in [f\"{COURSE_DATA_URL}/{name}\"] + ([fallback_url] if fallback_url else []):\n", " try:\n", " print(f\"downloading {name} from {url} ...\")\n", " urllib.request.urlretrieve(url, path + \".part\")\n", " os.replace(path + \".part\", path)\n", " break\n", " except OSError as err:\n", " print(f\" failed: {err}\")\n", " else:\n", " raise FileNotFoundError(f\"could not download {name}: put it into {os.path.abspath(DATA)}\")\n", " return path\n", "\n", "_places = [\".\", \"../hw7\", \"../../../2.5_NN/HW7\"] # next to this notebook, your HW7 folder, the course tree\n", "COMMON = next((p for p in _places if os.path.exists(f\"{p}/seq2seq_common.py\")), None)\n", "if COMMON is None:\n", " raise FileNotFoundError(\"lm_common.py / seq2seq_common.py (they come with the HW7 starter) not found in: \"\n", " + \", \".join(os.path.abspath(p) for p in _places))\n", "sys.path.insert(0, COMMON)\n", "import lm_common\n", "import seq2seq_common\n", "course_file(\"sentence_pairs_large.tsv\") # the Spanish-English pairs (24 MB, downloaded once)\n", "print(\"shared modules from\", os.path.abspath(COMMON), \"| data in\", os.path.abspath(DATA))" ] }, { "cell_type": "markdown", "id": "68be0081", "metadata": {}, "source": [ "---\n", "# Part A (second half of the extra lecture)\n", "\n", "## Segment 1 — The big picture: the book's Transformer on Spanish → English (note §1)\n", "**What to watch:** the parts are all familiar (embeddings, positional encoding, multi-head attention, residual +\n", "norm, an MLP per position, the output layer); the model is just their wiring. Every block keeps the shape\n", "(batch, positions, width)." ] }, { "cell_type": "code", "execution_count": 3, "id": "66a1991e", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:27.571168Z", "iopub.status.busy": "2026-10-18T03:35:27.571094Z", "iopub.status.idle": "2026-10-18T03:35:30.246661Z", "shell.execute_reply": "2026-10-18T03:35:30.246239Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tatoeba es->en, both sides <= 12 tokens: 243,067 pairs, 222,600 distinct Spanish sentences.\n", "train 10,000 pairs (of 237,612 available) | val 1,093 pairs | test 2,000 Spanish sentences (1.09 references each) | final test 2,000 (kept untouched)\n", "vocabularies: Spanish 3,566, English 2,884 tokens (min frequency 2); num_steps = 13\n", "batch shapes: src (128, 13) | decoder input (128, 13) | labels (128, 13)\n" ] } ], "source": [ "# [Changed from book §10.5, §11.7.6] HW7's Spanish-English pairs (dev-sized: 10,000 training pairs) instead of French\n", "mt = seq2seq_common.TatoebaEsEn(root=DATA, batch_size=128, num_train=10_000)\n", "print(mt.describe())\n", "src, tgt_in, src_len, label = next(iter(mt.train_dataloader()))\n", "print(\"batch shapes: src\", tuple(src.shape), \"| decoder input\", tuple(tgt_in.shape), \"| labels\", tuple(label.shape))" ] }, { "cell_type": "code", "execution_count": 4, "id": "2a7bd4ab", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.248115Z", "iopub.status.busy": "2026-10-18T03:35:30.248011Z", "iopub.status.idle": "2026-10-18T03:35:30.253094Z", "shell.execute_reply": "2026-10-18T03:35:30.252682Z" } }, "outputs": [], "source": [ "# [Book §11.7.5] the book's decoder block and decoder (in the book's text; the encoder is in the d2l package)\n", "class TransformerDecoderBlock(nn.Module):\n", " # The i-th block in the Transformer decoder\n", " def __init__(self, num_hiddens, ffn_num_hiddens, num_heads, dropout, i):\n", " super().__init__()\n", " self.i = i\n", " self.attention1 = d2l.MultiHeadAttention(num_hiddens, num_heads, dropout)\n", " self.addnorm1 = d2l.AddNorm(num_hiddens, dropout)\n", " self.attention2 = d2l.MultiHeadAttention(num_hiddens, num_heads, dropout)\n", " self.addnorm2 = d2l.AddNorm(num_hiddens, dropout)\n", " self.ffn = d2l.PositionWiseFFN(ffn_num_hiddens, num_hiddens)\n", " self.addnorm3 = d2l.AddNorm(num_hiddens, dropout)\n", "\n", " def forward(self, X, state):\n", " enc_outputs, enc_valid_lens = state[0], state[1]\n", " # During training all tokens of a target sentence are processed at once (state[2][i] is None); when\n", " # decoding token by token, state[2][i] holds this block's inputs for the positions decoded so far\n", " if state[2][self.i] is None:\n", " key_values = X\n", " else:\n", " key_values = torch.cat((state[2][self.i], X), dim=1)\n", " state[2][self.i] = key_values\n", " if self.training:\n", " batch_size, num_steps, _ = X.shape\n", " # (batch_size, num_steps), every row is [1, 2, ..., num_steps]: position t sees positions 1..t\n", " dec_valid_lens = torch.arange(1, num_steps + 1, device=X.device).repeat(batch_size, 1)\n", " else:\n", " dec_valid_lens = None\n", " X2 = self.attention1(X, key_values, key_values, dec_valid_lens) # masked self-attention\n", " Y = self.addnorm1(X, X2)\n", " Y2 = self.attention2(Y, enc_outputs, enc_outputs, enc_valid_lens) # cross-attention to the encoder\n", " Z = self.addnorm2(Y, Y2)\n", " return self.addnorm3(Z, self.ffn(Z)), state\n", "\n", "\n", "class TransformerDecoder(d2l.AttentionDecoder):\n", " def __init__(self, vocab_size, num_hiddens, ffn_num_hiddens, num_heads, num_blks, dropout):\n", " super().__init__()\n", " self.num_hiddens = num_hiddens\n", " self.num_blks = num_blks\n", " self.embedding = nn.Embedding(vocab_size, num_hiddens)\n", " self.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout)\n", " self.blks = nn.Sequential()\n", " for i in range(num_blks):\n", " self.blks.add_module(\"block\" + str(i), TransformerDecoderBlock(\n", " num_hiddens, ffn_num_hiddens, num_heads, dropout, i))\n", " self.dense = nn.LazyLinear(vocab_size)\n", "\n", " def init_state(self, enc_outputs, enc_valid_lens):\n", " return [enc_outputs, enc_valid_lens, [None] * self.num_blks]\n", "\n", " def forward(self, X, state):\n", " X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))\n", " self._attention_weights = [[None] * len(self.blks) for _ in range(2)]\n", " for i, blk in enumerate(self.blks):\n", " X, state = blk(X, state)\n", " self._attention_weights[0][i] = blk.attention1.attention.attention_weights # decoder self-attention\n", " self._attention_weights[1][i] = blk.attention2.attention.attention_weights # cross-attention\n", " return self.dense(X), state\n", "\n", " @property\n", " def attention_weights(self):\n", " return self._attention_weights" ] }, { "cell_type": "code", "execution_count": 5, "id": "9b148252", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.254282Z", "iopub.status.busy": "2026-10-18T03:35:30.254208Z", "iopub.status.idle": "2026-10-18T03:35:30.362257Z", "shell.execute_reply": "2026-10-18T03:35:30.361736Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "source embedding 456,448\n", "encoder blocks 263,936\n", "target embedding 369,152\n", "decoder blocks 395,520\n", "output layer 372,036\n", "total 1,857,092\n" ] } ], "source": [ "# [Changed from book §11.7.6] the book's model, width 128 / FFN 256 (book: 256 / 64), 2 + 2 blocks, 4 heads\n", "torch.manual_seed(0)\n", "num_hiddens, num_blks, dropout, ffn_num_hiddens, num_heads = 128, 2, 0.2, 256, 4\n", "encoder = d2l.TransformerEncoder(len(mt.src_vocab), num_hiddens, ffn_num_hiddens, num_heads, num_blks, dropout)\n", "decoder = TransformerDecoder(len(mt.tgt_vocab), num_hiddens, ffn_num_hiddens, num_heads, num_blks, dropout)\n", "model = seq2seq_common.BookSeq2Seq(encoder, decoder, tgt_pad=mt.tgt_vocab[\"\"], lr=0.001)\n", "with torch.no_grad():\n", " model(src[:2], tgt_in[:2], src_len[:2]) # one pass creates the lazy output layer\n", "parts = {\"source embedding\": encoder.embedding, \"encoder blocks\": encoder.blks,\n", " \"target embedding\": decoder.embedding, \"decoder blocks\": decoder.blks, \"output layer\": decoder.dense}\n", "for name, m in parts.items():\n", " print(f\"{name:17s} {sum(p.numel() for p in m.parameters()):>10,}\")\n", "print(f\"{'total':17s} {sum(p.numel() for p in model.parameters()):>10,}\")" ] }, { "cell_type": "code", "execution_count": 6, "id": "0e536ff0", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.363994Z", "iopub.status.busy": "2026-10-18T03:35:30.363871Z", "iopub.status.idle": "2026-10-18T03:35:30.378641Z", "shell.execute_reply": "2026-10-18T03:35:30.378079Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "token IDs (2, 13)\n", "embeddings x sqrt(d) + positions (2, 13, 128)\n", "after encoder block 0 (2, 13, 128)\n", "after encoder block 1 (2, 13, 128)\n" ] } ], "source": [ "# [Added for CS6140] the shape walk through the encoder: every block keeps (batch, positions, width)\n", "encoder.eval()\n", "with torch.no_grad():\n", " X = src[:2]\n", " print(\"token IDs \", tuple(X.shape))\n", " H = encoder.pos_encoding(encoder.embedding(X) * math.sqrt(num_hiddens))\n", " print(\"embeddings x sqrt(d) + positions \", tuple(H.shape))\n", " for i, blk in enumerate(encoder.blks):\n", " H = blk(H, src_len[:2])\n", " print(f\"after encoder block {i} \", tuple(H.shape))\n", "encoder.train();" ] }, { "cell_type": "markdown", "id": "3163722a", "metadata": {}, "source": [ "## Segment 2 — The wiring: three attentions and their masks (note §1)\n", "**What to watch:** encoder self-attention (Spanish looks at Spanish, padding hidden), decoder *masked*\n", "self-attention (English looks only backwards), and cross-attention (English queries, Spanish keys and values:\n", "\"the decoder asks, the encoder answers\"). Before training the weights are nearly uniform; the masks already\n", "show." ] }, { "cell_type": "code", "execution_count": 7, "id": "0600b433", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.380574Z", "iopub.status.busy": "2026-10-18T03:35:30.380461Z", "iopub.status.idle": "2026-10-18T03:35:30.387918Z", "shell.execute_reply": "2026-10-18T03:35:30.387470Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "encoder self-attention (8, 13, 13) queries = keys = Spanish positions\n", "decoder masked self-attn (8, 13, 13) queries = keys = English positions\n", "cross-attention (8, 13, 13) queries = English, keys/values = Spanish\n", "first sentence: a tom le empezó a dar hambre . \n" ] } ], "source": [ "# [Added for CS6140] the shapes of the three attentions' weights (d2l stores them as (batch x heads, queries, keys))\n", "model.train() # training mode: the decoder uses its causal valid_lens\n", "with torch.no_grad():\n", " model(src[:2], tgt_in[:2], src_len[:2])\n", "enc_w = encoder.attention_weights[0]\n", "dec_self_w, cross_w = decoder.attention_weights[0][0], decoder.attention_weights[1][0]\n", "print(\"encoder self-attention \", tuple(enc_w.shape), \" queries = keys = Spanish positions\")\n", "print(\"decoder masked self-attn \", tuple(dec_self_w.shape), \" queries = keys = English positions\")\n", "print(\"cross-attention \", tuple(cross_w.shape), \" queries = English, keys/values = Spanish\")\n", "print(\"first sentence:\", \" \".join(mt.src_vocab.to_tokens(src[0, :int(src_len[0])].tolist())))" ] }, { "cell_type": "code", "execution_count": 8, "id": "2f1175e1", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.389411Z", "iopub.status.busy": "2026-10-18T03:35:30.389290Z", "iopub.status.idle": "2026-10-18T03:35:30.519438Z", "shell.execute_reply": "2026-10-18T03:35:30.518820Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7wAAAElCAYAAADHte6VAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvgI3uAAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAXCxJREFUeJzt3QeYE1X79/Gb3ovSpHcUKQJSBURQkKKgqI+CgF1RRAQLoCIo0sSuSFOw8oiKBQsq0pSmqCCIgAgqVXpdOuS9fuf/Tp7sso3dzaZ9P9cV2Ewyk8kke3buOefcdxafz+czAAAAAACiTNZQ7wAAAAAAAMFAwAsAAAAAiEoEvAAAAACAqETACwAAAACISgS8AAAAAICoRMALAAAAAIhKBLwAAAAAgKhEwAsAAAAAiEoEvGHmxIkTtn//fgsX+/bts5MnT562/PDhw3bo0KEk1zt16pQdOHAgZMcps49jUscpHN+Pz+cLq+8YkBR9T/Xdj7XXTqy91S0h7d/evXuTXVftcHraplC8v4MHD4bNsQcARD4C3jAzd+5cq1atmoWL0qVL25IlS/z3//33X2vUqJEVK1bMevfuneR6zzzzjA0YMCBkxynh4+kNSFNSpEgRW7FiRZKPp/T6Z/p+0hvwNm7c2H7++ecM2R4QLJUqVbL58+fH3GsnpLZ24MCB8Zbdf//9VrhwYatQoUKS661bt85q1aoV9sFjwvf39NNP26OPPhrSfQIARA8CXpyRl19+2cqXL+96DV5//fVEn6Or86NGjbKHH344bI5ujRo1bOnSpTH7+oGyZs1qDz74oD322GOh3hUAaaCLa2+88YZt2LAh2R7eIUOGWK9evSxXrlwRdZwVzE+YMMG2bdsW6l0BAEQBAl6ckd9++831DmbJkiXJ50ydOtVq167tAuNwsWnTJqtfv37Mvn5C1157rc2bN8/++uuvUO8KgDS0wxrxcfbZZyf5nN27d9u0adOsW7duEXd89b5atmzpgnoAANKLgDfBvNPEhp0GDkc9fvx4skNFkxq2mtx6qXk8NfumOVB6XlpeI6ntJ5zTppMoPU+9Cklt75NPPrHWrVsnuZ+Jradta5txcXHJ7kNaj2PCIcWp/Uy9x85kDm1i20tqSHNmfC8S20bBggVdAP7ZZ58lu30gs6Vm+G16fm+S235aX1v5DI4cOeLfhvdzYo4dO3bG2w+kbWtqSbZs2ZJtM2fMmGFVqlSxkiVLpup1tF1tT22H/pYlpNc5evRoiu1aYu8vpW0n5rLLLrNPP/00Vc8FACA5BLz/f55Tu3btrFChQm5OVOfOnW3Hjh3x5mcOHz7cKleubPnz53dX1n/66Sf/42vXrrX27dtbgQIF3HP79evnP5n45Zdf3JxXb9s9e/aMd4KyaNEiO//88/3bnT179hnv21NPPeX2TSc2a9asibe+9uO2226zvHnz2llnnWUtWrSI95yUth+oWbNmbn8HDx7s5o0lFSz98MMPdsEFF8Rbpv188skn3Xp6r/Xq1Ys351Xb1mPFixe3c845x1555ZV466d0nFJ6POFc5JQ+0wULFth5553nHqtataoNGjQoxTm0Oh5JbS/h66f3/aT3Oyt169Z1nxUQDhQQXXPNNe77qnZAQ+4DgyMFW5rnWaJECdfWapqARikEBlV6XO2HHldb541g0MWqW2+91f2+6GJP06ZNXS9pRr32nXfe6drZSy+91OU3ePbZZ097f2+//baVKVPG7YNGv7z77rup3n4gTRfR/Fa1J2oz9dqpbYeTO0barrZXrlw591iXLl1cgOrR8Rk7dmy87el9qK1K6f2ltO3EqH1SmxXuCbcAABHAF+MOHz7sq1Kliq9///6+Q4cO+eLi4nzdunXzXX311f7nZMuWzdeyZUvftm3bfCdOnPDdfffdvqZNm7rHtE6lSpV8d911l2/fvn2+gwcP+p577jnfmjVrfPv37/edc845vkceecR39OhR34YNG3y1atXy3XvvvW5dPV6sWDHf8OHDfcePH/f98ccfblslSpQ4o31r3Lixb8uWLYm+v/fff99Xvnx539atW30nT570LViwwPfKK6+kevv58uXzLVq0yH+/RYsWvpdffjnJ46n3kSVLFt/ixYvjLdd+1q5d27dx40bfkSNHfPfff797rzouCf3000++okWL+reR0nFK6fHE3kdyn6k+R73+6NGj3fb+/PNPX+XKleNtL6Hktpfw9dP7ftL7nfUMHTrUPQcIBzfddJOvefPmvh07drjvdI8ePRRx+ubMmeMef+CBB3wNGzb0/fXXX75Tp065tu2ss85y33FRm1KjRg3fypUrXVs3b9483+uvv+4eu/XWW30NGjRw7aTanz59+vjKlSvnXicjXvvGG2/05cmTxzd79mz3eELHjh3z5cqVy/fRRx+5+5s3b/b17dvX/3hK27/tttvcPnvGjh172u9zQtdee607JoGSO0aBdu7c6Wvfvn28fbz88st9zz//fLznFSpUyPf999+n+P5S2nbC9ydr1651n4F3DAAASKuYD3g///xzF9zs3r3bt2fPHndbsWKFL2vWrO7EyAseZs2a5T9o8+fPd3/oZfr06b6zzz7b/cFPSH/8tW0FHJ4PP/zQV7hwYffztGnTfKVLl453gjRmzBh/YJPafdPzkjJz5kxf2bJlfatWrTrtsdRs/0wDXp1Eaf2FCxfGW679nDp1qv++gjadIH733Xfxnqfl2g+dgD7xxBOpOk4pPZ7Y+0juM9X2dMwCt/fqq6+mGPAmtb2Er5/e95Pe76xHx/fSSy9N8j0BmUXtRu7cuV0A5tm1a1e8oLNAgQK+Tz/91P+d101B4ttvv+1+V/Lnz59oW6jH1NZ89dVX/mX6PdHvpJal97W9gFcXnZJ7f8WLF/dNnjw53t8DT0rbT0vA+5///Md33333xTsOSR2jQNq/vXv3ur9t1atXT1XAm9L7S2nbiQW8umisz0AXIQAASI/sFuOU5XLPnj1WsWLFeMs17ErzpLzES4HJQZTx0pujpWREGsaVI0eO07a9detWN5RVc6082p6Gz2nOlx7XuoEJoDTk60z3Ta+R3Dyo0aNH20MPPWSbN292yaQ0HE7DdFO7/TPNAKzhctu3bz/tscD3ljt3bjd0UMdAF140THr8+PHu2OTJk8cd35tuusk9N6XjlNLjSUnqM/U+t8DtlS1bNs3bSyi97ye931mPMqAm990BMou+z/p+Bn7P9f3Nly+f/3Flhu/evftpCfPU1qjdUHZ4lRJKbNvKbxDYnun3Qe2U2sT0vrYnud8ltYtz5syx559/3saMGeO2r6HIGiac2u2fKbUhW7Zs8d9P7hjJN9984/5OrFq1yrXPape136mR3PtL67bVPulvQXKJuQAASI2Yn8OrP/6ac7Vr1y53QhB4S03Ap/U1B0onEglp/X/++Sdeog/9wdfraU6tHl+/fn28RCmB82vTu2+e66+/3s0vVd1VzUvt1KlThm4/oSZNmiRaguePP/7w/6w5dQrsdJI5c+ZMNzdMtWZ1rPT6N9xwg3/uVkrHKaXHz5T2SZ9pRm0vofS+n4z63DS/XJ8VEGqac675s4FthIJRL9+B8g/o9vHHH5/2nVfOBO/xxGphK2DSxaDA3yEFmNq+5pWm97VTS3PyJ06c6ObeKjBUgPvrr79m2PZTaoeTO0YKQLt27Wr33HOPe99qnz/44IN482f1N0sXaj36m6fjmNL7S822k2qfGjZsmOqgG0Dm0+91cudHyhmgRKahEMrXTkgZ89NaClLn72pTkT4x/5dESUZ0Zf6WW26xlStX2saNG90X86qrrkr1+gpAbrzxRncioZMm1RD8/fffXe+qTrbUICiA+v77713yo7vvvtutq0zGOono27evC4xnzZplzzzzTIbtmyj5k5JFaX1d7Q+84p8R20+q5I0yhCb0xBNP2MKFC10wp+RdCr51QuNlGtYJnoLgt956y9577z3/eikdp5QeP1Nt2rSxnDlz2gMPPOB6U9O7vYTS+34y4nPbuXOnLVu2LN2fNZBRbr/9dtcLqCBN7WjCZEx6TDVlv/32W9dOqC1Ru6vASPT7ovrSans08kZJlLyThDvuuMP69+/vEjkp4ZteS+32xRdfnCGvnRL9HutC43fffed6LvX3QAkFvR7d9G4/MW3btnXtfWDpseSOkdphJZJSJn4lohowYEC87TVo0MDeeecd1+b8+eef7ph6VQFSen8pbTsxX375pV133XVpfv8Aguurr75yF7jOPffcJJ+jx//++++QfBShfO2E1CYmTByaWvo7pcR/arORDukaEB0lNF9Lc52UzEOJTDT3SUk9PEWKFPEtX77cf3/p0qXx5nNu377dd8cdd/iqVq3q5iW99NJL/seUhOS6665z21XCqqeeesolIgqcp6S5UWXKlPG1bt3aJZSqVq1amvctISU10lxNra/XuPLKK32//fZbqrevuaRLlizx3+/QoYNvwoQJyR5PzY8rVapUvO1oTqnmnV188cVuPzp16uQSWHlzywYOHOirUKGCW69r164uyVLv3r1TfZxSejzh+0jpM9Wc5zZt2rjtaZ6rEkjp+CQlpe0lfP30vp/0fmdfeOEF3/XXX5/k+wEym5fMrmLFir46der4Jk6c6JK1aY6o104owVKTJk1cO9GsWTPflClT/OtrHqnyC9SvX9/Nwb/55pvdPHdRcrzHH3/c/b4oid8NN9zgb38y4rXvvPNO35AhQ5J9f5q3qhwISmSofXzzzTf9j6W0fbWFAwYM8N+fNGmSr23btike0169ern3nZpjNGPGDF+9evVcwjztx4svvujaLY8SMir5l9a78MIL3fxitT1eboLk3l9K2074/jZt2uTaMM1lBhCedG4yfvz4ZJ+j86eEc/8zSyhfOyElQdU5XVqp7VUCXKRdFv2TnoAZSMykSZPc1axXX33V3c+ePbu7X6dOnYg8YCr9pJ5WzVOLdBoqrRq8GlaoudwAopPm9Ldq1cq1vRo1Eik09E/ljdTzDSD8KL+ASmFqRIrysSRFIx2vuOIKN/Ixs4XytRPSKD2N4FGveFqHNffu3TtseqwjUcwnrUJwqOalbpFq5MiRbsi1hvFpGORzzz2XaG3NSKSLDxrODCC6KTGXptdE4gVGAP9HSfXUeaALV5qHr7n9lStXdlMJ7rrrLjctQUk/NWz2008/dXPnX3/9dReM1qxZ001lUP1x0dSxl19+2Z0DKGDV+rVq1UrxsYTmz5/vknkmDHa1fPLkyW4ag5e0LpCmiem9aFqEEm8qGA1M+KcpWt7jem1NwfAu1iX3vtL72poWpiktjRo1csdQyzUNL5Bqo48YMcL9rOmKqmOuqR1enoEzOX6aGvLII4+4aY7qgEhpXe2XPm+9h9QkZcXpYn4OLzJH4cKF42WrDneaP/fuu+9a48aNbfjw4S4A1pxZAACAzKBEngp2pk+fbu3bt3eBnpJ6ipLGff311y4pnCpwKA+A5v8rYZ0S8WkO/I8//uhypXhVGtQRoQShXbp0cesoYNuxY0eKjyWkXCzKBB9IeWouv/xylzxTOQTUq7l48WL/47r4dsEFF7icLdpnBaZ169Z1OUXkt99+sxo1arjM9f/5z3/c8+677z73WErvK72vrYsFL774oo0bN85tx0vuGkhzlRUk69auXTsXPCsg96T2+Gm0oHqfVYlEwW5q1tWFBVWDCczJgDOUjuHQAAAAAIJg3LhxLr+J8gx49u/f7/5fsWKFq1W9ePFi/2Pt2rVzOVA8yl+g+fLKSyDKYzB16lT/49qunpPSYwkpN4zmyAbSaz/44IP++/v27fMVLFjQP4+2c+fO8XKziPK5KLeNXHXVVb5bbrkl3uPee03pfaX3tTXHVnlOAo9zSpRrRTXclRshpePnzeFV/gXlaVi4cGG8baXm2Ov9fPnll6neP8THkGYAAAAgzGgYb9OmTV3tcI/KrAWqV69evNKX3bp1899XxQlNzdIQWlG2X1Wg+PDDD61Zs2auUoM3RDa5xxIqWrSo64kNpF5UZbz3aLixypV5NFx37dq1rgfWo/ve+9F7VVWRQN5jKb2v9L62qHc58Dgn1tuuIdMLFixwZSHVW6weZuVKKFWqVIrHTz3OqkmuaXLa90ApravcK+rR13FH2hDwAgAAAGE4HUzzNpOjoa6BgZ5KgAXSfSWBkw4dOribyq/NnDnTBctffPGFGzad3GMJKQGpnqehwd4cVr226mwHCryvxy+55BI3ZDhhrgHvvWoua2JSel/pfe2ExzExffr0caXiVFq0WLFirvSaAlivPnlKx09DsDV3WIG51itRooR/2ymtq4Be+VcUlCNtmMMLAAAAhBllGVYApDmsnvfeey/J57du3dpee+01i4uLc/eV6ErzW9u0aePujx071gVq1apVs3vuucf1InpzXZN7LCH1UKo3VImeAl974sSJ/nm1s2fPjpc0T72W6t1Uj7V6WnVT4Hjw4EH3+JVXXmljxozxz109duyYffzxx6l6X+l97dTW9dW8W+Vz0eei5FmBUjp+ujCg5GLaV9XWVYKu1K6r99q8efOIyrYfbqKyh1dXnLZs2eKGKniF7wGknaqXaTiNhu14V3MRWWgXgYxFu4hgu+iii+zpp592PZOqHKG/w+op9BJXJaSsvxoarCzOlSpVsuXLl9uQIUPcdmTz5s0umFJJQp0n58mTxyWISumxhNQbquBvypQpduGFF/rLiamXskqVKm47Ck6rV6/uX2fgwIGut1qJpdRTqcBWvbpvvvmm/3ENM9b6ylCsrMTeEOeU3ld6Xzs1lDlZCat0wUE90cpSHSi1x08JtRSfKIBVqSLtc0rr6jWVlAxpF5V1eDdt2nTaFxFA+umKZMLMjIgMtItAcNAuItg0PFc9luqVVNAn6p1UKZ7Aeakeld7R3FIFyQnnfSpYW7FihQv4lPU5sGMouccS0vYV7GoerZ4vmteq+7rAqsBSQar2WYGmR8Gmhu4q87CCvYSvoYBPWaAVsBYpUiTV7ys9r636tpqX6wXvSVFgqvU1HFn7pyzZ6q31el6TOn6JbV9zgRXYenOwk1pXGanVq6wgP5KqnYSbqAx4Na5fX5iNf6y0ggkm9wNyf8n/XflLrxe2ror6g7r/wAErW62Ga5C9OTOIzHbxRstrOS24I19i4XcCoF1ErNOcVtWsDZyPioylXm+NrPMudCBtonJIs3dVRMFuYFFqwJORJ/yx9B1jikDkf3b67gc74I2l3wmAdhGxKjBDNIJDw5yRfkzGAwAAAABEJQJeAAAAAEBUCtuAVxnJlEZcmelGjhzpCj4DAAAAABDRAe9zzz1nt912m6tVdeutt9qkSZOSTI0OAAAAAEBEJK1S0WjV1ho6dKi/5pRqbqnumApvezW3AAAAAACIqIB38eLFrrB2x44d/csaNGhgpUqVspkzZxLwAohZuiCoOoIqAZEzZ85Q7w4AZBjVT1UN1gIFCpD5GkCKVFlXMaNiRJVuiqiA959//nH/q65XIL2ZDRs2JLqO5vcGzvFVgW4AiCaDBw+20aNHu0L1x48ft2HDhlnv3r1DvVsAkCEU7JYtW5ajCeCMbNy40cqUKRNZAa9O5CRXrlzxlusk79ixY4muM2LECHviiScyZf8AILO9/fbbLtj99ttv3SiX6dOnW+fOne28885zuQ4AINKpZ1fmVihn+VPorUmril2aW7AdXrwiqNvP2+ceC7a458YEdfsHd8VZsJW4rVNQt7/rzc8s2M7u2DSo28/eM7Jjp/0HDljZajX8bUdEBbxnn322+3/37t1WtGhR//Jdu3ZZ3bp1E11n4MCB1q9fv3g9vFwlBBAtxo4da9dcc41/SoemfDRv3twtJ+AFEA2yZMni/lewmz9bcALegrmDPxUkR/ZsQd1+3nx5LNiyBfk9ZMkW3O1LwTzxO84y2rHMeA9B/r5mL1jQoqntiKgszV5Q+/PPP/uX7d2719auXZtkwKve4IIFC8a7AUC0zGv75ZdfrEmTJvGWN2vWzH766aeQ7RcAAEAkCLuAt2LFinbJJZe42rveEGb9nD9/fleXFwBiiRIyKEdBkSJF4i3XCJidO3cmuZ7W0WiXwBsAAECsCbuAV958803bs2ePS1SlkkSqw/v+++9b4cKFQ71rAJCpvMyDXn4Djy4IZktmSJVyGxQqVMh/Y5oHAACIRWE3h1fKlStny5Yts1WrVtnhw4etZs2alOAAEJOUjEEB67///htvue4nl5WQ3AYAAABh2sPrqV69utWrV49gF0BM0zSPr7/+Ot6yGTNmuOVJIbcBgMz08ccfW6tWrVwnRdeuXW39+vV8AADCQlgHvAAAs0cffdS+++47e/zxx23JkiXWq1cvV3fugQce4PAACLlPPvnE/vOf/9jVV19tr7/+upuCoUzySjoKAKFGwAsAYa5Bgwb2zTffuGD35ptvts2bN9u8efOsSpUqod41ALAhQ4a4tql3797WqFEje+edd+zQoUM2fvx4jg6AkAvLObwAgPhatGjhbgAQTtSL++uvv9pjjz0Wb0qFhjfPnTvX+vfvH9L9AwAC3lTqmS/p5DBnalzcJr55IcZnAABA+ml6hZxzzjnxluu+pmIkVzpNNw+l0wAEC0OaAQAAkCanTp1y/+fIkSPe8pw5c9rJkyeTXI/SaQAyCwEvAAAA0qRo0aLu/127dsVbvnPnTv9jSZVO27dvn//m9RQDQEYj4AUAAECalC5d2kqWLGmLFi2Kt3zBggVWv379JNejdBqAzELACwAAgDTr2bOnTZgwwf744w93/9VXX3U9trfffjtHFUDIkbQKAAAAafbII4+4cmm1atWyfPnyWfbs2W3KlCl2/vnnc1QBhBwBLwAAANJ+Mpk9u6u5++yzz9qePXusVKlSli1bNo4ogLBAwAsAMeSFrausYMGCEVPGLSWUGAPCR/78+d0NAMIJc3gBAAAAAFGJgBcAAAAAEJUIeAEAAAAAUYmAFwAAAAAQlQh4AQAAAABRiSzNAAAACAsVu11iBXPnDMq2t70/z4LtnEH3BHX7vS6924LtlRE3BnX7+YJcKUAOvPt5ULdf5MrGFnSHDgX/NWIEPbwAAAAAgKhEwAsAAAAAiEoEvAAAAACAqETACwAAAACISgS8AAAAAICoRJbmVBoXtym4n0QU6pmvTIZuj88AAAAAwJmghxcAAAAAEJUIeAEAAAAAUYmAFwAAAAAQlcI64D1+/HiodwEAAAAAEKHCLuA9evSojRkzxmrWrGkFCxa0QoUKWffu3W379u2h3jUAAAAAQAQJu4B3xYoVtmrVKnv//fctLi7O3V+9erVdf/31od41AAAAAEAECbuyRPXr13c3T7ly5WzAgAF27bXX2r59+1yPLwAAAAAAEdfDm5gNGzZYnjx5LH/+/KHeFQAAAABAhAi7Ht7Egt1hw4ZZz549LVu2bEnO+9XNs3///kzcQwAAAABAOArrHt6dO3da+/btrXbt2jZixIgkn6fHNNTZu5UtWzZT9xMAAAAAEH7CNuDdtWuXXXbZZVakSBH77LPPLFeuXEk+d+DAgW5+r3fbuHFjpu4rAAAAACD8hOWQ5t27d7tgV721X375peXLly/Z5ysYTi4gBgAAAADEnrALePfu3WutW7e27Nmz25QpU+z48eNumagub9asYdspDQAAAAAII2EX8P7www+2bt0693ONGjXiPbZo0SKrXr16iPYMAJAa4+I2ZdqB6pmvTFS+LwAAEKUB7+WXX+7v0QUAAAAAIK0YHwwAAAAAiEoEvAAAAACAqETACwAAAACISmE3hxfRI6MTvGRkchqSzwAAEIayZPm/WxAUb1jBgm1Vv+eDuv0XezS0oPP5grv5PXss2PJfUD64L3D4cHC3b2YbPv0pqNuvNMBiBj28AAAAAICoRMALAAAAAIhKBLwAAAAAgKhEwAsAYW79+vXWq1cvq1mzpp177rnWrVs3+/PPP0O9WwAAAGGPgBcAwtxNN93kgt333nvPPvnkEzty5Ig1b97ctm/fHupdAwAAiK6A96233rIZM2a4nz/66CM755xz7LrrrrNTp04FY/8AIOKpN3bQoEHu523btlmdOnXsvPPOs9WrV6dq/e+++87uvvtuF/RWr17dtcM7duywr7/+Osh7DiAWLFq0yF566SX38++//25VqlSxhg0buvYKAGIu4P3ll19szZo17medwL377ru2detW++abb4KxfwAQ8Xbu3GkzZ850P0+YMMGaNWtmd9xxh40cOTJV62dJUKLj8OHD5vP5LHfu3EHZXwCx5Z9//rGFCxe6n5999lm78847rUmTJjZ27NhUra/RJkOGDLE2bdpYhw4dbNSoURYXFxfkvQaAIAW8OsE6evSou+qnW6tWraxevXq2bt26M90UAMQEr92U2bNnW6dOndzJZFrbzQEDBljRokXdyWVS9Hr79++PdwOA1LRRV199tTVq1ChVbdSxY8escePGljVrVnvwwQft9ttvt7ffftvatm1rJ06c4IADCLnsZ7qChrg88cQTrpe3Y8eOrudBjdzx48eDs4cAEOGqVatmmzdvthEjRrjhgpp/u2zZsjS1m+p90cmkppYUKlQoyefptdRWA0BK1HGhUSfqpc2TJ49VrVrVfv7551S1UTly5LCVK1e69TwVKlRw29Q2FDgDQET18Hbu3Nn69OljZcqUsWeeecYfBJ9//vnB2D8AiHh58+a1zz//3Pbt2+eSTqk3pVixYm7o35l4+eWX7dFHH3X5E1q2bJnscwcOHOhez7tt3Lgxne8CQLQqV66cTZkyxSXEmzZtmltWsWJFa9GiRYrrquMjMNj12jyhMwRARPbwyq233hrvfteuXTNqfwAgKunCoG6eypUr+xNZpcaYMWPsoYceciej7du3T/H5uXLlcjcASI3WrVu7m0c9s2ntnR06dKiVLVvWGjRokORzNITaG0YtTLsAEDYBr+ZjPPnkkzZnzhw7cOCAf7lOxG688caM3j8AiApffPGFvfLKK/bvv/+6hFNSu3Ztl3E5JePGjbMHHnjABbtn2isMAKnx5ptvuukSu3bt8rdRmoeb2uR6ntGjR9uHH37oEvUld9GNaRcAwjbgnTx5shuap/T1hQsX9i8vWbJkRu8bAEQFnUB26dLFxo8fb7Vq1Tpt2F9K+vbt60q/qR5voIcfftjdACA9VqxY4S6qKegtX768f3ngeV5qp11o5IqmXShXQUrTLvr16xevh1e9wgAQ8oB3y5Ytru6uymoAAFJXlkjz4RT0psWmTZv8PS6BUhswA0BK53ZNmzZN1wgSpl0AiJqkVcrMvGDBAlLNA8AZZGnOly+frV+/Pk3HrEiRIq4MUcIbAS+AjKCyQio1uXv37jStr3q9TLsAEDU9vPPmzbNFixa5IS+BQ0805O7666/P6P0DgIinUkQa1qzhzMpony1bNre8Zs2a9tprr4V69wDEuCVLlrj8ApUqVXIX6FRuUpTESgmoUhrB0qtXLzf8efDgwe7mUZmjK664Iuj7DwAZGvCqGLmuBCakmmsAgNPpAqHmxiVUoEABDheAkKtbt6699957py3XSJKUKND98ccfE31MUzkAIOICXp24BSY0AAAkL3/+/IleKASAcKBpE7qlRfbs2a1+/foZvk8AELI5vPLpp5/axRdf7Ia+tGnTxr7//vsM2yEAiEaav3vDDTdY1apV3cnhs88+m2giKgAIhWXLltmVV17paoQ3adIkVSXTACAqe3g1bOX22293ZYk0/0wJrJTI6tdff7Vy5coFZy8B1SKN25Rhx6FnvjJhuV+ITidPnrT27du7k8kBAwbY9u3bXd6D3Llzu7lvABBKe/futcsvv9wefPBBGzZsmK1bt87uvfde1+tL7W8AMRfwzpgxw3r27Okvr6EkLPPnz7c5c+acViMSAGD2559/uqB39OjR/sMxatQoV8aDgBdAqP3www9Wu3Zte+ihh9x9/bx582b7/PPPCXgBxN6QZpXW2Lp1a7xlyuynOWoAgMTbzX379tnhw4dpNwGEZRulskSnTp3yL+PcDkDM9vDeeOONLptfjx49XHmNxYsX26ZNm6xdu3bB2UMAiHBlypSxFi1auNwHV111le3YscNlbf74449DvWsA4JLqKehVXpZLL73U/vrrL/vwww9dGUoAiLmAt2TJkrZ8+XKbPHmyG+7SsmVLe/vtty1v3rxBS6Lw22+/WfPmzckODSBiqeTHf//7X/v555+tUKFCbipIjRo1Qr1bAOAyLWtq2htvvOHqhisni86/QpGb5eiPK+xojjM+PU2VbAXyWLDtPHQsqNuv3qKZBZvv12VB3f7mOast2Ep3bhLcF8iVK7jbN7Nyt9OZmFHS1KIUL17c+vfvb8G2ZcsWV7BcgbVOFCmHBCBSZcuWzbp16+ZuABBulERPOVoAIGYDXiVbKVWqlAtC33333dMeV6IDDXfOKJpHohPDfv362QMPPJBh2wWAzLJixQrXdqp97N69+2mPKzEMpT8AhMrXX3/tenY1Wi+xjoy2bdvayJEjQ7JvAJDpAW/Xrl0tZ86cduzYMWvduvVpj5cuXdoyktLia5j0PffcQ8ALICKpnuWgQYNc+6ihgglpzhwAhEqjRo3svPPOc9MsEmujzjrrrJDsFwCEJEuzTtiKFStm77zzjptTW6dOHf9NVwh1yyia2zZu3DibNGlSqp5/9OhR279/f7wbAISaLtpVrVrVlSVST29gu5k1a1YbMmRIqHcRQAwrXLiwmy6mBKQavRfYRilL8yuvvBLqXQSAzC9LpJJEO3fujLds7dq1dujQofTvjZnt2bPHDY2eMGGCmyucGiNGjHBXJ71b2bJlM2RfACAjqH1ct25dvGXbt28/rcQbAITC3r17bePGjfGWqQLHrl27+EAAxM6Q5ueee86mTJniGsRcuXK5nl45ceKEbdiwwR5++OEM2aHBgwe7ITQKfPUax48f9/f6KphNrPzRwIED3Vxfj3p4CXoBhJpGw9x8880WFxfn2s769ev7H1PZj0cffTSk+wcgtn3zzTf2yCOPuHMu3bw2yufzuYt0EydODPUuAkDmBbzXXXedqyE5fvx4K1GihKslKZrXW7FiRStQoEDGpFuvXt12795tX331lbvvFUH/9ddf7eyzz0404FUArhsAhBO1jZqesXr1atd2Pv/88255lixZ7Jxzzsnw3AcAcCYaNGjg2qh58+a5mrsDBgzwZ5VX/XBNZQOAmAl41WOqW2APRTDcfffd7uY5cuSIK0nUq1cvu+GGG4L62gCQkZSUSm2mbpQjAhBuNKLOa6MAIFqFbVkiAIh0lCUCEM4oSwQgFoRtWSKPhtUokK5QoUJQtg8AwUJZIgDhjLJEAGJBqgPewIA2M+ed5ciRw58gCwAisSyRqMwHAIRbWSLdhDYKQLQ647JEb731ls2YMcP9/NFHH7nEK0po5SWXAgDEpzq8gwYNcj9v27bNnVied955LpkVAISaEla99NJL7ufff//dqlSpYg0bNnTtFQDEXMD7yy+/2Jo1a9zPOoHTfF7VklRqewDA6VS7fObMme5n1Rhv1qyZ3XHHHTZy5EgOF4CQ++eff2zhwoXu52effdbuvPNOa9KkiY0dOzbUuwYAmR/w5s6d244ePequ+unWqlUrq1evnqvXBgBIut2U2bNnW6dOndzJJO0mgHBso66++mo3v5c2CkBMzeH1aIjLE0884Xp5O3bs6OpJZs2a1Y4fP27RzJeBQ7bvLVA+w7Y1Jm5jhm0rloyL2xTqXUAMqVatmm3evNlGjBjhhgs2b97cli1bFvXtJoDIoI4LjToZMmSI5cmTx+Ue+Pnnn2mjAMRmD2/nzp2tT58+riD5M8884w+Czz///GDsHwBERfKqzz//3Pbt22effPKJ600pVqyYdejQIdS7BgBWrlw5mzJlih05csSmTZvmjkjFihWtRYsWHB0AsdfDK7feeutpJYsAAEnThUHdEpYsAoBwoJKTgWUnNaRZNwCIuR5e+fTTT+3iiy+2SpUqWZs2bez777/P+D0DgCiyfv16u+GGG9xQwfr167vEMD6fL9S7BQCOpllceeWV7mKccgyoKgcAxGTA++OPP9rtt99ud999twt8NcRZc3k3bNgQnD0EgAh38uRJa9++vZUtW9Y++OADGz58uE2aNMleffXVUO8aANjevXvt8ssvd50ZH3/8sT388MM2cOBA++KLLzg6AGJvSLNq8Pbs2dO6dOni7teqVcvmz59vc+bMsZtuuikY+wgAEV+HV0Hv6NGj/ctGjRplY8aMsV69eoV03wDghx9+sNq1a9tDDz3kDoZ+VqI95R4g1wCAmOvhzZcvn6u7G+jff/+1/PnzZ+R+AUDUULuphFWHDx/2L6PdBBBObZRKTZ4KqEhBGwUgZnt4b7zxRqtbt6716NHDZWZevHixbdq0ydq1axecPQSACKes9sp2quGCV111le3YscPefPNNN3QQAEKtcePGLuhVXpZLL73U/vrrL/vwww9t0aJFod41AMj8Ht6SJUva8uXLrUaNGrZlyxZr2bKlLVmyxJXdAAAk7r333nMl3Xbu3GmFChVyU0EuueQSDheAkMuePbubmnbttde6UXwqU6QkVueee26odw0AQlOWqHjx4u7Ebfv27S4AzpEjR/r3BACiWLZs2dwIGV0kLFCggBUsWDDUuwQAfqoPfscdd7jOjCJFitCRASB2A16V0bj//vvttddes8KFC9uBAwfskUcesQEDBgRnDwEgCiirvbLbqw1Vu6nhzVOmTHHtaGa6v2R1y2lZgvoa4+I2BXX7oXotIJpNmDDBncvlyZPH9uzZ43p7da6XM2fOTN2P3FdcZrnz5A7Ktje/MMWCrdlNFwV1+0e/nGXBlqtlk6Buv9Cyvy3odu8O6uaPbdtrQXfyf3PqgyF7T4sZZzyk+e2333Z1d//55x+XwU/Dm5VplFq8AJA4JaxS3gMFuBouuGvXLteD8uijj3LIAITc6tWrXRmi7777zp3bKWGV/n/xxRdDvWsAkPkB7/r1613t3aJFi7r7FSpUcLXbtBwAcDoFuarB683ZzZUrlyvjRrsJIBz8/fff1qRJE6tZs6a7rykXKj9JGwUgJgNe1WNTYgOvvMbu3btdLy/JVwAgcdWqVXPDBFesWOHua1izhjhfffXVHDIAIdeoUSPXo6uqG3L8+HGbMWOGderUKdS7BgCZP4d3wYIFLnNfqVKlrHz58vbnn3+6K4HXX3+9e7xv377+nwEAZqtWrXLDmuvVq2fVq1d3Q5qVrVkl3iZNmuR6VTRXDgBC4eeff3YdGFWrVnWZmRX8Hj161P0/ZMgQa926tQ0dOpQPB0BsBLzqkVC9tqRoiDMA4H90cfCtt95K8pAoazMAhIouvk2dOjXJx71pbAAQEwGvTtx0AwCkTv78+ZO9UAgAoaQkeroBQDQ64zm8AAAAQGI++OADu+yyy1yZIwAIBwS8ABBBFi1aZFdccYWNHDky1LsCAPEor0u/fv3st99+sz/++IOjAyAyhzTHqixZM+7awJi4jRm2LaRNz3xlMuzQjYv7v6yWQLDt3bvXunfvbidOnLDcuXNzwAGEjWPHjrmkpaNGjbLhw4eHencAwI8eXgCIELfddpsLeL1amQAQLvr37++y0Hft2jXUuwIA6Qt4ddVu0KBBtmXLljNdFQBituTHDTfc4Mq6pdWrr75qW7dutcceeyxD9w0Apk+fbnfccYe/VviZ+uKLL+yTTz6xMWPGpHodlT3av39/vBsAhEXAe9NNN1mWLFnsoosusi5durj5ZACApNWqVcvat29vDz30kNWvX9/efPNNd7KXWjoJVS3Md955x7Jly5aqdTiZBJBaLVu2tDp16li3bt3czx9//LGdPHkyVeuqA0SjT1R6rVChQql+zREjRrjne7eyZcvygQEIj4C3dOnS9uSTT7rEBJqr8cQTT1jDhg3t7bffdvM3AADx5cyZ03r06GELFy60yZMn25IlS9ywZI2WUa9tcg4dOuTa2qefftoqVaqU6kPLySSA1FIt8F69etmvv/5qw4YNc721NWrUcO3Onj17UszKfPjwYXc+qOzMuv3999/+bM1JGThwoO3bt89/27iR/CYAwixplYJbNYJqpHQyN3/+fBs6dKjNmjUrQ67SxcXFuRPDpUuXWpkyZeyee+6xEiVKpHu7ABBKajeVfOrUqVN24MABa9q0qTvB1IiZxMycOdPWrVtnH374obuJAuasWbO6bM2TJk2y4sWLJ3oyqWypHg0XpAcFQHLULql90k2j+XRBTj2/b7zxhuv5TUznzp1dcBxIw6MbNGhgd955Z5KvlStXLncDgLALeJVm/qWXXrKPPvrIOnToYGPHjnWNoagH4/PPP7e77747XTu1adMm17AqwNU2daLWpk0b++mnnyxHjhzp2jYAZDZdwFNNyokTJ7rgVD0pOoHMnj27a9uU1TSpgLdx48Y2bdq0eMu2b9/uLjT27NnT9cwkhpNJAKm1a9cuN/9WHQ0afaI26vLLL3dBb9WqVZMNeHUhLeHFtHz58lm5cuWS7eEFgLANeGfMmOGG1a1cudLOOuuseI/17dvXNXLppYBZ254zZ44/wL3llltSPXcNAMKJpoDoYuH7779/WoblRo0auaHNSdGFP/XkBho3bpwrS5RwOQCkhUaNHDx40J13VahQId5jamfq1q3LgQUQOwFv69atLU+ePKcFu5IRDaKSHyjbn5KzBPbmFilSJN3bBoBQKF++vEsGk1g5IbVt9IIACKXatWu7i2sJg11RT61uZ+K1117jvA1A5Aa8SkKQN29el200GJYvX24+n8/OO+88Gzx4sEtiUKVKFTcfpFixYklmIw3MeEpqewDhZNu2bXbfffe58kQZQW2j5vACQEZQJnhNu0g4fSKtNBUDAMLFGZ8xad6uElMFKyOzhtSICpcfP37clT/67rvvXDHzDRs2JLoO2UgBhDPNgdN83TVr1mTI9pQM5sILL8yQbQGAplb8888/Lj8AAFisB7zKxqy5HipPpAZSV/F0mzp1aobskFfDrXv37jZ8+HC7/fbbXSKsggULumRZiSG1PYBwtmrVKped+YILLnBTP7x2U+0bAISakoLu2LHD5WhRrXCvjUouvwAARO2Q5quvvjrRoSqJzftI6zwSZQXUkGaPekYqV67s5vcmhmykAMJ9Du9bb7112vKkMiwDQGbShbjEOi6KFi3KBwEg9gJenbjpFixKmqBU+Cp7pNpuCn41j/fHH390dX4BINLkz5+fOW0AwpaS55EcFEC0OuMhzUoo9eCDD9o555xjr7zyilumWrwKUDOKEif88ssvVqtWLbvyyitdr2/btm3tnnvuybDXAIDM9OWXX7pcBC1atPAPc1YiKwAIB6rBW7FiRevRo4e7P3fuXBs2bFiodwsAMr+HV4GtMo127NjRTpw44ZYpEUtgluT0UgFzZQxcuHCh7du3z5599lmrVq1ahm0fADLToUOH7NZbb3UJ9saPH++WqW3TyBUACDUlrHr88cetb9++/nbp33//dedimS5PPrO8uYOy6V27jliw/T55flC33/qJbhZsviPBPU7522ZCFu/du4O6+ZyVy1iwbf48Yyo7JCV443WjIOBVz2u7du3izadVxuacOXNm7I5lz24XX3xxhm4TAEJh3bp1VqpUKdfDG8x2EwDSWhJS2d/VTnloowDEbMCrxlAnb55Tp07ZggUL3HxbAMDpSpYs6cp9aEqI5/vvvw/JyJUXtq5yWe+DqWe+4F/59oyL25RprwVEK53bqUc3UKjaKAAIecDbrVs3a9asmeuZWLt2rRvirEyjrVq1yvCdA4BooEynykegEmo6qbzzzjvt/fffd9M2ACDUVNe7cOHCLieLplt06dLF5syZE5ohzQAQ6oBXdXJVi3fSpEn2999/u/JBmpuWNesZ578CgJihE8lp06a5IFc9rJoeopqXABAOpk+fbu+8844tW7bMJSZ98cUXrVixYqHeLQDI/IDXC3qV2ADITBk5TJJhkAiFa665xt0AINwod8rNN98c6t0AgNAHvCpF9OGHH562/N5777Vrr702o/YLAKLG77//nmhZtfPPP99effXVkOwTAHhmz55tTz755GkHRNPVlL0ZAGIq4O3QoYPVqVPH/awELEuXLrW33nrLGjfOhBTjABCBVGrtqaee8t/ftWuXDRkyxGW8B4BQq127drw2atOmTe7+5ZdfHtL9AoCQBLwqSq6bp3nz5rZ48WJbuXKllSmTeZk5ASBSKLGfkv0FypIli0v6p2RWABDqxHoJ2yhllv/qq6+sUaNGIdsvAMgIGZJp6sCBA6elswcA0G4CiEyc2wGI2R7eiRMn2qeffuq/v3XrVtuzZ49Nnjw5o/cNAKLCmjVr7IEHHvDfP3r0qC1ZsoR2E0BYUM3dUaNG+e8fPHjQfv31V5sxY0ZI9wsAQhLwagiz5qN5VLetbt26litXrgzZIQCINirxocR+HrWXSlhVokSJkO4XAEjVqlXjtVF58+a1WrVq2VlnncUBAhB7Aa/q7uoGAEh9Kbe2bdtyuACE7UU52igA0eqMA96XXnrJ3n///SQf79Onj1133XXp3S8AiKqyRHfeeWeSj9eoUcPGjx+fqfsEAJ5Zs2bZ4MGDkzwgl112mcssDwAxEfBeeumlrm5k79693ZA8ZWieOnWqjRkzxrJly2YVKlQIzp4CQIQqV66cG7589tln2w033GA7duxwNS8feeQRq1KlisviDAChcsEFF7j/L774YmvdurX99ddfNnLkSHvuuedcBmfdACBmAt45c+bY1Vdfbb169XL3W7ZsacuXL7e9e/e6Gr0AgPiOHDliy5Yts3Xr1vmX5cmTxz788EPr1q0bhwtASP3999+WI0cOGz58uP/cTlmaf/jhBxs6dCifDoDYKkukwFa3QMrSnHAZAOB/7WZcXJzLzky7CSAc26h9+/aZz+fzL+PcDkDM9vDeeOONrgj5yZMn/UOa165dS+8uACRBw5br1atnrVq1ss6dO7shzRMmTLB3332XYwYg5Jo1a+b+v+KKK9x8XQ1pfuedd2z27Nmh3jUAyPwe3ooVK7ohzJUrV3bD85o0aWJLly515YkAAImbPn269ezZ0zZu3Gg5c+a0efPmWbt27ThcAEIud+7ctmDBArvyyitdsFu8eHH75ZdfrE6dOqHeNQDI/B5eL319//790//qABAjsmfPbt27d3c3AAg3yiugi3IAYLHewwsAAAAAQCQg4AUAAAAARCUCXgAAAABAVCLgBQAAAABEpTQlrUL46JmvTIZub1zcJgtX4bxvAAAAAMIPAS8AAADCwprBEy1/1uAMQMyeLYsF20V1SwX3BarWCO72zcw37b2gbv/EgSMWbDmqlAvq9g//+LsFW5mXhgT9NWIFQ5oBAAAAAFGJgBcAAAAAEJUIeAEAAAAAUSks5/CuX7/eHn/8cVu8eLEdPXrUqlevbo888ohdcsklod41AEAKSDAXu4kPk8J3AgAQKmHXw+vz+axNmza2c+dO++yzz2zhwoVWu3Zta9u2rf3555+h3j0AAAAAQIQIu4B327Zttm7dOrv//vtdz27ZsmVt6NChrqf3559/DvXuAQAAAAAiRNgFvCVKlLCGDRvaf//7Xzt06JCdOnXKXn/9dTv77LOtefPmod49AAAAJHDixAl744037KabbrJevXrZL7/8wjECEBbCbg5vlixZ3FDm9u3bW4ECBSxbtmx21lln2ZdffmmlSiVe20y9v7p59u/fn4l7DAAAELsOHz5sl19+ue3YscP69OljuXPntt69e9u4ceOsVq1aod49ADEu7ALe48ePW4cOHaxo0aK2fPlyy58/v7366qt2xRVX2A8//GCVKlU6bZ0RI0bYE088EZL9BQAAiGXDhg2z1atX26pVq6xIkSJuWbdu3VwgDAChFnZDmr/99lv76aefbPz48VajRg0rX768jRo1ygW+EyZMSHSdgQMH2r59+/y3jRs3Zvp+AwAAxKJJkya5ocxesCvZs2d3I/UAINTCrodXc3ZFw2EC6b73WEK5cuVyNwAAAGQeDWPeunWr1a1b155++mk3d7d06dLWo0cPu+CCC5Jcj+loAGK2h7dx48buCmH//v3twIEDdvLkSTekec2aNW5eLwDEKtUo15w4jYDZvHlzqHcHACwuLs4dhYcfftiNsOvYsaMdO3bMLrzwQvvqq6+SPEKajlaoUCH/TVU5ACAmAl4Fu59//rmtXbvWihcvbvny5bMXXnjBZf675JJLQr17ABASmtqh5C/z5893c+XatWvn8hoAQCgVLlzY32Hx8ssvW9euXd3/V111lQ0fPjzJ9ZiOBiBmhzR7jeaCBQvcEGb18ObIkSPUuwQAIfP111+7k8NvvvnGLrvsMrfs4MGDrm45AIQ64FW+lSpVqsRbrvsrVqxIcj2mowGI2R7eQFmzZiXYBRDzNMqlVatW/mBXlMivcuXKMX9sAITeLbfc4kpKesOb9b/uN23aNNS7BgDh2cMLAPifxYsX2wMPPOAy2M+bN89N92jTpo2VKFEiycNEQhgAmWXAgAG2dOlS16urqRcqK1m1alUbPXo0HwKAkAvrHl4AiHU+n8/27t1rX3zxhd122222adMme+edd9zJ5OzZs5Ncj4QwADKLhid/8sknNmvWLOvXr59rm77//vt4ZYoAIFTo4Y1w4+I2Wazoma9Mhm0rlo4bIluWLFksb968rvTHypUr/SXYunfvbvfdd5/99ttvia6nOb868fTs37+fLKgAgur88893NwAIJwS8ABDmzjvvPDv33HPj1Rtv3ry5TZ061fUAKyhOiIQwAAAADGkGgLB33XXXufm7x48f9y9btGiRC4ITC3YBAADwf+jhBYAw16dPH5fxtEmTJi5ZlYY2z5kzxy0DAABA0khaBQBhLk+ePC47c//+/d183o4dO9ratWutRYsWod41AACAsEYPLwBEgOzZs7uhzQAAAEg9Al4AAACEhWpdL7KCuXIGZdsnN2y2YJv8wbKgbv/2b7+0YMuSP19Qt589WyYMMC1aNKibz3tlSwu2w8+9GNTt53+vvcUKhjQDAAAAAKISAS8AAAAAICoR8AIAAAAAohIBLwAAAAAgKhHwAgAAAACiEgEvAAAAACAqUZYIAGLI/SWrW07LEtTXGBe3KajbR8bhswIARDt6eAEAAAAAUYmAFwAAAAAQlQh4AQAAAABRiYAXAAAAABCVCHgBAAAAAFGJgBcAAAAAEJUIeAEAAAAAUYmAFwAAAAAQlQh4AQAAAABRiYAXAAAAABCVQhbw/vHHHzZ//nzz+XxJPmf16tW2dOlSO3bsWKbuGwAAAAAg8mV6wPvJJ59Y8+bNrXHjxu7/o0ePnvacjRs3Wp06daxZs2Z2zTXXWJkyZWzWrFmZvasAAAAAgAiW6QHvypUrbdiwYTZhwoQkn9OjRw8766yzbMuWLbZ+/Xq79dZb7brrrrO9e/dm6r4CAAAAACJXpge8jz76qF188cVJPv7XX3/Z3LlzbcCAAZYzZ063TD8fPHjQ9Q4DAAAAABCRSas0Z1cuvPBC/7LChQtb1apV/Y8lpGHR+/fvj3cDAAAAAMS2sAt4d+/e7f4/++yz4y0vUqSI/7GERowYYYUKFfLfypYtmyn7CgAAAAAIX9ktzOTIkcPfa5snTx7/8sOHD/uHOCc0cOBA69evn/++engJegEAACKDV7Vj/9HjQXuNk8dOWLAd9p0K6vaDeXz8Tp4M6uZ9x4L/HrIcORrc7WfNZsF2+Hhwv6+nInxE7P4DB9z/yVX8CduAt3z58u7/zZs3W5UqVfzLlcDqiiuuSHSdXLlyuRsAAAAiz4H/f/Ja8YVpod6VsNbvJfLZpM63Qf4kokDJchYtbYdG+EZUwNuoUSMrUKCATZ8+3d9ru2TJEhfwtm7dOtS7BwAAgAxWqlQpV5ZS54BZsmRJ8fneaD6tU7BgwYj8PHgP4YHPITI/B/XsKthV25GSTA94161bZ1u3brXVq1e7+wsXLnRDlWvWrOmSU2kY85AhQ2zQoEGWO3duK1q0qD322GPWsWNHu+iiizJ7dwEAABBkWbNmtTJlypzxejoxjtSA18N7CA98DpH3OaTUsxuygPfjjz/2lxdq2rSpPf744+7nZ5991vXuinp2S5Ysae+9956bu3vLLbdY3759M3tXAQAAAAARLNMD3gcffNDdUtKlSxd3AwAAAAAgKsoSAQAAAMlRstLBgwdHdNJS3kN44HOI/s8hiy81uZwjcNKzxnTv27oh4ud14H965jvzuT1JGRe3iUN7pr9TJcvZvn37+J2K8HbxFstnOS3lhDDpwe8XYgHtIgBEhrDL0pyhtdz+f4p7RIdj5svQExUEp9YZwpP32WXk71FS+P1CLKBdBIDIEJU9vJs2bXJprQFkLKWKT0sWTYQe7SIQHLSLABDeojLgPXXqlKvbm1Itt0ivuxXp+x8N7yHS9z+17yGw1plKRyB628Vo+34nxHuKHOH+WdEuAkBkiMohzWdayy3S625F+v5Hw3uI9P1PzXtIba0zRFeNy2j5fifEe4oc4fxZ0S5mvn///dcGDhxoCxYscBfwevToYX369LFIceLECZsyZYq7/fXXX1auXDm766677Nprr7VIdOjQIbv00kvdhan58+dbhQoVLJLMmzfPnnnmGVu9erVVr17dhg0bZrVq1bJIsXbtWnvqqafsl19+cRe2a9eubY8++qjVrFnTwtXChQtt7NixNnfuXFd2VuVoE1q6dKkNGjTIVq1a5c5dHnroIbviiivS/Jp01QAAACDsHT9+3Fq3bu2Cq//+97/uxF4nxSNHjrRI8cILL9isWbPcif5nn31m1113nXXt2tVee+01i0S9e/e27Nmz2+bNm10wH0k++OADa9OmjTVv3ty+/PJLd+Hkscces0hx+PBhu+SSS2zv3r329ttv29SpU93IEy3bs2ePhaNPPvnElae97LLLLFu2bInm/Pjnn3/ce9DFEz2/Y8eOdtVVV7nfmzTzxbB9+/ZpOLf7PxJF+v5Hw3uI9P2PlveA4IjG7wbvKXJE42eF9Jk6daova9asvn///de/bMSIEb7ChQv7jh49GhGH98SJE6ctu/XWW32NGjXyRZr//ve/vjp16vhmzZrlflfXrl3rixSHDh3yFSlSxDdw4MB4y0+ePOmLFMuWLXPHfenSpf5l69evd8vmzp3rC/fvf+XKlX2DBw8+7Tl9+/b1ValSJd6yTp06+Vq2bJnm143pHt5Ir7sV6fsfDe8h0vc/Wt4DgiMavxu8p8gRjZ8V0j/8VEM2S5Qo4V/Wtm1b18O1fPnyiDi86tVKrOc6R44cFknWr19v999/v7377ruWM2dOizTqLdy1a5fdcsst8ZZHUp6SatWqWfny5e3DDz90w5nVu/v++++7348LLrjAIuX7n9jvuXqAA+n3XEOhT548aWkROZ9qEOiP6JAhQyL2j2mk7380vIdI3/9oeQ8Ijmj8bvCeIkc0flZIf7b5c845J94yL/jVkNpI9Ouvv7qhqBrWHCkUoN9www1uOPn5559vkeiPP/6wvHnzusD9oosusvPOO8/No9bnESny5Mljs2fPdgGv8gnoNm7cOJszZ44VLlzYou33/OjRo7Zz5840bTOmA14AAABEBvViab5oIK9nNK09P6G0YcMGNz9Rt549e1qk0NzpYsWKWa9evSxSKWg/duyYS4A2atQomzZtmgsSNZ/377//tkhJGNapUyc799xzXa+okoY1adLELdOoh0h1Kgi/5wS8AAAACHsKshL28Hj39Vik9WK1atXK6tSp4zI2p7ZcXDiYPn26/fDDDy57rm6dO3d2yxUsKoCMBMWLF3dJtp599lm33zVq1HC9owq0FPxGAvXs/v777/bOO+9YvXr13HD/N954w5UgVBKraPs9z5o1q5199tlp2mZUliUCAABAdGnYsKG99957FhcXZ/ny5XPLvv/+ezeHNFznLCZGw69btmzpyuAoU3Ckzd9Vb6J6SD1LlixxQa/eSziXwwnUuHFj93/+/Pn9yxTsapiwsh9HAg3x9fbZo98FTQM5cuSIRfLv+YIFC+It0++5AvrcuXOnaZtRHfCqQdRk+nXr1lnFihXtxhtvdDXbMnqdYNm9e7dLx6190RU0pa4vWrRoks/XlSql+k5I8yy8X+zM9NZbb7m6YIEqVapk9913X4pXPfUHTVdzLrzwQrvmmmtCkkRAV1x//PHH05arYRkxYkSSc0JeffXV05Y/8sgj7mpisOk7oCuvahhUr0y18RJSI6jvuPZV9f/0HU9prkda1kFkW7x4sSvToF4HfZcaNGhgkUzDoD7//HPXJqlNV2mTSDpBTsnKlStt4sSJVr9+fevWrZtFOs2r++ijj1zJissvv9yaNm0a6l1CGLj++utd2RgNqR09erRt27bNDUft3r172NZqTmjr1q3+YFc9dJGY8CkwaZh4Q4A17zJSzg10/PV3QOdz6iHVfF7Vht2+fbt16NDBIoFK9+hv9JNPPukS/OlcWT3W+/btOy3pUyS5++67XZuvXmr9PVOyKsUFL774Ypq3GbVDmvVhN2rUyMaPH++uBkyaNMmdCCiIzMh1gkW12RTs6QqHfgl1v3LlyvbTTz8lG+zoy6Asbapd5d1CFbDrZFlDXgL3pWTJksmuo2QBujr43XffuatWDzzwgAt4Q0EXFwL3XTfV/dN7Sm4+jj6DhOtlxtVbNQj6juhCw+TJk90V18Tme6gR0T7qO66gvm7duu6kISlpWQeR7eWXX3YnZGoTlcWyWbNmLpiKVPquKrGKhnopQ6QuIuoi4DPPPGPRQL+jCgR0Ueqrr76ySKfPSX8HfvvtN9eL9/jjj7uhhsBZZ53lLlx9/fXXLkGPLqLrPC09J8KZ7fnnn7e1a9e6C+raf29YsHqvkLl0PqN5vBomqwsmTz/9tDvP0zlOJKhatarLyqxgUOf6uo0ZM8YFiuH6HjZt2uT/zutCyXPPPefv1PMoFtN5rGIAvSdd9OzXr5/dfvvtaX9hX5QaNGiQr3Tp0r4DBw64+wcPHvSVK1fO9/DDD2foOsGyZs0a/3542rVr52vVqlWS6xw+fNjV3vr+++994eD666/33XbbbWe0jmpsdejQwX9/5cqVvixZsvimT5/uCzV9Jjq+U6ZMSfI5M2fOdM8JBdVe27x5s/tZ32PVJkxo1KhRru7c7t273f0jR474zj33XN8999yT5HbTsg4i186dO3158uTxjR071r9s9OjRvgIFCkRsPdQ9e/b41q1bF2/ZmDFjfNmyZfPt37/fF+luv/1233333ee79NJLfTfeeKMvkq1atcqXI0cO38SJE+MtT/j5AWqr4uLiIu5A7N2717dx48bTblu2bPFFKtVA1ntIrMZwJND5fqT+ffPodyFh3BCOTpw4kej3f/v27ac9VzWRtfzYsWPpft2o7eH99NNP7aqrrvKPzddVYvUUaohwRq4TzNpagfMKpFatWqlKu6+r/EoaoB4Z9c6EepjdgAEDXE9Kcr3Tot6kuXPnxhuOp14ZXb0NxWeQkHr8dRXQS86QnGHDhrleCZUaCJznEkwagl+qVKlkn6PvuIan6iq5aJ7Hf/7znxR/L850HUSub775xl3x7tKli39Zjx497MCBA65uYSTSEDv1pCRsTzXMOdJHKujqvoafa1hnNNDfLU3/uO222+ItT/j5AUWKFHEj4CKNeqa9Hq7AW0oj4MKZhmXrPaSmxmo40vl+pAyJT4p+FxLGDeEoW7ZsiX7/E0s6pyHaWp4RoySjNuDVXEMN7wyk+xrKpiG/GbVOZg5ZUzKAiy++OMWGVPuqeaYaFqZU5ckNwQ0mzSvQiaYaEmWR07BYzbtJyp9//un2PbHPQMN/QknDxd9880134p9STUidmCl41zq68KBhJaEYFn8m33Fl9NN3LKPWQeTS560TSbUlHgUg+kMa6t/DjKT2sXTp0qd9tyPJX3/9Zb1793YXOdOayCPc6MKo/lYsW7bM1fgcOXJkyP6GAQCiQ1QmrVLQpCQ7Ceeu6uqNrugrq1nCk4O0rJOZ9ahuueUW1+vy1FNPJfk8XQFRj6pO4kQnC1dffbUb875ixQrLbDpRKV++vP/+lVde6XrM1YueWAIcL3hK7DMI9Yn2F198Yf/++2+K8wc0B0efgfddUcCrniQF+kqGEGo6xokdX++xxK6Wp2UdRK7EPm/vM4+WCxwKdnUBSwneIqkUSCCNHFFCQo2giaa5f7pYqF73m266ya699lo330vzyfX3LFLKnQAAwktUBrw6gVGvYsKiy3v27HFBYWKBa1rWyQwKxO+880431Fe35DL9apiAF+x670k9kgoy9b4yO3NeYLArCr7VS6REXIkFvN5JdmKfQagSb3lef/11u+iii1ydtuQk/Hy03xoCrWGi4UD7k9jxlaSGwqRlHUSuxD7vcPk9zAgaKaM29bXXXrP27dtbpFIiw+XLl7vkW/fff7+/d17TXnR/6NChEfl5aZ812kfJTLyqBGp31ZN97733RuR7AgCEVtQOadbcz9WrV8dbpvtKQ56R6wQ72O3Zs6frhZg9e3aa9kO9wpJZ80hT6qlWb3lS+6Jsc8rMnNhnoM8mlCUEZsyYYXfccUea1tdnEA7HP7nvuIZhJ3VRJy3rIHLp89YQfJVmCMw+rrqEofw9zAgqAaIcAcr4e/PNN1skUxZjldPQ3H0vG7ymsujCrX6O1Ll0Cm5V+iywBJ8qFuhvh76HAACcMV+UUlZRZZb9999/3X1l+SpWrJjvqaee8j9n7ty5vj59+pzROpnl1KlTvrvuustXvHhx32+//ZZkVjzt/6JFi9z9n3/+2Z9J18um27x5c1+dOnV8mU2Z4n744YfTsqLqK7d06VL/ssmTJ/tefPFF//1OnTr5mjVr5jt+/Li7r4zTWkefVago23HBggUTzQapjKL6DLZt2+buaz+9fZdNmzb5ihYt6rv33nszdZ+TytKszLvKtrthwwZ/9toyZcr4BgwY4H+Ovk96T/p+pXYdRA9lLS5cuLBv6NCh/mX9+/d3baEywUeqadOm+XLmzOl7/fXXfdEqGrI0qw3NlSuXyzrvGTZsmC9//vwR/f0DAIROFv1jUUhzbtu1a+cSTqkws+q6KguYard5cw5feOEF69u3rz8hVWrWySyvvPKKG8Kl2lPnnXeef7kSJnnZOA8ePOiGdymrpeaWKoOqhnzp+comPGfOHPd8ZfHUPNLMFBcX54YL6thqf3RMlUl0+PDh1qdPH//zNEdLc2Pnz5/v7msYW4sWLdzVffVoq96e5nKFssaeMmargPerr7562mOqeanvzKpVq9z7VH29CRMmuB4JvXfVIlY9MdVIC/aQcvXI6fh6Q7AvuOACl+Fax/Guu+5yy5VIq1OnTi4hjN6Thpcr+7K+O968XA31VG+2svJqyHJq1kF00bBfTYdo06aN61nTCBN9hzt27GiRSDkA1HOo3lD9vgZSm1mlShWLBvr9POecc+ydd96xSPbQQw+5edbK+7Bjxw73t0xtmmoNAwBwpqI24PWG0M6cOdMFWzrR0clb4DCvpUuX2rx58/zzn1KzTmZZtGhRopkplfr9nnvucT9rmKwKTLdu3do/t1RBiub6KumHhpwqq7OGCYeK3oMSZimA1VwznYwlTAalRDiBBacVyCtQVEklBY4NGza0UCZQUfFrnegnVhbjn3/+sY8//tgF5V7ZHmUv1sUSDWXW56L3kBk0x1KJeBLS8EYlCvPoV14BzJo1a9zQQV1UCUz5rqRb+h3o1auXf3lK6yD66Lv97bffulwAagd18S9S6aKaAvbE6KJbJL+3QB999JEb0qzfz0inC2z6+6F2tXnz5hFdsgUAEFpRHfACAAAASD2NrlOuA/0PRIOoTVoFAAAAILqpRJs3ikejdpSgMLHHELuisiwRAAAAgOiXXEBLsAuhhxcAAADAaVTbu0GDBi4ZnpLIqcdUiVGVk2XQoEHuOcrvobw3yoPjUfJXJfZMjPKK9O/f3+U50bb69evnTyCr/BHKQ6CkrOeee67LTeBR8tDSpUtb1qxZXX4JzfUP7MVV0s1p06a5vDB6XPsc2MOb3La1T4888ojVrl3bChUqZPfdd1+Kr4vIQcALAAAAIJ7ffvvNBYgjR450NcxVwUGZ7hUEK/Hr999/b5999pkLHpXtXlVNvABYyQKTS6CnpIgKiLUd/ewFpbfeequrLKIEoKpYoprpClRVMWDgwIEuMat+VoBcp06deNtUxZJrrrnGVRrQ49rnQElt26MqFDNmzLDff//dpk+fbr/88kuqXhfhj4AXAAAAgN/q1atdhYopU6bYpZde6koUKhhUkKhSnaVKlXKVTn799Vf3fFV2UFlGGT9+vLuv3tCk6PGyZcta+fLlXXk4Bb2qPqIqF0888YTrhVUVEr22lqliSuXKlV3P8NixY23jxo1n9Gklt23P3Xff7XpydWvSpImr2JLe10V4IOAFAAAA4KdSYBq2rJKE4g053r17t/vZuz322GNuuepmL1++3NavX+/KNSowTk5gMBxYMCZhkKzHvGXz5893w5U1pFglH9XDnNx2U3oscNuSO3du/88KdNWjm9rXRXgj4AUAAADgp3ms33zzjX366ac2fPhwy5Ejh7Vt29Z69+7thisnpADRG1KsnmH1oiZHQ4rVW6ohxWPGjLHLLrvMvYbqbg8ePNgOHDjggu05c+a4ZaKe5S5duthzzz1nzZo1c8OhE9tvBd0Jq66mtO3kpOZ1Ed4IeGNQ165d3dAOAIDZSy+9xBV7AEggf/789uWXX7phv48++qi9/vrrLnGT5sGqZ1Q3DUX2KOBduXKlG6KcEg0n1hzfunXrumBXyaVk0qRJtmLFCjdkWkOMJ0+e7IY9K8j2XlOP6f+bbrrptO3ecsstLsmUAnAlrQqU1LaTk9rXRXjL4kt4CQRRT0M29u7dG2/oRrhavHixSyLwwAMPuP+XLFli999//2mPAUBaKbGJei4SJjgJV16dSWVEVa/D1KlTT3sMADKTwgllSH7jjTfs888/T/a5yoisIc/6H8gM1OFFWFNCg6ZNm7qfNfRFQW5ijwFArPB6QhTw6gQzsccAIDPVq1fPdu3aZV988QUHHmGHgDfGaX6GbpqfsXbtWjfcQwkJVD+te/fubujKXXfdZUOGDHEJDEST9RV4PvTQQ6dtT49pboPmUGgIjCb3a9iIlxRAPbLK+OedmHlzJw4ePOiGFSoF/rFjx1wvhR5XkKt1VBfthRdecKnw9VjDhg3t4osvdo81btw42W17+1S0aFE39Ebb0hwUDXdJ6nUBxCYNX9PQPZWhUMIWzTNbtWqVVapUybUbqhmpHgy1cVdffbVb58iRI3bnnXfaxIkTLVeuXPG2p+kjGv6mOpMaPpcvXz7XdhYrVsw9vn37dtcGqW1TVlANCVS7K+q5VdkOzTdT1tAXX3zRLVf5DrVVw4YNc0Gvfpb333/f/1hy2/b26eGHH3ZDFDW3TfvktfFJvS4AJOVM5rV+8sknHEhkKubwxjCdtKlouIYE64ROQ+N0UteqVSt78803bdSoUe55OrFTIOx55pln3PMSo+QD2qYCYvW+qgi4UsB7We6UyEDp3XXr1KmTS2kvKvatxlKPK+DU/BDZtGmTLVq0yJ2Q6YRNcy28YNZ7LKVta58ef/xx+/HHH10gr5NOL3V+Uq8LIDbLcLRp08Zl46xQoYIrW3Ho0CE3z2z//v3+mpLnn3++Pfnkk/71VPNRQWTCYFeU5VMB5IABA6x+/fq2c+dON3dNgaoCZbVrWqYLeAqY+/bt69bTBUO1pWrr1DZ16NDBv02vV7dFixbuYqIe100/e48lt21vn5RdVfu0Z88e/3Du5F4XAIBIRA9vjHrqqafsp59+slmzZrmkBAoIc+bM6TLWeZSZT70c6qFVQgH9vHXrVjfhP7l5F9WqVXO1ykQnXFpXPcQKfrUN9XR48z10EqaTNvUkqDdFva86mUxIAW+jRo3c63u9F4Hz1JLbtjfURj3Eop5dFUfX+0rpdQHEhoULF9rIkSPtrbfecu2F2oi///7bfv/9d3cTJWPRhTaNMMmePbu7iKafVXNy9OjRSW5bwa16UnXBTglVatasaT/88IMb/qeRJ+PGjXPP08XG6tWru7ZKbVPBggWtSpUq7jUSy7mgbKEKcr02MZBGsyS1bW+fdGGzSJEidv3117v/JTWvCwBAJKGHN0Yp6556KxTsigJJnTx5PQX33HOPDR061D1WsWJFd6Kkoc8KIjXEWUFjUsqVK+f/WSd46mGQHTt2xMuGp+1qmag3WSdu6pnV+iklPEgouW2LMut5NKRQvTYZ8boAooNGpejil3fhS22iN6LEuykY1ogX6dWrlxspoukQR48edRf3kqI2pkyZMv77amvULqqNUk9yYF4CTa3QVIv27dtbjx493IU8DTU+0+R8yW1b1BvtBbkKavUeJL2vCwBAuCHgjVHqvVBhcG/YstLCa+hv586dXW+BbhrO51GKefXaao6s5oGldOLonVQpSK5Ro4b7Wf/rdT0zZsxwPR1SvHhxN9dM84ifffZZN6c4ocCTsoSS23ZyUvO6AKKfRodoyoMCPrVfderUsXXr1rmpGV6bqJtXW1IBsHpR1bN73333JbttjTjRaBrR8GGNrtFFRLVbyj3gtZczZ850c2b1GgqStV09/tdff7ke4i1btsTbri486nbixInTXjO5bScnNa8LAEAkYUhzjNJJj5KSaGhyXFycS17Srl07O/fcc12gqJMozWVTIhZRb7AStuiE0OsVSIoeVyIpnVxpyJ9XMkM9BZpLpmFyOqnSiZ83z1ZzxjSPWHPLAksPBdIwQ6Wx13xb9UYHziNObtvJSc3rAogNmtOaN29e195pFIxGuqg91HQKLddQXy+fgS7AaSiwciFo5EtyNCVjxIgR9txzz7kpIaqFriHDonJIXvCrNkgBpijTqfINiAJPPa4e6EBq67Rvavs0ikVJqzxqC5PadnJS87oAAEQS6vDGIPXsagivsnWqx1QnOC1btrTChQvbsmXL3Lw1BYBK/qSeX496OtTLq/muSVGRbwXSzz//vEsGpZMlBb4eJVJRECyB88Pmzp3rhvhpHrGGFHong8ouqkzNXiZm/ax91Nw0DQsMfCypbW/YsMG2bdtmDRo0cPfVW6FlWi+p1wUQOxQMlihRwj8dQxfLdOFOwa7aGAWpmgahYcBXXnmlfz0FsPv27fMn5kuM2iW1rXqe2idNqdDFu0DavtqlCy64wB9c/vnnn66tUzutocUKbL3szcpf4M3bPXz4sEvap+1fc801LmlV4JzexLat+bvK0eBlmZaPPvrIjfBJ7nUBAIhEBLxIkU6gNGxPJ0sppZL3Al79DwDRSAGkRr/oop5umhqRUsCr/wEAQOYj4EWqSnWsX7/eldJIrOxGoIS9qQAQbZTJWPN31c4FJqNKTGK9qQAAIPMQ8AIAAAAAohITcwAAAAAAUYmAFwAAAAAQlQh4AQAAAABRiYAXAAAAABCVCHgBAAAAAFGJgBcAAAAAEJUIeAEAAAAAUYmAFwAAAAAQlQh4AQAAAAAWjf4f441Yt0SoC2sAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Added for CS6140] the three masks, seen in the (untrained) weights of head 1, first sentence\n", "n_src, n_tgt = min(int(src_len[0]) + 2, mt.num_steps), 8\n", "fig, axes = plt.subplots(1, 3, figsize=(10, 3))\n", "for ax, (title, w) in zip(axes, [(\"encoder self (padding hidden)\", enc_w[0, :n_src, :n_src]),\n", " (\"decoder self (causal)\", dec_self_w[0, :n_tgt, :n_tgt]),\n", " (\"cross (decoder asks)\", cross_w[0, :n_tgt, :n_src])]):\n", " ax.imshow(w.cpu(), cmap=\"Reds\")\n", " ax.set_title(title, fontsize=9); ax.set_xlabel(\"key positions\", fontsize=8)\n", " ax.set_ylabel(\"query positions\", fontsize=8)\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "id": "e9058370", "metadata": {}, "source": [ "## Segment 3 — Layer norm = batch norm over the other axis (note §2)\n", "**What to watch:** layer norm normalizes each token (row) on its own; batch norm normalizes each feature\n", "(column) across the batch. LN removes each token's scale: both rows of the toy come out the same." ] }, { "cell_type": "code", "execution_count": 9, "id": "713ca572", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.520702Z", "iopub.status.busy": "2026-10-18T03:35:30.520611Z", "iopub.status.idle": "2026-10-18T03:35:30.524430Z", "shell.execute_reply": "2026-10-18T03:35:30.523918Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "layer norm (each row):\n", " tensor([[-1.2247, 0.0000, 1.2247],\n", " [-1.2247, 0.0000, 1.2247]])\n", "batch norm (each column):\n", " tensor([[-1.0000, -1.0000, -1.0000],\n", " [ 1.0000, 1.0000, 1.0000]])\n" ] } ], "source": [ "# [Changed from book §11.7.2] the note's toy X instead of the book's 2x2 (library layers, no learned scale/shift)\n", "X = torch.tensor([[1., 2., 3.], [2., 4., 6.]])\n", "ln = nn.LayerNorm(3, elementwise_affine=False)\n", "bn = nn.BatchNorm1d(3, affine=False) # training mode: statistics of this batch\n", "print(\"layer norm (each row):\\n\", ln(X))\n", "print(\"batch norm (each column):\\n\", bn(X))" ] }, { "cell_type": "markdown", "id": "938b8ebf", "metadata": {}, "source": [ "## Segment 4 — Add & Norm, the FFN, the encoder block; parameter counts (note §3–4)\n", "**What to watch:** Add & Norm and the FFN keep the shape; \"attention communicates, the FFN computes\". One block\n", "of width $d$ costs about $12d^2$ parameters: two thirds of them in the FFN." ] }, { "cell_type": "code", "execution_count": 10, "id": "b9cbc164", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.525690Z", "iopub.status.busy": "2026-10-18T03:35:30.525613Z", "iopub.status.idle": "2026-10-18T03:35:30.531021Z", "shell.execute_reply": "2026-10-18T03:35:30.530560Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "FFN on (2, 3, 4) -> (2, 3, 8)\n", "encoder block on (2, 100, 24) -> (2, 100, 24): shapes check\n" ] } ], "source": [ "# [Book §11.7.2–11.7.4] the book's FFN, Add & Norm and encoder block (d2l package) keep the shape\n", "ffn = d2l.PositionWiseFFN(4, 8)\n", "ffn.eval()\n", "print(\"FFN on (2, 3, 4) ->\", tuple(ffn(torch.ones((2, 3, 4))).shape))\n", "add_norm = d2l.AddNorm(4, 0.5)\n", "shape = (2, 3, 4)\n", "d2l.check_shape(add_norm(torch.ones(shape), torch.ones(shape)), shape)\n", "X = torch.ones((2, 100, 24))\n", "valid_lens = torch.tensor([3, 2])\n", "encoder_blk = d2l.TransformerEncoderBlock(24, 48, 8, 0.5)\n", "encoder_blk.eval()\n", "d2l.check_shape(encoder_blk(X, valid_lens), X.shape)\n", "print(\"encoder block on (2, 100, 24) -> (2, 100, 24): shapes check\")" ] }, { "cell_type": "code", "execution_count": 11, "id": "f5620f2e", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.532680Z", "iopub.status.busy": "2026-10-18T03:35:30.532589Z", "iopub.status.idle": "2026-10-18T03:35:30.538387Z", "shell.execute_reply": "2026-10-18T03:35:30.537862Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "d = 16: attention 1,088 + FFN 2,128 + layer norms 64 = 3,280 (12 d^2 = 3,072)\n", "d = 256: attention 263,168 + FFN 525,568 + layer norms 1,024 = 789,760 (12 d^2 = 786,432)\n" ] } ], "source": [ "# [Added for CS6140] parameters of one block (PyTorch's layer: attention 4(d^2+d), FFN 2*d*d_ff + d_ff + d, two LNs 4d)\n", "for d, h, d_ff in [(16, 4, 64), (256, 4, 1024)]:\n", " blk = nn.TransformerEncoderLayer(d, h, d_ff)\n", " att = sum(p.numel() for p in blk.self_attn.parameters())\n", " ffn_p = sum(p.numel() for p in [*blk.linear1.parameters(), *blk.linear2.parameters()])\n", " total = sum(p.numel() for p in blk.parameters())\n", " print(f\"d = {d:3d}: attention {att:,} + FFN {ffn_p:,} + layer norms {total - att - ffn_p:,} = {total:,}\"\n", " f\" (12 d^2 = {12 * d * d:,})\")" ] }, { "cell_type": "markdown", "id": "43299815", "metadata": {}, "source": [ "---\n", "# Part B (Thu 12/10)\n", "\n", "## Segment 5 — The decoder block: the future stays hidden (note §5)\n", "**What to watch:** with the causal mask, changing the *last* English word changes only the last output; the\n", "earlier positions cannot see it. Without the mask every position changes." ] }, { "cell_type": "code", "execution_count": 12, "id": "d7977842", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.539630Z", "iopub.status.busy": "2026-10-18T03:35:30.539554Z", "iopub.status.idle": "2026-10-18T03:35:30.546438Z", "shell.execute_reply": "2026-10-18T03:35:30.545888Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "with the causal mask change per position: 0.000 0.000 0.000 0.000 0.000 1.986\n", "without it change per position: 0.112 0.227 0.182 0.126 0.206 1.986\n" ] } ], "source": [ "# [Added for CS6140] \"change the last English word\" with PyTorch's decoder layer (dropout off)\n", "torch.manual_seed(0)\n", "layer = nn.TransformerDecoderLayer(d_model=16, nhead=4, dim_feedforward=32, dropout=0.0, batch_first=True).eval()\n", "memory = torch.randn(1, 5, 16) # the encoder's output for a 5-token Spanish sentence\n", "tgt = torch.randn(1, 6, 16) # 6 English positions\n", "tgt2 = tgt.clone()\n", "tgt2[0, -1] = torch.randn(16) # change only the last English word\n", "causal = nn.Transformer.generate_square_subsequent_mask(6)\n", "with torch.no_grad():\n", " for name, mask in [(\"with the causal mask\", causal), (\"without it\", None)]:\n", " diff = (layer(tgt, memory, tgt_mask=mask) - layer(tgt2, memory, tgt_mask=mask)).abs().amax(-1)[0]\n", " print(f\"{name:21s} change per position: \" + \" \".join(f\"{v:.3f}\" for v in diff.tolist()))" ] }, { "cell_type": "markdown", "id": "44fb5666", "metadata": {}, "source": [ "## Segment 6 — Train in parallel, generate in a loop (note §6)\n", "**What to watch:** training is one pass over all target positions at once (teacher forcing + the causal mask);\n", "generation is a loop, one word per pass. A minute of training on 10,000 pairs gives rough translations; HW7's\n", "full run (100,000 pairs, more steps, a careful embedding start) reaches about 45 BLEU." ] }, { "cell_type": "code", "execution_count": 13, "id": "0eb16d70", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:35:30.547611Z", "iopub.status.busy": "2026-10-18T03:35:30.547530Z", "iopub.status.idle": "2026-10-18T03:36:19.834222Z", "shell.execute_reply": "2026-10-18T03:36:19.833830Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "trained in 49 s\n" ] }, { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:36:19.819804\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Changed from book §11.7.6] 10 epochs instead of 30, on Spanish-English\n", "t0 = time.time()\n", "trainer = d2l.Trainer(max_epochs=10, gradient_clip_val=1, num_gpus=NUM_GPUS)\n", "trainer.fit(model, mt)\n", "print(f\"trained in {time.time() - t0:.0f} s\")" ] }, { "cell_type": "code", "execution_count": 14, "id": "c47a4c23", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:19.835643Z", "iopub.status.busy": "2026-10-18T03:36:19.835557Z", "iopub.status.idle": "2026-10-18T03:36:20.031355Z", "shell.execute_reply": "2026-10-18T03:36:20.030720Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "teacher forcing, all 13 positions at once: 14 ms | generation, one position per pass: 177 ms\n" ] } ], "source": [ "# [Added for CS6140] one teacher-forced pass vs the step-by-step loop, on the same 128 sentences\n", "model.eval()\n", "batch = [b.to(DEVICE) for b in (src, tgt_in, src_len, label)]\n", "with torch.no_grad():\n", " t0 = time.time(); model(batch[0], batch[1], batch[2]); t_forced = time.time() - t0\n", " t0 = time.time(); model.predict_step(batch, DEVICE, mt.num_steps); t_loop = time.time() - t0\n", "print(f\"teacher forcing, all {mt.num_steps} positions at once: {t_forced * 1000:.0f} ms | \"\n", " f\"generation, one position per pass: {t_loop * 1000:.0f} ms\")" ] }, { "cell_type": "code", "execution_count": 15, "id": "86bd68fa", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:20.032932Z", "iopub.status.busy": "2026-10-18T03:36:20.032835Z", "iopub.status.idle": "2026-10-18T03:36:20.195169Z", "shell.execute_reply": "2026-10-18T03:36:20.194542Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tengo hambre . => i have hungry .\n", "me gusta leer libros . => i like to read this books .\n", "¿ dónde está mi perro ? => where is my dog ?\n", "ella no sabe nadar . => she doesn't know her .\n", "mañana vamos a la playa con mis amigos . => i went to my friends .\n" ] } ], "source": [ "# [Changed from book §11.7.6] the book's predict_step on Spanish sentences (cut at )\n", "spanish = [\"tengo hambre .\", \"me gusta leer libros .\", \"¿ dónde está mi perro ?\", \"ella no sabe nadar .\",\n", " \"mañana vamos a la playa con mis amigos .\"]\n", "toks = [seq2seq_common.tokenize(x) for x in spanish]\n", "s, sl = mt.src_batch(toks)\n", "bos = torch.full((len(toks), mt.num_steps), mt.tgt_vocab[\"\"])\n", "preds, _ = model.predict_step([s.to(DEVICE), bos.to(DEVICE), sl.to(DEVICE), bos.to(DEVICE)], DEVICE, mt.num_steps)\n", "for es, p in zip(spanish, preds):\n", " print(f\"{es:42s} => {' '.join(seq2seq_common.ids_to_tokens(p.tolist(), mt.tgt_vocab))}\")" ] }, { "cell_type": "code", "execution_count": 16, "id": "d29d9338", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:20.196707Z", "iopub.status.busy": "2026-10-18T03:36:20.196613Z", "iopub.status.idle": "2026-10-18T03:36:20.565907Z", "shell.execute_reply": "2026-10-18T03:36:20.565407Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "corpus BLEU-4 after a minute: 10.4\n", "HW7 full runs (reference run): HW6 GRU about 27.9 -> attention GRU about 37.6 -> Transformer about 45\n" ] } ], "source": [ "# [Changed from book §11.7.6] corpus BLEU-4 on 500 test sentences (all their references), as in HW6/HW7\n", "n = 500\n", "s, sl = mt.src_batch(mt.test_src[:n])\n", "bos = torch.full((n, mt.num_steps), mt.tgt_vocab[\"\"])\n", "with torch.no_grad():\n", " preds, _ = model.predict_step([s.to(DEVICE), bos.to(DEVICE), sl.to(DEVICE), bos.to(DEVICE)], DEVICE,\n", " mt.num_steps)\n", "hyps = [seq2seq_common.ids_to_tokens(p.tolist(), mt.tgt_vocab) for p in preds]\n", "print(f\"corpus BLEU-4 after a minute: {seq2seq_common.corpus_bleu(hyps, mt.test_refs[:n]):.1f}\")\n", "print(\"HW7 full runs (reference run): HW6 GRU about 27.9 -> attention GRU about 37.6 -> Transformer about 45\")" ] }, { "cell_type": "code", "execution_count": 17, "id": "f0d34897", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:20.567583Z", "iopub.status.busy": "2026-10-18T03:36:20.567496Z", "iopub.status.idle": "2026-10-18T03:36:20.575713Z", "shell.execute_reply": "2026-10-18T03:36:20.575134Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "embedding x sqrt(d): typical size 11.3 | positional encoding: values in [-1, 1]\n", "at the start the word signal drowns the position signal; HW7 starts the embeddings at std 1/sqrt(d) (the original Transformer's choice): about 45 vs about 31 BLEU at the same budget (note §6)\n" ] } ], "source": [ "# [Added for CS6140] the embedding start: the book multiplies embeddings by sqrt(d); PyTorch starts them at N(0, 1)\n", "d = num_hiddens\n", "fresh = nn.Embedding(len(mt.src_vocab), d)\n", "pe = d2l.PositionalEncoding(d, 0).P[0, :mt.num_steps]\n", "print(f\"embedding x sqrt(d): typical size {fresh.weight.std().item() * math.sqrt(d):.1f}\"\n", " f\" | positional encoding: values in [{pe.min().item():.0f}, {pe.max().item():.0f}]\")\n", "print(\"at the start the word signal drowns the position signal; HW7 starts the embeddings at std 1/sqrt(d)\"\n", " \" (the original Transformer's choice): about 45 vs about 31 BLEU at the same budget (note §6)\")" ] }, { "cell_type": "markdown", "id": "0a2d86a2", "metadata": {}, "source": [ "## Segment 7 — Delete the encoder: a tiny GPT on TinyStories (note §7)\n", "**What to watch:** what is left of the Transformer is a stack of encoder-style blocks with the causal mask:\n", "GPT. Its task is HW6 Problem 3's (next word on TinyStories), and its output layer *is* the token embedding\n", "(weight tying). The book has no GPT code; this uses Hugging Face's GPT-2 class at a tiny size." ] }, { "cell_type": "code", "execution_count": 18, "id": "32d776b9", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:20.577305Z", "iopub.status.busy": "2026-10-18T03:36:20.577219Z", "iopub.status.idle": "2026-10-18T03:36:22.645264Z", "shell.execute_reply": "2026-10-18T03:36:22.644662Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "TinyStories: 20,891 training / 1,099 validation stories; 4,229,528 / 221,116 tokens; vocabulary 5,274 words (min frequency 10); 0.45% of training tokens are \n" ] } ], "source": [ "# [Added for CS6140] HW6/HW7's TinyStories split and word vocabulary (5,274 words), windows of 64 words\n", "lm = lm_common.TinyStories(root=f\"{DATA}/tinystories\", batch_size=32, num_steps=64, min_freq=10,\n", " batches_per_epoch=50)\n", "print(lm.describe())" ] }, { "cell_type": "code", "execution_count": 19, "id": "970de203", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:22.646531Z", "iopub.status.busy": "2026-10-18T03:36:22.646422Z", "iopub.status.idle": "2026-10-18T03:36:24.660959Z", "shell.execute_reply": "2026-10-18T03:36:24.660537Z" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/private/tmp/claude-501/-Users-vip-Dropbox-CS6140/7d0cce44-9e33-4ced-b542-36eb44979e55/scratchpad/env313/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "W1017 23:36:24.531000 3907 torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "parameters: 1,080,064\n", "output layer is the token embedding (tied): True\n" ] } ], "source": [ "# [Added for CS6140] Hugging Face's GPT-2, tiny: width 128, 2 blocks, 4 heads, context 64, tied output layer\n", "import transformers\n", "from transformers import GPT2Config, GPT2LMHeadModel\n", "transformers.logging.set_verbosity_error()\n", "\n", "\n", "class HFLanguageModel(nn.Module):\n", " \"\"\"Wraps GPT2LMHeadModel so that model(X) returns the logits, as lm_common.train_lm expects.\"\"\"\n", " def __init__(self, gpt):\n", " super().__init__()\n", " self.gpt = gpt\n", "\n", " def forward(self, X):\n", " return self.gpt(input_ids=X).logits\n", "\n", "\n", "torch.manual_seed(0)\n", "cfg = GPT2Config(vocab_size=len(lm.vocab), n_positions=64, n_embd=128, n_layer=2, n_head=4, n_inner=512,\n", " bos_token_id=None, eos_token_id=lm.vocab[\"\"], pad_token_id=lm.vocab[\"\"])\n", "gpt = GPT2LMHeadModel(cfg)\n", "print(f\"parameters: {sum(p.numel() for p in gpt.parameters()):,}\")\n", "print(\"output layer is the token embedding (tied):\", gpt.lm_head.weight is gpt.transformer.wte.weight)" ] }, { "cell_type": "code", "execution_count": 20, "id": "bc31379b", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:24.662323Z", "iopub.status.busy": "2026-10-18T03:36:24.662152Z", "iopub.status.idle": "2026-10-18T03:36:59.631027Z", "shell.execute_reply": "2026-10-18T03:36:59.630479Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "34 s | validation perplexity 23.3 (a uniform guess: 5,274)\n", "the HW ladder (reference runs): RNN about 23.8 -> LSTM 18.6 / GRU 17.5 -> GPT about 15.6 (HF GPT-2 14.8)\n" ] } ], "source": [ "# [Added for CS6140] train about a minute with HW6/HW7's training loop; perplexity on the validation windows\n", "torch.manual_seed(0)\n", "lm_model = HFLanguageModel(gpt)\n", "hist = lm_common.train_lm(lm_model, lm, DEVICE, steps=2000, lr=2e-3)\n", "print(f\"{hist['seconds']:.0f} s | validation perplexity {lm_common.perplexity(lm_model, lm, DEVICE):.1f} \"\n", " f\"(a uniform guess: {len(lm.vocab):,})\")\n", "print(\"the HW ladder (reference runs): RNN about 23.8 -> LSTM 18.6 / GRU 17.5 -> GPT about 15.6 (HF GPT-2 14.8)\")" ] }, { "cell_type": "markdown", "id": "be393a53", "metadata": {}, "source": [ "## Segment 8 — Pre-norm vs post-norm (note §8)\n", "**What to watch:** post-norm ends every block with a layer norm, so its output is normalized; pre-norm puts the\n", "norm *inside* the branch and leaves the residual stream $x + f(\\mathrm{LN}(x))$ untouched, which is why a\n", "pre-norm stack needs one final layer norm before the output layer." ] }, { "cell_type": "code", "execution_count": 21, "id": "0da5c1d1", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:59.632639Z", "iopub.status.busy": "2026-10-18T03:36:59.632519Z", "iopub.status.idle": "2026-10-18T03:36:59.638736Z", "shell.execute_reply": "2026-10-18T03:36:59.638297Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "post-norm LN(x + f(x)): output per token mean [0. 0. 0.], std [1. 1. 1.]\n", "pre-norm x + f(LN(x)): output per token mean [1.65 2.39 2.15], std [2.82 3.15 2.59]\n", "in HW7 (reference run, 4 blocks): post-norm about 15.6 vs pre-norm about 15.4 perplexity: about equal; pre-norm's advantage shows in deep stacks\n" ] } ], "source": [ "# [Added for CS6140] PyTorch's encoder layer in both placements, on the same input (dropout off)\n", "torch.manual_seed(0)\n", "x = 3 * torch.randn(1, 5, 16) + 2 # a residual stream that is not normalized\n", "for norm_first in [False, True]:\n", " layer = nn.TransformerEncoderLayer(16, 4, 32, dropout=0.0, batch_first=True, norm_first=norm_first).eval()\n", " with torch.no_grad():\n", " y = layer(x)\n", " name = \"pre-norm x + f(LN(x))\" if norm_first else \"post-norm LN(x + f(x))\"\n", " print(f\"{name}: output per token mean {y.mean(-1)[0, :3].numpy().round(2)}, \"\n", " f\"std {y.std(-1, unbiased=False)[0, :3].numpy().round(2)}\")\n", "print(\"in HW7 (reference run, 4 blocks): post-norm about 15.6 vs pre-norm about 15.4 perplexity: about equal;\"\n", " \" pre-norm's advantage shows in deep stacks\")" ] }, { "cell_type": "markdown", "id": "1d9b57ac", "metadata": {}, "source": [ "## Segment 9 — Sampling: greedy, temperature, top-k (note §9)\n", "**What to watch:** temperature reshapes the distribution (low = sharper, high = flatter); top-$k$ keeps only the\n", "$k$ best words. Check *probabilities*, not text. Greedy is safe but tends to repeat; high temperature makes\n", "word salad." ] }, { "cell_type": "code", "execution_count": 22, "id": "4ca177cf", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:59.640206Z", "iopub.status.busy": "2026-10-18T03:36:59.640082Z", "iopub.status.idle": "2026-10-18T03:36:59.643673Z", "shell.execute_reply": "2026-10-18T03:36:59.643174Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "tau = 1 [0.6439 0.2369 0.0871 0.0321]\n", "tau = 0.5 [0.865 0.1171 0.0158 0.0021]\n", "tau = 2 [0.4551 0.276 0.1674 0.1015]\n", "top-2, tau = 1 [0.7311 0.2689 0. 0. ]\n", "top-2, tau = 0.5 [0.8808 0.1192 0. 0. ]\n" ] } ], "source": [ "# [Added for CS6140] the note's table for logits z = (2, 1, 0, -1), with Hugging Face's logits warpers\n", "from transformers import TemperatureLogitsWarper, TopKLogitsWarper\n", "z = torch.tensor([[2.0, 1.0, 0.0, -1.0]])\n", "for name, warpers in [(\"tau = 1\", [TemperatureLogitsWarper(1.0)]), (\"tau = 0.5\", [TemperatureLogitsWarper(0.5)]),\n", " (\"tau = 2\", [TemperatureLogitsWarper(2.0)]),\n", " (\"top-2, tau = 1\", [TemperatureLogitsWarper(1.0), TopKLogitsWarper(2)]),\n", " (\"top-2, tau = 0.5\", [TemperatureLogitsWarper(0.5), TopKLogitsWarper(2)])]:\n", " scores = z.clone()\n", " for w in warpers:\n", " scores = w(None, scores)\n", " print(f\"{name:17s}\", (scores.softmax(-1)[0]).numpy().round(4))" ] }, { "cell_type": "code", "execution_count": 23, "id": "18f65468", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:36:59.644942Z", "iopub.status.busy": "2026-10-18T03:36:59.644853Z", "iopub.status.idle": "2026-10-18T03:37:00.385730Z", "shell.execute_reply": "2026-10-18T03:37:00.385226Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--- greedy:\n", "once upon a time, there was a little girl named lily. she loved to play with her toys. one day, she went to the park to play with her toys. she saw a big, red ball, and wanted to play with it\n", "--- temperature 0.5:\n", "once upon a time, there was a little girl named lily. she loved to play with her friends. one day, she was playing in the park with her friends. she went to the park with her friends. lily wanted to play with her\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "--- temperature 1.0:\n", "once upon a time, there was a little girl named lily. she had a big coat on her bed waiting for a bench. she was very intelligent and always made beautiful with it. one day, the fairy came to her feelings. the robot\n", "--- temperature 1.5:\n", "once upon a time, there was a little girl named mom. she had a regular kid named lily. it loved anyway tony. one reason abigail liked him one around him grandma advice and dreamed twice. \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "--- top-10:\n", "once upon a time, there was a little girl named lily. she had a big coat on her bed with her mom. she had a special toy that she had made of juice. she was sad and couldn't find her favorite toy to help.\n" ] } ], "source": [ "# [Added for CS6140] stories from the tiny GPT with Hugging Face's generate (same prompt, same seed)\n", "gpt.eval()\n", "prompt = lm.vocab[lm_common.tokenize(\"once upon a time , there was a little\")]\n", "ids = torch.tensor([prompt], device=DEVICE)\n", "settings = [(\"greedy\", dict(do_sample=False)), (\"temperature 0.5\", dict(do_sample=True, temperature=0.5, top_k=0)),\n", " (\"temperature 1.0\", dict(do_sample=True, temperature=1.0, top_k=0)),\n", " (\"temperature 1.5\", dict(do_sample=True, temperature=1.5, top_k=0)),\n", " (\"top-10\", dict(do_sample=True, temperature=1.0, top_k=10))]\n", "for name, kw in settings:\n", " torch.manual_seed(0)\n", " with torch.no_grad():\n", " out = gpt.generate(ids, attention_mask=torch.ones_like(ids), max_new_tokens=40, **kw)\n", " print(f\"--- {name}:\\n{lm_common.detokenize(lm.vocab.to_tokens(out[0].tolist()))}\")" ] }, { "cell_type": "markdown", "id": "cff3f889", "metadata": {}, "source": [ "---\n", "# Appendix (Mon 12/14)\n", "\n", "## Segment 10 — Three kinds of Transformer: attention patterns (note §10)\n", "**What to watch:** which positions may look at which. Encoder-only (BERT): everyone sees everyone. Decoder-only\n", "(GPT): only the past. Encoder–decoder (T5, translation): the source sees itself fully, the target sees its\n", "past, and every target position sees the whole source." ] }, { "cell_type": "code", "execution_count": 24, "id": "b773651e", "metadata": { "execution": { "iopub.execute_input": "2026-10-18T03:37:00.387734Z", "iopub.status.busy": "2026-10-18T03:37:00.387615Z", "iopub.status.idle": "2026-10-18T03:37:00.445664Z", "shell.execute_reply": "2026-10-18T03:37:00.445191Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2026-10-17T23:37:00.426770\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.11.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# [Added for CS6140] attention-pattern grids (white = may attend), a 6-token source and a 5-token target\n", "S, T = 6, 5\n", "full, causal = torch.ones(S, S), torch.tril(torch.ones(T, T))\n", "fig, axes = plt.subplots(1, 4, figsize=(11, 3))\n", "for ax, (title, m, xl, yl) in zip(axes, [(\"encoder-only (BERT)\", full, \"keys\", \"queries\"),\n", " (\"decoder-only (GPT)\", causal, \"keys\", \"queries\"),\n", " (\"enc-dec: decoder self\", causal, \"target keys\", \"target queries\"),\n", " (\"enc-dec: cross\", torch.ones(T, S), \"source keys\", \"target queries\")]):\n", " ax.imshow(m, cmap=\"gray\", vmin=0, vmax=1)\n", " ax.set_title(title, fontsize=9); ax.set_xlabel(xl, fontsize=8); ax.set_ylabel(yl, fontsize=8)\n", " ax.set_xticks([]); ax.set_yticks([])\n", "plt.tight_layout(); plt.show()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.16" } }, "nbformat": 4, "nbformat_minor": 5 }