{
"cells": [
{
"cell_type": "markdown",
"id": "803129d1",
"metadata": {},
"source": [
"# CNN Lecture Demos (Week 11)\n",
"\n",
"Laptop demos for the CNN lecture, in the order of the lecture plan. Each segment matches a block on the\n",
"board and a section of the note *Convolutional Networks: From Sliding Windows to ResNets*. The code\n",
"starts from the d2l book (chapters 7, 8 and 14.1–14.2); the first line of every code cell says where it\n",
"comes from:\n",
"\n",
"* `# [Book §x.y]`: the book's code as is;\n",
"* `# [Changed from book §x.y]`: the book's code with one change, named on that line;\n",
"* `# [Added for CS6140]`: something the book does not show.\n",
"\n",
"| Seg. | Board topic (note §) | Changed / added vs. the book | Time |\n",
"|---|---|---|---|\n",
"| 1 | Cross-correlation, padding, stride (§2) | — (book code) | <1 s |\n",
"| 2 | Channels, 1×1, convolution as one matrix product (§3) | Changed: multi-channel example through `F.conv2d`; Added: the patch matrix with `F.unfold` | <1 s |\n",
"| 3 | Pooling; small shifts (§4) | Changed: `nn` pooling layers instead of the book's loop; Added: the 1-D window-boundary example | <1 s |\n",
"| 4 | LeNet then and now (§5) | Changed: 3 epochs (book: 10); Added: modern LeNet vs MLP on shifted images | 33 s |\n",
"| 5 | Batch normalization (§6) | Changed: `nn.BatchNorm` LeNet only, 3 epochs; Added: BN vs LayerNorm toy matrix | 24 s |\n",
"| 6 | Residual blocks (§7) | Added: the note's 16 → 32-channel, stride-2 shape walk | <1 s |\n",
"| 7 | Augmentation and transfer (§8) | Changed: CIFAR-10 images (book: a cat photo); Added: pretrained ResNet-18 linear probe on 500 images | 4 s |\n",
"\n",
"Times: measured on an Apple M1 Max (Apple GPU where it helps), first run excluded (downloads).\n"
]
},
{
"cell_type": "markdown",
"id": "f4425dbe",
"metadata": {},
"source": [
"## Setup (run once)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "14398345",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:32.911950Z",
"iopub.status.busy": "2026-10-18T03:15:32.911825Z",
"iopub.status.idle": "2026-10-18T03:15:34.858736Z",
"shell.execute_reply": "2026-10-18T03:15:34.858213Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"device: mps | data: /Users/vip/Dropbox/CS6140/data\n"
]
}
],
"source": [
"# [Added for CS6140] device, the d2l Apple-GPU patch (as in the HW starters), and the course data folder\n",
"import os, time\n",
"import torch\n",
"from torch import nn\n",
"import torch.nn.functional as F\n",
"import torchvision\n",
"from torchvision.transforms import v2\n",
"from d2l import torch as d2l\n",
"import matplotlib.pyplot as plt\n",
"\n",
"DEVICE = (torch.device(\"cuda\") if torch.cuda.is_available() else\n",
" torch.device(\"mps\") if torch.backends.mps.is_available() else torch.device(\"cpu\"))\n",
"if DEVICE.type == \"mps\": # d2l only knows CUDA: point its GPU helpers at Apple's GPU\n",
" d2l.num_gpus = lambda: 1\n",
" d2l.gpu = lambda i=0: DEVICE\n",
" d2l.try_gpu = lambda i=0: DEVICE\n",
"DATA = next((d for d in [\"../data\", \"../../data\", \"../../../data\"] if os.path.isdir(d)), \"../data\")\n",
"TORCH_DATA = f\"{DATA}/torchvision\"\n",
"torch.hub.set_dir(f\"{TORCH_DATA}/hub\")\n",
"d2l.DataModule.__init__.__defaults__ = (TORCH_DATA, 4) # d2l's datasets download into the course folder\n",
"torch.manual_seed(0)\n",
"print(\"device:\", DEVICE, \"| data:\", os.path.abspath(DATA))"
]
},
{
"cell_type": "markdown",
"id": "0efe30a8",
"metadata": {},
"source": [
"## Segment 1 — Cross-correlation, padding, stride (note §2)\n",
"\n",
"**What to watch:** the board's 3×3 example gives 19, 25, 37, 43; a learned kernel recovers the edge detector\n",
"$[1,-1]$ from data alone; padding keeps the size, stride shrinks it."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "ebb88cfe",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.860229Z",
"iopub.status.busy": "2026-10-18T03:15:34.860050Z",
"iopub.status.idle": "2026-10-18T03:15:34.866050Z",
"shell.execute_reply": "2026-10-18T03:15:34.865212Z"
}
},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[19., 25.],\n",
" [37., 43.]])"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# [Book §7.2] the board's example: a 3x3 image, a 2x2 kernel (d2l.corr2d is the book's loop)\n",
"X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]])\n",
"K = torch.tensor([[0.0, 1.0], [2.0, 3.0]])\n",
"d2l.corr2d(X, K)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "f47870f3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.867745Z",
"iopub.status.busy": "2026-10-18T03:15:34.867587Z",
"iopub.status.idle": "2026-10-18T03:15:34.872231Z",
"shell.execute_reply": "2026-10-18T03:15:34.871832Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tensor([[1., 1., 0., 0., 0., 0., 1., 1.],\n",
" [1., 1., 0., 0., 0., 0., 1., 1.],\n",
" [1., 1., 0., 0., 0., 0., 1., 1.],\n",
" [1., 1., 0., 0., 0., 0., 1., 1.],\n",
" [1., 1., 0., 0., 0., 0., 1., 1.],\n",
" [1., 1., 0., 0., 0., 0., 1., 1.]])\n",
"tensor([[ 0., 1., 0., 0., 0., -1., 0.],\n",
" [ 0., 1., 0., 0., 0., -1., 0.],\n",
" [ 0., 1., 0., 0., 0., -1., 0.],\n",
" [ 0., 1., 0., 0., 0., -1., 0.],\n",
" [ 0., 1., 0., 0., 0., -1., 0.],\n",
" [ 0., 1., 0., 0., 0., -1., 0.]])\n",
"transposed image (horizontal edges only): 0.0\n"
]
}
],
"source": [
"# [Book §7.2] edge detection: a 6x8 image, black columns 2-5; the kernel [1, -1] marks the two edges\n",
"X = torch.ones((6, 8))\n",
"X[:, 2:6] = 0\n",
"K = torch.tensor([[1.0, -1.0]])\n",
"Y = d2l.corr2d(X, K)\n",
"print(X); print(Y)\n",
"print(\"transposed image (horizontal edges only):\", d2l.corr2d(X.t(), K).abs().sum().item())"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "78d6b9dc",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.873561Z",
"iopub.status.busy": "2026-10-18T03:15:34.873471Z",
"iopub.status.idle": "2026-10-18T03:15:34.879625Z",
"shell.execute_reply": "2026-10-18T03:15:34.879135Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"epoch 2, loss 8.330\n",
"epoch 4, loss 1.722\n",
"epoch 6, loss 0.422\n",
"epoch 8, loss 0.125\n",
"epoch 10, loss 0.043\n",
"learned kernel: tensor([[ 1.0063, -0.9662]])\n"
]
}
],
"source": [
"# [Book §7.2] learning the kernel from (X, Y) pairs: 10 gradient steps recover about [1, -1]\n",
"conv2d = nn.LazyConv2d(1, kernel_size=(1, 2), bias=False)\n",
"X4, Y4 = X.reshape((1, 1, 6, 8)), Y.reshape((1, 1, 6, 7))\n",
"lr = 3e-2\n",
"for i in range(10):\n",
" l = (conv2d(X4) - Y4) ** 2\n",
" conv2d.zero_grad()\n",
" l.sum().backward()\n",
" conv2d.weight.data[:] -= lr * conv2d.weight.grad\n",
" if (i + 1) % 2 == 0:\n",
" print(f\"epoch {i + 1}, loss {l.sum():.3f}\")\n",
"print(\"learned kernel:\", conv2d.weight.data.reshape((1, 2)))"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "9c28965e",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.881025Z",
"iopub.status.busy": "2026-10-18T03:15:34.880918Z",
"iopub.status.idle": "2026-10-18T03:15:34.885100Z",
"shell.execute_reply": "2026-10-18T03:15:34.884730Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'kernel_size': 3, 'padding': 1} -> (8, 8)\n",
"{'kernel_size': (5, 3), 'padding': (2, 1)} -> (8, 8)\n",
"{'kernel_size': 3, 'padding': 1, 'stride': 2} -> (4, 4)\n",
"{'kernel_size': (3, 5), 'padding': (0, 1), 'stride': (3, 4)} -> (2, 2)\n"
]
}
],
"source": [
"# [Book §7.3] padding and stride: output shapes for an 8x8 input\n",
"def comp_conv2d(conv2d, X):\n",
" X = X.reshape((1, 1) + X.shape)\n",
" Y = conv2d(X)\n",
" return Y.reshape(Y.shape[2:])\n",
"\n",
"X = torch.rand(size=(8, 8))\n",
"for kw in [dict(kernel_size=3, padding=1), dict(kernel_size=(5, 3), padding=(2, 1)),\n",
" dict(kernel_size=3, padding=1, stride=2), dict(kernel_size=(3, 5), padding=(0, 1), stride=(3, 4))]:\n",
" print(kw, \"->\", tuple(comp_conv2d(nn.LazyConv2d(1, **kw), X).shape))"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "bbc17e84",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.886814Z",
"iopub.status.busy": "2026-10-18T03:15:34.886676Z",
"iopub.status.idle": "2026-10-18T03:15:34.890058Z",
"shell.execute_reply": "2026-10-18T03:15:34.889665Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"padding 1:\n",
" tensor([[ 0., 3., 8., 4.],\n",
" [ 9., 19., 25., 10.],\n",
" [21., 37., 43., 16.],\n",
" [ 6., 7., 8., 0.]])\n",
"padding 1, stride (3, 2):\n",
" tensor([[0., 8.],\n",
" [6., 8.]])\n"
]
}
],
"source": [
"# [Added for CS6140] the board's padded and strided versions of the 3x3 example (note §2 numbers)\n",
"X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]]).reshape(1, 1, 3, 3)\n",
"K = torch.tensor([[0.0, 1.0], [2.0, 3.0]]).reshape(1, 1, 2, 2)\n",
"print(\"padding 1:\\n\", F.conv2d(X, K, padding=1)[0, 0])\n",
"print(\"padding 1, stride (3, 2):\\n\", F.conv2d(X, K, padding=1, stride=(3, 2))[0, 0])"
]
},
{
"cell_type": "markdown",
"id": "24894c43",
"metadata": {},
"source": [
"## Segment 2 — Channels, 1×1 convolutions, convolution as one matrix product (note §3)\n",
"\n",
"**What to watch:** two input channels add up (56, 72, 104, 120); a 1×1 convolution is a dense layer on\n",
"the channels at every pixel; unfolding the patches turns the whole convolution into one matrix product."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "52630b94",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.891460Z",
"iopub.status.busy": "2026-10-18T03:15:34.891364Z",
"iopub.status.idle": "2026-10-18T03:15:34.894861Z",
"shell.execute_reply": "2026-10-18T03:15:34.894353Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2 in, 1 out:\n",
" tensor([[ 56., 72.],\n",
" [104., 120.]])\n",
"2 in, 3 out: shape (3, 2, 2)\n"
]
}
],
"source": [
"# [Changed from book §7.4] the book's two-channel example, computed with PyTorch's F.conv2d instead of\n",
"# the book's loop (weight layout (c_out, c_in, k_h, k_w))\n",
"X = torch.tensor([[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]],\n",
" [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]])\n",
"K = torch.tensor([[[0.0, 1.0], [2.0, 3.0]], [[1.0, 2.0], [3.0, 4.0]]])\n",
"print(\"2 in, 1 out:\\n\", F.conv2d(X[None], K[None])[0, 0])\n",
"K3 = torch.stack((K, K + 1, K + 2), 0) # 3 output channels\n",
"print(\"2 in, 3 out: shape\", tuple(F.conv2d(X[None], K3)[0].shape))"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "f85f666b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.896224Z",
"iopub.status.busy": "2026-10-18T03:15:34.896078Z",
"iopub.status.idle": "2026-10-18T03:15:34.899026Z",
"shell.execute_reply": "2026-10-18T03:15:34.898657Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"max difference: 0.0\n"
]
}
],
"source": [
"# [Book §7.4] a 1x1 convolution = a matrix product over the channels at each pixel\n",
"X = torch.normal(0, 1, (3, 3, 3))\n",
"K = torch.normal(0, 1, (2, 3, 1, 1))\n",
"Y1 = (K.reshape(2, 3) @ X.reshape(3, 9)).reshape(2, 3, 3)\n",
"Y2 = F.conv2d(X[None], K)[0]\n",
"print(\"max difference:\", (Y1 - Y2).abs().max().item())"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "80cd6407",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.900179Z",
"iopub.status.busy": "2026-10-18T03:15:34.900098Z",
"iopub.status.idle": "2026-10-18T03:15:34.903013Z",
"shell.execute_reply": "2026-10-18T03:15:34.902661Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"patch matrix (one patch per column):\n",
" tensor([[0., 1., 3., 4.],\n",
" [1., 2., 4., 5.],\n",
" [3., 4., 6., 7.],\n",
" [4., 5., 7., 8.]])\n",
"kernel row x patches: tensor([[19., 25., 37., 43.]])\n"
]
}
],
"source": [
"# [Added for CS6140] convolution as ONE matrix product: PyTorch's F.unfold lists every 2x2 patch as a\n",
"# column; the board's 3x3 example has 4 patches, and kernel x patches = 19, 25, 37, 43\n",
"X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]]).reshape(1, 1, 3, 3)\n",
"K = torch.tensor([[0.0, 1.0], [2.0, 3.0]])\n",
"patches = F.unfold(X, kernel_size=2)[0] # (c*k_h*k_w, number of patches) = (4, 4)\n",
"print(\"patch matrix (one patch per column):\\n\", patches)\n",
"print(\"kernel row x patches:\", K.reshape(1, -1) @ patches)"
]
},
{
"cell_type": "markdown",
"id": "5c598536",
"metadata": {},
"source": [
"## Segment 3 — Pooling: shrink, and forgive small moves (note §4)\n",
"\n",
"**What to watch:** max and average pooling on the 3×3 example; a 1-pixel move *inside* a pooling window\n",
"disappears, a move *across* a window boundary does not."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "24bc1b9c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.904459Z",
"iopub.status.busy": "2026-10-18T03:15:34.904338Z",
"iopub.status.idle": "2026-10-18T03:15:34.908553Z",
"shell.execute_reply": "2026-10-18T03:15:34.908090Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"max:\n",
" tensor([[4., 5.],\n",
" [7., 8.]])\n",
"average:\n",
" tensor([[2., 3.],\n",
" [5., 6.]])\n",
"nn.MaxPool2d(3) on 4x4 (stride = window by default): tensor([[10.]])\n",
"nn.MaxPool2d(3, padding=1, stride=2):\n",
" tensor([[ 5., 7.],\n",
" [13., 15.]])\n"
]
}
],
"source": [
"# [Changed from book §7.5] max and average pooling on the 3x3 example with nn.MaxPool2d / nn.AvgPool2d\n",
"# (instead of the book's loop pool2d); then PyTorch's default stride = window size\n",
"X = torch.tensor([[0.0, 1.0, 2.0], [3.0, 4.0, 5.0], [6.0, 7.0, 8.0]]).reshape(1, 1, 3, 3)\n",
"print(\"max:\\n\", nn.MaxPool2d(2, stride=1)(X)[0, 0])\n",
"print(\"average:\\n\", nn.AvgPool2d(2, stride=1)(X)[0, 0])\n",
"X16 = torch.arange(16, dtype=torch.float32).reshape((1, 1, 4, 4))\n",
"print(\"nn.MaxPool2d(3) on 4x4 (stride = window by default):\", nn.MaxPool2d(3)(X16)[0, 0])\n",
"print(\"nn.MaxPool2d(3, padding=1, stride=2):\\n\", nn.MaxPool2d(3, padding=1, stride=2)(X16)[0, 0])"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "b13c159c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.909850Z",
"iopub.status.busy": "2026-10-18T03:15:34.909757Z",
"iopub.status.idle": "2026-10-18T03:15:34.912243Z",
"shell.execute_reply": "2026-10-18T03:15:34.911858Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"edge at 2: response [0, 0, 1, 0, 0, 0, 0, 0] max-pooled [0, 1, 0, 0]\n",
"edge at 3: response [0, 0, 0, 1, 0, 0, 0, 0] max-pooled [0, 1, 0, 0]\n",
"edge at 4: response [0, 0, 0, 0, 1, 0, 0, 0] max-pooled [0, 0, 1, 0]\n"
]
}
],
"source": [
"# [Added for CS6140] the note's 1-D example: an edge response at position 2, 3, 4, max-pooled (window 2, stride 2)\n",
"for pos in [2, 3, 4]:\n",
" r = torch.zeros(1, 1, 8)\n",
" r[0, 0, pos] = 1\n",
" print(f\"edge at {pos}: response {r[0, 0].int().tolist()} max-pooled {F.max_pool1d(r, 2)[0, 0].int().tolist()}\")"
]
},
{
"cell_type": "markdown",
"id": "28cd8ce8",
"metadata": {},
"source": [
"## Segment 4 — LeNet, then and now (note §5)\n",
"\n",
"**What to watch:** LeNet's layer shapes and 61,706 parameters; the 1998 version (sigmoid, average pooling)\n",
"learns slowly; with ReLU and max pooling it learns fast and holds up much better than an MLP when the images\n",
"move by a couple of pixels."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "99a3d7ae",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.913470Z",
"iopub.status.busy": "2026-10-18T03:15:34.913385Z",
"iopub.status.idle": "2026-10-18T03:15:34.917592Z",
"shell.execute_reply": "2026-10-18T03:15:34.917115Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Conv2d output shape:\t torch.Size([1, 6, 28, 28])\n",
"Sigmoid output shape:\t torch.Size([1, 6, 28, 28])\n",
"AvgPool2d output shape:\t torch.Size([1, 6, 14, 14])\n",
"Conv2d output shape:\t torch.Size([1, 16, 10, 10])\n",
"Sigmoid output shape:\t torch.Size([1, 16, 10, 10])\n",
"AvgPool2d output shape:\t torch.Size([1, 16, 5, 5])\n",
"Flatten output shape:\t torch.Size([1, 400])\n",
"Linear output shape:\t torch.Size([1, 120])\n",
"Sigmoid output shape:\t torch.Size([1, 120])\n",
"Linear output shape:\t torch.Size([1, 84])\n",
"Sigmoid output shape:\t torch.Size([1, 84])\n",
"Linear output shape:\t torch.Size([1, 10])\n",
"parameters: 61706\n"
]
}
],
"source": [
"# [Book §7.6] LeNet-5 (d2l.LeNet: sigmoid + average pooling) and its layer shapes\n",
"model = d2l.LeNet()\n",
"model.layer_summary((1, 1, 28, 28))\n",
"print(\"parameters:\", sum(p.numel() for p in model.parameters()))"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "6a0f2e55",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:34.918772Z",
"iopub.status.busy": "2026-10-18T03:15:34.918688Z",
"iopub.status.idle": "2026-10-18T03:15:59.291355Z",
"shell.execute_reply": "2026-10-18T03:15:59.290915Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"22 s\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# [Changed from book §7.6] train the 1998 LeNet: 3 epochs instead of 10\n",
"data = d2l.FashionMNIST(batch_size=128)\n",
"trainer = d2l.Trainer(max_epochs=3, num_gpus=1)\n",
"model = d2l.LeNet(lr=0.1)\n",
"model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)\n",
"t0 = time.time(); trainer.fit(model, data); print(f\"{time.time() - t0:.0f} s\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "994cef6c",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:15:59.293042Z",
"iopub.status.busy": "2026-10-18T03:15:59.292926Z",
"iopub.status.idle": "2026-10-18T03:16:07.682856Z",
"shell.execute_reply": "2026-10-18T03:16:07.682252Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"trained in 8 s | parameters: MLP 203,530, LeNet 61,706\n",
"shift (px): 0 1 2 3 4\n",
"MLP 0.849 0.776 0.553 0.334 0.241\n",
"LeNet 0.846 0.823 0.750 0.554 0.335\n"
]
}
],
"source": [
"# [Added for CS6140] modern LeNet (ReLU + max pooling) vs an MLP, both trained 2 epochs with Adam, then tested\n",
"# on images shifted right by 0-4 pixels (HW6 P1(C)'s experiment, small)\n",
"def load(train):\n",
" ds = torchvision.datasets.FashionMNIST(root=TORCH_DATA, train=train, download=True)\n",
" return ds.data.float().div(255).unsqueeze(1), ds.targets\n",
"\n",
"Xtr, ytr = load(True); Xte, yte = load(False)\n",
"mlp = nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))\n",
"lenet = d2l.LeNet().net # the book's LeNet, modernized: sigmoid -> ReLU, average -> max pooling\n",
"for i, m in enumerate(lenet):\n",
" if isinstance(m, nn.Sigmoid):\n",
" lenet[i] = nn.ReLU()\n",
" elif isinstance(m, nn.AvgPool2d):\n",
" lenet[i] = nn.MaxPool2d(kernel_size=2, stride=2)\n",
"lenet(torch.zeros(1, 1, 28, 28)) # materialize the lazy layers\n",
"\n",
"def fit(net, epochs=2):\n",
" net.to(DEVICE)\n",
" opt = torch.optim.Adam(net.parameters(), lr=1e-3)\n",
" for _ in range(epochs):\n",
" perm = torch.randperm(len(Xtr))\n",
" for i in range(0, len(Xtr), 128):\n",
" idx = perm[i:i + 128]\n",
" loss = F.cross_entropy(net(Xtr[idx].to(DEVICE)), ytr[idx].to(DEVICE))\n",
" opt.zero_grad(); loss.backward(); opt.step()\n",
" return net.eval()\n",
"\n",
"def shifted(X, s): # move right by s pixels, fill with background (0)\n",
" return X if s == 0 else F.pad(X, (s, 0))[..., :, :-s]\n",
"\n",
"@torch.no_grad()\n",
"def acc(net, X):\n",
" return (net(X.to(DEVICE)).argmax(1).cpu() == yte).float().mean().item()\n",
"\n",
"t0 = time.time()\n",
"fit(mlp); fit(lenet)\n",
"print(f\"trained in {time.time() - t0:.0f} s | parameters: MLP {sum(p.numel() for p in mlp.parameters()):,}, \"\n",
" f\"LeNet {sum(p.numel() for p in lenet.parameters()):,}\")\n",
"print(\"shift (px): \" + \" \".join(f\"{s:5d}\" for s in range(5)))\n",
"for name, net in [(\"MLP\", mlp), (\"LeNet\", lenet)]:\n",
" print(f\"{name:12s} \" + \" \".join(f\"{acc(net, shifted(Xte, s)):.3f}\" for s in range(5)))"
]
},
{
"cell_type": "markdown",
"id": "b807965f",
"metadata": {},
"source": [
"## Segment 5 — Batch normalization (note §6)\n",
"\n",
"**What to watch:** with batch norm the same sigmoid LeNet learns much faster (compare the first epoch with\n",
"Segment 4). The toy matrix: batch norm makes every *column* the same, layer norm every *row*."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "122393ba",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:16:07.684593Z",
"iopub.status.busy": "2026-10-18T03:16:07.684448Z",
"iopub.status.idle": "2026-10-18T03:16:31.302237Z",
"shell.execute_reply": "2026-10-18T03:16:31.301805Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"22 s\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# [Changed from book §8.5] the book's BNLeNet (PyTorch's nn.LazyBatchNorm layers), 3 epochs instead of 10\n",
"class BNLeNet(d2l.Classifier):\n",
" def __init__(self, lr=0.1, num_classes=10):\n",
" super().__init__()\n",
" self.save_hyperparameters()\n",
" self.net = nn.Sequential(\n",
" nn.LazyConv2d(6, kernel_size=5), nn.LazyBatchNorm2d(),\n",
" nn.Sigmoid(), nn.AvgPool2d(kernel_size=2, stride=2),\n",
" nn.LazyConv2d(16, kernel_size=5), nn.LazyBatchNorm2d(),\n",
" nn.Sigmoid(), nn.AvgPool2d(kernel_size=2, stride=2),\n",
" nn.Flatten(), nn.LazyLinear(120), nn.LazyBatchNorm1d(),\n",
" nn.Sigmoid(), nn.LazyLinear(84), nn.LazyBatchNorm1d(),\n",
" nn.Sigmoid(), nn.LazyLinear(num_classes))\n",
"\n",
"trainer = d2l.Trainer(max_epochs=3, num_gpus=1)\n",
"model = BNLeNet(lr=0.1)\n",
"model.apply_init([next(iter(data.get_dataloader(True)))[0]], d2l.init_cnn)\n",
"t0 = time.time(); trainer.fit(model, data); print(f\"{time.time() - t0:.0f} s\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "e01a2bb0",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:16:31.303711Z",
"iopub.status.busy": "2026-10-18T03:16:31.303608Z",
"iopub.status.idle": "2026-10-18T03:16:31.307887Z",
"shell.execute_reply": "2026-10-18T03:16:31.307515Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"BatchNorm (train mode):\n",
" tensor([[-1.336, -1.336, -1.336, -1.336, -1.336],\n",
" [ 1.069, 1.069, 1.069, 1.069, 1.069],\n",
" [ 0.267, 0.267, 0.267, 0.267, 0.267]])\n",
"LayerNorm:\n",
" tensor([[-1.414, -0.707, 0.000, 0.707, 1.414],\n",
" [-1.414, -0.707, 0.000, 0.707, 1.414],\n",
" [-1.414, -0.707, 0.000, 0.707, 1.414]])\n",
"running_mean after one step: tensor([0.600, 1.200, 1.800, 2.400, 3.000])\n",
"running_var[0] after one step: 3.0 (0.9 * 1 + 0.1 * unbiased variance 21)\n"
]
}
],
"source": [
"# [Added for CS6140] the note's toy matrix: BatchNorm normalizes each column (over the batch), LayerNorm each\n",
"# row (over the features); and what one training step does to the running statistics\n",
"x = torch.tensor([[1., 2., 3., 4., 5.], [10., 20., 30., 40., 50.], [7., 14., 21., 28., 35.]])\n",
"bn, ln = nn.BatchNorm1d(5), nn.LayerNorm(5)\n",
"torch.set_printoptions(precision=3, sci_mode=False)\n",
"print(\"BatchNorm (train mode):\\n\", bn(x).detach())\n",
"print(\"LayerNorm:\\n\", ln(x).detach())\n",
"print(\"running_mean after one step:\", bn.running_mean)\n",
"print(\"running_var[0] after one step:\", bn.running_var[0].item(), \"(0.9 * 1 + 0.1 * unbiased variance 21)\")"
]
},
{
"cell_type": "markdown",
"id": "f3d72bae",
"metadata": {},
"source": [
"## Segment 6 — Going deeper: residual blocks (note §7)\n",
"\n",
"**What to watch:** the block keeps the shape, or halves the size and changes the channels when the\n",
"shortcut gets a 1×1 convolution."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "b3755954",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:16:31.309130Z",
"iopub.status.busy": "2026-10-18T03:16:31.309042Z",
"iopub.status.idle": "2026-10-18T03:16:31.314672Z",
"shell.execute_reply": "2026-10-18T03:16:31.314069Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch.Size([4, 3, 6, 6])\n",
"torch.Size([4, 6, 3, 3])\n"
]
}
],
"source": [
"# [Book §8.6] the book's residual block (d2l.Residual): same shape, then halved size + new channels via 1x1\n",
"blk = d2l.Residual(3)\n",
"X = torch.randn(4, 3, 6, 6)\n",
"print(blk(X).shape)\n",
"blk = d2l.Residual(6, use_1x1conv=True, strides=2)\n",
"print(blk(X).shape)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "84750ca4",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:16:31.315892Z",
"iopub.status.busy": "2026-10-18T03:16:31.315795Z",
"iopub.status.idle": "2026-10-18T03:16:31.323006Z",
"shell.execute_reply": "2026-10-18T03:16:31.322497Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"main path conv1: (8, 32, 16, 16) | shortcut 1x1: (8, 32, 16, 16) | block output: (8, 32, 16, 16)\n"
]
}
],
"source": [
"# [Added for CS6140] the note's shape walk: 16 -> 32 channels, stride 2, on 32x32 feature maps\n",
"blk = d2l.Residual(32, use_1x1conv=True, strides=2)\n",
"X = torch.randn(8, 16, 32, 32)\n",
"print(\"main path conv1:\", tuple(blk.conv1(X).shape), \"| shortcut 1x1:\", tuple(blk.conv3(X).shape),\n",
" \"| block output:\", tuple(blk(X).shape))"
]
},
{
"cell_type": "markdown",
"id": "4ffa4084",
"metadata": {},
"source": [
"## Segment 7 — Little data: augmentation and transfer (note §8)\n",
"\n",
"**What to watch:** augmented copies of the same CIFAR-10 image (training data only); a pretrained\n",
"ResNet-18 with only a new last layer, trained on 500 images, against the same network with random weights."
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "22dd68a6",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:16:31.324287Z",
"iopub.status.busy": "2026-10-18T03:16:31.324187Z",
"iopub.status.idle": "2026-10-18T03:16:32.179494Z",
"shell.execute_reply": "2026-10-18T03:16:32.178816Z"
}
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# [Changed from book §14.1] random crops (4-pixel padding) and horizontal flips, on CIFAR-10 images instead of\n",
"# the book's cat photo\n",
"cifar = torchvision.datasets.CIFAR10(root=TORCH_DATA, train=True, download=True)\n",
"aug = v2.Compose([v2.RandomCrop(32, padding=4), v2.RandomHorizontalFlip()])\n",
"fig, axes = plt.subplots(3, 6, figsize=(9, 4.6))\n",
"for r, k in enumerate([0, 7, 29]):\n",
" img = v2.functional.to_image(cifar.data[k])\n",
" for c in range(6):\n",
" axes[r, c].imshow((img if c == 0 else aug(img)).permute(1, 2, 0))\n",
" axes[r, c].set_title(\"original\" if c == 0 else \"augmented\", fontsize=8)\n",
" axes[r, c].axis(\"off\")\n",
"plt.tight_layout(); plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "afd19762",
"metadata": {
"execution": {
"iopub.execute_input": "2026-10-18T03:16:32.181043Z",
"iopub.status.busy": "2026-10-18T03:16:32.180952Z",
"iopub.status.idle": "2026-10-18T03:16:35.309713Z",
"shell.execute_reply": "2026-10-18T03:16:35.309264Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"pretrained linear probe, 500 training images: test accuracy 0.768\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"random weights linear probe, 500 training images: test accuracy 0.283\n",
"2 s\n"
]
}
],
"source": [
"# [Changed from book §14.2] transfer, as a linear probe: frozen ImageNet ResNet-18 features + a new last layer,\n",
"# 50 CIFAR-10 images per class (book: fine-tuning on a hot-dog dataset); random weights for comparison\n",
"def subset(train, per_class):\n",
" ds = torchvision.datasets.CIFAR10(root=TORCH_DATA, train=train, download=True)\n",
" y = torch.tensor(ds.targets)\n",
" idx = torch.cat([torch.nonzero(y == c)[:per_class, 0] for c in range(10)])\n",
" X = torch.tensor(ds.data[idx.numpy()]).permute(0, 3, 1, 2).float().div(255)\n",
" return X, y[idx]\n",
"\n",
"Xs, ys = subset(True, 50)\n",
"Xt, yt = subset(False, 100)\n",
"norm = v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) # ImageNet statistics\n",
"\n",
"@torch.no_grad()\n",
"def features(net, X):\n",
" net.eval().to(DEVICE)\n",
" out = [net(norm(F.interpolate(X[i:i + 100], size=128, mode=\"bilinear\")).to(DEVICE)).cpu()\n",
" for i in range(0, len(X), 100)]\n",
" return torch.cat(out)\n",
"\n",
"t0 = time.time()\n",
"for name, weights in [(\"pretrained\", torchvision.models.ResNet18_Weights.DEFAULT), (\"random weights\", None)]:\n",
" net = torchvision.models.resnet18(weights=weights)\n",
" net.fc = nn.Identity() # the 512 features before the last layer\n",
" Ftr, Fte = features(net, Xs), features(net, Xt)\n",
" head = nn.Linear(512, 10)\n",
" opt = torch.optim.Adam(head.parameters(), lr=1e-3)\n",
" for _ in range(300):\n",
" loss = F.cross_entropy(head(Ftr), ys)\n",
" opt.zero_grad(); loss.backward(); opt.step()\n",
" print(f\"{name:15s} linear probe, 500 training images: test accuracy \"\n",
" f\"{(head(Fte).argmax(1) == yt).float().mean():.3f}\")\n",
"print(f\"{time.time() - t0:.0f} s\")"
]
},
{
"cell_type": "markdown",
"id": "bc8d1495",
"metadata": {},
"source": [
"---\n",
"*Book code: A. Zhang, Z. C. Lipton, M. Li, A. J. Smola, Dive into Deep Learning (d2l.ai), CC BY-SA 4.0.\n",
"Changes and additions for CS6140 are marked in each cell.*"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.14.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}