CS6140 Machine Learning

HW6 Modern Neural Networks: Convolution NN, Recurrent NN, LSTM

Make sure you check the syllabus for the due date. Please use the notations adopted in class.

Instructions. Submit your code (Jupyter notebook or scripts) together with a short report of results (accuracy/loss curves, example outputs). Libraries (PyTorch, numpy, matplotlib) are allowed for data handling, plotting, and standard layers; where a problem says "implement your own" or names a specific _scratch class, that piece must be your own, not a call to the PyTorch built-in it replaces.

Requirements.

Datasets. CIFAR and MNIST (PROBLEM 1, both via torchvision); the English–Spanish translation pairs data (PROBLEMS 2 and 3).

Starter code. PROBLEM 2 gives you the full encoder-decoder pipeline using PyTorch's built-in nn.RNN/nn.LSTM as a library-call baseline; PROBLEM 3 asks for the from-scratch counterpart, evaluated the same way — your from-scratch numbers should land on or near PROBLEM 2's. Answers to the (THEORY) problems may be written directly in your submitted notebook (e.g. a markdown cell) rather than a separate document, as long as they include any equation, plot, or illustration the answer depends on — not prose alone.

PROBLEM 1 CNN implementation in Pytorch/D2L [100 points]

Starting Code for Pb1 (modified LeNet from D2L book)
(A 5p) Run the basic LeNet design on CIFAR and MNIST datasets
(B 30p) Implement your own LazyConv2d_scratch() and AvgPool2D_scratch() and run LeNet_scratch design. You can do this by filling in the TODO sections, or code the cells on your own
(C 5p) Replace Softmax/Sigmoid + AvgPool with ReLU + MaxPool built in libraries: run LeNet_scratch_ReLU_MaxPool network
(D 5p) Add a third convolution layer, and change the channel numbers to 16,32, 64. Run the LeNet_scratch_ReLU_MaxPool_3Conv network
(E 30p) Implement batch normalization by filling the TODOs in the two respective cells. Run the BNLeNet network
(F 25p) Implement Residual Blocks by filling the TODO (or write your own). Run the BNLeNet_ResBlock design

PROBLEM 2 Recurrent NN : implement Encoder and Decoder [50 points]

Starting Code for Pb2
training script for Pb2
To setup: see the README, get the data, get the code: data_utils, evaluate. In this problem below you will train and evaluate for translation from English to Spanish
(A) Code up the Encoder and Decoder, and use the "nn.RNN" built in pipeline.
(B) Replace "nn.RNN" with the "nn.LSTM" built-in pipeline


PROBLEM 3 Recurrent NN : Implement the RNN and LSTM

Starting Code for Pb3
training script for Pb3
(A) [50 points] Encoder, Decoder: Replace "nn.RNN" with your own RNN pipeline, using code from the d2l book
(B) [optional, no credit] Encoder, Decoder: Replace "nn.LSTM" with your own LSTM, using LSTM code from the d2l book


PROBLEM 4 (THEORY) Why LSTM Handles Long-Range Dependencies Better Than a Plain RNN [25 points]

Your PROBLEM 3 plain RNN does noticeably worse on long sequences than on short ones — it seems to "forget" information from early in the sequence by the time it reaches the end. Switching the exact same architecture to an LSTM (same layer sizes, same training setup) substantially closes this gap on long sequences.

In a short written answer, explain, in terms of what happens during backpropagation through time, why the plain RNN struggles with long-range dependencies, and explain specifically what mechanism in the LSTM addresses that problem.

PROBLEM 5 (THEORY) CNN Invariance: What It Covers and What It Doesn't [25 points]

Your PROBLEM 1 CNN is trained only on images where the object of interest is centered and upright. On a held-out test set from the same distribution, it performs very well. But when the exact same trained model is evaluated on images where the object has been shifted to a corner of the frame, or rotated 90 degrees, accuracy drops sharply.

In a short written answer, explain what property of convolution + pooling gives CNNs some built-in invariance, and why it does not fully protect against the specific changes described here. Propose one concrete fix.