CS6140 Machine Learning

HW2B -Neural Networks

Make sure you check the syllabus for the due date. Please use the notations adopted in class, even if the problem is stated in the book using a different notation.

Starter code. Answers to the (THEORY) problems may be written directly in your submitted notebook (e.g. a markdown cell) rather than a separate document, as long as they include any equation, plot, or illustration the answer depends on — not prose alone.


PROBLEM 1 [40 points] Autoencoder Neural Network

Consider the following neural network (left graph), with 8 input units (for data with 8 features), 3 hidden units and 8 output units, and assume the nonlinear functions are all sigmoid.

picture

a)The 8 training data inputs are identical with the outputs, as shown in the right side table. Implement this network and the back-propagation algorithm (square loss, sigmoid function) to compute all the network weights; you should initialize the weights with nontrivial values (i.e not values that already minimize the error).

HINT: on the trained network, you should obtain values for the hidden units somehow similar with the ones shown in the table (up to symmetry). Feel free to make changes to the algorithm that suit your implementation, but briefly document them.

b) Since the outputs and inputs are identical for each datapoint, one can view this network as an encoder-decoder mechanism. (this is one of the uses of neural networks). In this context, explain the purpose of the training algorithm. (I expect a rather nontechnical --but documented-- answer).

Hints: Some students worked easier from this algorithm, rather than the one from Mitchell's book
Others found this BackPropagation post from Brilliant easier to code up.

PROBLEM 2 [20 points] Autoencoder Neural Network

Implement the Autoencoder Network in PB1 using PyTorch, and compare with your own implementation. You can use other NNet libraries, but TAs might not be able to help as much.

PROBLEM 3 [40 points] Classifier Square Loss Neural Network

Wine Dataset : train ; test
Implement a multi-class supervised Neural Network. Train and test on wine data (3 labels, 13 features). Layers should look like 13-input, k-hidden, 3-output. Use a square loss and a sigmoid activation function.

PROBLEM 4 [20 points] Classifier Square Loss Neural Network

Train and test the network in PB3 using PyTorch.

PROBLEM 5 (THEORY) [25 points] Why Dropout Lowers Train Accuracy but Raises Test Accuracy

Add dropout (dropping each hidden unit independently with probability p=0.5 during training) to your PROBLEM 1 or PROBLEM 3 network. Compared to the same network trained without dropout: training accuracy is now lower, but test accuracy is higher, and the train/test gap shrinks substantially.

In a short written answer, explain why dropout produces this pattern, and explain (conceptually) why dropout is turned off at test time rather than left on.

PROBLEM 6 (THEORY) [25 points] Diagnosing Learning Rate from a Loss Curve

Suppose two students train the same network with gradient descent and plot training loss vs. iteration. Student A's curve decreases for a few iterations, then starts oscillating wildly and trending upward. Student B's curve decreases smoothly but extremely slowly, flattening out after many iterations at a loss value still far above what a properly-tuned run achieves.

In a short written answer, diagnose each student's problem in terms of the learning rate, and explain the mechanism (not just "it's too big/small") behind each pattern.

PROBLEM 7 [optional, no credit] Classifier Max Likelihood Neural Network

Implement a multi-class supervised Neural Network using likelihood objective . Train and test on wine data.
Use the Maximum Likelihood (cross entropy) for loss function.
Use SoftMax activation function for output layer.
Use either sigmoid or RELU activation function for hidden layer

PROBLEM 8 [optional, no credit] Classifier Max Likelihood Neural Network

Train and test the network in PB7 using PyTorch.

PROBLEM 9 [40 points ] Train WordVectors

Implement the network discussed in class and this tutorial . Make positive word pairs (i,j) relate to frequency of cooccurrence; make negative pairs relate to 0 (you can downsample these for speed).
Train the word vectors on 20NG dataset , and separately on Text8 wikipedia dataset (also found with various sizes on MLdata folder )

Evaluate it by picking non-trivial words ("China", "computer", "phone", "God", "Napoleon", "Catholic" etc) and list for each the most similar 15 words by cosine-of-wordvectors.

PROBLEM 10 [20 points] Train WordVectors

Train the network in PB9 using PyTorch.

PROBLEM 11 [Optional, no credit]

Run your Max Likelihood Neural Network on a medium-size dataset
Digits_small

..and on a larger dataset
Digits Large Dataset (Training data, labels.  Testing data, labels): about 60,000 images, each 28x28 pixels representing digit scans. Each image is labeled with the digit represented, one of 10 classes: 0,1,2,...,9.