CS6140 Machine Learning

HW2B -Neural Networks

Make sure you check the syllabus for the due date. Please use the notations adopted in class, even if the problem is stated in the book using a different notation.


PROBLEM 1 [40 points] Autoencoder Neural Network

Consider the following neural network (left graph), with 8 input units (for data with 8 features), 3 hidden units and 8 output units, and assume the nonlinear functions are all sigmoid.

picture

a)The 8 training data inputs are identical with the outputs, as shown in the right side table. Implement this network and the back-propagation algorithm (square loss, sigmoid function) to compute all the network weights; you should initialize the weights with nontrivial values (i.e not values that already minimize the error).

HINT: on the trained network, you should obtain values for the hidden units somehow similar with the ones shown in the table (up to symmetry). Feel free to make changes to the algorithm that suit your implementation, but briefly document them.

b) Since the outputs and inputs are identical for each datapoint, one can view this network as an encoder-decoder mechanism. (this is one of the uses of neural networks). In this context, explain the purpose of the training algorithm. (I expect a rather nontechnical --but documented-- answer).

Hints: Some students worked easier from this algorithm, rather than the one from Mitchell's book
Others found this BackPropagation post from Brilliant easier to code up.

PROBLEM 2 [20 points] Autoencoder Neural Network

Implement the Autoencoder Network in PB1 using TensorFlow or PyTorch, and compare with your own implementation. You can use other NNet libraries, but TAs might not be able to help as much.

PROBLEM 3 [30 points] Classifier Square Loss Neural Network

Wine Dataset : train ; test
Implement a multi-class supervised Neural Network. Train and test on wine data (3 labels, 13 features). Layers should look like 13-input, k-hidden, 3-output. Use a square loss and a sigmoid activation function.

PROBLEM 4 [20 points] Classifier Square Loss Neural Network

Train and test the network in PB3 using TensorFlow or PyTorch.

PROBLEM 5 [30 points GR-ONLY] Classifier Max Likelihood Neural Network

Implement a multi-class supervised Neural Network using likelihod objective . Train and test on wine data.
Use the Maximum Likelihood (cross entropy) for loss function.
Use SoftMax activation function for output layer.
Use either sigmoid or RELU activation function for hidden layer

PROBLEM 6 [20 points GR-ONLY] Classifier Max Likelihood Neural Network

Train and test the network in PB5 using TensorFlow.

PROBLEM 7 [Extra Credit]

Run your Max Likelihood Neural Network on a medium-size dataset
Digits_small

..and on a larger dataset
Digits Large Dataset (Training data, labels.  Testing data, labels): about 60,000 images, each 28x28 pixels representing digit scans. Each image is labeled with the digit represented, one of 10 classes: 0,1,2,...,9.