Make sure you check the syllabus for the due date. Please use the notations adopted in class, even if the problem is stated in the book using a different notation.
We are not looking for very long answers (if you find yourself writing more than 1-2 page(s) of typed text per problem, you are probably on the wrong track). Try to be concise; also keep in mind that good ideas and explanations matter more than exact details.
Consider the following neural network (left graph), with 8 input units (for data with 8 features), 3 hidden units and 8 output units, and assume the nonlinear functions are all sigmoid.
a)The 8 training data inputs are identical with the outputs, as shown in the right side table. Implement this network and the backpropagation algorithm to compute all the network weights; you should initialize the weights with nontrivial values (i.e not values that already minimize the erorr).
HINT: on the trained network, you should obtain values for the hidden units somehow similar with the ones shown in the table (up to symmetry). Feel free to make changes to the algorithm that suit your implementation, but briefly document them.
b) Since the outputs and inputs are identical for each datapoint, one can view this network as an encoder-decoder mechanism. (this is one of the uses of neural networks). In this context, explain the purpose of the training algorithm. (I expect a rather nontechnical --but documented-- answer).
c) Can this encoder-decoder scheme work with 1 hidden unit? Can it work with 2 hidden units? What if there are more than 8 inputs and outputs? Justify your answers mathematically.
Tennis toy data is presented in a table.
a) Run the perceptron algorithm on the tennis data (each day is a datapoint). Use a one-out-of-three encoding for Outlook and for Temperature, and a single node for each of Humidity and Wind. Is the data linearly separable ?
b) Write a program that takes as input any subset of the 14 tennis records ad implements the corresponding naive Bayes classifier. Design it so that it estimate each conditional probability using a bayesian approach based on a uniform prior. Split the data randomly into K=7 folds of 2 recods each and run your algorithm in a K-fold cross validation fashion. Report the cross-validated training and testing errors.
a) Prove that
a) DHS CH2, Pb 44
b) DHS CH2, Pb 45
A good introduction for your project, read prof Andrew Ng's lecture on ML practice advice. Write a brief summary (1 page) explaining the quantities in the lecture and the advice.
For a function f(x1,x2,..., xn) with real values, the "Hessian" is the matrix of partial second derivatives
Consider the log-likelihood function for logistic regression
Show that its Hessian matrix H is negative semidefinite, i.e. for any vector z satisfies
Remark: This fact is sometimes written and implies the log-likelihood function is concave.
Hint: 