DS 5220: Supervised Machine Learning and Learning Theory

Fall 2019

Class description:

Machine learning is a fast-pacing and exciting field achieving human-level performance in tasks such as image classification, speech recognition. machine translation, precision medicine, and self-driving cars. Machine learning has already impacted greatly our daily lives and has the potential to transform the world even more in the near future. This course will provide a broad introduction to machine learning and cover the fundamental algorithms for supervised and unsupervised learning. We will cover topics related to regression, classification, ensemble learning, neural networks, and deep learning. The class will also provide an introduction into adversarial machine learning, an emerging area that studies the fundamental security issues of machine learning,

 

Instructors:

Class Schedule:

·      Monday and Wednesday, 2:50-4:30pm

·      Location: WVH 108

Office Hours:

·      Alina: Wednesday, 4:30-6:00pm, ISEC 625

·      Ewen: Thursday, 5:00-6:00pm, ISEC 605

·      Christopher: Monday, 5:00-6:00pm, ISEC 605

Class policies: 

·      We will use a Piazza forum and Gradescope for homework submission.

·      Academic integrity policy is strictly enforced.

 

Pre-requisites: 

·      Probability

·      Statistics

·      Linear algebra

(Materials covered in class DS 5020)

Textbooks

·      [ISL] Gareth JamesDaniela WittenTrevor Hastie and Robert Tibshirani. An Introduction to Statistical Learning with Applications in R. [PDF]

·      [ESL] Trevor Hastie, Rob Tibshirani, and Jerry Friedman, Elements of Statistical Learning, Second Edition, Springer, 2009.

·      [Bishop] Christopher Bishop. Pattern Recognition and Machine Learning. Springer, 2006.

·      [D2L] A. Zhang, Z. Lipton, and A. Smola. Dive into Deep Learning  

 

Grading

The grade will be based on:

 

-       Assignments – 20%

-       Final project report and presentation – 25%

-       Midterm exam – 25%

-       Final exam – 25%

-       Class participation – 5%

 

Calendar (Tentative)



Unit

Week

Date

Topic

Readings

Introduction and review

1

Wed

09/04

Course outline (syllabus, grading, policies)

Introduction to supervised learning [PDF]

[ISL] Chapters 1 and 2

2

Mon

09/09

Learning problems and challenges

Linear algebra and probability review. [PDF]

Linear algebra review from Stanford

Probability review from Stanford

[Bishop] 1.2

Linear regression

Wed

09/11

Bivariate normal distribution.

Simple linear regression.

Maximum likelihood for linear regression [PDF]

[RW] Chapter 1

[ESL] Chapters 3.1-3.2

[RW] Chapter 5.1

3

Mon

09/16

Multiple linear regression.

Gradient descent for linear regression [PDF]

[RW] Chapter 5.2

[RW] Chapter 2

 

Wed

09/18

Linear classification

Perceptron, LDA

Guest lecture [PDF]

[ESL] Chapter 4.3

4

Mon

09/23

LDA.

Bias-variance decomposition [PDF]

[RW] Chapter 5.3

[ESL] Chapter 4.3

Linear classification and evaluation Metrics

Wed

09/25

Gradient descent

Regularization

Ridge regression [PDF]

[RW] Chapter 2

[ESL] Chapter 3.4

 

5

Mon

09/30

k-Nearest Neighbors (kNN).

Cross validation [PDF]

Logistic regression

Lecture notes from Stanford

 

Wed

10/02

CLASS CANCELLED

Python Programming Session in ISEC 655, 5-6pm

6

Mon

10/07

Gradient Descent for Logistic Regression.

Metrics, ROC curves

Evaluation of ML [PDF]

[RW] Chapter 8.1

Decision trees and ensembles

Wed

10/09

Density estimation.

Naïve Bayes. KDE  [PDF]

[ESL] Chapter 6.6.3

[RW] Chapter 8.2

 

7

Mon

10/14

Columbus Day, no class

 

Wed

10/16

Feature selection

Information Gain

Decision trees [PDF]

[ESL] Chapter 9.2.3

Tree handout

 

88

Mon

10/21

Decision trees, cont.

[PDF]

Project proposal due

 

Wed

10/23

Midterm prep [PDF]

[

 

9

Mon

10/28

Midterm exam

 

Non-linear classifiers

Wed

10/30

Ensemble learning

Bagging; Random forests

Boosting; AdaBoost [PDF]

[ESL] Chapter 10.1

[ESL] Chapter 15.1-15.3

 

10

Mon

11/04

 

AdaBoost

Neural networks and deep learning. [PDF]

http://cs229.stanford.edu/notes/cs229-notes-deep_learning.pdf

 

Neural networks and deep learning

Wed

11/06

Feed-Forward Networks. Forward Propagation. [PDF]

Optional: [D2L] Chapter 4

 

11

Mon

11/11

Veterans Day, no class

 

Wed

11/13

Multi-class classification

Convolutional neural networks. [PDF]

Optional: [D2L] Chapter 6

12

Mon

11/18

CNN

Regularization [PDF]

Project milestone due

 

Wed

11/20

Backpropagation [PDF]

 

Security of ML

13

Mon

11/25

SVM

Maximum margin classifier.

Kernels [PDF]

[ESL] Chapter 4.5.2

[ESL] Chapter 12.1, 12.2, 12.3.1

 

Wed

11/27

Thanksgiving; no class

 

Review

Presentations

Exam

14

Mon

12/02

Review and exam preparation [PDF]

 

 

Wed

12/04

Final exam

 

 

 

12/09

Project presentations (tentative)

 

 

 

12/10

Project report due