CY 7790: Special Topics in Security and Privacy: Machine Learning Security and Privacy

Fall 2021

Class Information

Calendar

Review Materials

Other Resources

 

Instructors:

  • Instructor: Alina Oprea (alinao)
  • TA: Giorgio Severi (severi.g@northeastern.edu)

Class Schedule:

  • Monday and Thursday, 11:45am-1:25pm EST
  • Location: Snell Library 125

Office Hours:

  • Alina: Thursday, 4-5pm EST and by appointment
  • Giorgio: Monday, 4-5pm EST and by appointment

Class forum:  Piazza

Class policies:  Academic integrity policy is strictly enforced

 

Class description: Machine learning is increasingly being used for automated decisions in applications such as health care, finance, autonomous vehicles, personalized recommendations, and cyber security. These critical applications require strong guarantees on both the integrity of the machine learning models and the privacy of the user data used to train these models. The area of adversarial machine learning studies the effect of adversarial attacks against machine learning models and aims to design robust defense algorithms. In this course, we will study a variety of adversarial attacks on machine learning and deep learning systems that impact the security and privacy of these systems, and we will discuss the challenges of designing robust models. The objectives of the course are the following:

·       Provide an overview of several machine learning models for classification and regression, including logistic regression, SVM, decision trees, ensemble learning, and deep neural network architectures. 

·       Discuss generalization in machine learning, the bias-variance tradeoff, and the underlying assumptions that most algorithms rely on. 

·       Provide an in-depth coverage of adversarial attacks on machine learning systems, including evasion attacks at inference time, poisoning attacks at training time, and privacy attacks. Learn how to classify the attacks according to the adversarial objective, knowledge, and capability. 

·       Discuss adversarial attacks in real-world applications, including cyber security, autonomous vehicles, and natural language processing. 

·       Understand existing methods for training robust models and the challenges of achieving both robustness and accuracy. 

·       Discuss fairness issues in machine learning that might exacerbate existing risks of adversarial attacks. 

·       Read research papers from both security and machine learning conferences and discuss them in class. Students will participate in class discussions, lead discussion on selected papers in teams, and write notes about the class discussion. 

·       Provide students the opportunity to work on a semester-long research project on a topic of their choice, as well as complete several assignments on machine learning security and privacy.

 

Pre-requisites:

  • Probability, calculus, and linear algebra
  • Basic knowledge of machine learning is preferred

Grading

The grade will be based on:

 

-       Assignments – 10%

-       Paper summaries– 10%

-       Discussion leading – 15%

-       Scribing – 15%

-       Final project – 50%

-        

 Calendar (Tentative)

 

Unit

Week

Date

Topic

Readings

 

 

Introduction and Review

1

Thu

09/09

 No class

PhD Hooding Ceremony

 

Mon

09/13

Course outline (syllabus, grading, policies)

Introduction [Slides]

2

Thu

09/16

Review of machine learning (regression, classification, gradient descent) [Slides] [Annotations]

Mon

09/20

Review of deep learning [Slides] [Annotations]

3

Thu

09/23

Introduction to Adversarial ML 

[Slides] [Annotations]

Papernot et al. SoK: Security and Privacy in Machine Learning. IEEE Euro S&P 2018

 

Biggio and Roli. Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Mon

09/27

Evasion attacks Gradient-based attacks [Annotations] [Lecture Notes]

Biggio et al. Evasion attacks against machine learning at test time

 

 

Szegedy et al. Intriguing properties of neural networks

 

Optional read: Goodfellow et al. Explaining and Harnessing Adversarial Examples

4

Thu

09/30

Evasion attacks: Carlini-Wagner, Transferability of attacks [Slides] [Lecture Notes]

Carlini and Wagner. Towards Evaluating the Robustness of Neural Networks

 

Papernot et al. Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

Evasion Attacks and Defenses

Mon

10/04

Evasion attacks: Black-box attacks, adversarial training as a defense [Slides] [Lecture Notes]

Guo et al. Simple Black-box Adversarial Attacks

 

Madry et al. Towards Deep Learning Models Resistant to Adversarial Attacks

 

 

5

Thu

10/07

Evasion attacks: Explaining adversarial examples; certified defenses [Slides] [Lecture Notes]

Ilyas et al. Adversarial Examples Are Not Bugs, They Are Features

 

Cohen et al. Certified Adversarial Robustness via Randomized Smoothing

 

Mon

10/11

No class

University holiday

 

 

6

Thu

10/14

Poisoning attacks: Availability, backdoor, subpopulation attacks [Slides] [Lecture Notes]

Gu et al. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv 2017

 

Jagielski et al. Subpopulation Data Poisoning Attacks. arXiv 2020.

 

Optional read: Jagielski et al. Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning

 

Poisoning Attacks and Defenses

Mon

10/18

Poisoning attacks: Targeted attacks, poisoning semi-supervised learning [Slides] [Lecture Notes]

Shafahi et al. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. NIPS 2018

 

Carlini et al. Poisoning the Unlabeled Dataset of Semi-Supervised Learning

 

7

Thu

10/21

Project proposal

 

Mon

10/25

Poisoning defenses [Slides] [Lecture Notes]

Steinhardt et al. Certified Defenses for Data Poisoning Attacks

 

Wang et al. Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks. IEEE S&P 2019

 

8

Thu

10/28

Application domains for evasion attacks: object detectors; security [Slides] [Lecture Notes]

Wu et al. Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectors. ECCV 2020

 

Chen et al. Cost-Aware Robust Tree Ensembles for Security Applications. USENIX Security 2021

Application Domains

Mon

11/01

Application domains for poisoning attacks: malware and NLP [Slides] [Lecture Notes]

Severi et al. Explanation-Guided Backdoor Poisoning Attacks Against Malware Classifiers. USENIX Security 2021

 

Schuster et al. Humpty Dumpty: Controlling Word Meanings via Corpus Poisoning. IEEE S&P 2020

 

9

Thu

11/04

Federated learning poisoning [Slides] [Lecture Notes]

Bagdasaryan et al. How To Backdoor Federated Learning. AISTATS 2020

 

Shejwalkar et al. Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning. NDSS 2021

 

 

Mon

11/08

Privacy risks in ML. Membership Inference attacks [Slides] [Lecture Notes]

Shokri et al. Membership Inference Attacks Against Machine Learning Models. IEEE S&P 2017

 

Yeom et al. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. CSF 2018

 

10

Thu

11/11

No class

University holiday

Mon

11/15

Model extraction attacks and memorization in machine learning [Slides] [Lecture Notes]

 

Tramer et al. Stealing Machine Learning Models via Prediction APIs

 

Carlini et al. Extracting Training Data from Large Language Models. USENIX Security 2021

Privacy Attacks and Defenses

11

Thu

11/18

Differentially private SGD and auditing DP-SGD [Slides] [Lecture Notes]

Abadi et al. Deep Learning with Differential Privacy. ACM CCS 2016

 

Jagielski et al. Auditing Differentially Private Machine Learning: How Private is Private SGD? NeurIPS 2020

 

Project Milestone due on 11/19

 

Mon

11/22

Machine unlearning and model updates [Slides] [Lecture Notes]

Bourtoule et al. Machine unlearning. IEEE S&P 2021

 

Brockschmidt et al. Analyzing Information Leakage of Updates to Natural Language Models. ACM CCS 2020

     12

Thu

11/25

No class

University holiday (Thanksgiving)

Mon

11/29

Fairness in ML [Slides] [Lecture Notes]

Zafar et al. Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment. In WWW 2017

 

Hardt et al. Equality of Opportunity in Supervised Learning. In NeurIPS 2016

 

 

 

Fairness in ML

13

Thu

12/02

Fairness in ML and privacy [Slides] [Lecture Notes]

Chang and Shokri. On the Privacy Risks of Algorithmic Fairness. In Euro S&P 2021

 

Bagdasaryan and Shmatikov. Differential Privacy Has Disparate Impact on Model Accuracy

 

Mon

12/06

Project presentations

 

 

14

Thu

12/09

Project presentations

 

 

Mon

12/13

Final project reports due

 

 

 

Review materials

 

 

Other resources

 

Books: