Machine Learning and Privacy Vitaly Shmatikov
Typical Task: Classification Training Set Query Classification result airplane automobile ship truck slide 4
Deep Neural Networks input output slide 5
activation signals Deep Neural Networks bias 1 w_0 neurons in the previous layer a_1 a_2 a_k w_1 w_2 w_k parameters f(wa) activation function next layer Learn parameters using Stochastic Gradient Descent (SGD) slide 6
Parameter Training using SGD w_13 w_11 w_12 w_23 w_21 w_22 w_33 w_31 w_32 w_41 w_42 w_43 w_1i w_2j w_3k w_4n Find parameters that minimize the classification error slide 7
Parameter Training using SGD ) 1Feed-Forward input output slide 8
Parameter Training using SGD ) 2Back-propagation error slide 9
Parameter Training using SGD ) 3Gradient Descent error E slide 10
Parameter Training using SGD Parameter Update Repeat for new batches of training data slide 11
2014 Users data Services Threats Collection of sensitive personal data Anonymization and re-identification Inference attacks Side channels slide 12
2018 Users data Machine learning Services Do trained models leak sensitive data? Is it possible to train a good model while respecting privacy of training data? Is it possible to keep the model itself private? slide 13
Model Inversion Fredrikson et al. Given an output of a machine learning model, infer something about the input unexpected attributes slide 14
Model Inversion in Action Model given patient s genome determine correct warfarin dosage Privacy breach : given patient s warfarin dosage, infer information about patient s genome What does this chart measure? slide 15
Does Inference Breach Privacy? Model training set slide 16
Recommended Reading Frank McSherry. Statistical inference considered harmful https://github.com/frankmcsherry/blog/blob/master/posts/2016-06-14.md slide 17
Machine Learning as a Service Model Prediction API Training API Input from users, apps Classification DATA Sensitive! Transactions, preferences, online and offline behavior slide 18
Exploiting Trained Models Model Prediction API Training API Input from the training set Classification DATA Input not from the training set Classification recognize the difference slide 19
ML Against ML Model without knowing the specifics of the actual model! Prediction API Training API Input from the training set Classification DATA Train a model to Input not from the training set Classification recognize the difference slide 20
Training Attack Model using Shadow Models Target Model Shadow Model 1 Shadow Model 2 Shadow Model k Train 1 classification classification Test 1 Train 2 Test 2 Train k Test k classification IN OUT IN OUT IN OUT Train the attack model to predict if an input was a member of the training set (in ) or a non-member (out) slide 21
Training Data for Shadow Models Real: must be similar to training data of the target model (drawn from same distribution) Synthetic: sample feature values from (known) marginal distributions Synthetic: exploit target model Sample from inputs classified by the target model with high confidence Confidence of target model s predictions input space target s training inputs slide 22
Synthesizing Shadow Training Data Hill-climb the space of possible inputs to find those classified by the target model with high confidence Sample from these inputs to synthesize the training dataset for shadow models If many candidate inputs rejected by the target model, re-randomize some features and try again slide 23
Membership Inference Attack Input (data) Output (classes and confidence values) airplane automobile ship truck Was this image part of the training set? slide 24
Model Prediction API Training API target data record DATA Membership Inference Attack Was this record in the training set? Training Set slide 25
Minimum Attack Accuracy on 75% of classes 0.8 0.9 Purchase Dataset Classify Customers slide 26
Next Step: Reconstruction Model Prediction API Training API Partial record????? DATA Auxiliary information, public databases, accidentally revealed data INFER hidden parts of the customer record Example: store purchases or mobile phone locations slide 27
Why Do These Attacks Work? Model Overfitted! Prediction API Training API Membership Inference Reconstruction DATA slide 28
Attack Success vs. Test-Train Gap More overfitted slide 29
Privacy : Does the model leak information about data in the training set? Learning : Does the model generalize to data outside the training set? Model training set Overfitting is the common enemy data universe slide 30
Does Inference Breach Privacy? SCIENCE! Model training set PRIVACY BREACH! Privacy breach = risk of membership: Gap between what can be inferred from the model about a member of the training set and an arbitrary input from the population slide 31
Non-Members Risk of membership Baseline (use statistics) Members of Training Set Purchase Dataset Classify Customers Google API slide 32
Future Modern machine learning is both a threat and an opportunity for data privacy For once, privacy and utility are not in conflict: overfitting is the common enemy Overfitted models leak training data Overfitted models lack predictive power Need generalizability and accuracy slide 33
Utility Privacy-preserving machine learning Privacy slide 34