![Page 1: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/1.jpg)
Differential PrivacyStudy Group March 2017
![Page 2: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/2.jpg)
Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K. and Zhang, L., 2016, October. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (pp. 308-318). ACM.
Deep Learning with Differential Privacy
![Page 3: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/3.jpg)
MotivationAbadi, M. et al. Deep learning with differential privacy. 2016
![Page 4: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/4.jpg)
![Page 5: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/5.jpg)
Fredrikson, M., Jha, S. and Ristenpart, T., 2015, October. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (pp. 1322-1333). ACM.
![Page 6: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/6.jpg)
![Page 7: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/7.jpg)
![Page 8: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/8.jpg)
ε,𝛿
![Page 9: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/9.jpg)
ε,𝛿
Theorem 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 10: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/10.jpg)
Algorithm
![Page 11: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/11.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 12: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/12.jpg)
N
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 13: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/13.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
N
![Page 14: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/14.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
N
gt(x3)gt(x3)gt(x8)gt(x11)
![Page 15: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/15.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
N
gt(x3)gt(x3)gt(x8)gt(x11)
clipping also seen in normal SGD for non-privacy reasons — but then done on the batch level
![Page 16: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/16.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
N
gt(x3)gt(x3)gt(x8)gt(x11)
![Page 17: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/17.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
N
lot
batch
![Page 18: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/18.jpg)
Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016
N
lotbatch
![Page 19: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/19.jpg)
Implementation
![Page 20: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/20.jpg)
class DPSGD_Optimizer(): def __init__(self, accountant, sanitizer): self._accountant = accountant self._sanitizer = sanitizer
def Minimize(self, loss, params, batch_size, noise_options): # Accumulate privacy spending before computing # and using the gradients. priv_accum_op = self._accountant.AccumulatePrivacySpending( batch_size, noise_options) with tf.control_dependencies(priv_accum_op): # Compute per example gradients px_grads = per_example_gradients(loss, params) # Sanitize gradients sanitized_grads = self._sanitizer.Sanitize( px_grads, noise_options) # Take a gradient descent step return apply_gradients(params, sanitized_grads)
def DPTrain(loss, params, batch_size, noise_options): accountant = PrivacyAccountant() sanitizer = Sanitizer() dp_opt = DPSGD_Optimizer(accountant, sanitizer) sgd_op = dp_opt.Minimize( loss, params, batch_size, noise_options) eps, delta = (0, 0) # Carry out the training as long as the privacy # is within the pre-set limit. while within_limit(eps, delta): sgd_op.run() eps, delta = accountant.GetSpentPrivacy()
Figure 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 21: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/21.jpg)
class DPSGD_Optimizer(): def __init__(self, accountant, sanitizer): self._accountant = accountant self._sanitizer = sanitizer
def Minimize(self, loss, params, batch_size, noise_options): # Accumulate privacy spending before computing # and using the gradients. priv_accum_op = self._accountant.AccumulatePrivacySpending( batch_size, noise_options) with tf.control_dependencies(priv_accum_op): # Compute per example gradients px_grads = per_example_gradients(loss, params) # Sanitize gradients sanitized_grads = self._sanitizer.Sanitize( px_grads, noise_options) # Take a gradient descent step return apply_gradients(params, sanitized_grads)
def DPTrain(loss, params, batch_size, noise_options): accountant = PrivacyAccountant() sanitizer = Sanitizer() dp_opt = DPSGD_Optimizer(accountant, sanitizer) sgd_op = dp_opt.Minimize( loss, params, batch_size, noise_options) eps, delta = (0, 0) # Carry out the training as long as the privacy # is within the pre-set limit. while within_limit(eps, delta): sgd_op.run() eps, delta = accountant.GetSpentPrivacy()
Figure 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 22: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/22.jpg)
class DPSGD_Optimizer(): def __init__(self, accountant, sanitizer): self._accountant = accountant self._sanitizer = sanitizer
def Minimize(self, loss, params, batch_size, noise_options): # Accumulate privacy spending before computing # and using the gradients. priv_accum_op = self._accountant.AccumulatePrivacySpending( batch_size, noise_options) with tf.control_dependencies(priv_accum_op): # Compute per example gradients px_grads = per_example_gradients(loss, params) # Sanitize gradients sanitized_grads = self._sanitizer.Sanitize( px_grads, noise_options) # Take a gradient descent step return apply_gradients(params, sanitized_grads)
def DPTrain(loss, params, batch_size, noise_options): accountant = PrivacyAccountant() sanitizer = Sanitizer() dp_opt = DPSGD_Optimizer(accountant, sanitizer) sgd_op = dp_opt.Minimize( loss, params, batch_size, noise_options) eps, delta = (0, 0) # Carry out the training as long as the privacy # is within the pre-set limit. while within_limit(eps, delta): sgd_op.run() eps, delta = accountant.GetSpentPrivacy()
Figure 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 23: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/23.jpg)
class DPSGD_Optimizer(): def __init__(self, accountant, sanitizer): self._accountant = accountant self._sanitizer = sanitizer
def Minimize(self, loss, params, batch_size, noise_options): # Accumulate privacy spending before computing # and using the gradients. priv_accum_op = self._accountant.AccumulatePrivacySpending( batch_size, noise_options) with tf.control_dependencies(priv_accum_op): # Compute per example gradients px_grads = per_example_gradients(loss, params) # Sanitize gradients sanitized_grads = self._sanitizer.Sanitize( px_grads, noise_options) # Take a gradient descent step return apply_gradients(params, sanitized_grads)
def DPTrain(loss, params, batch_size, noise_options): accountant = PrivacyAccountant() sanitizer = Sanitizer() dp_opt = DPSGD_Optimizer(accountant, sanitizer) sgd_op = dp_opt.Minimize( loss, params, batch_size, noise_options) eps, delta = (0, 0) # Carry out the training as long as the privacy # is within the pre-set limit. while within_limit(eps, delta): sgd_op.run() eps, delta = accountant.GetSpentPrivacy()
Figure 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 24: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/24.jpg)
class DPSGD_Optimizer(): def __init__(self, accountant, sanitizer): self._accountant = accountant self._sanitizer = sanitizer
def Minimize(self, loss, params, batch_size, noise_options): # Accumulate privacy spending before computing # and using the gradients. priv_accum_op = self._accountant.AccumulatePrivacySpending( batch_size, noise_options) with tf.control_dependencies(priv_accum_op): # Compute per example gradients px_grads = per_example_gradients(loss, params) # Sanitize gradients sanitized_grads = self._sanitizer.Sanitize( px_grads, noise_options) # Take a gradient descent step return apply_gradients(params, sanitized_grads)
def DPTrain(loss, params, batch_size, noise_options): accountant = PrivacyAccountant() sanitizer = Sanitizer() dp_opt = DPSGD_Optimizer(accountant, sanitizer) sgd_op = dp_opt.Minimize( loss, params, batch_size, noise_options) eps, delta = (0, 0) # Carry out the training as long as the privacy # is within the pre-set limit. while within_limit(eps, delta): sgd_op.run() eps, delta = accountant.GetSpentPrivacy()
Figure 1: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 25: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/25.jpg)
class DPSGD_Optimizer(): def __init__(self, accountant, sanitizer): self._accountant = accountant self._sanitizer = sanitizer
def Minimize(self, loss, params, batch_size, noise_options): # Accumulate privacy spending before computing # and using the gradients. priv_accum_op = self._accountant.AccumulatePrivacySpending( batch_size, noise_options) with tf.control_dependencies(priv_accum_op): # Compute per example gradients px_grads = per_example_gradients(loss, params) # Sanitize gradients sanitized_grads = self._sanitizer.Sanitize( px_grads, noise_options) # Take a gradient descent step return apply_gradients(params, sanitized_grads)
def DPTrain(loss, params, batch_size, noise_options): accountant = PrivacyAccountant() sanitizer = Sanitizer() dp_opt = DPSGD_Optimizer(accountant, sanitizer) sgd_op = dp_opt.Minimize( loss, params, batch_size, noise_options) eps, delta = (0, 0) # Carry out the training as long as the privacy # is within the pre-set limit. while within_limit(eps, delta): sgd_op.run() eps, delta = accountant.GetSpentPrivacy()
Figure 1: Abadi, M. et al. Deep learning with differential privacy. 2016
Code on GitHub: • More logic for dealing with batches • Two accountants:
• AmortizedAccountant • GaussianMomentsAccountant
• Per example gradient code (including for convolutional layers)
• MNIST example
Also has code for Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data https://github.com/tensorflow/models/tree/master/differential_privacy
![Page 26: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/26.jpg)
Results
![Page 27: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/27.jpg)
Figure 2: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 28: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/28.jpg)
![Page 29: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/29.jpg)
Figure 3: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 30: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/30.jpg)
Figure 3: Abadi, M. et al. Deep learning with differential privacy. 2016
“compare to 98.3% without
![Page 31: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/31.jpg)
Figure 3: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 32: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/32.jpg)
Figure 4: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 33: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/33.jpg)
Figure 5: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 34: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/34.jpg)
86% Accuracy
![Page 35: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/35.jpg)
80 % accuracy
retrain
fix from cifar-100
![Page 36: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/36.jpg)
Figure 6: Abadi, M. et al. Deep learning with differential privacy. 2016
![Page 37: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/37.jpg)
Hall, R., Rinaldo, A. and Wasserman, L., 2013. Differential privacy for functions and functional data. Journal of Machine Learning Research, 14(Feb), pp.703-727.
Differential Privacy for Functions and Functional Data
![Page 38: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/38.jpg)
Motivation
![Page 39: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/39.jpg)
![Page 40: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/40.jpg)
![Page 41: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/41.jpg)
1. Useful if results naturally function valued
2. May want to have a data summary that is a function
![Page 42: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/42.jpg)
1. Useful if results naturally function valued
2. May want to have a data summary that is a function
![Page 43: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/43.jpg)
1. Useful if results naturally function valued
2. May want to have a data summary that is a function
x xx x x xx
![Page 44: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/44.jpg)
Proofs
![Page 45: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/45.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 46: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/46.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
Skip rest of proof of Prop3
![Page 47: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/47.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 48: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/48.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 49: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/49.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 50: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/50.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 51: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/51.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 52: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/52.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 53: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/53.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 54: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/54.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 55: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/55.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 56: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/56.jpg)
Applied to KDEs
![Page 57: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/57.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 58: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/58.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 59: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/59.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 60: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/60.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 61: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/61.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 62: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/62.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 63: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/63.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 64: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/64.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 65: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/65.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 66: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/66.jpg)
Hall, R., et. al. Differential privacy for functions and functional data. 2016
![Page 67: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/67.jpg)
![Page 68: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/68.jpg)
varying alpha
![Page 69: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/69.jpg)
varying alpha
![Page 70: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/70.jpg)
varying beta
![Page 71: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/71.jpg)
varying beta
![Page 72: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/72.jpg)
varying beta
![Page 73: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/73.jpg)
demo maybe…
https://gist.github.com/john-bradshaw/e63d2a20537beda035b32224a1be8831
![Page 74: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/74.jpg)
references
Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K. and Zhang, L., 2016, October. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (pp. 308-318). ACM.
Hall, R., Rinaldo, A. and Wasserman, L., 2013. Differential privacy for functions and functional data. Journal of Machine Learning Research, 14(Feb), pp.703-727.
![Page 75: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/75.jpg)
appendix
![Page 76: Differential Privacy - cbl.eng.cam.ac.ukcbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/... · Algorithm 1: Abadi, M. et al. Deep learning with differential privacy. 2016 N g t(x](https://reader035.vdocuments.site/reader035/viewer/2022080721/5f7a36247ef0c35dc54bbc4a/html5/thumbnails/76.jpg)