Privacy-Preserving Machine Learning in

TensorFlow with TF Encrypted

Morten Dahl

O’Reilly AI Conference, New York, April 2019

 Why?
Privacy in machine learning
Machine Learning Process

data set training prediction service

clinical photos transfer learning prediction service

machine learning positioned to have huge impact on health care

 Potential Bottlenecks
data access incentive
(liability and controlled use) (accuracy and exposure)

data set training prediction service

leakage
(model and training data)
risk management

(store and process)
 Sanitization
(Differential Privacy)
data access incentive
(liability and controlled use) (accuracy and exposure)

sanitised data set training sanitised prediction

leakage
(model and training data)
risk management

(store and process)
 Encryption
(Secure Computation)
data access incentive
(liability and controlled use) (accuracy and exposure)

encrypted data set encrypted training encrypted prediction

leakage
(model and training data)
risk management

(store and process)
 Hybrid
data access incentive
(liability and controlled use) (accuracy and exposure)
privacy mitigates bottlenecks

encrypted data set encrypted training encrypted + sanitised


prediction

leakage
(model and training data)
risk management

(store and process)
 How?
Computing on encrypted data
 Prediction with Linear Model
w x x

dot(x, w)

w1

dot( x1 x2 x3 , w2 ) = x1*w1 + x2*w2 + x3*w3

w3
 … using Homomorphic Encryption
w Enc(x) x

Enc(dot(x, w))
homomorphic

encryption scheme

public multiplication
private addition
w1

dot( Enc(x1) Enc(x2) Enc(x3) , w2 ) = Enc(x1)*w1 + Enc(x2)*w2 + Enc(x3)*w3

w3 = Enc(x1*w1) + Enc(x2*w2) + Enc(x3*w3)

= Enc(x1*w1 + x2*w2 + x3*w3)

Paillier Homomorphic Encryption
typically ~4000 bits:
computation is significantly

public encryption key more expensive

c = Enc(x, r) = g^x * r^n mod n^2

g = 36
 Enc(5, 2) = 36^5 * 2^35 mod 1225 = 718

n = 35

n^2 = 1225 Enc(5, 4) = 36^5 * 4^35 mod 1225 = 674
Private Addition in Paillier

Enc(x, r) * Enc(y, s)

= (g^x * r^n mod n^2) * (g^y * s^n mod n^2)

= g^(x + y) * (r * s)^n mod n^2

= Enc(x+y, r*s)

Enc(5, 2) * Enc(5, 4)
= 718 * 674

= 57

= 36^10 * 8^35
= Enc(10, 8)
Public Multiplication in Paillier

Enc(x, r) ^ w

= (g^x * r^n mod n^2) ^ w

= g^(x*w) * (r^w)^n mod n^2

= Enc(x*w, r^w)

Enc(5, 2) ^ 2
= 718 * 718
= 1024

= 36^10 * 4^35
= Enc(10, 4)
… using Secret Sharing
w Share1(x)

Share1(dot(x, w))

w
Share2(x)

Share2(dot(x, w))
 Secret Sharing

Share1(x, r) = r mod m

Share2(x, r) = x - r mod m

x = Share1(x, r) + Share2(x, r) mod m

Share1(5, 7) = 7 mod 10 = 7
m = 10
Share2(5, 7) = 5 - 7 mod 10 = 8

7 + 8 = 15 = 5 mod 10
Private Addition with Secret Sharing

x1 y1 z1 = x1 + y1

x2 y2 z2 = x2 + y2

x+y
= (x1 + x2) + (y1 + y2)

x = x1 + x2 y = y1 + y2
= (x1 + y1) + (x2 + y2)
= z1 + z2
Public Multiplication with Secret Sharing

x1 w z1 = x1 * w

x2 w z2 = x2 * w

x*w
= (x1 + x2) * w

x = x1 + x2
= (x1 * w) + (x2 * w)
= z1 + z2
… using Secret Sharing, with Private Model
Share1(w) Share1(x)

Share1(dot(x, w))

Share2(w) Share2(x)

Share2(dot(x, w))
 … using Secret Sharing, with Private Model
Share1(w) Share1(x)

Share1(dot(x, w))

private multiplication

Share(w0)
Share2(w) Share2(x)
dot( Share1(x0) Share1(x1) Share1(x2) , Share(w1) ) = Share1(x0)*Share(w0) + …

Share2(dot(x, =w)) Share1(x0*w0) + …

Share(w2)

= Share1(x0*w0 + x1*w1 + x2*w2)

Private Multiplication with Secret Sharing
z1 = alpha*beta
( a1 , b1 , c1 ) x1 y1 alpha beta + alpha*b1

+ beta*a1

+ c1

z2 = alpha*b2
( a2 , b2 , c2 ) x2 y2 alpha beta + beta*a2
+ c2

a = a1 + a2 x * y

alpha = x - a
b = b1 + b2 x = x1 + x2 y = y1 + y2 =…
beta = y - b = z1 + z2
c = a*b = c1 + c2
 Multidisciplinary Challenge

Data science

(use-cases, workflow, monitoring)

Cryptography
 Machine learning


(techniques, protocols, trust) (models, approx, precision)

Engineering

(distributed, multi-core, readability)

need common language

TF Encrypted
Making it accessible
 TensorFlow
platform for research and production-level training and deployment

popular and backed by Google

 TF Encrypted Architecture
standard operations

(matmul, relu, sigmoid, tanh, etc)
easily mix ordinary and
encrypted computations
App

MPC ML
secure computation

directly using TensorFlow
TF Encrypted
ordinary TensorFlow

Dist Tensor ML
HE MPC

TensorFlow
third party libraries for
secure computation
Prediction
Encouraging use
 Participants
Share1(x)
Share1(w0, b0, …)

Share1(logits)

w0, b0, … x

Share2(x)
Share2(w0, b0, …)

Share2(logits)
Private Prediction with TF Encrypted
 Overall Computation

compute servers

prediction client

model owner
 Local Processing

TF Data pipeline
Joint Prediction
Combining knowledge for nuance
 Participants
Share1(x_age) x_age
Share1(x_gender)
Share1(w0, b0, …) Share1(x_income)

Share1(res)
w0, b0, …
x_gender

Share2(x_age)
Share2(x_gender)
Share2(w0, b0, …) x_income
Share2(x_income)

Share2(res)
Private Joint Prediction with TF Encrypted
Training
Learning without seeing
 Participants

Share1(x, y) Share1(w)

x, y

Share2(x, y) Share2(w)
 Participants

Share1(x, y) Share1(w)

x, y

Share2(x, y) Share2(w)
 Participants

Share1(x, y) Share1(w)

x, y

Share2(x, y) Share2(w)
Private Training with TF Encrypted
 Overall Computation
data owner

model owner
Joint Training
Combining insights for better models
 Participants
x_0, y_0

Share1(x_0, y_0)
Share1(x_1, y_1) Share1(w)

x_1, y_1 Share2(x_0, y_0)

Share2(x_1, y_1) Share2(w)
Private Joint Training with TF Encrypted
Federated Learning
Keeping data decentralized
 Participants
weights

x_0, y_0

Share1(update_0)
Share1(update_1)
Share1(update_2) Share1(aggregated-update)
x_1, y_1 weights

Share2(update_0) Share2(aggregated-update)
x_2, y_2 Share2(update_1)
Share2(update_2)
Secure Federated Learning in TF Encrypted
 Overall Computation
compute servers

data owners

model owner
 Local Optimization

TF optimization
 Roadmap

High-level API (Private Keras, Pre-trained Models, Owned Data)

Tighter integration (TF Data, TF 2.0, TF Privacy, TF Federated)

Third-party cryptographic libraries (HE, MPC)

Improved performance
 Wrap-Up
You can compute on encrypted data,

without the ability to decrypt

Privacy-preserving ML mitigate bottlenecks and

enable access to sensitive information
Thank you!
Secure computation distributes trust and control,

and is complementary to e.g. differential privacy
github.com/tf-encrypted/

Privacy-preserving ML is a multidisciplinary field
 @mortendahlcs

benefitting from adaptations on both sides
@dropoutlabsai

TF Encrypted focuses on


usability and integration