Red OctoberImplementing the Two-man Rule for Keeping Secrets

July 23, 2014 !

Nick Sullivan @grittygrease

github.com/cloudflare/redoctober

Red October• A deployment story

• Secrets and threats

• Two-person rule as a service

• Software design

• Future directions

2

Deployment Story

Sneakernets and evidence bags

3

Sneakernet Deployment

4

Sneakernet Deployment• Trusted engineer as build engineer

• Secret kept on build machine

• Check out tag

• Compile

• Burn to writable CD/DVD - Gold Master

• Deploy via sneakernet

5

Sneakernet Deployment - Pros• High amount of physical security

• Deniability

• Exercise?

6

Sneakernet Deployment - Cons• Inconvenient and slow

• “Trusted” engineers can leave

• Some secrets too sensitive to be revoked

7

Secrets

Can they be kept?

8

Secret Types• Credentials

• Cryptographic keys

9

Threats to secrets

10

Threats to secrets• Insider threat — don’t trust access control

!

• Insecure build machine

• Insecure production environment

• Binary disclosure

11

Suggestions from compliance• PCI DSS requirement 3.5.2

• Encrypt them with a key-encrypting key that is at least as strong as the data-encrypting key and stored in a separate location

• Store them within a secure cryptographic device

• Store them in two pieces

12

Improving the secret management• Encrypt with PGP

• Check into SCM

!

• Problem: single admin can walk off with secrets

13

Multi-person build

14

• Two person rule

• Also called m of n

Improving the secret management• Double-encrypt

• Two engineers need to PGP decrypt secret

!

!

• Hard to use in practice: no remote PGP decrypt

• PGP/GnuPG not the right tool for the job

15

Double-encrypt as a service

aka Red October

16

WARNING1. Don’t roll your own crypto

2. Or your own key management software

!

• But if you do, open source it and ask for help

17

What the service needs to do• Encrypt secrets

• Only decrypt secrets if the right people approve it

• Fit into an automated workflow

18

What the service does not need to do• Store encrypted data

19

Red October

20

Cryptography

21

• No new crypto

• AES, RSA, scrypt

• Elliptic curve cryptography (ECC)

It’s about layers

22

23

24

Passwords are fundamental

25

• In login: hashed+salted passwords are compared

• In Red October: hashed+salted passwords are the key

!

• Server can’t decrypt secrets without password

Usage

26

• Run Red October

• pick a port

• use a TLS certificate + key

• JSON API or Web interface

• Create admin account

• Create user accounts

Usage

27

• Encrypt data to a set of users

• only needs public key

• Users delegate their key for time or number of usages

• Admins can decrypt if enough users are delegated

Web interface

28

Web interface demo

29

Why is this in Go?

And how does the code work?

30

Why Go?

31

• easy and fun to write

• JSON serialization a snap

• simple to set up TLS 1.2 server

• simple to make servers multi-threaded

• crypto baked in

• simplified deployment

Golang features used

32

• Structs are serialized and deserialized to JSON automatically

• Caps means public, missing means native zero

• json.Marshal type delegate struct {!! Name string!! Password string!!! Time string! Uses int! admin bool!}

{“Name":"Bob",! “Password":"Rob",! “Time":"2h",! "Uses":1}

Golang features used

33

• Built in TLS support (tls.NewListener)

config := tls.Config{! Certificates: []tls.Certificate{cert},! Rand: rand.Reader,! PreferServerCipherSuites: true,! SessionTicketsDisabled: true,! }!!

!

lstnr := tls.NewListener(conn, &config)

Golang features used

34

• goroutines and channels for multi-processor support

runtime.GOMAXPROCS(runtime.NumCPU())!! process := make(chan userRequest)! go func() {! for {! req := <-process! if f, ok := functions[req.rt]; ok {! r, err := f(req.in)! if err == nil {! req.resp <- r! } else {! log.Printf("Error handling %s: %s\n", req.rt, err)! }

Golang features used

35

• go crypto

• Support for RSA, AES, ECC, HMAC in standard library

// encrypt! aesSession, err := aes.NewCipher(aesKey)! if err != nil {! return! }! out = make([]byte, 16)! aesSession.Encrypt(out, in)

Golang features used

36

• Deployment

• no dependencies!

• single binary

Code Structure

37

• cryptor: encryption and decryption of data

• keycache: storage of live encryption/decryption keys

• passvault: management of disk vault

• core: API interface

• redoctober: https server

Who uses it?

38

• TheGoodData (https://thegooddata.org:81)

• U.S. Navy

• More people and projects (let me know!)

Drawbacks

i.e. what to fix

39

Design Drawbacks

40

• No password recovery (the password is the key)

Current Implementation Drawbacks

41

• 2 of m only

• No two-factor auth, or key-based authentication (like ssh)

• Awkward workflow

• No delegation granularity

• No secure hardware support

Red October 2

42

2 of m only

43

• Adding support for Shamir’s scheme

Key-based authentication

44

• Add support for PGP-based replacement of passwords

• Sign a challenge instead of providing a password

Awkward workflow

45

• Delegation has to happen before build — bad workflow

!

• New push-based system

• Decryption request triggers push notification to file owners

• Delegation request in a mobile app or email

• Details visible to delegators

Granularity of delegation

46

• All-or-nothing right now, good for one secret per server

• Only admins can encrypt/decrypt

!

• Delegators can choose which users can decrypt which files

Secure hardware device

47

• Build into HSM

• Keys backed by TPM

Solving the insider threat

48

Conclusions

49

• Does this solve the insider threat?

!

• Red October server does not get secrets without passwords

• Admin of build machine gets temporary access — automate secret deletion?

• Who created the secret in the first place?

• What about build artifacts or binary disassembly?

Open Questions

50

• How to securely create secrets

• Secure multi-party computation?

• How to adapt Red October to other types of services

• API keys

• SSL private keys

Red OctoberImplementing the Two-man Rule for Keeping Secrets

July 23, 2014 !

Nick Sullivan @grittygrease

github.com/cloudflare/redoctober