The BlackBox projectSafely storing secrets and credentials in Git

for use by Puppet

Tom Limoncelli, SRE, StackExchange.comBlog: EverythingSysadmin.com

125+ Q&A CommunitiesServerFault.comStackOverflow.com

(We <3 Puppet!)

StackExchange.com

What are secrets?

Anything you don’t want exposed externally.

● SSL Certificates (the private bits)● Passwords● API keys

Puppet manages secrets

If you store secrets in git, you’re gonna have a bad time.

● Laptops get stolen.● Workstations have guest accounts● “Circle of Trust” now includes:

○ Everyone with admin access to workstations.■ Your desktop support people?

○ Everyone with admin access to your git server:■ Server team, storage team, backup team

○ Everyone you collaborate with that wants read-only access to Puppet manifests.

You have 3 bad options:

1. Deny git access. (Hurts collaboration)2. Permit git access. (Hurts security)3. Email individual files. (Hurts… just hurts)

Option 4: Encrypt secret parts

● If a file contains secrets, encrypt before checking into Git.

● Need to edit a secret?○ Decrypt - Edit - Encrypt

What about Puppet master?

● After “git pull”, decrypt all files.○ Automate this as part of CI.

● Files are unencrypted “at rest”.● This does not decrease security:

○ No worse than what we were doing before.○ If you can break into root or puppet on the master,

you’ve already won.

Easy, right?

Decrypt:gpg -q --decrypt -o secret.crt secret.crt.gpg

Encrypt:gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt

Easy, right?

Decrypt:gpg -q --decrypt -o secret.crt secret.crt.gpg

Encrypt:gpg --yes --trust-model=always --encrypt -o secret.crt.gpg $(<keynames) secret.crt

● ...and don’t make any typos when entering the command● ...and don't accidentally check in the unencrypted version

Security is 1% technology plus 99% following the procedures correctly.

Any process with more than 1 step probably won't be followed consistently most of the time.

Related reading: "Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0”, Alma Whitten", Usenix Security 1999

Therefore…. we automate

Introducing: Blackbox

Scripts for keeping Puppet secrets in git/hg.

User commands:

Decrypt for editing:blackbox_edit_start.sh file

Encrypt when done:blackbox_edit_end.sh file

First time a file is encrypted:

Enroll a file into the system:blackbox_register_new_file.sh file

Commands that act on all GPG files:

Decrypt all files: (for use on puppet master)blackbox_postdeploy.sh

Re-encrypt all files: (after new users added)blackbox_update_all_files.sh

Everyone has their own key

This doesn’t use “symmetric encryption” where there is one passphrase to decrypt/encrypt all files.

We maintain a keyring of:● Each person that should have access.● A key for the Puppet master.

Indoctrinate a new user:

1. New user does this:

● Create GPG key.● Add their username@host to blackbox-admins.

txt● git commit -a

(Currently a doc, not a script. Patches gladly accepted.)

Indoctrinate a new user:

2. Existing admin does this:

$ gpg --import keyrings/live/pubring.gpg

$ blackbox_update_all_files.sh

$ git commit -a

Demo: Edit a file

Demo: Edit a file

Demo: Edit a file

Demo: Edit a file

Demo: Edit a file

Demo: Edit a file

Demo: Edit a file

Code is open source as of TODAY

● Entirely written in bash.● MIT License.● Download it now:

○ https://github.com/StackExchange/blackbox

In the project’s first 9 months:

StackExchange/ServerFault has eliminated plaintext secrets in our Puppet git repo.● 7 SREs+Devs sharing the repo securely.● 50+ files now stored encrypted.

○ Mostly SSL certs and SSH private keys.● 40+ individual passwords/API keys:

○ Everything from SNMP communities, SaaS API keys, and many many passwords.

Future plans

❏ Open source scripts.❏ More usability enhancements.❏ Better setup documentation.

Join the open source projecthttp://github.com/StackExchange/blackbox

Q&A

URLs from this talk:https://github.com/StackExchange/blackbox

EverythingSysadmin.com

Shameless plugPre-order now! Save 35%

Ships in September.

informit.com/TPOSADiscount code TPOSA35

Read “rough cuts” today:safaribooksonline.com

Q&A

URLs from this talk:https://github.com/StackExchange/blackbox

EverythingSysadmin.cominformit.com/TPOSA (code TPOSA35)

● Easier transition. No Puppet code changes for big files like SSL certs.

● Faster. Zero run-time performance impact on master.

● eyaml didn’t exist when we started.

Why didn’t we use eyaml?