dynamic database credentials: security contingency planning

Dynamic Database Credentials:Security Contingency Planning

Sean ChittendenEngineering, HashiCorp@SeanChittendensean@hashicorp.comhttps://keybase.io/seanc

Vault

Vault Manages Secure Information

Risk Assessment

Security is an Operational Concern

Loss Aversion byproduct of Security

Real security threats often differ from theoretical security.

Practical security implies loss aversion. Against what threat?

How do you bound exposure?

Types of Security Risk

Leaked secret

Tightly guarded master keys leak

Inability to audit access

Inappropriate access to secrets or data leakage

Human-scale response to compromise (slow mitigation)

Opportunity Assessment

Benefits of Low Friction Security

Low friction Security Systems imply:

• Security Automation

• Workflow Integration

• Secure by Default

• Principle of Least Privilege

PostgreSQL Security Friction

Sticking Points

Statically Managed Defense in Depth

• pg_hba.conf: network, auth, username, database, transport

• ROLE: passwords, expirations, connection limits

• Data Control Language: GRANT, RLS

Configuration Management To The Rescue… ?

Keys to the Kingdom

• Centrally Stored

• Eventually Consistent

• No Access Control

• No Auditing

• No Revocation

Meatspace Operational Concerns

• How do applications get secrets?

• How do humans acquire secrets?

• How are secrets updated?

• How is a secret revoked?

Why not use PostgreSQL for secrets?

• Not designed for dynamic secrets

• Typically plaintext storage by default

• Limited auditing capabilities

• No revocation abilities

• Homegrown RLS access controls

What if…

Embraced Requirements

• Centralized key management

• Grant temporary leases to secured resources

• Trust memory (not disk)

• Embrace automation (and the necessary APIs)

• Assume cyphered data at rest is trustworthy

• Decoupled an HA storage backend from the secrets management

Secure like a Bank…

…convenient like a drive thru…

…and built like a tank!

Secrets Lifecycle in Dynamic Environments

Vault Architecture

Audit Broker

Audit Backend

Audit BackendCredentialBackend

SecretBackend

SystemBackend

Path Routing

Rollback Mgr. Expiration Mgr.

Token Store Policy Store

HTTP API

Storage Backend

Core

Barri

er

Glossary

Glossary

Storage backend

The storage backend is responsible for durable storage of encrypted data. There is only one storage backend per Vault cluster.

Data is encrypted in transit and at rest with 256bit AES.

Examples: in-mem, file, consul, and postgresql

Glossary

Secret backend

A secret backend is responsible for managing secrets. Some secret backends behave like encrypted key-value stores, while others dynamically generate secrets when queried. There can be multiple secret backends in a Vault cluster.

Examples: generic, transit, postgresql

Glossary

Secret backend

Secret backends can perform almost any function, not just return static data or hand out credentials.

PKI – Acts as a full CA, leveraging Vault’s auth

Transit – Allows round-tripping data through Vault for "encryption as a service", without ever divulging the key

Glossary

Auth backend

An auth backend is a credential-based backend that can be used as a way to authenticate humans or machines against Vault.

Machine-oriented: approle, tls, tokensOperator-oriented:  github, ldap, userpass

Glossary

Vault token

A vault token is a conceptually similar to a session cookie on a website. Once a user authenticates via an auth backend, Vault returns a token which is to be used for future requests.

Glossary

Secret

A secret is anything stored or returned by Vault that contains confidential material.

A secret is anything that, if acquired by an unauthorized party, would cause political, financial, or appearance harm to an organization.

Glossary

Server

The Vault server provides an HTTP API which clients interact with and manages the interaction between all the backends, ACL enforcement, and secret lease revocation.

Vault Architecture

Audit Broker

Audit Backend

Audit BackendCredentialBackend

SecretBackend

SystemBackend

Path Routing

Rollback Mgr. Expiration Mgr.

Token Store Policy Store

HTTP API

Storage Backend

Core

Barri

er

Initializing Vault

Meatspace Integration

Key Shares

Master Key Encryption Key

Terminal

$ vault init -key-shares=5 -key-threshold=2 \ -pgp-keys="keybase:seanc,keybase:jefferai,keybase:vishalnayak,alice.asc,bob.asc"... Key 1: c1c04c03d5f43b6432ea77f3010800...Key 2: 612b611295f255baa2eb702a5e254f...Key 3: ebfd78302325e2631bcc21e11cae00......

Obtaining a Vault Token

Hypothetical Model

User1) userpass auth

Server

PostgreSQL

app1

(or pgbouncer)

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN

Server

PostgreSQL

app1

(or pgbouncer)

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

Server

PostgreSQL

app1

(or pgbouncer)

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

4) policy-scoped VAULT_TOKEN

Server

PostgreSQL

app1

(or pgbouncer)

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

4) policy-scoped VAULT_TOKEN

Server

PostgreSQL

app1

(or pgbouncer)

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

4) policy-scoped VAULT_TOKEN

Server 5) VAULT_TOKEN + cred/read/app1

PostgreSQL

app1

(or pgbouncer)

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

4) policy-scoped VAULT_TOKEN

Server 5) VAULT_TOKEN + cred/read/app1

PostgreSQL

app1

(or pgbouncer)

6) CREATE ROLE…

GRANT…

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

4) policy-scoped VAULT_TOKEN

Server 5) VAULT_TOKEN + cred/read/app17) Dyn PG Creds

PostgreSQL

app1

(or pgbouncer)

6) CREATE ROLE…

GRANT…

Hypothetical Model

User1) userpass auth

2) user VAULT_TOKEN3) VAULT_TOKEN + token-create + policy

4) policy-scoped VAULT_TOKEN

Server 5) VAULT_TOKEN + cred/read/app17) Dyn PG Creds

PostgreSQL

8) PG user/pass

app1

(or pgbouncer)

6) CREATE ROLE…

GRANT…

Terminal

pgopen1 % psql -d postgrespostgres=# CREATE DATABASE app1;postgres=# \c app1postgres=# CREATE SCHEMA my_app1;postgres=# CREATE TABLE my_app1.myfoo (i INT);postgres=# INSERT INTO my_app1.myfoo VALUES (6*9);

Terminal

pgopen1 % vault mount postgresqlSuccessfully mounted 'postgresql' at 'postgresql'!pgopen1 % vault path-help postgresql/ | head -8## DESCRIPTION

The PostgreSQL backend dynamically generates database users.

After mounting this backend, configure it using the endpoints withinthe "config/" path.

## PATHSpgopen1 % vault path-help postgresql/ | grep /the "config/" path. ^config/connection$ ^config/lease$ ^creds/(?P<name>\w[\w-.]+\w)$ ^roles/(?P<name>\w[\w-.]+\w)$ ^roles/?$

Terminal

pgopen1 % vault path-help postgresql/config/connection | head -15Request: config/connectionMatching Route: ^config/connection$

Configure the connection string to talk to PostgreSQL.

## PARAMETERS

connection_url (string) DB connection string

max_idle_connections (int) Maximum number of idle connections to the database; a zero uses the value of max_open_connections and a negative value disables idle connections. If larger than max_open_connections it will be

Terminal

pgopen1 % vault write postgresql/config/connection \ connection_url="postgresql://postgres@127.0.0.1/app1?sslmode=disable"Success! Data written to: postgresql/config/connection

pgopen1 % vault write postgresql/config/lease \ lease=1h \ lease_max=24hSuccess! Data written to: postgresql/config/lease

Terminal

pgopen1 % vault write postgresql/roles/app1 \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}'; GRANT USAGE ON SCHEMA my_app1 TO \"{{name}}\"; GRANT SELECT ON ALL TABLES IN SCHEMA my_app1 TO \"{{name}}\"; GRANT SELECT,USAGE ON ALL SEQUENCES IN SCHEMA my_app1 TO \"{{name}}\";"Success! Data written to: postgresql/roles/app1

Terminal

pgopen1 % vault write postgresql/roles/app1 \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}'; GRANT USAGE ON SCHEMA my_app1 TO \"{{name}}\"; GRANT SELECT ON ALL TABLES IN SCHEMA my_app1 TO \"{{name}}\"; GRANT SELECT,USAGE ON ALL SEQUENCES IN SCHEMA my_app1 TO \"{{name}}\";"Success! Data written to: postgresql/roles/app1

pgopen1 % vault read postgresql/creds/app1Key Valuelease_id postgresql/creds/app1/6b614cec-ff9c-1fe2-968a-be300f8bf434lease_duration 3600lease_renewable truepassword 773104c8-aa09-88e9-6a33-e86750239dd3username userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575

Terminal

pgopen1 % vault read postgresql/creds/app1Key Valuelease_id postgresql/creds/app1/6b614cec-ff9c-1fe2-968a-be300f8bf434lease_duration 3600lease_renewable truepassword 773104c8-aa09-88e9-6a33-e86750239dd3username userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575

pgopen1 % psql -q -d app1app1=# \xapp1=# \dn+List of schemas-[ RECORD 1 ]-----+-------------------------------------------------------------Name | my_app1Owner | pgsqlAccess privileges | pgsql=UC/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=U/pgsqlDescription |

Terminal

pgopen1 % psql -q -d app1app1=# \xapp1=# \dn+List of schemas-[ RECORD 1 ]-----+-------------------------------------------------------------Name | my_app1Owner | pgsqlAccess privileges | pgsql=UC/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=U/pgsqlDescription |

app1=# \dp+Access privileges-[ RECORD 1 ]-----+-------------------------------------------------------------Schema | my_app1Name | myfooType | tableAccess privileges | pgsql=arwdDxt/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=r/pgsqlColumn privileges |Policies |

Terminal

app1=# CREATE ROLE my_app1_web_tier;app1=# GRANT USAGE ON SCHEMA my_app1 TO my_app1_web_tier;app1=# GRANT SELECT ON ALL TABLES IN SCHEMA my_app1 TO my_app1_web_tier;app1=# \dp+Access privileges-[ RECORD 1 ]-----+-------------------------Schema | my_app1Name | myfooType | tableAccess privileges | pgsql=arwdDxt/pgsql + | my_app1_web_tier=r/pgsqlColumn privileges |Policies |

Terminal

pgopen1 % vault write postgresql/roles/app1 \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}'; ALTER GROUP my_app1_web_tier ADD USER \"{{name}}\";"Success! Data written to: postgresql/roles/app1

pgopen1 % vault read postgresql/creds/app1Key Valuelease_id postgresql/creds/app1/b4d81724-68e8-b31c-4a81-2f1afde5ace8lease_duration 3600lease_renewable truepassword cf6627de-0ef9-1d2c-4968-b5fa703e6af2username userpass-sean-0bf4abe8-5193-7d6c-ed7a-766d705dda66

Terminal

pgopen1 % psql -q -U userpass-sean-0bf4abe8-5193-7d6c-ed7a-766d705dda66 app1app1=> SET search_path = my_app1; app1=> \dn+List of schemas-[ RECORD 1 ]-----+-------------------------------------------------------------Name | my_app1Owner | pgsqlAccess privileges | pgsql=UC/pgsql + | "userpass-sean-bd0f4f3b-2f86-b79b-db3d-7b3d39e76575"=U/pgsql+ | my_app1_web_tier=U/pgsqlDescription |app1=> \dp+Access privileges-[ RECORD 1 ]-----+-------------------------Schema | my_app1Name | myfooType | tableAccess privileges | pgsql=arwdDxt/pgsql + | my_app1_web_tier=r/pgsqlColumn privileges |Policies |

Terminal

app1=> SELECT * FROM my_app1.myfoo ;-[ RECORD 1 ]i | 1

Terminal

pgopen1 % # Revoke one leasepgopen1 % vault revoke postgresql/creds/app1/b4d81724-68e8-b31c-4a81-2f1afde5ace8Key revoked with ID 'postgresql/creds/app1/b4d81724-68e8-b31c-4a81-2f1afde5ace8'.

Terminal

pgopen1 % # Revoke all of postgresql/ ’s leasespgopen1 % vault revoke -prefix postgresql/creds

app1=# \duList of roles-[ RECORD 1 ]----------------------------------------------------------Role name | my_app1_web_tierAttributes | Cannot loginMember of | {}-[ RECORD 2 ]----------------------------------------------------------Role name | pgsqlAttributes | Superuser, Create role, Create DB, Replication, Bypass RLSMember of | {}

Terminal

pgopen1 % # Create some new creds from a child tokenpgopen1 % vault token-createKey Valuetoken fc20f365-4250-e840-739e-2e658dba8678token_accessor 046f5be2-a98d-4608-4398-4c232b3afca2token_duration 0token_renewable truetoken_policies [root]$ env VAULT_TOKEN=fc20f365-4250-e840-739e-2e658dba8678 vault read postgresql/creds/app1Key Valuelease_id postgresql/creds/app1/d6e01c35-ff11-365f-5bce-d78dbe9fd995lease_duration 2592000lease_renewable truepassword 2828a500-f813-b785-5d49-0da565de2938username token-8921e3d4-9a0a-302c-e6c2-cfb86d1107a4$ env VAULT_TOKEN=fc20f365-4250-e840-739e-2e658dba8678 vault read postgresql/creds/app1Key Valuelease_id postgresql/creds/app1/7532550e-e5cb-501d-a8e5-3578a71a40c3lease_duration 2592000lease_renewable truepassword 2a9f3169-c49d-f83f-e0d7-b261c4aadcc9username token-8ef907c8-e619-4b8f-01f4-372dbcac51ca

Terminal

postgres=# SELECT usename FROM pg_catalog.pg_user; usename-------------------------------------------- pgsql token-8921e3d4-9a0a-302c-e6c2-cfb86d1107a4 token-8ef907c8-e619-4b8f-01f4-372dbcac51ca(3 rows)

pgopen1 % vault token-revoke fc20f365-4250-e840-739e-2e658dba8678Revocation successful.

postgres=# SELECT usename FROM pg_catalog.pg_user; usename------------------------------------------- pgsql(1 rows)

How 'bout that API?

Audit Broker

Audit Backend

Audit BackendCredentialBackend

SecretBackend

SystemBackend

Path Routing

Rollback Mgr. Expiration Mgr.

Token Store Policy Store

HTTP API

Storage Backend

Core

Barri

er

Terminal

pgopen1 % curl --request GET \ —header 'X-Vault-Token: 35173305-8c33-9d5d-633d-8b31fc8127d5' \ http://127.0.0.1:8200/v1/postgresql/creds/app1 | jq .{ "lease_id": "postgresql/creds/app1/798c7488-917f-c12f-6b95-38d06cd9b689", "renewable": true, "lease_duration": 3600, "data": { "password": "eb4d64f5-4bf4-56d7-1ecd-e5f10ed008f4", "username": "userpass-sean-d558fa25-e1ed-10cc-1b00-927ac25ef3d9" }, "warnings": null, "auth": null}

Integrations: consul-template

"Process-manager" renders templates out using consul or Vault as data sources, executes or signals child process dynamically.

$ consul-template -template "in.ctmpl:out.txt:command"

https://github.com/hashicorp/consul-template

Integrations: envconsul

envconsul:

See consul-template, but limited to environment variables

https://github.com/hashicorp/envconsul

Integrations: fabio

fabio:

HTTP load balancer. Uses consul service discovery, can pull SSL certs from Vault.

TCP-SNI support experimental (hellooo pq+TLS!)

https://github.com/eBay/fabio/wiki/Certificate-Stores

Questions?

@SeanChittendensean@hashicorp.com

https://keybase.io/seanc

https://github.com/sean-/pgopen16-dyn-creds