PostgreSQL andLinux Containers

Jignesh Shah Founding Team @appOrbit

SF Bay Area PostgreSQL User Group – Jan 5th 2015

Disclaimer

The views expressed in this presentation are my own and do not necessarily reflect the views of appOrbit.

About Jignesh (@jkshah)

• appOrbit• My focus is on managing Application Data running in Containers

• VMware• Lead and manage Postgres and Data Management teams at VMware for various

products embedding PostgreSQL running in virtualized embedded instances

• Sun Microsystems• Team Member of first published SpecJAppServer 2004 benchmark with PostgreSQL• Performance of PostgreSQL on Solaris/Sun Servers

• Working with PostgreSQL community since 2005• http://jkshah.blogspot.com/2005/04/profiling-postgresql-using-dtrace-on_22.html

• Working with Container technologies (Solaris Zones) since 2004• http://jkshah.blogspot.com/2004/08/db2-working-under-solaris-10-zones_30.html

Agenda• Containers

• Definition• Early examples

• Linux Containers• Underlying Technologies • LXC• Systemd-nspawn

• Docker Containers• Installation• Images• Volumes

• PostgreSQL in Docker Container• What it means• Best practices

What are Containers?• OS Level virtualization where kernel allows for multiple isolated user-

space instances

Operating System

Bare Metal Server

OS

Bare Metal Server

Hypervisor

OS

Operating System

Bare Metal Server

C C C C C OS

Bare Metal Server

Hypervisor

OS

C C C C

Advantages of Containers• Lower footprint• Very Quick Startup and Shutdown time• Density• Nesting

Disadvantages of Containers• Same Kernel version• Cannot run other OS natively• Security (to be improved)

Where to use container?• Recreate identical environment (cookie-cutter)• Resource Grouping of specific processes in heavily loaded server• Handling multiple versions of software applications • Ephemeral application instances (Dev/Test)• Many more

Implementations of Containers• Chroot circa 1982 • FreeBSD Jails circa 2000• Solaris Zones circa 2004• Meiosys – MetaClusters with Checkpoint/Restore 2004-05• Linux OpenVZ circa 2005 (not in mainstream Linux)• AIX WPARs circa 2007• LXC circa 2008 • Systemd-nspawn circa 2010-2013 • Docker circa 2013

What makes containers possible?• Process Group Isolation• Filesystem Isolation• Network Isolation• CPU Isolation• Memory Isolation

• Example – Solaris Containers• Base implementation provided Process, Filesystem and Network

Isolation• Resource pools consisting of CPU, memory was originally used • Branded Zones – Userland Library Isolation

What makes Linux containers possible ?• cgroups• Allows limitation and prioritization of resources (CPU, memory,

block I/O, network, etc.) • Namespace isolation• Mount namespace• PID namespace• Network namespace• UTS (Allows changing hostname, domainname)• IPC Namespace• User namespace

• LXC• Combines kernel’s cgroup and namespaces to provide an isolated

environment

LXC• CentOS 7 + EPEL Repository set

• Commands available

• Quick Guide to use an LXC based container of busybox

lxc-attach lxc-clone lxc-destroy lxc-ls lxc-stop lxc-usernsexeclxc-autostart lxc-config lxc-execute lxc-monitor lxc-top lxc-waitlxc-cgroup lxc-console lxc-freeze lxc-snapshot lxc-unfreezelxc-checkconfig lxc-create lxc-info lxc-start lxc-unshare

wget https://www.busybox.net/downloads/binaries/busybox-x86_64 -O busyboxchmod a+x busyboxPATH=$(pwd):$PATH lxc-create -t busybox -n mycontainerlxc-start -d -n mycontainerlxc-console –n mycontainer # (Use CTRL-A Q to exit console mode)lxc-stop -n mycontainerlxc-destroy -n mycontainer

yum install epel-releaseyum install bridge-utils libvirt lxc lxc-templates

Systemd-nspawn• Systemd

• Replacement of SysV init scripts

• Systemd-nspawn• Used to run a command or OS in light weight namespace container

• Installed on most newer distros by default

• Commands availablesystemd-analyze systemd-delta systemd-nspawnsystemd-ask-password systemd-detect-virt systemd-runsystemd-cat systemd-cgls systemd-loginctl systemd-sysv-convert systemd-cgtop systemd-machine-id-setup systemd-coredumpctl systemd-notify systemd-tty-ask-password-agentsystemd-inhibit systemd-stdio-bridge systemd-tmpfilessystemctl machinectl hostnamectl journalctl

yum install systemd

Systemd-nspawn• Quick guide to a container deployment using systemd-nspawn

• Useful toolsmachinectl status mycontainersystemd-cglssystemd-cgtop

# Create an Imageyum -y --releasever=7 --nogpg --installroot=/mycontainers/centos7 install \ systemd passwd yum fedora-release vim-minimal

# Change the root password in the imagesystemd-nspawn -D /mycontainers/centos7passwdexit

# Start the container as if booting into the container imagesystemd-nspawn -D /mycontainers/centos7 –M mycontainer –b

# Get into the containernsenter -m -u -i -n -p –t $PIDmachinectl login mycontainer

Trends of Container Technologies

Disruption of trends by Docker

Docker• Installation

• Commands available with docker binary

• Quick Guide to use a docker based container

attach build commit cp create diff eventsexec export history images import info inspectkill load login logout logs pause portps pull push rename restart rm rmirun save search start stats stop tagtop unpause version wait

docker run --name mycontainer -e POSTGRES_PASSWORD=mysecretpassword -d postgresdocker exec -ti mycontainer psql -U postgresdocker stop mycontainerdocker rm mycontainerdocker rmi postgres

yum install dockersystemctl start docker

Dockerfile – Custom Recipe• Container images are created using “Dockerfile”

• Build an image using the recipe file

• Push to a public or private registry (hub account login or private registry needed)

FROM centos:centos7.0.1406RUN yum install -y epel-releaseRUN yum install -y nginxEXPOSE 80 CMD ["bash", "-l", "-c", "/usr/sbin/nginx -g \"daemon off;\""]

docker build –t jkshah/nginx –t .

docker push jkshah/nginx:latest

Docker Images• Docker Images are layered templates used by containers instances

• Container instance images are layered Copy on Write Images based on Docker images

• You can mutate your local container images (unless you use --read-only flag)

• Not good for fast and frequent changing data• Can select a different underlying type using --storage-driver

docker push jkshah/nginx:latest

Do you really want to push to public registry? [y/n]: yThe push refers to a repository [docker.io/jkshah/nginx] (len: 1)d498b5680966: Pushed4b1d16518ce1: Pushed3185bef36db4: Pushed2f7013eef4b8: Pushed539eca37bade: Pushedf1b10cd84249: Pushedlatest: digest: sha256:46208d1f0393946b33c2bdce498168de507b7186a897f332bab0cffc1ea601c7 size: 10608

Docker Volumes • Persists beyond the life of a Docker container

• VOLUME command in Dockerfile or• Using –v using docker run command • Automatically created if not already present during docker run• Not part of docker push/pull operations• Can select a non-local directory using --volume-driver• Third party components required to get multi-host support (NFS, etc )

• On CentOS with SELinux enabled need to set security context

• Different options using –v• -v /hostsrc/data:/opt/data:ro # for read only volumes (default rw)• -v /hostsrc/data:/opt/data:Z # Z – private volume, z – shared volume• -v /etc/nginx.conf:/etc/nginx.conf # for mounting a single file only

• Volumes can be shared from another container using --volumes-from on same host

• Docker 1.9 gives first class status to Docker Volumes

chcon -Rt svirt_sandbox_file_t /hostpath/pgdata

PostgreSQL in Docker container• Quick Deployment:

• Check Deployment:

• Use inspect command to get more information about the container

• To check PostgreSQL system logs

docker run --name mycontainer -v /hostpath/pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword -d postgres

docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES8d720b31c1fe postgres "/docker-entrypoint.s" 5 minutes ago Up 5 minutes 5432/tcp mycontainer

docker logs mycontainerThe files belonging to this database system will be owned by user "postgres".This user must also own the server process.…LOG: database system was shut down at 2016-01-04 22:58:40 UTCLOG: MultiXact member wraparound protections are now enabledLOG: database system is ready to accept connectionsLOG: autovacuum launcher started

docker inspect mycontainer

PostgreSQL in docker container• Part of Postgres Dockerfile has

• Impacts• Port is exposed only to other containers directly linking with this container • Data is persistent only for the life of container (docker rm) but not easily accessible outside the

container• -v enables to access the data outside container

• Linking to PostgreSQL Server container

• Ports are exposed externally using –p option• Verify using

docker run --name myapp \ --link mycontainer:ds2db \ -e POSTGRES_USER=postgres \ -e POSTGRES_PASSWORD=mysecretpassword \ -p 8080:80 \ -d jkshah/dvdstore2

EXPOSE 5432VOLUME /var/lib/postgresql/data

docker port myapp80/tcp -> 0.0.0.0:8080

PostgreSQL in docker container• Check for statistics using

• Check for top processes in a container

docker stats mycontainer myappCONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/Omyapp 0.01% 51.51 MB/1.924 GB 2.68% 54.42 kB/7.576 MBmycontainer 0.01% 133.3 MB/1.924 GB 6.93% 7.577 MB/54.42 kB

docker top mycontainerUID PID PPID C STIME TTY TIME CMDpolkitd 25313 10698 0 16:58 ? 00:00:00 postgrespolkitd 25385 25313 0 16:58 ? 00:00:00 postgres: checkpointer processpolkitd 25386 25313 0 16:58 ? 00:00:00 postgres: writer processpolkitd 25387 25313 0 16:58 ? 00:00:00 postgres: wal writer processpolkitd 25388 25313 0 16:58 ? 00:00:00 postgres: autovacuum launcher processpolkitd 25389 25313 0 16:58 ? 00:00:00 postgres: stats collector process

docker top myappUID PID PPID C STIME TTY TIME CMDroot 30747 10698 0 18:56 ? 00:00:00 httpd -D FOREGROUND48 30796 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND48 30797 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND48 30798 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND48 30799 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND48 30800 30747 0 18:56 ? 00:00:00 httpd -D FOREGROUND

PostgreSQL in docker container• Alternate way to see Process Tree using systemd-cgls tool

systemd-cgls├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21├─user.slice│ └─user-0.slice│ └─session-34.scope│ ├─25129 sshd: root@pts/0│ ├─25131 -bash│ └─31397 systemd-cgls└─system.slice ├─docker-9457652d7f6ec24ebd95305e788fe39030b049deb22f240ee2b7383488d0c215.scope │ ├─30747 httpd -D FOREGROUND │ ├─30796 httpd -D FOREGROUND │ ├─30797 httpd -D FOREGROUND │ ├─30798 httpd -D FOREGROUND │ ├─30799 httpd -D FOREGROUND │ └─30800 httpd -D FOREGROUND ├─docker-8d720b31c1fe0de77d3cd89942c1a72902b67d466f821ce63ee1271561a36451.scope │ ├─25313 postgres │ ├─25385 postgres: checkpointer process │ ├─25386 postgres: writer process │ ├─25387 postgres: wal writer process │ ├─25388 postgres: autovacuum launcher process │ └─25389 postgres: stats collector process ├─docker.service │ ├─10698 /usr/bin/docker daemon --selinux-enabled │ └─30739 docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -contain...

PostgreSQL in Docker container• Alternate way to see top containers using systemd-cgtop

systemd-cgtopPath Tasks %CPU Memory Input/s Output/s/ 100 0.9 1.5G - -/user.slice 3 0.9 70.7M - -/system.slice - 0.0 268.5M - -/system.slice/docker.service 2 0.0 31.9M - -/system.slice/nimbus.service 5 0.0 20.0K - -/system.slice/tuned.service 1 0.0 36.0K - -/system.slice/dock...49deb22f240ee2b7383488d0c215.scope 6 0.0 49.1M - -/system.slice/auditd.service 1 - 11.9M - -/system.slice/avahi-daemon.service 2 - 12.0K - -/system.slice/crond.service 1 - 92.0K - -/system.slice/dbus.service 1 - - - -/system.slice/dock...7d466f821ce63ee1271561a36451.scope 6 - 127.3M - -/system.slice/gssproxy.service 1 - - - -/system.slice/libvirtd.service 3 - - - -/system.slice/lvm2-lvmetad.service 1 - - - -/system.slice/polkit.service 1 - 16.0K - -/system.slice/postfix.service 3 - 3.5M - -/system.slice/rsyslog.service 1 - 2.1M - -/system.slice/sshd.service 1 - 3.0M - -/system.slice/system-getty.slice/getty@tty1.service 1 - - - -/system.slice/systemd-journald.service 1 - 16.0M - -/system.slice/systemd-logind.service 1 - 20.0K - -/system.slice/systemd-udevd.service 1 - 520.0K - -/system.slice/xe-linux-distribution.service 2 - 1.0M - -/user.slice/user-0.slice/session-34.scope 3 - - - -

PostgreSQL as a database server container• Maybe you want a database server standalone

• Not all database clients will be on docker containers in the same host• Need to limit memory usage• Need different layout of how files are distributed (separage XLOG )

• Use the –p option to make the port available even to non containers clients

• Use –m to limit memory usage by the DB server (by default it can see and use all)• Note this does not set shared buffers automatically with the library image

docker run --name mycontainer \-m 4g \-e POSTGRES_PASSWORD=mysecretpassword \-v /hostpath/pgdata:/var/lib/postgresql/data \-p 5432:5432 -d postgres

PostgreSQL in an enterprise environment• However for a real production use case we would need

• Bigger shared memory configurations • Need different layout of how files are distributed (separage XLOG )• Ability to backup the database• Ability to setup replication • etc

• In short we need a more custom image of PostgreSQL

Best Practices for custom image• For production install customize the docker image

• Allocate proper memory limits - example 8GB• All pagecache usage shows up as docker container memory usage

• Bump up shared buffers and other parameters as required• Hint: use PostgreSQL 9.3 or later otherwise have to privileged containers

• http://jkshah.blogspot.com/2015/09/is-it-privilege-to-run-container-in.html• Support multiple volumes in your image

• Pg_xlog• PITR archives• Tablespaces as required• Full Backup directory

• PostgreSQL Extensions• Setup replication support

• Out of box replication setup• Monitoring Tool

• Your favorite monitoring agent

References• http://www.freedesktop.org/wiki/Software/systemd/• https://linuxcontainers.org/• http://www.haifux.org/lectures/299/netLec7.pdf• http://haifux.org/lectures/320/netLec8_final.pdf

Revolutionary platform to encapsulate and manage

both legacy and new micro-services based applications

with data and configurations running on containers and

virtual machines in a private, public or hybrid cloud.

Put your DevOps on hyper-drive!

Efficiency and Velocity for Apps & InfrastructureWe are

HIRING !!!