docker to the rescue of an ops team
TRANSCRIPT
AgendaThis talk is about:
● Monitoring (a bit)● Private Registry● CI/CD● Security● Docker experience
Once upon a time An ops team starts a new project
Rebuild everything !June 2014A monitoring system from scratch“Microservices” orientedReplaceable parts
Some rules first !SimpleEfficientExtendable Python based
Components Collectd
Collectd proxy
Graphite
Grafana
Cabot (alerting)
Test your might !Python 2.6 along with python 2.7Different version of “some” librariesWhisper backend (I/O storm)Upstream repositories issues
Docker to the rescueSave our project
Docker ? way too soon !Barely no skillsUsed only to do some testingPretty serious concerns
Ok let’s gamble !Grow our docker fuWrite some DockerfilesBuild images locallySpawn PoC platform
…..
RUN echo "deb http://mirror.debian.ikoula.com/debian wheezy-backports main" >> /etc/apt/sources.listRUN apt-get -qq updateRUN apt-get -qqy dist-upgrade
RUN apt-get -qqy --force-yes install vim python-cairo gunicorn supervisor (...)RUN pip install whitenoise txamqp whisper==0.9.13 carbonateRUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon==0.9.13RUN pip install --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp" graphite-web==0.9.13
ADD conf/nginx.conf /etc/nginx/nginx.confADD conf/supervisord.conf /etc/supervisor/conf.d/grafana.conf
ADD initial_data.json /var/lib/graphite/webapp/graphite/initial_data.jsonADD conf/local_settings.py /var/lib/graphite/webapp/graphite/local_settings.pyADD conf/carbon.conf /var/lib/graphite/conf/carbon.confADD conf/storage-schemas.conf /var/lib/graphite/conf/storage-schemas.confRUN mkdir -p /var/lib/graphite/storage/whisperRUN touch /var/lib/graphite/storage/graphite.db /var/lib/graphite/storage/indexRUN chmod 0775 /var/lib/graphite/storage /var/lib/graphite/storage/whisperRUN python /var/lib/graphite/webapp/graphite/manage.py syncdb --noinput --pythonpath=/var/lib/graphite/webapp/graphite --settings=settingsRUN chmod 0664 /var/lib/graphite/storage/graphite.dbRUN chown -R www-data /var/lib/graphite/storage
…..
WHAT ???
Container = OS …. Wait !
Too many layers ( 121+ layer issue)
Build time …. (20 to 30 minutes at best)
Huge Images (800+ Mo)
Unnecessary tools and libs
Bye Bye !!!!
We can do better !Apply best Practices (@abbyfuller)
Implement simple CI/CD
Dockerfile Linting
Build a private registry
Deal with security concerns
…
RUN echo "APT::Install-Recommends false;" >> /etc/apt/apt.conf.d/00recommends \ && echo "APT::Install-Suggests false;" >> /etc/apt/apt.conf.d/00recommends \ && echo "APT::AutoRemove::RecommendsImportant false;" >> /etc/apt/apt.conf.d/00recommends \ && echo "APT::AutoRemove::SuggestsImportant false;" >> /etc/apt/apt.conf.d/00recommends
ENV DEBIAN_FRONTEND noninteractiveENV GRAPHITE_VERS 0.9.13
RUN apt-get -qqy update \ && apt-get -qqy install python-cairo gunicorn git python2.7-dev wget ca-certificates python-flup expect sqlite3 libcairo2 libcairo2-dev pkg-config nodejs sqlite3 memcached python-ldap make gcc libffi-dev
RUN wget https://bootstrap.pypa.io/get-pip.py \ && python get-pip.py \ && pip install --no-cache-dir --upgrade setuptools \ && pip install --no-cache-dir django django-admin-tools \ && pip install --no-cache-dir whitenoise txamqp whisper==${GRAPHITE_VERS} carbonate \
&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/lib" carbon==${GRAPHITE_VERS} \
&& pip install --no-cache-dir --install-option="--prefix=/var/lib/graphite" --install-option="--install-lib=/var/lib/graphite/webapp" graphite-web==${GRAPHITE_VERS}
RUN apt-get purge gcc make python2.7-dev libcairo2-dev libffi-dev python2.7-dev pkg-config -qqy \ && apt-get clean\ && apt-get autoremove -qqy \ && rm -rf /root/.cache /var/lib/apt/lists/* /tmp/* /var/tmp/*
...
Best practices :)Few image layers ( < 20 )Small image (~ 400Mo)Lower footprint (100Mo)Faster build time (~5Min)
We did it , we did it Yeah !
CI/CD Diagram
CI/CD Recipe
build: image: registry.synolia.com/synolia/dockerunitest:latest
publish: docker: repo: synomon_datastor tag: $${BRANCH/master/latest} file: Dockerfile insecure: true when: repo: synolia/systeam-monitoring_datastor branch: [develop, master]
notify: hipchat: from: "synoci" room_id_or_name: "$$ROOM_ID" auth_token: "$$AUTH_TOKEN" notify: true when: success: false failure: true
Dockerfile Linting
Build
Push
Notify (fail only)
Docker to the rescue IIThe return of the hero moby
This isn’t over yet !Docker udp issuesTricky iptable filteringUnstable data volumeConfiguration management
Round 2: FIGHT !Metric proxy (Collectd) on the host!!Simplify iptables rulesMount directories (metrics)
A new path opensTo a brighter future
Epic loots !No more dependency issuesReplaceable and movable partsGreater security levelClustering ready (Swarm/K8S)
Lessons learnedCaution when using udp IPv4Config files out of the containerDon’t use env variables (security)Use (abuse) automation
What’s next ?Greater Docker challenges
Many rooms to growReduce (even more) Image sizeSign Images (notary to the rescue)Vulnerability scanningImplement rolling upgrades
New docker based projectsMigrate Development platform (2015) Swarm clustering (*)Full scale Docker (*)(*) Work In Progress
Thank You DockerCon !PS: don’t forget to rate my talk :)