how openstack is built
TRANSCRIPT
How Big is OpenStack?
265,850 commits
5,744 developers
54 project teams
585 git repos
2
OpenStack CI System
3
• Developed and managed by the OpenStack Infra team
Ansible+Puppet - two invisible hands.
■ All configuration of CI infra nodes is managed by Puppet manifests ■ The ‘Puppetmaster’ is actually an Ansible control machine■ Puppetmaster holds playbook + hiera data■ Ansible playbooks trigger ‘puppet apply’ on the nodes
4
+
Git - the mighty archivist
■ the most widely used distributed open-source VCS
■ Served by ‘cgit’■ Changes submitted with the help
of ‘git-review’ tool
5
Gerrit – the gatekeeper
■ Originally developed by Shawn Pearce at Google for Android development.
■ Manages git repositories■ Provides code review and
approval rules■ Collaborates with human and non-
human gatekeepers
6
Zuul – the dispatcher
■ Written by James E. Blair■ Defines and enforces the gating
process■ Listens to Gerrit events■ Manages the queue and
dependencies■ Dispatches the verification
process to Jenkins (through Gearman)
7
Zuul – the beautiful dispatcher
■ Written by James E. Blair■ Defines and enforces the gating
process■ Listens to Gerrit events■ Manages the queue and
dependencies■ Dispatches the verification
process to Jenkins (through Gearman)
8
Gearman - the outsourcer■ a framework to farm out work to
other machines or processes
9
■ sends jobs to Jenkins through jenkins gearman plugin (by Khai Do & James E. Blair)
■ manages the job queues
Jenkins - the butler :)
■ Continuous Integration Server■ Executes testing jobs, records executions results■ Jobs performed on slaves■ “Our jenkins servers, at peak load, runs 20,000+ jobs
per day. At that load we require many jenkins slaves (900+)…” (From Jenkins Gearman plugin page)
■ Jobs created and managed by Jenkins Job Builder■ devstack slaves provisioned by Nodepool
10
Jenkins Job Builder■ Configures Jenkins jobs using YAML files stored in git■ Defaults, macros and job configs
- job: name: example-docs node: node-label
triggers: - zuul
builders: - git-prep - docs
publishers: - scp: site: 'scp-server' files: - target: 'dir/ectory' source: 'build/html/foo' keep-hierarchy: true - console-log 11
Nodepool - the provisioner
■ Manages a pool of Devstack images to use in project testing.■ Provisions nodes on HP and Rackspace clouds. (using novaclient)■ Makes sure there are always available nodes for each provider type.■ Listens to events from Jenkins ZeroMQ publisher plugin. (by C.Boylan)■ Tears down old instances when tests are completed.
12
DevStack Gate■ The integration test that verifies that all projects still work together after the
changes■ Provides a full OpenStack installation on a single machine with Devstack■ Installs Devstack and runs Tempest (the OpenStack integration test suite)
13
Logstash
■ All jenkins job logs are indexed and stored by Logstash
■ Logs pushing also handled by Gearman jobs
14
■ Elastic Search helps analyze the test results
Wrap Up
■ Git■ Gerrit■ Zuul■ Gearman■ Jenkins Job Builder■ Jenkins■ Devstack■ Nodepool■ Logstash
15
Only 9 out of 32 projects/tools managed by the Openstack InfraTeam
Some References:
■ Scaling the OpenStack Test Environment - http://amo-probos.org/post/15■ Zuul: a Pipelining Trunk Gating System - http://amo-probos.org/post/14■ Multiple Jenkins Masters with Jenkins Gearman Plugin - http://
docs.openstack.org/infra/publications/gearman-plugin■ Understanding the OpenStack CI System - www.joinfu.com/2014/01/
understanding-the-openstack-ci-system/■ cgit - https://git.zx2c4.com/cgit/about/
16