cloudstack eu user group - trillian
TRANSCRIPT
The Cloud Specialists
THE HITCH-HIKERS GUIDE TO CI/CD
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus
The Cloud Specialists ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public & private clouds. They are the leading
independent global CloudStack services company”
A b o u t S h a p e B l u e
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Paul Angus – VP Technology• Global authority on CloudStack & cloud infrastructure design.
• 10+ years C-Level experience.
• Apache CloudStack project committer
• Specialising in deployment of CloudStackand surrounding infrastructure
• USP, Georgian Ministry of Justice, Orange, TomTom, PaddyPower, Ascenty, BSkyB
A b o u t M e
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n
Why?
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h y
“Building CloudStack environments using CloudStackand Ansible? Aren’t you just having fun?”
CEO
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h y
“Building CloudStack environments using CloudStack?Are you just having fun?”
Talented Cloud Architect
“If we can automate the building of environments including the networking, and generate any topology we want quickly, we can achieve the tasks that I’ve highlighted in these slides that I have with me, just in case you asked.”
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h y
building of environments including the networking, and generate any topology we want quickly, we can achieve the tasks that I’ve highlighted in these slides that I have with me, just in case you asked.”
CEO“That would be excellent.Go ahead.Oh, and here’s a pay rise”
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
We need to test:• New features• Hotfixes/patches• Full Versions• Physical integrations• New hypervisors• Upgrades
• Against a broad range of deployment architectures• Against a broad range of hypervisor types and
versions
W h y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Many, many, MANY permutations of a CloudStack deployment….• Basic / Advanced• Local / shared / mixed storage• More than 8 common hypervisor types/versions• 4 or 5 Management server OS possiblilities
• That’s 144 combinations only looking the basics.
• Previous attempts at test automation haven’t been able to get coverage or capacity.
W h y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Discovery projects:• New Features
• CloudStack integrations (ie Ceph)
• CloudStack ecosystem components (ie ELK stack)
W h y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Lab v1.0
• Templates to build nested hypervisor hosts
• Cloudmonkey scripts to configure zone
• Manual creation of storage mounts
Simple but pretty effective
H u m b l e B e g i n n i n g s
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Lab v1.0
• A lot of small edits scripts to fit use cases.
• Human error still a big factor
• Can’t ‘set and forget’
• A lot of individual knowledge required
• Loose definition of ‘testing’.
H u m b l e B e g i n n i n g s
The Cloud Specialists ShapeBlue.com @ShapeBlue
Trillian Tenets
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Fully automated deployment of CloudStack environments from any CloudStack package repo
Tr i l l i a n Te n e n t s
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Ability to support multiple concurrent environments• No hardcoding to a specific (our) lab/environment• As few separate components as possible• Separation of ‘code’ and ‘configuration’• Scripted command-line use fully supported• Ability to support as many hypervisors as possible• As flexible as possible• Best ‘performance’ possible• Best utilization of hardware resources possible
Tr i l l i a n Te n a n t s
The Cloud Specialists ShapeBlue.com @ShapeBlue
Meet The Family
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
T h e C I F a m i l y
Trillian
MarvinWinston
JenkinsBlueOrangutan
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
TrillianAnsible based environment builder
• Multi-tenant (and multiple concurrent environments)• CloudStack builds the Virtual Infrastructure• Hypervisor hosts are CloudStack templates• Supports multiple and mixed hypervisors• Supports Advanced and Basic networks• Open source
T h e C I F a m i l y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
TrillianInstalls & Configures:
• CloudStack• MySQL• XenServer (configures)• vCenter (configures)• ESXi (configures)• KVM• Marvin hosts and Marvin test suites
T h e C I F a m i l y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
MarvinPython based test framework based on ‘nose’ test platform. Configuration files define environment to be tested.
• Integral part of the CloudStack codebase• Recently given its own install package• Smoke and Component tests• 53 smoke test suites• 154 component test suites• Additional tests for plugins such as Nuage and SolidFire
T h e C I F a m i l y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
JenkinsCI Platform
• Runs build jobs to produce RPMs from source code• Presents an integration point to wrap up the creation of
environments using Trillian and the running of tests using Marvin.
• Gives a UI to Trillian.
Click icon to add picture
T h e C I F a m i l y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Blue OrangutanBuild-bot
• Automation endpoint to send build jobs to Jenkins for CentOS 6/7 and Debian packages from CloudStack source
• Provides an integrated way to create packages from pull requests in Github.
• Can kick-off Jenkins jobs to build and test environments based on packages which it builds.
Click icon to add picture
T h e C I F a m i l y
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
WinstonTest Orchestrator (Future project)
• Receives ‘offers’ of test type/capacity from remote test labs• Takes request from Blue Orangutan to test packages and
distributes tests to suitable remote labs.Offers might include:• Use of local storage• Use of basic vs advanced networks• Use of particular hypervisors/versions
T h e C I F a m i l y
The Cloud Specialists ShapeBlue.com @ShapeBlue
How its used
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Developers doDark Magic
Developer creates pull request on
GitHubInitial code review
Blue Orangutan calls Jenkins to build packages
based on PR
Blue Orangutan calls Jenkins to run
a test matrix
Jenkins calls Trillian to build
environments
Trillian builds environments (inc
Marvin host)
Jenkins calls respective Marvin hosts to run tests
Jenkins attaches results to pull
request on GitHub
The crowd goes WILD.
P r o c e s s u s e d i n A p a c h e C l o u d S t a c k P r o j e c t
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Test Pull Requests to add XenServer 7 support• XenServer won’t run on ESXi 5.5
• Create XenServer 7 VMs on stand-alone ESXi 6.0 host and ‘point’ Trillian at them.
• vSphere 6.5 – Create vCenter and ESXi templates and run tests.
X e n S e r v e r 7 & v S p h e r e 6 . 5
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Build base CloudStack environment quickly
• Leverage CloudStack to create other infrastructure components just as quickly.ie• HostBill server• Cloudian manager server and storage nodes• Leverage CloudStack networking to give external access
to environments – Static NAT, port forwarding or VPN.
C l o u d i a n & H o s t B i l l
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Build required configuration with required components• Specific versions of CloudStack• Specific hypervisor types• Specific hypervisor versions
• Manual creation of additional components still allowed• ie second zone with different hypervisor type or version
S o l i d F i r e P l u g i n
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
We can do all of these things
AT THE SAME TIME
C o u p D e G r â c e
The Cloud Specialists ShapeBlue.com @ShapeBlue
Trillian
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
https://github.com/shapeblue/Trillian
Tr i l l i a n
• Single zone• Use pre-set or custom Repos• Local or shared storage• ESXi, KVM (C6,C7,U14.04),
XenServer hosts• Customisable number of primary
and secondary storage pools• Multiple Management servers• Selectable Management server
OS (C6.x, C7, U14.04)
• Multiple hypervisors• dvSwitch (public/guest traffic)• Configurable Java versions• Option to use pre-built hosts• Multiple MySQL configurations
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
A n s i b l e i n T r i l l i a n
• All driven from Ansible
• A global group_vars file stores all local configuration data such as:
• Passwords• All defaults ie basic vs
advanced or network naming
• EVERYTHING can be customised at runtime.
(except that)(and that)
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
ansible-playbook generate-cloudconfig.yml -i localhost --extra-vars "env_name='localstoragetest‘env_version=cs49mgmt=1db=0hvtype=x xs_ver=xs65sp1hv=2env_accounts=allpri=2sec=1build_marvin=yesmgmt_os=6use_local_storage=true
use_shared_storage_systemvms=trueuse_shared_storage=false"
ansible-playbook deployvms.yml -i ./hosts_localstoragetest
Click icon to add picture
A n s i b l e i n T r i l l i a n
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n i n J e n k i n s
• Different Jenkins’ jobs with specific‘assumptions’.
• Pull Requests receive URL from Blue Orangutan and automatically Run test suites, then clean up.
• Reference_Trillian is a reference implementation of Jenkins and has more corner-case options exposed.
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n i n J e n k i n s
• Allows user to force a version of Java
• Allows use of ‘pre-built’ hypervisors and hypervisors which cannot themselves be virtualised.
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n i n J e n k i n s
• If environment requires the hypervisor to use local storage, then CloudStack adds a data volume to hypervisor and configures hypervisor to use it for local storage. This keeps template size down
• Mixed local and shared storage is supported
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Click icon to add pictureInternal ShapeBlue summary page UI plugin.
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n & J e n k i n s i n C l o u d S t a c k
• UI gives overall status/health of ‘Lab’ and environments
• UI links to a job’s Jenkins output as well as UI of built environment and SSH session onto specific VMs
C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
I n S u m m a r y
The Cloud Specialists
THE HITCH-HIKERS GUIDE TO CI/CD
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus