towards automated testing -...
TRANSCRIPT
![Page 2: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/2.jpg)
Agenda • Motivation • Architecture • Implementation • Replication • Proposal • Enhancements
![Page 3: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/3.jpg)
Agenda • Motivation • Architecture • Implementation • Replication • Proposal • Enhancements
![Page 4: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/4.jpg)
Motivation • Master is invariably unstable • Unchecked commits • No provision for developer to test feature branch • Testing is not yet a community effort • Not easy to test plug-ins privately • CloudStack set-up is involved – Management server, hypervisor,
network, storage
![Page 5: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/5.jpg)
Just for laughs Cloudstack Community Who are we
What do we want Testing and Code Quality
How do we want do it ………
![Page 6: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/6.jpg)
Goals • Continuously running BVTs • Enable community testing • Keep master stable • Isolate test-runs from each other, and from other infrastructures • Easy to replicate and set-up • Provide commits to cherry pick / buggy commits
![Page 7: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/7.jpg)
Agenda • Motivation
• Architecture • Implementation • Replication • Proposal • Enhancements
![Page 8: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/8.jpg)
Architecture
![Page 9: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/9.jpg)
Architecture Phase 1 : Set-up infrastructure • Provision management server from scratch • Provision hypervisor from scratch • Storage and networking set-up
Phase 2 : Run Marvin tests • Track bad commits • Cherry-pick good commits
Need to isolate test-runs Based on Prasanna’s effort (Big Thanks!!)
![Page 10: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/10.jpg)
Agenda • Motivation • Architecture
• Implementation • Replication • Proposal • Enhancements
![Page 11: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/11.jpg)
Implementation • Jenkins for integration We all like Jenkins
• Glue to hold all together • Reporting, alerting etc. • Separate jobs for phases 1 and 2
![Page 12: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/12.jpg)
Implementation Provisioning management server & hypervisors • Need a way to deploy an OS • Need a way to configure the systems • Need a way to provision storage and networking in an isolated manner
![Page 13: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/13.jpg)
Implementation Quick overview of technologies involved • Cobbler • IPMI tool • Puppet • DNSMASQ • Squid
![Page 14: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/14.jpg)
Cobbler • Open source OS installation server • Provides baremetal provisioning via PXE • Components
Ø Distro – the OS to provision Ø Profile – associate kickstart / configuration with OS Ø System – MAC to profile mapping. Additional configs
• Snippets to run post install scripts etc • Manages TFTP and PXE seamlessly
![Page 15: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/15.jpg)
Others… Puppet • Management server configuration • Hypervisor configuration like KVM agent etc. IPMI tool • To bootstrap baremetal provisioning Squid • HTTP proxy server (needed for isolated network requirement) DNSMASQ • DHCP / DNS management underneath Cobbler
![Page 16: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/16.jpg)
Implementation • Put all these and related technologies on a single VM • May contain additional storage (NFS server) • Seed system VMs • Python code acting as launcher script
![Page 17: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/17.jpg)
Implementation
![Page 18: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/18.jpg)
Implementation Minimum Requirements • XenServer to host VM and management server • Hosts for hypervisors • Isolated network • IPMI network
![Page 19: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/19.jpg)
Implementation Phase 2 • Run Marvin tests • Track commits from previous stable build to current build • If fails, one of the commits broke BVT • If pass, auto cherry-pick commits to (stable) master • All code in Python
![Page 20: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/20.jpg)
Agenda • Motivation • Architecture • Implementation
• Replication • Proposal • Enhancements
![Page 21: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/21.jpg)
Replication Setting up from scratch requires • Familiarity with CI tools • Familiarity with CloudStack specifics • Lots of time • Many DHCP / network configs
![Page 22: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/22.jpg)
Replication Propose to push VM like DevCloud (QACloud?) Thus, steps become • Load VM on XenServer • Provision network using conf provided • Add IPMI network • Change MAC addresses for Cobbler (has easy to use interface) • Add to Jenkins master • Optionally add NFS volumes • Optionally change domain name for hosts
![Page 23: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/23.jpg)
Replication To set-up from scratch https://cwiki.apache.org/confluence/display/CLOUDSTACK/QA+Infra+from+scratch
![Page 24: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/24.jpg)
Agenda • Motivation • Architecture • Implementation • Replication
• Proposal • Enhancements
![Page 25: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/25.jpg)
Proposal To keep master stable • Maintain a staging branch • Developer commits go to staging • If all BVTs pass, push commits • May need more BVTs • Reduce BVT running time (Simulator?)
![Page 26: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/26.jpg)
Agenda • Motivation • Architecture • Implementation • Replication • Proposal
• Enhancements
![Page 27: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/27.jpg)
Enhancements 1. Better integration of VMWare hypervisor 2. Pull latest systemvm templates 3. Make VM size manageable 4. Scheduling mechanism for resource pools 5. Throw open to developers via a service
(Thanks to Bharath for 2,3,4,5)
![Page 28: Towards automated testing - events17.linuxfoundation.orgevents17.linuxfoundation.org/sites/events/files/slides/Towards...• Proposal • Enhancements. Agenda • Motivation • Architecture](https://reader034.vdocuments.site/reader034/viewer/2022050715/5d59a32b88c99380578bd64c/html5/thumbnails/28.jpg)
Only the first step Lets all keep master stable!! Lets all test together!! Big Thanks to AlexH for guiding J