apache cloudstack qa strategy
TRANSCRIPT
ASF CloudStack – QA StrategyApache Cloudstack Collaboration ConferenceDec 1, 2012
Sudha R Ponnaganti, Sr Manager, SQA Engineering, Citrix Systems Inc.,
Email:[email protected]
QA Strategy - Introduction
Need for QA strategy
Goals for current presentation Transparency
Expected outcome Community participation
Superior Quality Releases
Expected release cycles
Agenda
Releases and QA Overview
QA strategy
Future
Community participation
Q & A and Feedback
Releases and QA Overview
Releases and QA overview
QA Activity Major Releases ASF CS Minor Releases ASF CS
Installation and Upgrade validation
New feature validation
Regression validation Automated
Integration validation
Performance and scalability testing
Solution validation
Security review
Recovery validation
Usability validation
L10N validation
Defect validation
Documentation review RN only
Continuous Release Cycles
Currently we are doing minor and major releases Cycles are longer
Time based vs. feature based release discussion
There is a need for
Continuous integration
Continuous validation
Continuous Integration
Every check-in gets tested as and when it is submitted
Builds upfront quality faster
Should take less than 5 minutes to complete
Below is the process that is being worked up on currently
Jenkins kicks off a job after each
checkin
That job spins up a devcloud appliance
devcloud checks out the commit that kicked off
the job;
devcloud builds cloudstack
devcloud adds itself as a host
devcloud runs the smoke test
Continuous Validation
Run all the tests on nightly builds
With current structure, below are the sequence of tests that run nightly
RAT ( release
audit tool)
Compile + junit
Docs + apidocs
Package building
Install tests Smoke testsRegression
Tests
QA Strategy
Installation and Upgrade Validation Compatibility Matrix
Manual Validation
Priority Type Type of Dist Base Upgrade V OS Networking Network Devices Storage Guest VMs Browsers Features to Cover
Xen KVM Vmware OVM Bare Metal
P1 Open Source ASF CS 4.1 CentOS 6.3 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic
VR, F5, SRX,
Netscaler VPX ISCSI, NFS, Local
Template: CentOS5.6PV 64bit,
CentOS5.5PV 64bit, IE8, FF 10, Chrome, SafariBVTs
P1 ASF CS 4.1 Ubuntu 12.04 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic
VR, F5, SRX,
Netscaler VPX ISCSI, NFS, Local
Template: OVM System.img
ISO: OVM System.img IE8, FF 10, Chrome, SafariBVTs
Multinode install
P1 Multiple MS ASF CS 4.1 CentOS 6.3 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic
VR, F5, SRX,
Netscaler VPX
(NS10,build 54.7.nc) ISCSI, NFS, Local
Template: CentOS5.6PV 64bit,
CentOS5.5PV 64bit,
Windows2008R2 64bit, IE8, FF 10, Chrome, SafariBVTS
P1 ASF CS 4.0 ASF CS 4.1 CentOS 6.3 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic
VR, F5, SRX,
Netscaler VPX ISCSI, NFS, Local
Template: CentOS5.6PV 64bit,
CentOS5.5PV 64bit, IE8, FF 10, Chrome, SafariBVTs
P1 ASF CS 4.0 ASF CS 4.1 Ubuntu 12.04 6.0.2+HF, 6.1 RHEL 6.2 5.1 2.2.2 N/A Advanced/Basic
VR, F5, SRX,
Netscaler VPX ISCSI, NFS, Local
Template: CentOS5.6PV 64bit,
CentOS5.5PV 64bit, IE8, FF 10, Chrome, SafariBVTs
Compatibility Matrix ASF 4.1- Supported Vs Tested
Upgrade
Fresh
Hypervisors
Installation and Upgrade Validation
Install and Deployment validation Specifics:
Install and deployment Validation need to be done in context of release artifacts to be in alignment with voting process.
Step by step instructions are posted here.
https://cwiki.apache.org/CLOUDSTACK/cloudstack-40-test-procedure.html
New Feature and Regression Validation
New Feature Validation Regression Validation
• QA team writes test cases for new
features based on FS.
• QA posts test plans to community to get
feedback
• QA automates and executes new test
cases for current release. Adds TCs to
regression for future releases
• We can provide template if you need to
add test cases.
• You can find test suites on wiki
• There are around 3000 + test cases in
the regression suite currently.
• This suite includes Automated and
Manual tests
• QA reviews overall feature list and
identifies regression coverage based on
impact for each release – not sustainable
• To be successful in open source env,
need to have 100% automated coverage
New Features and Regression Validation
Various Frameworks are available based on Junit, Python, Selenium to automate test cases
A combination of all of these provide comprehensive coverage
Unit Tests BVT Tests Regression Suite
Other Integration Suites ( EC2 and
S3)UI Automation
New Features and Regression Validation
• Unit Test framework ( JUnit )
• Provides early feedback and fixes issues close to source
• How to Write Unit tests - https://cwiki.apache.org/CLOUDSTACK/unit-
testing-101.html
• Currently all unit tests running with 100% pass rate
• Runs on every build
• Code coverage results are posted here
New feature and Regression Validation BVTs and Functional Automation - Marvin
600+ test cases running on multiple configurations and covering multiple hypervisors
How to write functional automated TC : https://cwiki.apache.org/CLOUDSTACK/testing-with-python.html
BVT
Hypervisor Configuration
Xen ( multiple versions) Advanced, Basic
VM ware Advanced, Basic
KVM Advanced, Basic
OVM Planning
Functional Regression
Hypervisor Configuration
Xen ( multiple versions)Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic with and
without SG
VM ware Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic w/o SG
KVM Advanced, Advanced - Netscaler, EIP/ELB Rules, Basic with and
w/o SG
OVM Planning
New features and Regression Validation
Multiple test pods are being setup to run suites in parallel– around 15 test pods are required
Porting to ASF CS env has been in progress. BVTs are done.
Prasanna and Edison are covering more detailed talk on Automation on Sunday morning.
New Features and Regression Validation Integration frameworks
Currently we are in the process of submitting test suite that covers EC2 Query API validation – JCloud framework
Around 40 APIs are automated In the process of covering all APIs
Failed test cases are being fixed
Need to automate complex scenarios.
Details on test implementation are posted below https://cwiki.apache.org/confluence/display/CLOUDSTACK/EC2+Qu
ery+API+Test+Suite+Automation
New Features and Regression Validation
UI Automation
Selenium
Faster and Easy
Compatible with python backend
Plan to Deliver Smoke Tests by – Early FY 13
POC and initial submits – coming soon
Performance and Scalability Testing
Native Tool - Simulator
Simulator - mock hypervisor and will react to every command sent to it by the MS similar to a regular hypervisor - easy to setup and run
Simulator and Perf resources: http://wiki.cloudstack.org/pages/viewpage.action?pageId=1344953
2 papers have been published Covering Basic Zone
Advanced Zone
Performance Use cases
• Time taken to propagate security group rules in basic zone • Deploy VM time, CPU, load averages and # of DB connections• Steady state CPU, load average and # of DB Connections• Restart MS measures: Time to stop MS, Start MS and rebalance
with 30K Hosts across 4 MS (agent load size to 500 and 1000)• Restart MS measures : with Host in maintenance mode• Bandwidth usage collection frequency for routers• Router Status poll metrics in RVR env
Performance Testing
What Next Holistic Coverage
Router
Storage
Network
Realistic Scale Testing
Security Validation
Security validation has been important aspect of ASF CS releases
All FS should include security considerations so that QA can identify and cover these areas
Security Coverage done included the following along with manual validation and other security reviews
Solution Validation
o Current solution Validation use cases include
o reference architecture deployment and
o additional complex deployment scenarios.
o Test plans are posted on wiki
o There is no need to adopt these right away in to Cloudstack but will monitor if there is a need in future
Recovery, Usability and L10N
Recovery scenarios, Usability Reviews/ L10N - Not prioritized Jessica Wang has been doing L10N review along with few other
contributors If you have expertise in these areas, do participate in QA forums
Documentation review Focused validation at the end of the cycles Watch for QA requests for review
Defect Triage and Validation
You can participate by testing product logging defects here ( ASF JIRA)
Always log defects when you start discussion on mailing lists
Add automation for defects that you validate. Tests can be checked in to appropriate suites based on type of test –unit/smoke/regression suites
Participate in Bug fest activities
Future
Future - Status
Some improvements that QA has been doing Q3/ Q4 FY ‘12
Automation
Functional – BVTs and Regression (Python framework) - goal 700+ TCs
Unit Test Framework ( Junit ) – Add TCs for ongoing releases + improve legacy by adding TC for patches
Integration automation frameworks (ec2 and s3) –implementation and make it part of regression suite ( Jcloud Test framework)
Future - status
• TCM Implementation and loading test cases –Address Test Case gaps ( total inventory) – Currently 3000 test cases exist. Goal is to get to 5000
• Performance White paper – Use cases collected and release V3 with known use cases – backlog has been built.
• Tools Static code analyzer – In progressCode coverage – Cobertura Code inspection -
Future - status Infrastructure improvements
Test Env improvements ( Fresh Install Automation for Functional testing)
Acquire external devices and standardize test env
Build infrastructure improvements
Continuous integration implementation
Continuous validation
Standardized certifications
New versions of HV, NS and other integrations
Standardized templates
Test plan/Test case / Defect writing Guidelines, Test case execution reports, baseline metrics
Future - status
Initiatives for FY ‘13
Move towards enterprise quality standards Unit Test Coverage + Automation – 70% + Code coverage Regression coverage – 90% + coverage New features – 100% coverage Defects regressed through automation Install/Upgrade Automation UI automation - selenium Standardize QA coverage for enterprise readiness
Regression New functionality Automation Install/upgrade ( compatibility matrix automation) Usability Performance ( tools and use case expansion) Reliability ( longevity under load) L10N ( internal or vendor based)
Future - status
508 ( Accessibility)
Integration
Security
Real world use cases
Install Upgrade automation
Infrastructure improvements to run automation across
multiple configurations
Community Participation
Community Participation
Scheduled QA Meetings on weekly basis – Will be publishing meeting details for CS 4.1 and expect it to be starting soon. Everyone is welcomed to join and participate in QA Activities for ASF CS 4.1 release
David has planned IRC Sessions to drive Quality as a continuous process. Please do participate and help to drive this continuous test process - timings 1800 UTC in #cloudstack-dev on Wednesdays
ASF Cloudstack Dev mailing list is great way to share your ideas and contributions [email protected]
All Dev, Build and QA documents are posted here Defects can be logged here
Q & A
ASF CloudStack – QA StrategyApache Cloudstack Collaboration ConferenceDec 1, 2012
Sudha R Ponnaganti, Sr Manager, SQA Engineering, Citrix Systems Inc.,
Email:[email protected]