ensuring openstack version up compatibility...wrap up the tests are very important for ensuring...
TRANSCRIPT
![Page 1: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/1.jpg)
Page 1
Ensuring OpenStack Version up Compatibility
CloudOpen Japan 2013-05-31
Masayuki Igawa : [email protected]
![Page 2: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/2.jpg)
Agenda
1. Overview of the OpenStack
2. Overview of Compatibility Issues
3. Overview of the OpenStack Tests 1. Unit Test
2. Integration Test
4. Overview of Scenario Tests
5. Proposal for the Implementation at Havana Summit
6. Current Implementation of Scenario Tests
7. Current Status 1. Tempest
2. My proposal
8. Wrap-up
9. Appendix
Page 2
![Page 3: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/3.jpg)
Overview of the OpenStack
▌One of the most popular OSS IaaS infrastructure Software
▌Consists of several loosely-coupled components
▌Many features are being evolved with six month release cycle
Page 3
Version up compatibility is one of the greatest concerns for user’s viewpoint!
![Page 4: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/4.jpg)
Release Cycle - Overview of the OpenStack
▌Many Features are being developed with 6 months release cycle.
Page 4
Now
Now
6months
6months
6months
6months
2013.2
6months
![Page 5: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/5.jpg)
Overview of Compatibility issues
▌Databases There are many changes between Every Release Cycle
• ex) 35 DB schemas were changed, and 33 new DB Tables were added(*)
OpenStack components have database migration mechanisms.
Nova, Glance and Cinder have a test framework for this issue.
• In Nova, this framework discovered a data loosing bug.
• Other components need volunteers.
Page 5
* between Essex and Folsom
![Page 6: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/6.jpg)
Overview of Compatibility issues
▌Configurations OpenStack components have many configuration setting parameters.
In the public document, the number is about over 600! in Nova only.
• However, I think there are many undocumented configuration parameters.
The number of Configuration changes is over 130!(*)
Page 6
* between Essex and Folsom
http://docs.openstack.org/trunk/openstack-compute/admin/content/list-of-compute-config-options.html
![Page 7: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/7.jpg)
Overview of Compatibility issues
▌APIs APIs are versioning in the OpenStack.
Same version APIs should have backward compatibility.
However, most of APIs parameters are not validated.
• This can caused compatibility issues by the fluctuation of input values.
The comprehensive and more strict input validation can avoid this issue.
There are some works in this area.
These works are very nice! And we need to test for ensuring the compatibility.
Page 7
for Nova for Cinder
![Page 8: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/8.jpg)
Overview of Compatibility issues
▌How can we ensure compatibility? - Source Code Review? Yes! We are already doing on https://review.openstack.org.
All commits are reviewed by core reviewers
Page 8
![Page 9: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/9.jpg)
Overview of Compatibility issues
▌How can we ensure compatibility? – Testing? Yes! We are already doing it.
Manually? No! Manual testing is painful!
• Jenkins does it: https://jenkins.openstack.org
• All commits are tested with Jenkins CI.
Page 9
![Page 10: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/10.jpg)
Overview of Compatibility issues (Development work flow)
▌Gerrit Workflow Quick Reference
Page 10
https://wiki.openstack.org/wiki/File:Contribution_path.png
OpenStack Community Environment
approve
merge
By Sdague
![Page 11: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/11.jpg)
Page 11
Tests are very important!
http://www.flickr.com/photos/sidelong/246816211/ By David Bleasdale:
![Page 12: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/12.jpg)
Overview of the OpenStack Tests
▌Basically, test cases are always tested automatically by the community.
-> Developers need to write the test code for their feature implementation.
We have two types of the tests.
Page 12
Integration Testing Unit Testing
By David Goehring : http://www.flickr.com/photos/carbonnyc/6415460111/ By INTVGene :http://www.flickr.com/photos/intvgene/370973576/
![Page 13: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/13.jpg)
Unit Test
▌All Developers must write Unit Test codes. The test code makes the implemented code behavior clear.
And, ensures the feature code quality.
▌Tools testr/nose
nose
• is nicer testing for Python
• extends unittest to make testing easier.
testr
• will run tests in parallel(so they go faster)
• it keeps robust logs of the results.
Jenkins
• Continuous Integration environment.
• Jenkins runs unit tests every ‘git review’ing.
Page 13
![Page 14: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/14.jpg)
Integration Test
▌Tempest Tempest is the OpenStack Integration Test Suite
Runs and validates on the every ‘git review’(*) automatically
Page 14
(*) OpenStack Community uses the gerrit system.
![Page 15: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/15.jpg)
Integration Test - Tempest
▌When the test is failed, you get ’-1’ from Jenkins.
▌Then, you need to fix your code, and do ‘git review’ again
Page 15
![Page 16: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/16.jpg)
Hard to impl, 9.6%
Unnecessary,
25.4%
Implemented,
36.3%
We've implemented,
8.3%
Suspended, 7.1%
Ongoing, 13.3%
0
50
100
150
200
250
# o
f APIs
Test coverage of Nova APIs with Tempest
Status of Integration Testing
▌Test coverage of APIs(Nova)
Page 16
A little more! About 20% APIs are remained
Still not Tested
(Our research at 2013-05-24)
![Page 17: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/17.jpg)
Overview of Scenario Tests
▌What are Scenario Tests? Testing across the components such as Nova, Keystone, Glance and so on.
Top-down testing from the user’s perspective.
Page 17
![Page 18: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/18.jpg)
Overview of Scenario Tests(continued)
▌Tempest tests until Grizzly
Page 18
Network tests • create keypairs • create security
groups • create networks
:
Block Storage tests
• create a volume • get the volume • delete the volume
:
Compute tests • create a keypair • create a security
group • boot a instance
:
Identity tests :
Image tests :
Individual component based testing
Object Storage tests
:
![Page 19: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/19.jpg)
Overview of Scenario Tests(continued)
▌What are Scenario Tests?
Page 19
Scenario 1 1. create a flavor 2. create a image 3. create a network 4. create &
configure a project, a quota, a role, a user
5. create a keypair 6. boot a instance 7. list & show the
instance 8. create a volume 9. list & show the
volume 10. attach the
volume :
Across the multiple components & sequential testing
![Page 20: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/20.jpg)
Overview of Scenario Tests(continued)
▌Effects of Scenario Tests For developers: We’ll be able to check whether new code (bug-fixes or
features) will cause side effects on the other components with the top-down method.
For users: We’ll be able to increase the use case coverage which is important from the user’s perspective.
We’ll be able to cover the user’s use case by adding various scenarios according to the usage scene.
▌Points to Consider for the Scenario Category of the scenario: Private cloud, Public cloud, VPC and so on
Scale: # of tenants, users, networks and so on
Validation method: REST API, ping, ssh and so on
Page 20
![Page 21: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/21.jpg)
Proposal for the Implementation at Havana Summit
▌The Test Scenario of the First Implementation Basic
Minimum
Across the multiple components
▌Access Client and Directory Options
Page 21
No Access Client Directory
1 CLI (cli.ClientTestBase) tempest/scenario
2 Client Library tempest/tempest/tests/scenario
3 RestClient tempest/tempest/tests/scenario
![Page 22: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/22.jpg)
Current Implementation of Scenario Tests
▌The Test Scenario of the First Implementation Basic
Minimum
Across the multiple components
▌Access Client and Directory Options
Page 22
No Access Client Directory
1 CLI (cli.ClientTestBase) tempest/scenario
2 Client Library tempest/tempest/scenario
3 RestClient tempest/tempest/tests/scenario
<- The concepts are not changed.
![Page 23: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/23.jpg)
Current Implementation of Scenario Tests - Detail
▌Basic – All OpenStack Administrators can understand
▌Small – Main code is only 208 lines
▌Besides, testing across the multiple components
Page 23
https://review.openstack.org/#/c/26403/
![Page 24: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/24.jpg)
Current status of Tempest
▌Restructuring directory tree and well documented
Page 24
./ +-- bin/ +-- cli/ | +-- simple_read_only/ +-- etc/ +-- include/ +-- stress/ | +-- tests/ | +-- tools/ +-- tempest/ | +-- common/ | +-- openstack/ | +-- services/ | +-- tests/ +-- tools/
./ +-- bin/ +-- doc/ | +-- source/ +-- etc/ +-- include/ +-- tempest/ | +-- api/ | +-- cli/ | +-- common/ | +-- hacking/ | +-- openstack/ | +-- scenario/ | +-- services/ | +-- stress/ | +-- thirdparty/ | +-- whitebox/ +-- tools/
Trunk Grizzly
Restructuring and well
documented
![Page 25: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/25.jpg)
Current status of my proposal
▌Blueprint Not approved yet..
▌Code Core reviewer reviewed and +1ed. Almost merged!
Page 25
![Page 26: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/26.jpg)
Wrap up
▌The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective.
▌The scenario test directory is ready now
▌Please share your scenario tests. -> That makes all of us happy.
Page 26
![Page 27: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/27.jpg)
APPENDIX
Page 27
![Page 28: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/28.jpg)
proposal for tempest directory restructure
Page 28
https://review.openstack.org/#/c/28453/
![Page 29: Ensuring OpenStack Version up Compatibility...Wrap up The tests are very important for ensuring compatibility Especially, scenario tests are very important for user’s perspective](https://reader034.vdocuments.site/reader034/viewer/2022042115/5e91fea5f2abb16fff20f195/html5/thumbnails/29.jpg)
Appendix
▌Related Blueprints/Havana Summit Sessions/code Add Scenario Tests to Tempest (by me):
https://blueprints.launchpad.net/tempest/+spec/add-scenario-tests
https://review.openstack.org/#/c/26403/
Nova VM Lifecycle Test (by Nithya Ganesan):
https://blueprints.launchpad.net/tempest/+spec/nova-vmlifecycle-test
Strategies for Gating in a Growing Project(by David Kranz):
http://summit.openstack.org/cfp/details/23
Tempest Scope (by Sean Dague):
http://summit.openstack.org/cfp/details/165
Tempest - Gap Analysis - Identify new testsdevelop (by Ravikumar Venkatesan):
http://summit.openstack.org/cfp/details/32
Page 29