diablo vs havana - openstack 622 tests to run and 154 (27%) to pass.* ... automated gate ......
TRANSCRIPT
![Page 1: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/1.jpg)
DIABLO VS HAVANAHOW OPENSTACK HAS MATURED
Joe Gordon
![Page 2: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/2.jpg)
ABOUT ME Developer @
Full time upstream OpenStack developernova-coreATC since EssexInterested in making OpenStack scale and workcontact information
jog0 on freenodegithub.com/jogolaunchpad.net/~jogo
![Page 3: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/3.jpg)
OPENSTACK FROM DIABLO TO HAVANAA BRIEF HISTORY OF THE INTEGRATED RELEASE
![Page 4: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/4.jpg)
DIABLOSwiftNova
Nova-volumeNova-networkIdentity
GlanceKeystone*
![Page 5: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/5.jpg)
ESSEXSwiftNova
Nova-volumeNova-networkIdentity
GlanceKeystoneHorizon
![Page 6: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/6.jpg)
FOLSOMSwiftNova
Nova-volumeNova-network
GlanceKeystoneHorizonQuantum (Neutron)Cinder
![Page 7: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/7.jpg)
GRIZZLYSwiftNova
Nova-networkGlanceKeystoneHorizonQuantum (Neutron)Cinder
![Page 8: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/8.jpg)
![Page 9: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/9.jpg)
Total number of OpenStack services: 32
HAVANASwift (4)Nova (6)
Nova-networkGlance (2)Keystone (1)Horizon (1)Neutron (6)Cinder (4)Ceilometer (4)Heat (4)
![Page 10: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/10.jpg)
HAVANA FEATURES SUPPORTED BY DIABLOSwiftNova
Nova-networkGlanceKeystoneHorizonNeutron (Quantum)CinderCeilometerHeat
![Page 11: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/11.jpg)
HAVANA FEATURES SUPPORTED BY DIABLOSwiftNova
Nova-networkGlanceKeystoneCinder
![Page 12: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/12.jpg)
OPENSTACK HAS GROWN
![Page 13: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/13.jpg)
GROWN OUTNEW SERVICESHeatCeilometerMore in pipeline
IcehouseTrove
IncubationIronicMarconiSavanna
![Page 14: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/14.jpg)
GROWN UPExpand existing APIs
Split out projectsExtensions
![Page 15: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/15.jpg)
DIABLO NOVA EXTENSIONS| Hosts | Host administration| Volumes | Volumes support| Quotas | Quotas management support| SimpleTenantUsage | Simple tenant usage extension| FlavorExtraData | Provide additional data for flavors| SecurityGroups | Security group support| FlavorExtraSpecs | Instance type (flavor) extra specs| Createserverext | Extended support to the Create Server v1.1 API| Multinic | Multiple network support| VSAs | Virtual Storage Arrays support| VolumeTypes | Volume types support| Keypairs | Keypair Support| Rescue | Instance rescue mode| VirtualInterfaces | Virtual interface support| Floating_ips | Floating IPs support
![Page 16: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/16.jpg)
POST DIABLO NOVA EXTENSIONS AdminActions Agents Aggregates AssistedVolumeSnapshots AttachInterfaces AvailabilityZone BareMetalExtStatus BareMetalNodes BlockDeviceMappingV2Boot CellCapacities Cells Certificates Cloudpipe CloudpipeUpdate ConfigDrive ConsoleOutput Consoles Coverage DeferredDelete DiskConfig Evacuate ExtendedAvailabilityZone ExtendedFloatingIps ExtendedIps ExtendedIpsMac ExtendedQuotas ExtendedServerAttributes ExtendedServices ExtendedStatus ExtendedVIFNet ExtendedVolumes FixedIPs FlavorAccess FlavorDisabled FlavorManage FlavorRxtx FlavorSwap FloatingIpDns FloatingIpPools FloatingIps FloatingIpsBulk Fping HideServerAddresses Hypervisors ImageSize InstanceActions Migrations MultipleCreate NetworkAssociationSupport Networks OSInstanceUsageAuditLog OSTenantNetworks QuotaClasses SchedulerHints SecurityGroupDefaultRules ServerDiagnostics ServerPassword ServerStartStop ServerUsage Services Shelve UsedLimits UsedLimitsForAdmin UserData UserQuotas VolumeAttachmentUpdate
![Page 17: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/17.jpg)
SO ONCE WE NORMALIZEFOR THE GROWTH IN
OPENSTACK WHAT ELSEHAVE WE DONE IN THEPAST FOUR RELEASES?
![Page 18: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/18.jpg)
CHANGESDriversTesting and bugsPerformance and scalabilityForward facing internal changesProcess
![Page 19: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/19.jpg)
MORE DRIVERSNova: baremetal, docker, hyperv, powervm
Cinder: coraid, emc, glusterfs, gpfs, nexenta, rbd, sheepdog,soldfire, ...
![Page 20: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/20.jpg)
BETTER TESTING, FEWER BUGSUnit testsIntegration tests (Tempest)Lots of fixed bugs
![Page 21: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/21.jpg)
UNIT TESTSPERCENT UNIT TESTS BY LOC
Project Diablo HavanaNova 43% 56%Glance 62% 62%Swift 65% 71%Keystone 41% 45%
![Page 22: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/22.jpg)
TEMPESTTHE OPENSTACK INTEGRATION TEST SUITE
![Page 23: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/23.jpg)
DIABLO TEMPESTOrignally named 'openstack-integration-tests'Just 107 tests*
extensionsflavorsfloating ipsimage metadataimageskeypairsserversvolumes
2,655 lines of python codeFirst release of Tempest
![Page 24: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/24.jpg)
HAVANA TEMPEST1275* tests
1150 api testscompute, image, identity, network, object_storage,orchestration, volume102 test files
92 cli tests11 scenario tests18 thirdparty (boto) tests4 other
Parallel20-30 minutes to run in gate24,837 lines of python code
![Page 25: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/25.jpg)
HAVANA API TESTS785 compute216 identity24 image17 network50 object_storage54 volume
![Page 26: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/26.jpg)
HAVANA TEMPEST VS OPENSTACK DIABLOSame APIs as Diablo plus new onesDiablo has been 'end of life'edMany tests cover new features. Ignore thoseTempest not designed to be version independentGot 622 tests to run and 154 (27%) to pass.*
110 of which were in compute
![Page 27: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/27.jpg)
HAVANA TEMPEST VS OPENSTACK DIABLORESULTS
Diablo has poor parameter validationInvalid key namesMin/max count for multiple creates
Changed return codes for negative casesReturn empty list and 200 instead of 400 when invalid filtervalue in nova list serversWe shouldn't have done this
Poor pagination in DiabloJSON only tests in Diablo, Havana tests XML tooMany false negatives in results due to slight changes in nowAPIs and metadata are used
![Page 28: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/28.jpg)
CRITICAL BUGSProject Diablo Essex Folsom Grizzly HavanaNova 14 22 30 31 33Glance 5 15 30 13 8Keystone 31 11 9 8Cinder 10 3 12Neutron 27 14 24
CategoriesNon-gated featuresDependenciesRace conditionsPerformance
![Page 29: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/29.jpg)
PERFORMANCE ANDSCALABILITY
![Page 30: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/30.jpg)
PERFORMANCE AND SCALABILITYNova-scheduler
Active-activeNo RPC fanout
DBIndicesSmarter queriesFewer queries
service groupHeartbeat to DB every 10 seconds1k DB calls a second if have 10K nodes.Adjustable intervalAlternatives memcache and zookeeper
![Page 31: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/31.jpg)
PERFORMANCE AND SCALABILITYMore efficient image work flow
Glance, Nova, CinderMove large files around less
PKI TokensFewer round trips to keystone
![Page 32: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/32.jpg)
PERFORMANCE AND SCALABILITYCASE STUDY FROM HP PUBLIC CLOUD
Diablo40,836 LOC changed to 239 files in nova2,265 LOC to manage DB+100s of hours of DB tuning to make it work with morethen just a few nodes
HavanaDB just worked1,167 LOC changed in 35 filesMost of that is in patches in review today
Thanks to Thom Leggett for these statistics
![Page 33: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/33.jpg)
FORWARD FACING INTERNAL CHANGESSplit out projectsOsloRoad to live and continuous upgradesIterate on APIs
![Page 34: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/34.jpg)
PROCESS
![Page 35: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/35.jpg)
DRIVING PRINCIPLESNever break trunk
Master branch is always greenDevelopers are never blocked
TransparencyAutomate everythingEgalitarianBe strict. Reduce burden on reviewers
![Page 36: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/36.jpg)
DIABLOAutomated Gate87 Contributors in September 2011 (release of Diablo)No GateNo Git or GerritBzr and Launchpad
![Page 37: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/37.jpg)
HAVANA346 Contributors in October 2013 (release of Havana)
Scale development processIntegrated Gate
9 Projects30+ separate services1 Merge pipeline
New tools for an asynchronous worldTests not pass fail, rather classified with a race percentageFix that race condition we hit 0.5% of the timeDebugging is hardelastic-recheck
![Page 38: DIABLO VS HAVANA - OpenStack 622 tests to run and 154 (27%) to pass.* ... Automated Gate ... Limiting factor to the whole system is reviewers (and not just](https://reader031.vdocuments.site/reader031/viewer/2022022012/5b1c055f7f8b9a19258f63bb/html5/thumbnails/38.jpg)
HOW WILL OPENSTACK KEEP MATURINGWHAT CHALLENGES ARE WE FACING TODAY?
More testingLive rolling upgradesLimiting factor to the whole system is reviewers (and not justcore)Improving quality while growing the project as a wholeScalability