how to use opnfv testing tools to test/validate an ... · intel ericsson huawei opendaylight onos...
TRANSCRIPT
How to use OPNFV Testing tools to test/validate an Openstack
provided by a vendor
Morgan Richomme
NFV architect at Orange Labs
OPNFV ambassador
ONAP & OPNFV Functest committer
Speakers
Valentin Boucher
Software engineer at Kontron Canada
OPNFV Functest committer: https://wiki.opnfv.org/display/functest
Agenda
● OPNFV testing overview • OPNFV Testing chain
• OPNFV Testing tools
● Reusing those tools• Kontron
• Orange
OPNFV Testing OverviewTesting chain and tools …..
OPNFV CI complexity
Addons / Features
SDN Controller
Installer / Deployer
Lab/POD
SFC BGPVPN DPDK FDio
APEX Compass Daisy Joid
Linux Foundation Intel Ericsson Huawei
OpenDayLight ONOS OpenContrail
Fuel XCI
OPNFV Testing chain
ScenarioHA/noHA, SDN,
DPDK etc..
DeployerAPEX, Compass,
Daisy, Fuel and JOID
PodLF, Huawei, Intel etc ..
VersionMaster, Fraser
DeployInstaller project
ReportTest api, reporting
TestFunctest,
Yardstick etc
Test result
OPNFV Testing ecosystem
Functional testing -> Functest
Functest● Answer the question: “Does my infrastructure simply work as expected?”
● Provide tests from basic connectivity (<1s) to VNF testing (>1h)
● Leverage Upstream test suites from the integrated components (OpenStack/Rally & Tempest, Kubernetes/Default test suites, OpenDaylight, …)
● Allow “A la carte” test combination through rich, light and modular Alpine based framework, adapted to CI/CD chains
● PTL: C.Ollivier, Orange
Functest testcases for OpenStackC
ompl
exity
Gating
Official R
elease validation
Target validation
Full stack
Performance testing -> Yardstick & cie
Yardstick● Answer the question: “Qualify my NFVI and give me valuable figures?”
● Qualify Compute, Network and Storage domains through elementary tests
● Leverage OPNFv community to build baseline and define acceptable thresholds for Telco Cloud solution
● Allow lab owner to compare their Telco Cloud KPI with the other OPNFV labs
● In line with ETSI NFV
● PTL: R. Brattain, Intel
Yardstick
More than 50 available tests covering different domains
(compute, network, storage, HA, IPv6, KVM, ..)
Src: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/15-list-of-tcs.html
Yardstick
Src: Alassane Samba (Orange) https://wiki.opnfv.org/display/testing/R+post-processing+of+the+Yardstick+results
TC_012 post processing
Yardstick testing capacity
• Generic test cases
• Compute
• Storage
• Network
• HA test cases
• Service HA
• Node HA
• Function test cases
• SRIOV
• Dpdk
• Hugepages
Certification -> Dovetail
Dovetail● Answer to question: “Do these products conform to the interfaces,
behaviors and functionalities defined by OPNFV?”
● Main objective: reduce VNF onboarding cost
● Do not implement test cases in their own code base but reuse the test cases from the OPNFV test projects (Functest, Yardstick etc..)
● Allow vendors to use the OPNFV verified logo
● PTL: Georg Kunz, Ericsson
Dovetail – Test scopeWhole OPNFV testsuite
OPNFV Verified 2018.01
Mandatory test cases● OpenStack API tests (205 tests)● Layer 2 packet forwarding (2 tests)● OpenStack control service HA (8 tests)
Optional test cases● IPv6 tenant networks (25 tests)● BGPVPNs (4 tests)● Fundamental VIM capabilities (30 tests)
Future test case candidates● Neutron Trunk ports ● BGPVPN Tempest tests● SDN Controller HA● VNF lifecycle management● ….
Future ..
Certification workflow
1. Submission of participation form
2. Testing
3. Submission of results
4. Notification of reviewers
5. Review of test results
6. Grant of use of program marks
Reuse those tools
Kontron communications unit (Canada): part ODM, part integrator
SYMKLOUD hw and solution portfolio: integrate open source with high density converged platforms + cloud servers + white box switchesUsed for: NFVi at Edge + Core, SDN, DC clouds for the telco and video/broadcast, and cloud providers We make it easy: removed complexities to integrate software (CI/CD open source or not) on Symkloud hw; plus can ‘mix-n-match’ composable modules (GPU, FPGA, Crypto, Storage).
Ramp up services faster and with lower risk: easy to Configure, Deploy and Manage services on our platforms (automated switch configuration and deployment, integrate addons,etc.)
Context
Problematics
How to test/validate that our deployment of Openstack or Kubernetes on SymKloud is fine ?
● Different SymKloud Platforms○ Different type of switches (40Gb, 100Gb/SDN-ready etc..)○ Different CPU (12, 16 .. cores)○ Servers with/without GPU or FPGA etc…
● Differents software type and version(OpenStack Newton/Queens, Kubernetes 1.9/1.10 etc..)
● Differents add-ons (6wind network acceleration for OpenStack, OpenStack Ironic support etc..)
More features can be added in the future (New deployer, SDN controller, New hardware)
Current CI
ScenarioHA/noHA, ironic,
6wind etc..
Cloud softwareOpenStack, Kubernetes
Hardware
VersionArnold, Britney, Develop etc..
DeployMAAS/Juju
ReportTestrail
TestHomemade
test suite using Behave
Experimental CI
ScenarioHA/noHA, ironic,
6wind etc..
Cloud softwareOpenStack, Kubernetes
Hardware
VersionArnold, Britney, Develop etc..
DeployMAAS/Juju
ReportTest api, reportingTestrail
Test
Test result
TestHomemade
test suiteFunctest,Yardstick
KontronOPNFV tools
Reporting
Our goals
Why reuse those tools in addition of our internal QA test ?• Our QA test suite has a more diverse test coverage than the OPNFV
tools (not only focusing on the NFVI+VIM test but also installer, add-ons, configuration and hardware test)
• The OPNFV test suite has more tests related to NFV so it completes our test coverage
• The goal is to be better and close to the customer target usage• We ensure that next Dovetail verification will work if all opnfv tests pass
on our platform.
The Orange CI/CD view (Preproduction)
CI/CD viewn ONAP OpenLABBare Metal O/S management
Infrastructure Installation
ONAP Installation
Openstack BIFROST / OPNFV XCI
Kolla Ansible
ONAP Heat
Kubespray
Functest Xtesting(Healthchecks/moke/vnf //
smoke/conformance)
ONAP OOM
Functest Xtesting/onap-robot
Run daily
Run daily
Test DB
Test
AP
I
ONAP Heat (Community lab)
OPNFV Orange Integration Center
Installers 5: Apex, Compass, Daisy, Fuel, Joid Preproduction: 1 (RedHat Director)Orange ONAP OpenLab: 1 Kolla Ansible
Scenarios Mores than 30 Preproduction: 1 OpenStack + ContrailOrange ONAP OpenLab: 2 OpenStack / Kubernetes + ONAP
Tests Tests (automation): relying on several test projects:
● Functest* for functional testing● Yardstick* for performance testing● Qtip for bench HW● Storeperf for ceph● Vsperf for Vswitch qualification● Bottleneck to optimize Yardstick runs
* used for release validation
Preproduction: Functest + YardstickOrange ONAP OpenLab: Functest + ONAP Robot test suites
Reuse of OPNFV tooling for ONAP
Reuse of Functest Xtesting framework to onboard ONAP robot tests• Only 3 files needed (Robot natively supported)• Docker size from 1 Go to 100Mo (Alpines)• Inherit Test reporting capabilities• Consistent view to manage Infra and orchestration testing (reuse
DB, Test API)
Reuse of OPNFV tooling for ONAP
● Test framework● Test API/Test DB● CI/CD methodology
Conclusions
• OPNFV testing tool is useful, evolutive, promoting Upstream First (Openstack/ODL testing projects), it can easily complete existing vendor suites with a good focus on NFV
• Automation is key: Xtesting framework is very well adapted to this context (light (Alpines), simple, modular)
• Tooling is easily reusable out of OPNFV context (ONAP, Openstack/Kubernetes - non OPNFV - solutions) including frameworks, DB, test API
• OPNFV release/Dovetail criteria should be stronger as test frameworks offer more than what is used as release criteria: infrastructure trustability is key for Telco Cloud/NFV adoption
Thank You!
How does Functest work?
Yardstick Architecture
• User Interface
• Main Framework
• Context
• Scenario
• Runner
• Dispatcher
• Plugin
• Container Manager
• Services
OPNFV Test result API
http://artifacts.opnfv.org/releng/docs/testapi.html
case_name: "cloudify_ims",pod_name: "huawei-pod2",version: "fraser",installer: "compass",scenario: "os-odl_l3-nofeature-ha",start_date: "2018-04-18 04:58:10",stop_date: "2018-04-18 05:38:50",criteria: “PASS” ,build_tag: "jenkins-functest-baremetal-22",details: { }
name: "cloudify",version: "4.0",status: "PASS",duration: 349
name: "clearwater"descriptor_version: "122",version: "107",status: "PASS",duration: 1330
name: "clearwater-live-test”,version: "1.0",status: "PASS",duration: 273,result: {failures: 0, skipped: 30, passed: 46}
orchestrator
VNF
VNF Test
Result of each run in OPNFV test API
OPNFV Reporting
http://testresults.opnfv.org/reporting/
Trend, score, iteration by installer, version, scenario
OPNFV Reporting
http://testresults.opnfv.org/reporting/
Last test result by installer, version, scenario