June 20–23, 2016 | Berlin, Germany

Carrier Grade Testing Integration

J.Lausuch, Ericsson

M.Richomme ,Orange

Carrier grade?

Carrier grade?

Carrier grade?

Not a single definition

The power of the IT cloud meeting Telco constraintsOr

How to 99.999% with 99.999% subsystems

¿Carrier grade?

A question

of

confidence ?

Robust

Manageable

Scalable

Agile

Legalconstraints

Secure Efficient

Monitoring/SupervisionTroubleshootingBackup/restoreLegacy support

HAGeographic redundancyDowntime / SLA

Resources optimizationIntelligent placement

VNFs easy integrationMulti Mano stacksSoft/hard update

For our customersFor the system

Critical real-time applications QoS control

Legal InterceptionData retentionEmergency services

Carrier grade?

Our first answers

OPNFV Testing community

Sanity testing

Performance testing

Maintainability testing

Reliability testing

System testing

Black box testing

Regression testingIntegration testing

Security testing

Acceptance testing

Unit testing

Load testing

Compatibility testing Recovery testing

Usability testing

Comparison testing

Stress testingEnd-to-end testing

Scalability testing

Compliance testing

Efficiency testing

Functional testing

OPNFV Testing community

Functional testing

Performance testing

Feature testing

OPNFV Testing community

From Telco issue to OPNFV projects

Fault management

Resource management

Smooth Upgrade

Multisite

HA

Doctor

Promise

Escalator

Multisite

HA

Focus on feature projects

Open vSwitch

SFC

BGPVPN

Copper…

Focus on feature projects

Problem detected

OPNFV Feature Project created

Dev in Upstream project 1

Dev in Upstream project N

Feature integrated in

OPNFV scenario(s)

Feature tested in CI

Feature integrated in

OPNFV releaseInternal devFeature tests dev

Focus on CI

automation, tooling and software infrastructure support for OPNFV projects

Why?

What for?

What? software development infrastructure

repeatability and stability control

Focus on Infrastructure

Define the min. HW requirements

to deploy and test OPNFV.

Available to the community for

development and automation.

Used to validate OPNFV releases.

The challenges

Beyond CI, repeatability

Automating tests is not enough

Understanding behavior changes is mandatory to

Save time (not run systematically all the tests)

Adapt the test strategy according to the environment

More features, more scenarios….

Problems:More Features to be tested…

More Scenarios to be tested…

Days are limited to 24h…

Hardware resources for tests may increase but not exponentially

Solutions:Reduce the number of tests

Select/promote Telco grade scenarios

Telco testing means:

repeat the same test again and again.

stress the system over days/weeks.

KPI compliance.

Reliability

Stability

Performant

Here comes MANO…

NFVI and VIM Telco grade foundations are still questionable…still lots to do

MANO game is very open (descriptors, pragmatic versus formal approach,..)

Lots of Open Source MANO initiatives• More tests, more combinations, …

• Time needed before possible momentum

What about O&M?

• O&M tests not addressed yet in OPNFV

• Several possible options (nagios, centreon, zabix, skydive,…)

• Shall we test the integration of such tools as part of OPNFV

Conclusions

Thank you

Credits

• Earthquake by Irfan Syler, FreeImages.com Content License

• FIREMEN AT WORK II by Ossian Engmark, FreeImages.com Content License