using the mu test suite, the vendor’s ip service testing

4
SECURITY FUNCTIONAL INTEROPERABILITY RESILIENCE

Upload: others

Post on 17-Mar-2022

2 views

Category:

Documents


0 download

TRANSCRIPT

“Using the Mu solution for a combination of functional, interoperability, security and resilience testing enables us to provide our customers with the industry’s highest quality IP services.”Tier 1 Network Operator

“Mu helps our development, QA and support teams throughout Juniper’s software development lifecycle and ultimately protects our customer’s revenue streams by ensuring Juniper products deliver a reliable and secure IP service foundation.”Avi Avivi, Senior Director of High End Security Systems, Juniper Networks

“We’ve been a Mu user for several years and have increasingly found the Mu solution to be a key part of our ability to uncover issues that no other tool provides our QA team.”Tushar Patel, VP SW & QA, Force10 Networks

“Mu’s approach delivers measurable ROI - a 10x or more improvement - over using existing test solutions.”Peter Fetterolf, Principal Analyst, NSP Partners

“Mu is continuing to execute with customers in delivering integrated IP service testing on a single platform that accurately reflects the logic of the service, drives multiple data sets through the services, and executes multiple test types all at the same time.”Olga Yashkova, Research Analyst, Test & Measurement, Frost & Sullivan

Copyright © 2009 Mu Dynamics. All rights reserved. Mu Dynamics, Mu Test Suite, Mu-4000, Mu-8000 and the Mu Dynamics logo are trademarks of Mu Dynamics. SB0900110

web: www.mudynamics.com | email: [email protected]: 686 W. Maude Ave., Suite 104, Sunnyvale, CA 94085, USAphone: (866) 276-4640 or (408) 329-6330 | fax: (408) 329-6317

CASE STUDY | LEADING NETWORK INFRASTRUCTURE VENDOR

A leading network infrastructure vendor adopted the Mu Test Suite, and achieved MORE THAN 10X improvement in test effectiveness in less than 6 months. The business benefits included faster product deployment, greater customer satisfaction due to increased product quality levels and lower testing costs.

Using the Mu Test Suite, the vendor’s IP Service testing was:• Better: Improved Test Coverage and Accuracy• Faster: Reduced Time to Market• Less Expensive: Lowered Testing Costs

Before Mu:Before using the Mu, the average test cycle for this leading vendor ran more than 12 months. The vendor realized that too much time was spent on manual testing efforts around the custom, dynamic protocol components of the service - off the shelf test tools were unavailable. Also, both development and test teams spent a lot of time fighting field fires on quality-related issues. In particular, large quantities of time was wasted simply reproducing the exact problem rather than implementing and testing a fix.

In the face of tightening engineering budgets from both an Opex and a Capex Perspective, this vendor needed a change.

After Mu:Using the Mu Test Suite, this vendor rapidly generates an extensive set of Functional tests tailor-made for its customers’ unique IP Products and Services. The vendor leverages Mu's sharable and reusable XML-based test assets, and has built a comprehensive library of Functional, Interoperability, Resilience, and Security test suites.

With Mu, the vendor efficiently replicates and isolates issues using packet captures from the field, and has achieved significant savings in customer support. Thanks to the higher quality, there are far fewer critical customer issues in the first year of new product deployments. Mu's unified approach to testing has helped this vendor view all the testing activities from a holistic perspective, rather than as a set of discrete activities – a welcome first. Using Mu's single platform throughout its software development lifecycle (SDLC), the vendor's average test time and test costs are reduced more than 50%, while the test coverage has improved 100%. The overall improvement in test effectiveness is more than 10x!

Average test time of new IP product or service

Approximate cost of testing (opex and capex)one major release

Test coverage on first release

Number of critical issues within the first year

Time spent in solving field issues

$4M (assuming team of 20+and costs of tools, labs)

> 12 months

> 2 man-weeks per issue

< 40%

> 25

BEFORE MU

> 80%

< 1 month

< 2 days per issue

< 10

< $250K

AFTER MU

SECURITY

FUNCTIONAL

INTE

ROPE

RABIL

ITY

RESI

LIEN

CE

About Mu DynamicsMu Dynamics proactively eliminates costly service and product weaknesses by testing for the unexpected. Mu’s solution automates a systematic and repeatable process that identifies hard-to-detect sources of potential downtime within IP services and underlying networks. The award-winning Mu solution is deployed at more than 100 locations, primarily at leading global service providers, government agencies, and network product vendors. Headquartered in Sunnyvale, Calif., Mu is backed by leading venture capital firms that include Accel Partners, Benchmark Capital, DAG Ventures and Focus Ventures.

10X+ IMPROVEMENT

NETWORK INFRASTRUCTURE

VPN IPv6 Carrier EthernetBackhaul

CHALLENGE

Product Vendors and Service Providers are struggling to deliver IP Services to the market

with high quality, on schedule and within budget. Current approaches to testing IP

Services and products either rely on manually written test scripts or on commercially

available protocol testing tools. These approaches are fundamentally broken because IP

Services are:

1. Dynamic: IP Services are built on constantly evolving standards and custom extensions.

2. Complex: Every function of an IP Service involves complex interactions with multiple

building blocks and protocols.

3. Unique: Every deployment scenario has unique components and configurations.

WHAT’S NEEDED

The industry needs a new approach to testing that recognizes the above aspects of IP

Services, and is designed with the following capabilities in mind:

1. Accurately modeling the logic of a complex service

2. Quickly driving multiple datasets through the service to exhaustively test it

3. Systematically and consistently testing multiple aspects of the service

SOLUTION

For the first time ever, the Mu Test Suite from Mu Dynamics enables engineering organizations to overcome the challenge of

effectively testing IP Services. Mu's solution enables customers to use their actual service traffic as the basis for IP service

testing to:

• Model the actual service for the purpose of testing. This enables service flow testing, not just individual components.

• Use the service traffic as the basis for the automated test creation. This dramatically accelerates time to test and ensures

that the test cases are designed to test the customer's own unique service implementation vs. a generic standard.

This solution can be leveraged across multiple testing disciplines such as:

• Functional Testing. Development and Test teams quickly create tests that check if individual features and functionality of the

service behave as expected.

• Interoperability Testing. Test and Customer Support teams effectively test for interoperability against other vendor systems

or run regression tests against older versions of their own software.

• Resilience Testing. Test teams rapidly generate thousands of tests to measure how the service behaves from a resilience

perspective. The Mu Test Suite delivers carefully chosen fuzzing tests designed to expose unknown weaknesses in an IP

Service.

• Security Testing. Test teams perform comprehensive tests for exploitable software weaknesses, published or known security

attacks, and Denial of Service (DoS) attacks.

IP SERVICES ARE MAINSTREAM

IP Service offerings are woven into the fabric of our business and social lives. We rely on these innovative services for business productivity, communication, entertainment, education and national security.

Almost all new applications delivered today by service providers are built on IP Services. Examples include wired and wireless services, network infrastructure, smart grid, critical infrastructure, cloud computing and enterprise services.

Behind the scenes, IP Services are characterized by enormous complexity, constant change, and custom building blocks. Each IP Service has its own unique configuration which makes the testing of them an unprecedented challenge.

CLOUD AND ENTERPRISE SERVICES SMART GRID AND CRITICAL INFRASTRUCTUREWIRED AND MOBILE SERVICES

LTE IPTV VoIP IMS IPv6 Auto DR SCADAStorage SaaS IPv6

IPSERVICES

SOFTWAREVENDOR/

EQUIPMENTMAKER

SERVICEPROVIDER/NETWORKOPERATOR

IP SERVICEEND USER

FIELD ISSUES

NEW DEVICES, UPGRADES &

PATCHES

FIELD ISSUES

NEW SERVICES, UPGRADES &

PATCHES

NETWORK INFRASTRUCTURE

VPN IPv6 Carrier EthernetBackhaul

CHALLENGE

Product Vendors and Service Providers are struggling to deliver IP Services to the market

with high quality, on schedule and within budget. Current approaches to testing IP

Services and products either rely on manually written test scripts or on commercially

available protocol testing tools. These approaches are fundamentally broken because IP

Services are:

1. Dynamic: IP Services are built on constantly evolving standards and custom extensions.

2. Complex: Every function of an IP Service involves complex interactions with multiple

building blocks and protocols.

3. Unique: Every deployment scenario has unique components and configurations.

WHAT’S NEEDED

The industry needs a new approach to testing that recognizes the above aspects of IP

Services, and is designed with the following capabilities in mind:

1. Accurately modeling the logic of a complex service

2. Quickly driving multiple datasets through the service to exhaustively test it

3. Systematically and consistently testing multiple aspects of the service

SOLUTION

For the first time ever, the Mu Test Suite from Mu Dynamics enables engineering organizations to overcome the challenge of

effectively testing IP Services. Mu's solution enables customers to use their actual service traffic as the basis for IP service

testing to:

• Model the actual service for the purpose of testing. This enables service flow testing, not just individual components.

• Use the service traffic as the basis for the automated test creation. This dramatically accelerates time to test and ensures

that the test cases are designed to test the customer's own unique service implementation vs. a generic standard.

This solution can be leveraged across multiple testing disciplines such as:

• Functional Testing. Development and Test teams quickly create tests that check if individual features and functionality of the

service behave as expected.

• Interoperability Testing. Test and Customer Support teams effectively test for interoperability against other vendor systems

or run regression tests against older versions of their own software.

• Resilience Testing. Test teams rapidly generate thousands of tests to measure how the service behaves from a resilience

perspective. The Mu Test Suite delivers carefully chosen fuzzing tests designed to expose unknown weaknesses in an IP

Service.

• Security Testing. Test teams perform comprehensive tests for exploitable software weaknesses, published or known security

attacks, and Denial of Service (DoS) attacks.

IP SERVICES ARE MAINSTREAM

IP Service offerings are woven into the fabric of our business and social lives. We rely on these innovative services for business productivity, communication, entertainment, education and national security.

Almost all new applications delivered today by service providers are built on IP Services. Examples include wired and wireless services, network infrastructure, smart grid, critical infrastructure, cloud computing and enterprise services.

Behind the scenes, IP Services are characterized by enormous complexity, constant change, and custom building blocks. Each IP Service has its own unique configuration which makes the testing of them an unprecedented challenge.

CLOUD AND ENTERPRISE SERVICES SMART GRID AND CRITICAL INFRASTRUCTUREWIRED AND MOBILE SERVICES

LTE IPTV VoIP IMS IPv6 Auto DR SCADAStorage SaaS IPv6

IPSERVICES

SOFTWAREVENDOR/

EQUIPMENTMAKER

SERVICEPROVIDER/NETWORKOPERATOR

IP SERVICEEND USER

FIELD ISSUES

NEW DEVICES, UPGRADES &

PATCHES

FIELD ISSUES

NEW SERVICES, UPGRADES &

PATCHES

“Using the Mu solution for a combination of functional, interoperability, security and resilience testing enables us to provide our customers with the industry’s highest quality IP services.”Tier 1 Network Operator

“Mu helps our development, QA and support teams throughout Juniper’s software development lifecycle and ultimately protects our customer’s revenue streams by ensuring Juniper products deliver a reliable and secure IP service foundation.”Avi Avivi, Senior Director of High End Security Systems, Juniper Networks

“We’ve been a Mu user for several years and have increasingly found the Mu solution to be a key part of our ability to uncover issues that no other tool provides our QA team.”Tushar Patel, VP SW & QA, Force10 Networks

“Mu’s approach delivers measurable ROI - a 10x or more improvement - over using existing test solutions.”Peter Fetterolf, Principal Analyst, NSP Partners

“Mu is continuing to execute with customers in delivering integrated IP service testing on a single platform that accurately reflects the logic of the service, drives multiple data sets through the services, and executes multiple test types all at the same time.”Olga Yashkova, Research Analyst, Test & Measurement, Frost & Sullivan

Copyright © 2009 Mu Dynamics. All rights reserved. Mu Dynamics, Mu Test Suite, Mu-4000, Mu-8000 and the Mu Dynamics logo are trademarks of Mu Dynamics. SB0900110

web: www.mudynamics.com | email: [email protected]: 686 W. Maude Ave., Suite 104, Sunnyvale, CA 94085, USAphone: (866) 276-4640 or (408) 329-6330 | fax: (408) 329-6317

CASE STUDY | LEADING NETWORK INFRASTRUCTURE VENDOR

A leading network infrastructure vendor adopted the Mu Test Suite, and achieved MORE THAN 10X improvement in test effectiveness in less than 6 months. The business benefits included faster product deployment, greater customer satisfaction due to increased product quality levels and lower testing costs.

Using the Mu Test Suite, the vendor’s IP Service testing was:• Better: Improved Test Coverage and Accuracy• Faster: Reduced Time to Market• Less Expensive: Lowered Testing Costs

Before Mu:Before using the Mu, the average test cycle for this leading vendor ran more than 12 months. The vendor realized that too much time was spent on manual testing efforts around the custom, dynamic protocol components of the service - off the shelf test tools were unavailable. Also, both development and test teams spent a lot of time fighting field fires on quality-related issues. In particular, large quantities of time was wasted simply reproducing the exact problem rather than implementing and testing a fix.

In the face of tightening engineering budgets from both an Opex and a Capex Perspective, this vendor needed a change.

After Mu:Using the Mu Test Suite, this vendor rapidly generates an extensive set of Functional tests tailor-made for its customers’ unique IP Products and Services. The vendor leverages Mu's sharable and reusable XML-based test assets, and has built a comprehensive library of Functional, Interoperability, Resilience, and Security test suites.

With Mu, the vendor efficiently replicates and isolates issues using packet captures from the field, and has achieved significant savings in customer support. Thanks to the higher quality, there are far fewer critical customer issues in the first year of new product deployments. Mu's unified approach to testing has helped this vendor view all the testing activities from a holistic perspective, rather than as a set of discrete activities – a welcome first. Using Mu's single platform throughout its software development lifecycle (SDLC), the vendor's average test time and test costs are reduced more than 50%, while the test coverage has improved 100%. The overall improvement in test effectiveness is more than 10x!

Average test time of new IP product or service

Approximate cost of testing (opex and capex)one major release

Test coverage on first release

Number of critical issues within the first year

Time spent in solving field issues

$4M (assuming team of 20+and costs of tools, labs)

> 12 months

> 2 man-weeks per issue

< 40%

> 25

BEFORE MU

> 80%

< 1 month

< 2 days per issue

< 10

< $250K

AFTER MU

SECURITY

FUNCTIONAL

INTE

ROPE

RABIL

ITY

RESI

LIEN

CE

About Mu DynamicsMu Dynamics proactively eliminates costly service and product weaknesses by testing for the unexpected. Mu’s solution automates a systematic and repeatable process that identifies hard-to-detect sources of potential downtime within IP services and underlying networks. The award-winning Mu solution is deployed at more than 100 locations, primarily at leading global service providers, government agencies, and network product vendors. Headquartered in Sunnyvale, Calif., Mu is backed by leading venture capital firms that include Accel Partners, Benchmark Capital, DAG Ventures and Focus Ventures.

10X+ IMPROVEMENT