using the mu test suite, the vendor’s ip service testing
TRANSCRIPT
“Using the Mu solution for a combination of functional, interoperability, security and resilience testing enables us to provide our customers with the industry’s highest quality IP services.”Tier 1 Network Operator
“Mu helps our development, QA and support teams throughout Juniper’s software development lifecycle and ultimately protects our customer’s revenue streams by ensuring Juniper products deliver a reliable and secure IP service foundation.”Avi Avivi, Senior Director of High End Security Systems, Juniper Networks
“We’ve been a Mu user for several years and have increasingly found the Mu solution to be a key part of our ability to uncover issues that no other tool provides our QA team.”Tushar Patel, VP SW & QA, Force10 Networks
“Mu’s approach delivers measurable ROI - a 10x or more improvement - over using existing test solutions.”Peter Fetterolf, Principal Analyst, NSP Partners
“Mu is continuing to execute with customers in delivering integrated IP service testing on a single platform that accurately reflects the logic of the service, drives multiple data sets through the services, and executes multiple test types all at the same time.”Olga Yashkova, Research Analyst, Test & Measurement, Frost & Sullivan
Copyright © 2009 Mu Dynamics. All rights reserved. Mu Dynamics, Mu Test Suite, Mu-4000, Mu-8000 and the Mu Dynamics logo are trademarks of Mu Dynamics. SB0900110
web: www.mudynamics.com | email: [email protected]: 686 W. Maude Ave., Suite 104, Sunnyvale, CA 94085, USAphone: (866) 276-4640 or (408) 329-6330 | fax: (408) 329-6317
CASE STUDY | LEADING NETWORK INFRASTRUCTURE VENDOR
A leading network infrastructure vendor adopted the Mu Test Suite, and achieved MORE THAN 10X improvement in test effectiveness in less than 6 months. The business benefits included faster product deployment, greater customer satisfaction due to increased product quality levels and lower testing costs.
Using the Mu Test Suite, the vendor’s IP Service testing was:• Better: Improved Test Coverage and Accuracy• Faster: Reduced Time to Market• Less Expensive: Lowered Testing Costs
Before Mu:Before using the Mu, the average test cycle for this leading vendor ran more than 12 months. The vendor realized that too much time was spent on manual testing efforts around the custom, dynamic protocol components of the service - off the shelf test tools were unavailable. Also, both development and test teams spent a lot of time fighting field fires on quality-related issues. In particular, large quantities of time was wasted simply reproducing the exact problem rather than implementing and testing a fix.
In the face of tightening engineering budgets from both an Opex and a Capex Perspective, this vendor needed a change.
After Mu:Using the Mu Test Suite, this vendor rapidly generates an extensive set of Functional tests tailor-made for its customers’ unique IP Products and Services. The vendor leverages Mu's sharable and reusable XML-based test assets, and has built a comprehensive library of Functional, Interoperability, Resilience, and Security test suites.
With Mu, the vendor efficiently replicates and isolates issues using packet captures from the field, and has achieved significant savings in customer support. Thanks to the higher quality, there are far fewer critical customer issues in the first year of new product deployments. Mu's unified approach to testing has helped this vendor view all the testing activities from a holistic perspective, rather than as a set of discrete activities – a welcome first. Using Mu's single platform throughout its software development lifecycle (SDLC), the vendor's average test time and test costs are reduced more than 50%, while the test coverage has improved 100%. The overall improvement in test effectiveness is more than 10x!
Average test time of new IP product or service
Approximate cost of testing (opex and capex)one major release
Test coverage on first release
Number of critical issues within the first year
Time spent in solving field issues
$4M (assuming team of 20+and costs of tools, labs)
> 12 months
> 2 man-weeks per issue
< 40%
> 25
BEFORE MU
> 80%
< 1 month
< 2 days per issue
< 10
< $250K
AFTER MU
SECURITY
FUNCTIONAL
INTE
ROPE
RABIL
ITY
RESI
LIEN
CE
About Mu DynamicsMu Dynamics proactively eliminates costly service and product weaknesses by testing for the unexpected. Mu’s solution automates a systematic and repeatable process that identifies hard-to-detect sources of potential downtime within IP services and underlying networks. The award-winning Mu solution is deployed at more than 100 locations, primarily at leading global service providers, government agencies, and network product vendors. Headquartered in Sunnyvale, Calif., Mu is backed by leading venture capital firms that include Accel Partners, Benchmark Capital, DAG Ventures and Focus Ventures.
10X+ IMPROVEMENT
NETWORK INFRASTRUCTURE
VPN IPv6 Carrier EthernetBackhaul
CHALLENGE
Product Vendors and Service Providers are struggling to deliver IP Services to the market
with high quality, on schedule and within budget. Current approaches to testing IP
Services and products either rely on manually written test scripts or on commercially
available protocol testing tools. These approaches are fundamentally broken because IP
Services are:
1. Dynamic: IP Services are built on constantly evolving standards and custom extensions.
2. Complex: Every function of an IP Service involves complex interactions with multiple
building blocks and protocols.
3. Unique: Every deployment scenario has unique components and configurations.
WHAT’S NEEDED
The industry needs a new approach to testing that recognizes the above aspects of IP
Services, and is designed with the following capabilities in mind:
1. Accurately modeling the logic of a complex service
2. Quickly driving multiple datasets through the service to exhaustively test it
3. Systematically and consistently testing multiple aspects of the service
SOLUTION
For the first time ever, the Mu Test Suite from Mu Dynamics enables engineering organizations to overcome the challenge of
effectively testing IP Services. Mu's solution enables customers to use their actual service traffic as the basis for IP service
testing to:
• Model the actual service for the purpose of testing. This enables service flow testing, not just individual components.
• Use the service traffic as the basis for the automated test creation. This dramatically accelerates time to test and ensures
that the test cases are designed to test the customer's own unique service implementation vs. a generic standard.
This solution can be leveraged across multiple testing disciplines such as:
• Functional Testing. Development and Test teams quickly create tests that check if individual features and functionality of the
service behave as expected.
• Interoperability Testing. Test and Customer Support teams effectively test for interoperability against other vendor systems
or run regression tests against older versions of their own software.
• Resilience Testing. Test teams rapidly generate thousands of tests to measure how the service behaves from a resilience
perspective. The Mu Test Suite delivers carefully chosen fuzzing tests designed to expose unknown weaknesses in an IP
Service.
• Security Testing. Test teams perform comprehensive tests for exploitable software weaknesses, published or known security
attacks, and Denial of Service (DoS) attacks.
IP SERVICES ARE MAINSTREAM
IP Service offerings are woven into the fabric of our business and social lives. We rely on these innovative services for business productivity, communication, entertainment, education and national security.
Almost all new applications delivered today by service providers are built on IP Services. Examples include wired and wireless services, network infrastructure, smart grid, critical infrastructure, cloud computing and enterprise services.
Behind the scenes, IP Services are characterized by enormous complexity, constant change, and custom building blocks. Each IP Service has its own unique configuration which makes the testing of them an unprecedented challenge.
CLOUD AND ENTERPRISE SERVICES SMART GRID AND CRITICAL INFRASTRUCTUREWIRED AND MOBILE SERVICES
LTE IPTV VoIP IMS IPv6 Auto DR SCADAStorage SaaS IPv6
IPSERVICES
SOFTWAREVENDOR/
EQUIPMENTMAKER
SERVICEPROVIDER/NETWORKOPERATOR
IP SERVICEEND USER
FIELD ISSUES
NEW DEVICES, UPGRADES &
PATCHES
FIELD ISSUES
NEW SERVICES, UPGRADES &
PATCHES
NETWORK INFRASTRUCTURE
VPN IPv6 Carrier EthernetBackhaul
CHALLENGE
Product Vendors and Service Providers are struggling to deliver IP Services to the market
with high quality, on schedule and within budget. Current approaches to testing IP
Services and products either rely on manually written test scripts or on commercially
available protocol testing tools. These approaches are fundamentally broken because IP
Services are:
1. Dynamic: IP Services are built on constantly evolving standards and custom extensions.
2. Complex: Every function of an IP Service involves complex interactions with multiple
building blocks and protocols.
3. Unique: Every deployment scenario has unique components and configurations.
WHAT’S NEEDED
The industry needs a new approach to testing that recognizes the above aspects of IP
Services, and is designed with the following capabilities in mind:
1. Accurately modeling the logic of a complex service
2. Quickly driving multiple datasets through the service to exhaustively test it
3. Systematically and consistently testing multiple aspects of the service
SOLUTION
For the first time ever, the Mu Test Suite from Mu Dynamics enables engineering organizations to overcome the challenge of
effectively testing IP Services. Mu's solution enables customers to use their actual service traffic as the basis for IP service
testing to:
• Model the actual service for the purpose of testing. This enables service flow testing, not just individual components.
• Use the service traffic as the basis for the automated test creation. This dramatically accelerates time to test and ensures
that the test cases are designed to test the customer's own unique service implementation vs. a generic standard.
This solution can be leveraged across multiple testing disciplines such as:
• Functional Testing. Development and Test teams quickly create tests that check if individual features and functionality of the
service behave as expected.
• Interoperability Testing. Test and Customer Support teams effectively test for interoperability against other vendor systems
or run regression tests against older versions of their own software.
• Resilience Testing. Test teams rapidly generate thousands of tests to measure how the service behaves from a resilience
perspective. The Mu Test Suite delivers carefully chosen fuzzing tests designed to expose unknown weaknesses in an IP
Service.
• Security Testing. Test teams perform comprehensive tests for exploitable software weaknesses, published or known security
attacks, and Denial of Service (DoS) attacks.
IP SERVICES ARE MAINSTREAM
IP Service offerings are woven into the fabric of our business and social lives. We rely on these innovative services for business productivity, communication, entertainment, education and national security.
Almost all new applications delivered today by service providers are built on IP Services. Examples include wired and wireless services, network infrastructure, smart grid, critical infrastructure, cloud computing and enterprise services.
Behind the scenes, IP Services are characterized by enormous complexity, constant change, and custom building blocks. Each IP Service has its own unique configuration which makes the testing of them an unprecedented challenge.
CLOUD AND ENTERPRISE SERVICES SMART GRID AND CRITICAL INFRASTRUCTUREWIRED AND MOBILE SERVICES
LTE IPTV VoIP IMS IPv6 Auto DR SCADAStorage SaaS IPv6
IPSERVICES
SOFTWAREVENDOR/
EQUIPMENTMAKER
SERVICEPROVIDER/NETWORKOPERATOR
IP SERVICEEND USER
FIELD ISSUES
NEW DEVICES, UPGRADES &
PATCHES
FIELD ISSUES
NEW SERVICES, UPGRADES &
PATCHES
“Using the Mu solution for a combination of functional, interoperability, security and resilience testing enables us to provide our customers with the industry’s highest quality IP services.”Tier 1 Network Operator
“Mu helps our development, QA and support teams throughout Juniper’s software development lifecycle and ultimately protects our customer’s revenue streams by ensuring Juniper products deliver a reliable and secure IP service foundation.”Avi Avivi, Senior Director of High End Security Systems, Juniper Networks
“We’ve been a Mu user for several years and have increasingly found the Mu solution to be a key part of our ability to uncover issues that no other tool provides our QA team.”Tushar Patel, VP SW & QA, Force10 Networks
“Mu’s approach delivers measurable ROI - a 10x or more improvement - over using existing test solutions.”Peter Fetterolf, Principal Analyst, NSP Partners
“Mu is continuing to execute with customers in delivering integrated IP service testing on a single platform that accurately reflects the logic of the service, drives multiple data sets through the services, and executes multiple test types all at the same time.”Olga Yashkova, Research Analyst, Test & Measurement, Frost & Sullivan
Copyright © 2009 Mu Dynamics. All rights reserved. Mu Dynamics, Mu Test Suite, Mu-4000, Mu-8000 and the Mu Dynamics logo are trademarks of Mu Dynamics. SB0900110
web: www.mudynamics.com | email: [email protected]: 686 W. Maude Ave., Suite 104, Sunnyvale, CA 94085, USAphone: (866) 276-4640 or (408) 329-6330 | fax: (408) 329-6317
CASE STUDY | LEADING NETWORK INFRASTRUCTURE VENDOR
A leading network infrastructure vendor adopted the Mu Test Suite, and achieved MORE THAN 10X improvement in test effectiveness in less than 6 months. The business benefits included faster product deployment, greater customer satisfaction due to increased product quality levels and lower testing costs.
Using the Mu Test Suite, the vendor’s IP Service testing was:• Better: Improved Test Coverage and Accuracy• Faster: Reduced Time to Market• Less Expensive: Lowered Testing Costs
Before Mu:Before using the Mu, the average test cycle for this leading vendor ran more than 12 months. The vendor realized that too much time was spent on manual testing efforts around the custom, dynamic protocol components of the service - off the shelf test tools were unavailable. Also, both development and test teams spent a lot of time fighting field fires on quality-related issues. In particular, large quantities of time was wasted simply reproducing the exact problem rather than implementing and testing a fix.
In the face of tightening engineering budgets from both an Opex and a Capex Perspective, this vendor needed a change.
After Mu:Using the Mu Test Suite, this vendor rapidly generates an extensive set of Functional tests tailor-made for its customers’ unique IP Products and Services. The vendor leverages Mu's sharable and reusable XML-based test assets, and has built a comprehensive library of Functional, Interoperability, Resilience, and Security test suites.
With Mu, the vendor efficiently replicates and isolates issues using packet captures from the field, and has achieved significant savings in customer support. Thanks to the higher quality, there are far fewer critical customer issues in the first year of new product deployments. Mu's unified approach to testing has helped this vendor view all the testing activities from a holistic perspective, rather than as a set of discrete activities – a welcome first. Using Mu's single platform throughout its software development lifecycle (SDLC), the vendor's average test time and test costs are reduced more than 50%, while the test coverage has improved 100%. The overall improvement in test effectiveness is more than 10x!
Average test time of new IP product or service
Approximate cost of testing (opex and capex)one major release
Test coverage on first release
Number of critical issues within the first year
Time spent in solving field issues
$4M (assuming team of 20+and costs of tools, labs)
> 12 months
> 2 man-weeks per issue
< 40%
> 25
BEFORE MU
> 80%
< 1 month
< 2 days per issue
< 10
< $250K
AFTER MU
SECURITY
FUNCTIONAL
INTE
ROPE
RABIL
ITY
RESI
LIEN
CE
About Mu DynamicsMu Dynamics proactively eliminates costly service and product weaknesses by testing for the unexpected. Mu’s solution automates a systematic and repeatable process that identifies hard-to-detect sources of potential downtime within IP services and underlying networks. The award-winning Mu solution is deployed at more than 100 locations, primarily at leading global service providers, government agencies, and network product vendors. Headquartered in Sunnyvale, Calif., Mu is backed by leading venture capital firms that include Accel Partners, Benchmark Capital, DAG Ventures and Focus Ventures.
10X+ IMPROVEMENT