outsourced product development: product qa services
DESCRIPTION
TRANSCRIPT
Overview of Product QA
Product QA: Why it is tough
• Product is more than software– Software application– Platform and variations– Documentation: online help, manuals,
tech briefs, examples, training– Add-ins and third party product
integrations– Installation experience– Handling upgrades and updates
Product QA: Why it is tough
• Quality is more than lack of bugs– Does it solve the problem when it works?– Is it practical to use? Is it designed for
efficient use?– Does it keep working or works only
under ideal conditions?– Is it economical to build, maintain and
respond rapidly to customer/user needs?
• QA is everything you do to minimize risk of failure and promote excellence
• Help producing products that satisfy target customer base
Role of QA in Product Company
Role of QA in Product Company
• Broad functions of QA– Risk management: release planning, release
approval– Bring customer involvement: early release programs– Act as skillful developers: mimic tough customer!– Process definition and improvement– Inspection and testing– Experience-based improvement: contribute to
product definition, many finer points
• Should play an important role throughout The Life of a Product
5 4
3
21
6
Planning Design
Coding
TestingRelease
Post-Release
Process Definition
Process Improvement
QA Planning
Risk Assessment
QA Certification
Release Inspection
Test Development
Test Suite Design
Product DefinitionInputs
Early ReleaseProgram
Test Execution
Code Review
Design Review
Structure and Function of QA
Non-testing Functions
• What QA is required to do• QA procedures, expectations from
other teams• Effort estimation and team
planning• Resource planning, tools and
software procurement
QA Planning
• Design the tests in parallel to software design
• Tests need organization into independent test suites
• Build the tests before you build the software
• If you find a bug, write a test that proves the bug exists, fix the bug and then rerun the test to prove it is fixed
Test Suite Design
• Goal: Clarification and accept/reject design decisions
• Starts with a presentation of the design for the component/module to a selected panel of developers
• Peer review is commonly used, with most relevant engineers from other parts of the product included
• Discussion and coordination of comments incorporation
Test Suite Design Review
• Check-in review done with peers of the dev team– Including new features and bug fixes
• Code walkthrough by dev across streams, including QA– review meeting with prepared participants– Sample source files are selected for review
• Focus is on finding defects, before they slip through to testing phases– Lower costs by catching them early
Check-in Review, Walkthrough
• Need to ensure smooth porting of current customers
• Test ISV applications available internally
• Public program to increase the test base– Alpha and Beta testing
Early Release Programs
• QA evaluates quality, independent of developers– Developers should not determine how
much quality is sufficient
• QA acts as internal customers and verifies release readiness
Release Readiness
Product Testing
Based on Class Description
How a test is written
Black box Test does not use knowledge of product code and design
White box Test uses product internals, even calls unexposed functions
When a test is being run
Sanity testing Basic set of tests run after every nightly build, to ensure that nothing is broken
Alpha testing Testing done on code release to QA
Beta testing Testing done with the help of external partners, early release customers, etc.
Final testing Final set of testing done by QA to approve GA release
What purpose the test is serving
Functional Verify that functional requirements of the product are as per specifications
Compliance Test suite for compliance to industry standards
Load Verify performance and scalability of the product; also produce data on installation requirements
Reliability Negative testing to ensure that the product functions gracefully under unexpected conditions
Usability Testing the product for practicality, efficiency of use and ease of learning of the product interface
Our Test Classification
• Key aspects to cover (they are endless!)– Correctness– Interoperability– Compatibility– Compliance– Security
• Think of permutations– Functions vs. input data vs. states– Product vs. platforms– System vs. external factors– Testing vs. versions of product
Functional Testing
• Emphasis on when product is expected to be declared “compliant” with Industry Standard (such as J2EE™)
• Compliance test suite (TCK for J2EE) might be made available
• Compliance to standards does not mean the product is functionally tested for use
Compliance Testing
• Performance testing– Performance that can be achieved on a configuration setup– Deterioration with load– Variations across platforms– Writing a sizing guide with suggested deployment
configurations– Generating a near real-world load
• Scalability testing– How does the system scale with increase in platform capacity– Fail over and load balancing under clustered configurations
• Benchmarking– Comparative performance testing and analysis with competitor
products– Public benchmarking, such as SPECjAppserver™ or ECperf™
Load Testing
Performance Reports
• Maturity: how the product expects erroneous user handling
• Fault-tolerance: Ability to perform under faulty hardware/platform conditions with degraded performance but near full functionality
• Integrity: How the system cannot be broken by malicious users
Reliability Testing Aspects
Reliability Testing Aspects...
• Recoverability: ability of the product to recover to a consistent state when power outage, network fault or other disaster happens
• Safety: protection of user data, backup, logging, etc.
• Understandability: clarity of interface, intuitiveness
• Learnability: ease of learning, matches user’s current understanding of other products
• Consistency: across versions, all across the product
Usability Testing Aspects
Usability Testing Aspects...
• Its not only about GUI, its also about file formats, XML, alternatives such as command line tools
• We normally work with external consultants who are specialized in usability engineering
• Efficiency– Storage– Processing
• Maintainability– Code organization, branching– Modularity for ease of changes, maintenance,
re-factoring
• Portability– Platform support– Reusability of code
Other Aspects of Testing
Some of these aspects may not be important to end-customers, but very important for engineering
• Document the design
• Use internal error checking
• Test units before integrating them
• Tell the tester what’s new or flaky
• Test the build yourself, first
• Evolve product in functional layers
Making Testing Easier
Tools and Infrastructure
• Test automation helps repeatability of testing
• Useful test automation is a major software project
• Engineers are able to detect more problems, more accurately
Testing Automation
Test Case Management System
J2EE™ Compliance Testing
• J2EE CTS Kit– From Sun
• Standards Verification• Automated Harness
• Augment CTS– For Complex scenarios
• Built on Studio Framework– Homegrown
• Ease of writing new tests• Automated Discovery of new tests• Extensive tests metadata
– Compose new plans/suites
• Integrated into the Build System– For automated post-build testing
Test Management Framework
Test Management Framework
Reliability Testing
Distributed Stress Testing System
Thank you.