architecture testing: wrongly ignored!' by peter zimmerer
DESCRIPTION
State-of-the-art testing approaches typically include different testing levels like reviews, unit testing, component testing, integration testing, system testing, and acceptance testing. There is also common sense that typically unit testing is done by developers (they are responsible to check the quality of their units at least to some extent) and system testing is done by professional independent testers. But, who is responsible to adequately test the architecture which is one of the key artifacts in developing and maintaining flexible, powerful, and sustainable products and systems? History has shown that too many project failures and troubles are caused by deficiencies in the architecture.Furthermore, what does the term architecture testing mean and why is this term seldom used? To answer these questions, Peter describes what architecture testing is all about and explains a list of pragmatic practices and experiences to implement it successfully. He offers practical advice on the required tasks and activities as well as the needed involvement, contributions, and responsibilities of software architects in the area of testing – because a close cooperation between testers and architects is the key to drive and sustain a culture of prevention rather than detection across the lifecycle. Finally, if we claim to be in pursuit of quality then adequate architecture testing is not only a lever for success but a necessity. And this results not only in better quality but also speeds up development by facilitating change and decreasing maintenance efforts.TRANSCRIPT
Copyright © Siemens AG 2011. All rights reserved.
Corporate Technology
EuroSTAR 2011Manchester, EnglandNovember 24, 2011
Peter ZimmererPrincipal Engineer
Siemens AGCorporate Technology
81739 Munich, [email protected]
http://www.siemens.com/corporate-technology/
Architecture Testing: Wrongly Ignored!
Contents
Motivation
Architecture testing
Practices and tasks in architecture testing Involvement of Software Architects in testing activities
Summary
Motivation
The system‘s architecture is important …
Vehicle for communication among stakeholders – including testers! Providing system context, scoping, and partitioning of development work Foundation for many lifecycle artifacts – including the test basis! Representing early design decisions and constraints:
most critical to get right, hardest to change First design artifact addressing (enabling or inhibiting) objectives,
non-functional requirements, and quality attributes Key for systematic reuse, integration with legacy / third party software,
managing change, cost-effective maintenance and evolution
Poor architectures are a major cause for project failures and disaster
… so, we should test it thoroughly!
Architecture testing Glossary V2.1, April 2010@Glossary V2.0, Dec 2007
A search in the glossary for “architecture“ results in only one hit:
design-based testing:An approach to testing in which test cases are designed based on the architecture and/or detailed design of a component or system (e.g. tests of interfaces between components or systems).
Architecture testing
Requirements testing
Architecture testing
Document testing
Unit testing
Integration testing
System testing
Acceptance testing
1,010,000
102,000
191,000
4,900,000
1,630,000
2,770,000
1,840,000
Result hits* @ :
@
*Hits counted on September 12, 2011 – ratio still about the same since 2005 …
Testing strategy – Based on risks
Base the testing strategy on business goals and priorities Risk-based testing strategy (RBT)
Risk identificationRisk = P × D P Probability of failure
Frequency of use Chance of failure:
criticality & complexity at implementation & usage, lack of quality D Damage (consequence & cost for business & test & usage)
Risk analysis – Product risk analysis workshop
Risk response – Test objectives, test levels, test design methods …
Product risk analysis for software architects
Identify especially all risks related to the architecture Architectural requirements (functional and non-functional) Architectural quality attributes Architectural use cases, features, and functions Architectural decisions Design decisions Technology selections Third-party component selections (open source) Bug categories
Actively participate in a product risk analysis workshop as one
important stakeholder
Test levels – Example V model
System Requirements
Architecture,Design
UnitSpecification
Coding
AcceptanceTesting
SystemTesting
IntegrationTesting
UnitTesting
User Requirements
based on
Test levels – Example V model with architecture testing
System Requirements
Architecture,Design
UnitSpecification
AcceptanceTesting
SystemTesting
IntegrationTesting
UnitTesting
User Requirements
Code QualityManagement
Architecture interrogation:interviews,interactive workshops
Architecture testingis any testing of architecture and architectural artifacts mapping of architectural risks to test levels
Test case design
Analysis,reviews,previews,inspections
Load modelspecification
based on
Coding
Static testing
MTW, QAW,ATAM
Prototyping,simulation
From SEI, seewww.sei.cmu.edu/
Test-driven development (TDD) Preventive Testing
Preventive testing is built upon the observation thatone of the most effective ways of specifying something isto describe (in detail) how you would accept (test) itif someone gave it to you.
David Gelperin, Bill Hetzel (<1990)
Real TDD means to let testing drive and influence the architecture and design as well
Do not use TDD only on the level of unit and acceptance testing
Testability – Factors
Control(lability) The better we can control the system,
the more and better testing can be done, automated, and optimized Ability to apply and control the inputs to the system under test
or place it in specified states (for example reset to start state) Interaction with the system under test (SUT) through control points
Visibility / observability What you see is what can be tested Ability to observe the inputs, outputs, states, internals, error conditions,
resource utilization, and other side effects of the system under test Interaction with the system under test (SUT) through observation points
Control(lability)
Visibility / observability
Design for Testability (DfT) – Who?
Joint venture between architects (+ developers) and testers Collaboration between different stakeholders
Testability must be built-in by architects (+ developers) Pro-active strategy:
design the system with testability as a key design criterion Testers (test automation engineers) must define testability
requirements to enable effective and efficient test automation
Contractual agreement between design and test
Balancing production code and test code Production code and test code must fit together Manage the design of production code and test code
as codependent assetsEd
uca
te s
tak
eho
lde
rs o
n b
en
efi
t o
f D
fT
Test design methods
Test design methods are not for testers only …
Check and foster the usage of test design methods in the development team
Use test design methods To provide an adequate test basis In architecture reviews, for example scenario testing To design test cases for the architecture,
especially for integration testing To prevent bugs (think test-first …)
The act of designing tests is one of themost effective bug preventers known.
Boris Beizer, 1983
Integration testing
The goal of integration testing is to test in a grey-box manner The interaction of components and subsystems The interaction and embedding with the environment and system
configuration The dynamic behavior and communication of the system Control flow and data flow The architecture and design as specified in the
Software Architecture Description document
Select an appropriate integration strategythat supports and drives the goals andbenefits of integration testing
Address and follow integration testing needsin the architecture (including testability)
Integration testing
Integration
Architecture
Test architectures, test automation
The architecture of a test system may be even more complex than the architecture of the system under test
Use good architecture practices to define, specify, create, realize, and maintain the test (automation) system as well
An efficient quality check of software is not possiblein real project life without the right test automation! Incremental, iterative, agile processes Regression testing Maintenance – refactoring, redesign, reengineering
Regression testing – The big questions are always …
Software Architect:Which parts of the architecture must be tested againafter a (code) change?
Test Manager:Which test cases must be executed againafter a (code) change?
Remark:Change can be a code changebut also any change in the environment or system configuration
Use knowledge about the architecture and design of the system during change and impact analysis to identify risky areas to select required regression test cases effectively
Architectural quality – internal software qualityand code quality management
Checking of architectural conformance Validate the formalized description Evaluate reported architecture violations Simulate a restructuring of a system (refactoring, redesign,
reengineering)
Identification of suspicious components and trends (software aging) Detect potential problems early enough
Analyzing the impact of design changes and design alternatives Use it as input for well-founded decisions
Code comprehension Get a well-rounded knowledge
of the real state of the system
Avoid this:
Practices and tasks in architecture testing
Understand the mission and the value of testing and promote it
Risk-based testing strategy (test levels)
Test-driven development
Design for testability
Test design methods
Integration testing
Test architectures, test automation
Regression testing
Architectural quality – internal software quality and code quality management
Summary
The Software Architect cooperates closely with the Test Manager To define, motivate, drive, and enforce a comprehensive
understanding of and attitude to testing and quality in the whole team To sustain a culture of prevention rather than detection across the
lifecycle
The Software Architect and the Test Manager address "quality"
from different viewpoints: Software Architect:
Build quality into the system (or rather into the architecture)
and do architecture testing Test Manager:
Provide information related to quality (feedback)