introduction to software testing chapter 9.1 challenges in testing software – testing for emergent...

Introduction to Software TestingChapter 9.1

Challenges in Testing Software –Testing for Emergent Properties: Safety and Security

Introduction to Software Testing (Ch 9.2) © Ammann & Offutt 2

Chapter 9 Outline

1. Testing for Emergent Properties: Safety and Security

2. Software Testability

3. Test Criteria and the Future of Software Testing


Emergent Property Overview

How do we address such properties?

A property that arises as a result of assembling A property that arises as a result of assembling components together into a systemcomponents together into a system

• Emergent properties exist at system level

• The key is the interaction of a system with its environment

• Emergent properties do not exist at component level – But individual component design can have a profound effect on emergent

properties

– Safety and Security are classic emergent properties

• General definition:

Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

Example

Sample Security Property: Outsiders only have Sample Security Property: Outsiders only have access through intended interfaceaccess through intended interface

…gets (buf)…

P

Web Application

Internet

Property Violation: Buffer Property Violation: Buffer overflow vulnerability leads to overflow vulnerability leads to shell access inside componentshell access inside component

4


Why Emergent Properties Are Hard

• Fundamentally different than analyzing intended function– Trying to show software lacks certain “features”

– Trying to show absence of certain behaviors.

– This is really hard!

• Alternative approach– Catalogue typical problem areas

– Systematically work through catalog.

– Not complete!


High Level Steps• Capture relevant safety/security properties

– Often well-understood by system engineers

• Hazard model for safety domain

• Threat model for security domain

• Identify high risk areas– Relates system properties to component properties

• Example: Fault tree analysis for safety

• Mitigate risk– Testing is only one possible approach

– Often redesign is a better option

– It helps to understand the issues as early as possible!


Test Cases For Emergent Properties• Develop misuse cases

– Helps developers think about ways in which system can be misused

• Identify assumptions, and devise test cases that violate them– Can a critical object reach an inconsistent state?– What ways beyond the explicit API exist to alter the state?

• What happens when objects are deserialized?• What happens when a database file is accessed outside the DBMS?• What “normal” checks can be easily evaded?

• Identify configuration issues, and devise tests to check them• Develop invalid input tests

– Often the unsafe or insecure behavior exists outside the expected domain of inputs

– See discussion of bypass testing in Chapter 7

• Don’t forget about static analysis:– Avoidance/removal of unsafe library calls


Summary• Most “real” systems have safety and/or security requirements

• Emergent properties only exist at the system level– Think about the interaction between a system and its environment

– Components, by themselves, don’t exhibit emergent properties

• Emergent property requirements are better understood by domain experts than by software developers

– Communication is essential

• Successfully addressing emergent properties requires careful attention at ALL phases of the software development life cycle

– Safety and Security cannot be “tested in” at the end

– Testing is only one tool

9

Chapter 9 Outline



3. Test Criteria and the Future of Software TestingTest Criteria and the Future of Software Testing


10

Testing in the 1980s and 1990s• Software engineering was very different

– Low quality software was normal and expected

– The cost of building better software outweighed the benefits

• Software was smaller– Most software was bundled, shrink-wrapped, contracted

– Industry became dominated by one monopolistic vendor

• About a dozen books– A few broad survey books, one or two on testing, design, requirements,

etc

• In the early 1980s, one semester course was enough to learn most of the knowledge in software engineering

• Safety critical software was a tiny part of the industry

Testing, especially high-end testing, Testing, especially high-end testing, simply was not very importantsimply was not very important


11

Cost of Late Testing

RequirementsTest Planning

DesignReview

DevelopmentUnit Testing

FunctionalTesting

SystemTesting

Production

Cost per Fault 1X 1X 1X 5X 10X 50X

$6K $13K $20K $101K $363K $252KCurrent State

FaultDiscovery

20%13%6% 20% 5%36%

10%Fault Origination 40% 50%

Planning &Requirements

Design Development Testing UATDeploy to

Production

Source – Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002; page 56-58


12

Testing in the 21st Century• The field has dramatically changed

• Today’s software market :– is much bigger

– is more competitive

– has more users

– used in more places

• The web offers a new deployment platform– Very competitive and very available to more users

• Enterprise applications means bigger programs and more users

• Embedded software is now ubiquitous … check your pockets !

• Paradoxically, free software increases our expectations !

• Security has now become essential …


13

Problems Leading to Security Vulnerabilities

• 1980 : Security was mostly about math– Cryptography

• 1990 : Security was mostly about protecting the database

• 2000 : Security was mostly about bullet-proofing the network

• Today : Most security vulnerabilities are a result of faults in the software

Software testing is fast becoming essential to an Software testing is fast becoming essential to an essential problem in the software industryessential problem in the software industry

Software SecuritySoftware Security


14

Testing Research• 1980s : Criteria and algorithms for unit testing

• 1990s : Test criteria for other technologies and “levels”

• 2000s : More automation for test criteria and more technologies

• 2010s : Practical solutions to industry problems

A major point of this book is that we have enough criteriaA major point of this book is that we have enough criteria

• Test criteria are uniform across different software artifacts– A graph is a graph is a graph …

– 36 criteria on 4 structures

• So are we done with testing research ?– Can I retire now …


15

Testing Research – Are We Done ?

• We need more work on how to test modern technologies– The ideas in chapter 7 are still developing

– OO, web, GUIs, real-time, embedded, …

– How to create the structures and how to adapt the criteria ?

• Which criteria to use when ?– Cost / benefit tradeoffs among criteria are not known

• Which structure should be used when ?

• Technology transition to industry– How best to automate the testing research ideas ?

– How to insert new testing techniques into the development process ?

– How to effectively and efficiently balance research theory with practical needs ?

No !No !


16

Testing Research – Going Forward• Automatic test data generation is one of the hardest problems

– But also the core of testing – getting values to satisfy criteria– Research started in the 1970s – major advances in the 1980s, 1990s and

2000s– Still very little help from industry-quality tools

• Testability, reliability, and other related areas are wide open• We know a lot about testing new software – much less about

regression testing or testing evolving software– Which happens to account for most of the effort in industry …

• Research in software testing is increasing– Currently lots of funding in Europe and Asia – from both government

and industry sources– New, well attended venues (eg ICST)


Future of Software Testing

1. Increased specialization in testing teams will lead to more efficient and effective testing

2. Testing and QA teams will have more technical expertise

3. Developers will have more knowledge about testing and motivation to test better

4. Agile processes puts testing first—putting pressure on both testers and developers to test better

5. Testing and security are starting to merge

17Introduction to Software Testing (Ch 9.2) © Ammann & Offutt

18

Summary

We have entered a golden age for software We have entered a golden age for software testingtesting

• With more attention from– Industry

– Research

– Education …

Finally we have the knowledge, resources, Finally we have the knowledge, resources, and motivation to make testing aand motivation to make testing a

technical disciplinetechnical discipline



Chapter 9 Outline



3. Test Criteria and the Future of Software Testing

Software Testability


Testability Overview• Testability is distinct from software testing

• General definition :

An estimate or measurement of a conditional probability An estimate or measurement of a conditional probability – assuming that a software artifact contains a fault, how – assuming that a software artifact contains a fault, how likely is it that testing will reveal that fault ?likely is it that testing will reveal that fault ?

• What do we do with testability estimates ?– Pay more attention to components with low testability – code reviews,

formal analysis, stronger test criteria

– Modify low testability components to increase their testability

– We have more confidence in components with high testability

– Risk analysis – low testability components represent risk that management needs to be aware of


Model of Testability

If a program has a fault, how difficult will it be to If a program has a fault, how difficult will it be to find a test that causes a failure ?find a test that causes a failure ?

Simple model

Outfailurecausing

Inputs

P

Testability =| failure causing |

| Input | X 100 %Impractical to measure


Approximating Testability

Testability can be approximated with the RIP Testability can be approximated with the RIP model and mutationmodel and mutation

Given a location X in a program P

P: entry

X

exit

Induce faults(mutants)

R = % inputs from some distribution that reach X

I = % inputs that cause a fault to infect (average over N faults)

P = % infected states that propagate to output

Sensitivity (X) = R * I * P

Testability (P) = F (Sensitivity (X)), for all X in P


Issues in Approximating Testability

• Reasonable input distribution ?

• How to induce faults ?– What faults ?

• How to measure propagation ?– Expensive!

• Information hiding reduces propagation

• Assertion checking can be used to increase testability


Testability of OO Software• Information hiding decreases testability

– State information is saved in instance variables

– No direct access to instance variables

• Inheritance compounds the problem– Instance variables are defined in ancestor classes – harder to get to

• These are primarily issues of observability

• Increasing observability in OO software :– Require additional get ( ) methods – must be done during development

– Use Java reflection to access internal variables – hard to interpret the data values

• This is an area of ongoing research


Testability of Web Applications• Both controllability and observability are very low

• User interface and most software components distributed on different computers

– Server software may be distributed even further

• Most communication is through message passing

• Much of the inter-component communication goes through the client

– Stateless HTTP messages

• State is kept in an unusual combination of technologies– Cookies and session objects

Testability in web applications is still very Testability in web applications is still very much an open research areamuch an open research area


Testability Summary• Testability can give valuable information to testers, managers

and developers

• Testability is often thought of as combining two characteristics of software

– Controllability and observability

• Measuring testability is a very technical task

How best to measure testability is still an How best to measure testability is still an open research problemopen research problem

introduction to software testing chapter 9.1 challenges in testing software – testing for emergent...

Documents

testing software testing

system emergent properties

system properties

emergent properties

software testing chapter

risk testing

software testability

software deve