CS527: (Advanced) Topics in Software Engineering

(Software Testing and Analysis)

Darko Marinov

August 23, 2011

Course Overview• Graduate seminar on program analysis (for

bug finding), emphasis: systematic testing (for concurrent code)

• Focus on a (research) project– Papers: reading in advance, writing reports,

presenting, and discussing– Project: initial proposal, progress report, [final

presentation, based on enrollment size], paper• One homework assignment (maybe two) to

help with projects

Administrative Info

• Meetings: TR 2-3:15pm, 1302 SC

• Credit: 4 graduate hours• Auditors welcome for discussions

– Can come to any lecture, mostly self-contained

• Prerequisites: some software engineering and programming languages

Evaluation

• Grading– Project [40%]– Presentation [20%]– Participation (reports and discussion) [20%]– Homework assignment(s) [20%]

• Distribution– Grades will be A- centered– No guarantee for A (or even a passing grade)!

• Project is the most important

Project

• Proposal (due in late September)• Progress report (in November)• [Presentation (after Thanksgiving Break)]• Paper (last day of classes)

• 15 students who took similar classes published 12 papers based on their projects– I’m happy to help, no co-authorship required

Fair Warnings

• This class will differ from most you take– Seminar style, reading papers– Centered around (research) projects

• Projects are NOT easy– Require that you explore a topic in great depth– The topic can/should be fairly narrow

Repeated Warning

• Project matters the MOST– If you like open-ended projects, do take this

course– If you don’t like open-ended projects, please

drop this course now– If you’re unsure, please discuss with me

• The worst scenario: take the course but realize you don’t like projects

Project Overview

• Testing/analysis of some open-source code– We will use mostly Java this semester

• You can use C#/C/C++/C--/Clojure/Cecil…http://en.wikipedia.org/wiki/List_of_programming_languages#C

• Sample topics– Automated unit testing– Test coverage and adequacy criteria– Test-input generation, test oracles– Test maintenance– …

Course Communication

• Wikihttp://wiki.engr.illinois.edu/display/cs527fa11

• Mailing listcs527-fa11 AT cs.illinois.edu

Personnel

• Instructor: Darko Marinov– Office: 3116 SC, hours: by appointment– Phone number: 217-265-6117– NetID: marinov

• TA: Shin Hwei Tan– NetID: stan6

• Please use your netid@illinois.edu email addresses for communication

Signup Sheet

• Name• NetID (please write it legibly)• Program/Year• Group• Interests

• Please also sign up on Wiki (if possible,as there are some delays with netids)

This Lecture: Overview

• Why look for bugs?• What are bugs?• Where they come from?• How to detect them?

Some Costly “Bugs”

• NASA Mars space missions– Priority inversion (2004)– Different metric systems (1999)

• BMW airbag problems (1999)– Recall of >15000 cars

• Ariane 5 crash (1996)– Uncaught exception of numerical overflow– http://www.youtube.com/watch?v=kYUrqdUyEpI

• Your own (in)famous example?

Some “Bugging” Bugs

• An example bug on my laptop– “Jumping” file after changing properties

• Put a read-only file on the desktop• Change properties: rename and make not read-only

• Your own favorite example?

Economic Impact

• “The Economic Impact of Inadequate Infrastructure for Software Testing”NIST Report, May 2002

• $59.5B annual cost of inadequate software testing infrastructure

• $22.2B annual potential cost reduction from feasible infrastructure improvements

Estimates

• Extrapolated from two studies (5% of total)– Manufacturing: transportation equipment– Services: financial institutions

• Number of simplifying assumptions• “…should be considered approximations”

• What is important for you?– Correctness, performance, functionality

Terminology• Anomaly• Bug• Crash• Defect• Error• Failure, fault • Glitch• Hangup• Incorrectness• J…

Dynamic vs. Static

• Incorrect (observed) behavior– Failure, fault

• Incorrect (unobserved) state– Error, latent error

• Incorrect lines of code– Fault, error

“Bugs” in IEEE 610.12-1990

• Fault– Incorrect lines of code

• Error– Faults cause incorrect (unobserved) state

• Failure– Errors cause incorrect (observed) behavior

• Not used consistently in literature!

Correctness

• Common (partial) properties– Segfaults, uncaught exceptions– Resource leaks– Data races, deadlocks– Statistics based

• Specific properties– Requirements– Specification

Traditional Waterfall ModelRequirements

Analysis

DesignChecking

ImplementationUnit Testing

IntegrationSystem Testing

MaintenanceVerification

Phases (1)

• Requirements– Specify what the software should do– Analysis: eliminate/reduce ambiguities,

inconsistencies, and incompleteness• Design

– Specify how the software should work– Split software into modules, write specifications– Checking: check conformance to requirements

Phases (2)

• Implementation– Specify how the modules work– Unit testing: test each module in isolation

• Integration– Specify how the modules interact– System testing: test module interactions

• Maintenance– Evolve software as requirements change– Verification: test changes, regression testing

Testing Effort

• Reported to be >50% of development cost [e.g., Beizer 1990]

• Microsoft: 75% time spent testing– 50% testers who spend all time testing– 50% developers who spend half time testing

When to Test

• The later a bug is found, the higher the cost– Orders of magnitude increase in later phases– Also the smaller chance of a proper fix

• Old saying: test often, test early• New methodology: test-driven development

(write tests before code)

Software is Complex

• Malleable• Intangible• Abstract• Solves complex problems• Interacts with other software and hardware• Not continuous

Software Still Buggy

• Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing)

• Consensus: total correctness impossibleto achieve for (complex) software– Risk-driven finding/elimination of bugs– Focus on specific correctness properties

Approaches for Detecting Bugs

• Software testing• Model checking• (Static) program analysis

Software Testing

• Dynamic approach• Run code for some inputs, check outputs• Checks correctness for some executions

• Main questions– Test-input generation– Test-suite adequacy– Test oracles

Other Testing Questions

• Maintenance• Selection• Minimization• Prioritization• Augmentation • Evaluation• Fault Characterization• …

Model Checking

• Typically hybrid dynamic/static approach• Checks correctness for all executions

• Some techniques– Explicit-state model checking– Symbolic model checking– Abstraction-based model checking

Static Analysis

• Static approach• Checks correctness for all executions

• Some techniques– Abstract interpretation– Dataflow analysis– Verification-condition generation

Comparison

• Level of automation– Push-button vs. manual

• Type of bugs found– Hard vs. easy to reproduce– High vs. low probability– Common vs. specific properties

• Type of bugs (not) found

Soundness and Completeness

• Do we detect all bugs?– Impossible for dynamic analysis

• Are reported bugs real bugs?– Easy for dynamic analysis

• Most practical techniques and tools are both unsound and incomplete!– False positives– False negatives

Analysis for Performance

• Static compiler analysis, profiling• Must be sound

– Correctness of transformation: equivalence• Improves execution time• Programmer time is more important

• Programmer productivity– Not only finding bugs

Combining Dynamic and Static

• Dynamic and static analyses equal in limit– Dynamic: try exhaustively all possible inputs– Static: model precisely every possible state

• Synergistic opportunities– Static analysis can optimize dynamic analysis– Dynamic analysis can focus static analysis– More discussions than results

Current Status

• Testing remains the most widely used approach for finding bugs

• A lot of recent progress (within last decade) on model checking and static analysis– Model checking: from hardware to software– Static analysis: from sound to practical

• Vibrant research in the area

• Gap between research and practice

Topics Related to Finding Bugs

• How to eliminate bugs?– Debugging

• How to prevent bugs?– Programming language design– Software development processes

• How to show absence of bugs?– Theorem proving– Model checking, program analysis

Next Lecture

• Thursday, August 25, at 2pm, 1302 SC• Texts to read (listed on Wiki)

– How to Read an Engineering Research Paperby William G. Griswold

– Writing Good Software Engineering Research Papers

by Mary Shaw (ICSE 2003)• If you have read that paper, read on another area

• You don’t have to write any report now• Assignment 0: Modify “People” on Wiki