presentation on software testing: a research travelogue ... · overview 1 introduction 2 research...

Presentation on Software Testing: A research Travelogue( 2000 - 2014)

Presented by : Ji Gao

1Department of Computer Science, University of Virginiahttps://qdata.github.io/deep2Read/

August 26, 2018

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 1 / 26

Overview

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 2 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 3 / 26

Introduction

Testing: Most popular and practised way to assess software quality

Many research progress in past years

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 4 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 5 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 6 / 26

Automated test input generation

Formally, generating a set of input values for a program or programcomponent, typically with the aim of achieving some coverage goal orreaching a particular state.

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 7 / 26

Symbolic ExecutionAutomated test input generation

Use symbols to replace variable in the code.

Express the execution of the program as a bunch of paths.

Test the validity of paths, and its corresponding range of inputsymbols.

Symbolic state as a mapping S :M→ E , M is the set of memories,and E is the set of symbolic values.

Exectuion: S ′ = S ⊕ [m→ e ′]

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 8 / 26

White box fuzzing

Concolic testing

In the execution of a certain path, try to cover other path, generatingnew test cases by flipping the original input.

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 9 / 26

Symbolic ExecutionAutomated test input generation

Machine learning models can be expressed as symbolic operation.

However, the optimization process gives a huge tree that can’t bechecked.

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 10 / 26

Search-Based TestingAutomated test input generation

Use heuristic search-based optimization techniques (for example, hillclimbing) to solve optimization problems.

Have a variety of forms and wide applications.

Challenge and opportunities:- Oracle problem- Combining SBST with symbolic testing- Co-evolutionary computation- Hyper-heuristic software engineering

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 11 / 26

Random testingAutomated test input generation

Randomly generate test inputs

Or not that randomly

Adaptive Random Testing: Generate input that is the most ’distant’from previous inputs

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 12 / 26

Combined techniquesAutomated test input generation

Combine testing with other verification skills: static verification,abstraction, ...

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 13 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 14 / 26

Testing strategies

Strategies of testing that can lead to a better result

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 15 / 26

Combinatorial testingTesting strategies

Combinatorial testing: Model the problem as a finite set ofparameters. Find a combination of test inputs that covers all inputs.

Weighted, biased, guided way of CIT?

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 16 / 26

Model-based TestingTesting strategies

Based on the specialty of every model, develop testing suites.

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 17 / 26

Mining and Learning from Field DataTesting strategies

Use large data to help testing

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 18 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 19 / 26

Regression Testing

Verify whether a software previously tested still performs correctlyafter changed

Attempt to select/modify a smaller test suite to complete the quest

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 20 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 21 / 26

Emperical studies

Experiments

Frameworks

Continuous integration

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 22 / 26

Outline

1 Introduction

2 Research Topics1. Automated test input generation2. Testing strategies3. Regression Testing4. Emperical studies

3 Challenges and Opportunities

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 23 / 26

Testing modern systems

Heterogeneous software systems

Rich environments

High configuability

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 24 / 26

Oracle problem

Inferring invariants from programs

Rich environments

High configuability

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 25 / 26

Other opportunities

Probabilistic program analysis

Testing non-functional properties

Domain-based testing

Cloud and crowd computation

Presented by : Ji Gao (University of Virginia)Presentation on Software Testing: A research Travelogue ( 2000 - 2014)August 26, 2018 26 / 26