testing database applications
DESCRIPTION
Testing Database Applications. Donald Kossmann http://www.dbis.ethz.ch. Joint work with: Carsten Binnig, Eric Lo. Thanks: i-TV-T AG, Porsche, Microsoft, Baden-Würtemberg. Quotes. „50% of our cost is on testing (QA)“ (Bill Gates @ Opening of Gates Building) - PowerPoint PPT PresentationTRANSCRIPT
Testing Database Applications
Donald Kossmann
http://www.dbis.ethz.ch
Joint work with: Carsten Binnig, Eric Lo
Thanks: i-TV-T AG, Porsche, Microsoft, Baden-Würtemberg
Quotes• „50% of our cost is on testing (QA)“
(Bill Gates @ Opening of Gates Building)
• „Testing alone makes up for six months of the 18 month product release cycle“(Anonymous SAP Executive)
• Estimated damage of USD 60 bln per year in USA caused by software bugs (US Department of Commerce, 2004)
• Mercury: 30% of testing can be automated
• HP: Buys Mercury for $4.5 bln
Observations• Everybody loves writing code
Everybody hates testing it– more work on new models etc. than on testing– solution: automate the testing!– (Computers are cheap and do not complain)
• Test Automation is a DB Problem– several optimizations in different flavors– it is all about logical data independence– it is a far cry from being solved!
• Research on Testing (Automation) is fun!
Test Automation• Idea: Testing ~ Programming
– Testprg = Actions + (desired) Responses– Examples: JUnit, Caputre & Replay
• But ...– Programming is expensive; Tests aren‘t any better– Maintenance of (Test-) Programs is expensive– Test-programs often have more bugs than the
systems they test– Test-programs are not enough: test databases – Test-programs must be optimized
• Idea of Test Automation is good!Risk: Going from bad to worse.
Project Goals
• Higher level of Abstraction of Tests (Prgs+DBs)– Avoid „over-specification“ of test components– Automate testing: execution, generation, evol., ...
• Automate Generation of Test Databases– Generate relevant Test Databases– Generate scalable Test Databases
• Automate Execution of Regression Tests– Optimization and Parallelization
• (Automate Evolution of Test-DBs + Programs)
Project Goals
• Higher level of Abstraction of Tests (Prgs+DBs)– Avoid „over-specification“ of test components– Automate testing: execution, generation, evol., ...
• Automate Generation of Test Databases– Generate relevant Test Databases– Generate scalable Test Databases
• Automate Execution of Regression Tests– Optimization and Parallelization
There are tools for some of these goals available. But they do not fit together, and have limitations.
Approach• Bottom-up: Various ideas and tools
– model-based testing (not invented by ETH)– Reverse Query Processing (ETH)– HTPar (ETH)
• Exploit „Standards“– Databases everywhere: „Fluch & Segen“– Web-based apps (simplifies tooling)
• Prototypes and Industry Collaboration– Canoo, i-TV-T, Microsoft, Porsche– Everything is implemented and „tested“
Agenda
• Motivation and Overview
• Database Regression Testing: Overview
• RQP: Generating Test Databases
• Related Work
• Conclusion
Structure of a Test Program
• Phase 1: Setup– Initialize Variables / State (= DB)– (long) sequence of SQL „insert“ statements
• Phase 2: Execute– Execute test step by step– Check responses of the system; compute ‘s
• Phase 3: Cleanup– Release resources– e.g., SQL „drop table“ statements
• Phase 4: Report– Report ‘s from execute
What is our contribution?• Get rid of setup und cleanup
– Only specify the id of a test-DB used for initialization– implicit setup / cleanup by testing infrastructure– reduces size of test code by up to 80% (IBM)– optimize the setup and cleanup
improve testing performance by a factor of 500 (Unilever)
• Execute– Model-based testing (HTTrace: Web-based C&R Toolkit)
• Only specify behavior you want to test; ignore randomness
– flexible, app-dependent function– XML representation of test runs for better evolution / queries
• Report– Understand the HTML page (tables, keys, same errors, etc.)
How to optimize regression tests
• Test 1: Insert a new order(Load Test-DB 1)insertOrder(Schmitz, 1000, Staples)showAllOrders()
• Test 2: Show all pending orders(Load Test-DB 1)showAllOrders()
How to optimize regression tests
• Test 1: Insert a new order(Load Test-DB 1)insertOrder(Schmitz, 1000, Staples)showAllOrders()
• Test 2: Show all pending orders(Load Test-DB 1)showAllOrders()
• How do you execute these 2 tests efficiently?– How do you do that with 1 million tests?– IBM manually defines test buckets! (bad!)
Solution Overview [Haftmann et al. 2007]
• Optimistic Execution of Test Programs– execute a test (setup = cleanup = NOP)– if it fails, reset the database and try again– if it fails again, then it really fails– (watch out for „false negatives“)
• Slice Algorithm– remember conflicts between test runs– create a conflict graph between test runs– order execution of test runs according to graph– (more smarts in the fineprint of the algo)
• Parallelization– shared nothing vs. shared DB– clever scheduling and reset strategies
Model-based Testing<project name="SimpleTest“ basedir=".“ default="main">
<property name="webtest.home“ location="C:/java/webtest"/>
<import file="${webtest.home}/lib/taskdef.xml"/>
<target name="main"> <webtest name="myTest">
<config host=www.myserver.com port="8080“ protocol="http“ basepath="myApp"/>
<steps><invoke description="getLoginPage“ url="login"/>
<verifyTitle description=“blabla“ text="Login Page"/>
</steps>
</webtest> </target></project>
Understanding HTML
Agenda
• Motivation and Overview
• Database Regression Testing: Overview
• RQP: Generating Test Databases
• Related Work
• Conclusion
State-of-the Art: DB Generation• Commercial Products and Open Source Tools
– Input: •DB Schema (Tables + Constraints)•Scaling Factor•(Constants)
– Output: SQL „insert“ Statements
• Result of the following query on generated DB?SELECT c.name, o.priceFROM Customer c, Order o, Region r, Product pWHERE c.region = r.id AND r.name = Asia ...
State-of-the Art: DB Generation• Commercial Product and Open Source Tools
– Input: •DB Schema (Tables + Constraints)•Scaling Factor•(Constants)
– Output: SQL „insert“ Statements
• What is the result of the following query?SELECT c.name, o.priceFROM Customer c, Order o, Region r, Product pWHERE c.region = r.id AND r.name = Asia ...
c.name o.price
The Solution
• Reverse Query Processing– Input:
•DB Schema (Tables + Constraints) •Scaling Factor•(Constants)•Application Program (SQL Queries)•Meaningful Query Results (Tables)
– Output: SQL „insert“ Statements
• Generate „Relevant“ Test-DBs– a Test-DB must be specific to the application– Evolution: Create new or extend Test-DB when the
schema evolves and application has new queries
Finding the right query• In practice, application has many queries
– workflow might involve sequence of queries– independent workflows have multiple queries
• Idea: generate one generic query that „encodes“ needs of a whole workflow– logically a „join/union“ of all queries
• Status: Create this query manually– still better than state-of-the-art: higher abstraction– but, indeed sometimes difficult to find
• Future Work: semi-automatic process– symbolic computation helps (SIGMOD 07)
RQP: Problem Statement• Given:
– Query Q, Table R– Schema S (including integrity constraints)
• Generate a database instance D such that:
R = Q(D)R = Q(D)
such that D matches S and its constraints
• Yes, the problem is undecidable (Q with “-”)– even undecidable whether such a D exists
• Who cares? You can always check D?– semi-automatic approach, if check fails
RQP Example
select c.name, sum(amount) as revenuefrom order o, customer cwhere o.cid = c.cid and c.age > 18group by c.name;
Database Constraint: Order.amount <= 70
c.name revenue
Paul 130 R
Q
S
RQP Example
select c.name, sum(amount) as revenuefrom Order o, Customer cwhere o.cid = c.cid and c.age > 18group by c.name;Database Constraint: Order.amount <= 70
c.name revenue
Paul 130 R
Q
S
cid amount
1 70
1 60
Order
cid name age salary
1 Paul 23 5000
Customer
D
Trichotomy (thanks to MJF)
Answers
Database
Queries
Trichotomy (thanks to MJF)
Answers
Database
QueriesQuery Processing
Trichotomy (thanks to MJF)
Answers
Database
Queries
Reverse Query Processing
Trichotomy (thanks to MJF)
Answers
Database
QueriesProgramming By Example
Architecture
Reverse Query Processor
run-time
compile-time
Bottom-up query
annotation
Query parser and
translator
Query optimizer
Modelchecker
Top-downdata
instantiation
Formula L Instantiation I
RTable R Parameter values
Database D
Query Q
Database Schema S
Reverse query tree TQ
Annotated T+Q
Optimized T’Q
Example• SQL Query Q:
SELECT SUM(price)
FROM Lineitem, Orders
WHERE l_oid=oid
GROUP BY orderdate
HAVING AVG(price)<=100;
• Schema S:CREATE TABLE Lineitem (
lid INTEGER PRIMARY KEY,
name VARCHAR(20),
price FLOAT,
discount FLOAT
CHECK (1>= discount >=0),
l_oid INTEGER);
CREATE TABLE Orders(
oid INTEGER PRIMARY KEY,
orderdate DATE);
Parser - RRA Tree
orderdateχ-1SUM(price), AVG(price)
σ-1AVG(price)<=100
П-1SUM(price)
-1l_oid=oid
Lineitem Orders
Traditional SQL Parsing; 1:1 relationship from RA to RRA
Parser - RRA Tree
orderdateχ-1SUM(price), AVG(price)
σ-1AVG(price)<=100
П-1SUM(price)
-1l_oid=oid
Lineitem Orders
Dat
a F
low
Traditional SQL Parsing; 1:1 relationship from RA to RRA
Reverse Projection
SUM(price)
100
120
orderdate SUM(price) AVG(price)
1990-01-02 100 100
2006-07-31 120 60
П-1SUM(price)
orderdateχ-1SUM(price), AVG(price)
σ-1AVG(price)<=100
П-1SUM(price)
-1l_oid=oid
Lineitem Orders
Reverse Projection
1st attempt (count=1):orderdate!=19900102 & sum_price=120 & avg_price<=100 &sum_price=price1 & avg_price=sum_price/1 Not Satisfiable!
2nd trial (count=2):orderdate!=19900102 & sum_price=120 & avg_price<=100 &sum_price=price1+price2 & avg_price=sum_price/2 Satisfiable!
Instantiationsum_price=120, avg_price=60,price1=80, price2=40, orderdate=20060731
Reverse Selection
σ-1AVG(price)<=100
orderdate SUM(price) AVG(price)
1990-01-02 100 100
2006-07-31 120 60
orderdate SUM(price) AVG(price)
1990-01-02 100 100
2006-07-31 120 60
Reverse Aggregation
orderdateχ-1SUM(price), AVG(price)
lid name price discount L_oid oid orderdate
1 A 100 0.0 1 1 1990-01-02
2 B 70 0.0 2 2 2006-07-31
3 C 50 0.0 2 2 2006-07-31
orderdate SUM(price) AVG(price)
1990-01-02 100 100
2006-07-31 120 60
Reverse Equi Join
-1l_oid=oid
oid orderdate
1 1990-01-02
2 2006-07-31
lid name price discount L_oid
1 A 100 0.0 1
2 B 70 0.0 2
3 C 50 0.0 2
lid name price discount L_oid oid orderdate
1 A 100 0.0 1 1 1990-01-02
2 B 70 0.0 2 2 2006-07-31
3 C 50 0.0 2 2 2006-07-31
Architecture
Reverse Query Processor
run-time
compile-time
Bottom-up query
annotation
Query parser and
translator
Query optimizer
Modelchecker
Top-downdata
instantiation
Formula L Instantiation I
RTable R Parameter values
Database D
Query Q
Database Schema S
Reverse query tree TQ
Annotated T+Q
Optimized T’Q
Reverse Query Optimization• Some observations
– projections and group by‘s are expensive– (equi-) joins and selections are cheap– nested queries are expensive– calls to the model checker are expensive
• depend on number of free variables, types of vars
• Conclusions– apply „smarts“ to avoid model checker calls– apply smarts to simplify model checker calls– do aggressive query rewriting– but do not worry about join ordering, push-down
Correctness Criterion
• Rewrite of Plan P1 into Plan P2 allowed iff
Q(P1(R) = Q(P2(R)) = R
• That is, P1 and P2 may produce different databases!!! That is okay.
• Goal (here): generate large databases fast. (Alternative goal: „good“ DBs)
R = Q(D)R = Q(D)
Query Unnestingselect name from lineitemwhere l_oid not in (select max(cid)
from ordersgroup by odate)
becomes
select name from lineitem
• (An empty „orders“ table is generated!)
Query Unnestingselect name, price from lineitemwhere price = (select min(price)
from lineitem)
becomes
select name, price from lineitem
• (Precise definition of rules in the Tech.Rep.)
• (Of course, all traditional rules are applicable.)
Some Tricks (see TechRep for complete list)
• (Constrictive) Independent Attributes– can take random values (avoid model checker)– or can take fixed values (in „distinct“ queries)
• Infer cardinalities from AVG and SUM– avoid trial-error algorithm
• Bound cardinalities from MAX, MIN, SUM– limit trial-error algorithm
• Simplify constraint formulae– use SUM(a) / n for aggregations
• Memoization: cache model checker calls
Performance Experiments
• Use dbgen in order to generate TPC-H DBs– use three scaling factors: 100 MB, 1 GB, 10 GB
• Run 22 TPC-H Queries on DBs (PostGres)– get 22 x 3 RTables
• Run RQP on 22x3 RTables– get 22x3 different DBs
• Compare original DB with generated DBs
• Measure Running Time of RQP
Results (DB Size)
100 MB 1 GB 10 GB
Query RTable Generated RTable Generated RTable Generated
1 4 600.572 4 6.001.215 4 59.986.052
2 44 220 460 2.300 4.667 23.335
3 1216 3.648 11.620 34,86 114.003 342.009
4 5 10.186 5 105.046 5 1.052.080
5 5 30 5 30 5 30
6 1 1 1 1 1 1
7 4 24 4 24 4 24
8 2 32 2 32 2 32
… … … … … … …
Results (hh:mm:ss)
Query 100 MB 1 GB 10 GB
1 26:51:00 207:11:00 2054:19:00
2 00:24 00:47 04:02
3 19:20 183:49:00 1819:48:00
4 00:20 02:26 24:15:00
5 00:12 00:12 00:12
6 00:02 00:01 00:01
7 00:10 00:10 00:09
8 00:15 00:17 00:14
… … … …
Other RQP Applications
• Updating (non-updateable) Views– find all possible update scenarios– define a policy that selects update scenario
• Privacy / Security– what can be inferred from the published data
• SQL Debugger– determine operator that screws up result
• Program Verification (weakest pre-condition)
• Database Compression / Sampling– real DB -> queries -> results -> queries -> small DB
Symbolic SQL Computation• Goal: control of intermediate results
– selectivity of query operators, cardinality, distr., ...– test database system (not app); e.g., optimizer
• Idea: Process tuples with variables as values
– Put constraints on variables: e.g., $z > 1000– (Reverse/Forward) process variables with query – instantiate variables at the end -> test database
Customer Product Volume
Paul Smith $x1 5000
$y $x2 $z
Agenda
• Motivation and Overview
• Database Regression Testing: Overview
• RQP: Generating Test Databases
• Related Work
• Conclusion
Related Work• Testing owned by Software Eng. Community
– JUnit: Mother of regression testing for Java– focus on processes and methodology– database often simulated using mock objects :-)
• noticeable exception: AGENDA Project (Chays et al.)
– no mention of „optimization“, „data independence“
• Generating Test Databases– Gray et al. (SIGMOD 94), ...– Bruno, Chaudhuri, Thomas (TKDE 06)
• Generating SQL Test Queries– Slutz (VLDB 98), Poess, Stephens (VLDB 04)
Conclusion• Automated testing has many hidden costs
– Definition of test modules / buckets – Definition of the order of test execution (manual parallel.)– Generierating Test-DBs (adjusting Test-DBs)– Evolution of tests and Test-DBs with new releases– Writing code for setup and cleanup– Definition of delta function– ...
• Vendors solve one problem at cost of another• We don‘t have a good solution, but ...
– we have some fun ideas– and we are honest
Research Challenges (CIDR 05)
• Test Run Generation (in progress)– automatic (robot), teach-in, monitoring,
decl. specification
• Test Database Generation (in progress)• Test Run, DB Management and Evolution (uns.)• Execution Strategies (solved), Incremental (uns.)• Computation and visualization of (solved)• Quality parameters (in progress)
– functionality (solved)– performance (in progress)– availability, concurrency, scalability, security (unsolved)
• Cost Model, Test Economy (unsolved)