Generating Shifting Workloads to Benchmark Adaptability inRelational Database Systems

Tilmann Rabl, Andreas Lang, Thomas Hackl,Bernhard Sick, Harald Kosch

Faculty of Computer ScienceUniversity of Passau

TPC Technology Conference on Performance Evaluation & BenchmarkingAugust 24, 2009

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Introduction

Hot database research topics:

Adaptability

Automatic and autonomic tuning

How can they be benchmarked?

Today’s solution

Starting from scratch

Changing the workload completely

2 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Introduction

Hot database research topics:

Adaptability

Automatic and autonomic tuning

How can they be benchmarked?

Today’s solution

Starting from scratch

Changing the workload completely

2 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Motivation

How does real-world workload look like?Homogeneous workload:

Daily and weekly patterns

3 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Motivation

How does real-world workload look like?Special purpose workload:

November December January February11th March 16th April May June21st0

0.5

1

1.5

2

2.5

3

3.5

4x 10

4

Dai

ly A

cces

ses

Daily Website Accesses 24 October 08 − 11 June 09

pluginsexternseminare_mainmeine_seminaredetails

Daily and weekly patterns

Workload classes: working days, holidays, outliers

Trends

3 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Benchmark Design Challenges

Design Challenges

Realistic data & workload patterns

Scalable

Configurable

Random generators

Basis

Stud.IP eLearning management system

Online application

1 year web server log data

Complete database dump

4 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Benchmark Design

25 tables (original 200)

30 queries

Only course management

Most important:seminar - seminar user - user

user_idpasswordusernamevornamenachnameemailpermission

users

seminar_idnameweekdaystarttimeendtimestartdateenddateects

seminar

seminar_iduser_idstudiengang_id

seminar_user

course_idseminar_idweekdaystarttimeendtimeroommax_participants

courses

user_idseminar_idcourse_idteam_id

courses_user

course_iduser_id

course_lecturer

studiengang_idname

studiengaengeuser_idstudiengang_id

user_studiengang user_idaddresshobbyphonecellphonepicture_path

user_info

object_iduser_idcontenttimestampvalid_until

objectsuser_idobject_idtimestamp

object_user_visit

institute_idnameaddress

institute

seminar_idinstitute_id

seminar_institute

topic_idseminar_iduser_idroot_idparent_idtimestamplast_editedcontent

topics

seminar_idsem_hierarchy_id

seminar_sem_hierarchy

sem_hierarchy_idparent_idnamepo_idpo_version

sem_hierarchy

dokument_iduser_idseminar_idfilepathtimestampvalid_untilclick_counter

dokumente

folder_idname

folder

permission_iditem_idrange_id

permissions

plugin_idpluginpathpluginnameplugindescenabled

plugins

team_idcourse_idpasswordname

teams

source_iddestination_idtype

eigeneDateien_links

message_idtimestampsender_idreceiver_idsubjectcontent

messages

user_idmessage_id

inbox

user_idmessage_id

outbox

5 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Data Generation

Based on original data

Maximum likelihood estimation

Standard probability distributions

Scalable

020

4060

80100

020

4060

80100

0

20

40

60

80

100

#Users per Seminar

Reference distribution in table seminar_user

#Seminars per User

#Ent

ries

10 20 30 40 50 60 70 80 90 1000

500

1000

1500

2000

2500

3000

3500

4000

4500Reference Distribution of Users per Seminar in Table seminar_user

Users per Seminar

Ent

ries

User References

Log−Normal Distribution

10 20 30 40 50 60 70 80 90 1000

500

1000

1500

2000

2500

3000

3500

4000

4500Reference Distribution of Seminars per User in Table seminar_user

Seminars per User

Ent

ries

Seminar ReferencesLog−Normal DistributionGamma Distribution

6 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Workload Modeling

Workload interpreted as time series

Classes of queries (meine seminare)

Classes of days (Monday during lecture period)

Daily approximation by polynomials

0 1 2 3 4 50

2000

4000

6000

8000

10000

12000

Weeks

Pag

e H

its

June 1 − July 6, 2008

pluginsexternseminar_mainmeine_seminare

7 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Polynomial Approximation

Optimal approximating polynomial of degree K :

pa(x) =K∑

k=0

akpk(x)

Properties of pk :

different ascending degrees

leading coefficient one

pair wise orthogonal (inner product)

6 8 10 12 14 16 18 20 22 0 2 40

200

400

600

800

1000

1200Most Likely Approximating Polynomial

Hours

Pag

e H

its

Most Likely PolynomialAverage MondayMonday 17−11−08Monday 12−01−09

8 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Workload Generation

Unchanging Behavior

Evaluate polynomial at certain points in time

Random Generation

Time series as point in shape space

Weighting factors ak are normally distributed

Multivariate Gaussian to generate weighting vector a

−5 0 5 10 15 20 25−1

−0.5

0

0.5

1

1.5

2Monomial Coefficients

x0

x1

0 200 400 600 800 1000 1200−40

−30

−20

−10

0Orthogonial Coefficients

average

slop

e

9 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Benchmarking Objectives

Basic Performance

Shifting workloadsMeasure peak performance

Adaptivity

Different daily patternsMeasure adaptability

Robustness

Introduction of outliersMeasure over adaptation

Energy and Space Efficiency

Shifting workloadsMeasure energy / spaceefficiency

November December January February11th March 16th April May June21st0

0.5

1

1.5

2

2.5

3

3.5

4x 10

4

Dai

ly A

cces

ses

Daily Website Accesses 24 October 08 − 11 June 09

pluginsexternseminare_mainmeine_seminaredetails

10 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Conclusion

Conclusion

Online eLearning management benchmark

New generator model for query workloads

More realistic benchmarking for automatic / autonomic tuning

Future Work

Tune and test

Apply techniques to standard benchmarks

Trends in workloads

Schema evolution

11 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Conclusion

Conclusion

Online eLearning management benchmark

New generator model for query workloads

More realistic benchmarking for automatic / autonomic tuning

Future Work

Tune and test

Apply techniques to standard benchmarks

Trends in workloads

Schema evolution

11 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems

Introduction Benchmark Design Workload Generation Benchmarking Objectives Conclusion

Questions?

Thank you.

12 / 12

Generating Shifting Workloads to Benchmark Adaptability in Relational Database Systems