Interpreting and Reporting Performance Test Results

ERIC PROEGLER

• From Mountain View, California• @ericproegler• Performance Engineer, (soasta.com)• 13 years in (back-end) performance, 20 in software• Speak Easy Mentor (speakeas.ie)• AST Board (associationforsoftwaretesting.org)• WOPR Organizer (performance-workshop.org).

…Think “CAVIAR”C ollectingA ggregating V isualizing I nterpretingA ssessingR eporting

For Actionable Results…

Collecting• Objective: Gather all results from test that• help gain confidence in results validity• Portray system scalability, throughput & capacity

• provide bottleneck / resource limit diagnostics

• help formulate hypotheses

Aggregating• Objective: Summarize measurements using

• Various sized time-buckets to provide tree & forest views

• Consistent time-buckets to enable accurate correlation

• Meaningful statistics: scatter, min-max range, percentiles

• Multiple metrics to confirm (or invalidate) hypotheses

Visualizing• Objective: “forest view” of metrics

relative to load• Turn barrels of numbers into a few pictures• Vary graph scale & summarization granularity to

expose hidden facts• ID load point where degradation begins• ID system tier(s) where bottlenecks appear, limiting

resources

Visualizing• Errors over load (“results valid?”)• Bandwidth throughput over load (“system

bottleneck?”)• Response time over load (“how does system scale?”)

• Business process end-to-end• Page level (min-avg-max-SD-90th percentile)

• System resources (“how’s the infrastructure capacity?”)• Server cpu over load• JVM heap memory/GC• DB lock contention, I/O Latency

Interpreting• Objective: Draw conclusions from

observations, hypotheses• Make objective, quantitative observations from

graphs / data • Correlate / triangulate graphs / data• Develop hypotheses from correlated observations • Test hypotheses and achieve consensus among tech

teams• Turn validated hypotheses into conclusions

Interpreting• Observations:

• “I observe that…”; no evaluation at this point!• Correlations:

• “Comparing graph A to graph B…” – relate observations• Hypotheses:

• “It appears as though…” – test with extended team; corroborate w/other information (anecdotal observations, manual tests)

• Conclusions:• “From observations a, b, c, corroborated by d, I conclude

that…”

Monitor resources liberally, provide (and annotate!) graphs selectively: which resources tell the main story?

Assessing• Objective: Turn conclusions into

recommendations• Tie conclusions to test objectives – were objectives

met?• Determine remediation options at appropriate level –

business, middleware, application, infrastructure, network

• Perform agreed-to remediation, Re-test

Assessing• Recommendations:

• Should be specific and actionable at a business or technical level

• Should be reviewed (and if possible, supported) by the teams that need to perform the actions

• Should quantify benefit, if possible cost, and risk

• Final outcome is management’s judgment, not yours

Step 1: *DO NOT* press “Print” of tool’s default Report

•Who is your audience? •Why do they want to see 50 graphs and 20 tables? What will they be able to see?

•Data + Analysis = INFORMATION

Reporting

•Step 2: Understand What is Important•What did you learn? Study your results, look for correlations.

•What are the 3 things you need to convey?•What info is needed to support these 3 things?•Discuss findings with technical team members: “What does this look like to you?”

Reporting

•Step 3: So, What is Important?•Prepare three paragraph email summary

•Prepare a 30 second Elevator Summary for when someone asks you about the testing

•More consume these than any test report

•Get feedback

Reporting

•Step 4: Preparing Final Report•Your primary audience is usually executive sponsors and the business.

•Write the Summary for them:• Language, Acronyms, and Jargon • Level of Detail• Correlation to business objectives

Reporting

•Step 5: Audience (cont.)•Rich Technical Detail within:

•Observations, including selected graphs• Include Feedback from Technical Team•Conclusions •Recommendations

Reporting

Reporting•Step 6: Present!

• Remember, no one is going to read the report.• Gather audience: executive, business, and technical.

• Present your results: Help shape the narrative. Explain the risks. Earn your keep.

• Call to action! Recommend solutions

…Think “CAVIAR”C ollectingA ggregating V isualizing I nterpretingA ssessingR eporting

For Actionable Results…

Thanks for Attending!

@ericproegler