from zero to performance hero in minutes - agile testing days 2014 potsdam

Performance Clinic Workshop

From Zero to Performance HeroAndreas Grabner (@grabnerandi)

agrabner@dynatrace.comhttp://bit.ly/atd2014challenge

Why Applications Fail and/or are Slow!

Performance areas we cover today

• Frontend

• Backend

• Deployment

Frontend

http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/

#Overloaded Web Pages• #1: Too many resources loaded on web page

– Images: can be “sprited”– CSS and JS: can be merged

• #2: Large Content– Images: do you need high-res? Compress!– CSS and JS: Minify, remove comments, …

• #3: 3rd Party Content– Slow or too much

• #4: AJAX to the MAX!– Too many AJAX Calls requesting too much data

Examples on Overloaded

This is a heavy page

This is a light page

http://apmblog.compuware.com/2011/11/29/5-things-to-learn-from-jc-penney-and-other-strong-black-friday-and-cyber-monday-performers/

softdrink.com during SuperBowl

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

http://apmblog.compuware.com/2014/01/31/technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/

Fifa.com during Worldcup

http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

Kia.com during SuperBowl

Ad

on

air

GoDaddy.com during SuperBowl

1h before SuperBowl KickOff

1h after Game ended

Kia vs GoDaddy: The Facts!

http://apmblog.compuware.com/2014/02/19/dns-tcp-and-size-application-performance-best-practices-of-super-bowl-advertisers/

3rd Party Content

Do you really need all bells and whistles?

# of Domains # of Resources Total Bytes DNS [ms] Connect [ms]

With Third Party Content

26 176 2856 Kb 1286,82 1176,09

Without Third Party Content

2 59 897 Kb 0,91 22,25

Too heavy AJAX/JavaScript

1.1s on my IE 10 to

execute magicSpanLinks()

The each loop calls this block of JavaScript for every span node

759 span nodes are processed

by the anonymous function

It adds the dynamically generated link and removes the

old spanWe can see all the DOM

Modifications and how this sums up in execution time

3rd Party JavaScript Problems

Slow 3rd Party

Libraries

Slow jQuery

Lookukps

15 Minute Sanity Check – Live Demo

• Dynatrace Performance Test Center– http://www.dynatrace.com/en_us/application-performance-

management/products/performance-center.html

• Dynatrace Browser Agent (formerly AJAX Edition)– http://apmblog.compuware.com/2014/07/01/can-monitor-web-

performance-free/

• Fiddler – simulate other browsers– http://www.telerik.com/fiddler

• PerfMap – HeatMap for a Website– https://github.com/zeman/perfmap

Summary – WPO Best Practices• Additional Blog Posts

– http://apmblog.compuware.com/2013/12/02/the-terrible-website-performance-mistakes-of-mobile-shopping-sites-in-2013/

– http://apmblog.compuware.com/2011/11/29/5-things-to-learn-from-jc-penney-and-other-strong-black-friday-and-cyber-monday-performers/

– http://apmblog.compuware.com/2010/08/25/top-10-client-side-performance-problems-in-web-2-0/

• Recommended Books from Steve Souders covering things like– Make fewer HTTP Requests

– Proper Cache Settings

– Optimize/Compress Content

– Use CDNs

– Watch out for 3rd Parties

Tooling Support• http://bit.ly/dttrial• http://ajax.dynatrace.com• http://yslow.org• https://developers.google.com/speed/pagespeed• http://www.webpagetest.org/• http://www.sitespeed.io/• http://www.showslow.org• http://www.telerik.com/fiddler• https://github.com/zeman/perfmap• https://chrome.google.com/webstore/detail/user-agent-switcher-for-

c/djflhoibgkdhkhhcedjiklpkjnoahfmg

Hands-On

Backend Performance

http://bit.ly/atd2014challenge@grabnerandi

Bad Architectural Decisions#1: Database Access

N+1 Query Problem

Loading Too Much Data

Connection Leaks

#2: External Frameworks

Bad Configuration of O/R Mappers, e.g: Hibernate

Worked well in Sample App Attitude

#3: Excessive Logging & Exceptions

Debug Logging turned on

Using outdated logging libraries

Exception Overload

#4: Memory Leaks

Keeping objects for too long

Bad Cache Implementations

#5: Bad Coding

Synchronization to Death

High on CPU

Examples

Database Access

Database: N+1 Query Problem

Querying too much DataAll of these requests with large size have the same problem in common:

DB Access

24889 Calls to the Database!

Tomcat needs to process all this data!SideEffect: High Memory Usage to process

data -> results in high GC

High GC is not the problem. It is just the symptom of too much data

loaded!

Too many connections 12444 individual connections to execute 12444 individual SQL

Statements

Classical N+1 Query Problem. The same SQL is executed many times

with different WHERE ClauseOptimize this by only calling it once

with a better WHERE Clause

Individual executions are fast. But VOLUME is

killing you

Filtering Data in Memory instead of DB

Most lookups are done by getRoomCapacity

Assumption: All the data loaded ends up in Hashtable. The lookups are then very expensive because these Tables

are so huge

External Frameworks

Non Optimized Hibernate

http://apmblog.compuware.com/2014/04/23/database-access-quality-metrics-for-your-continuous-delivery-pipeline/

Non Optimized Telerik Controls

http://apmblog.compuware.com/2014/04/03/database-access-patterns-gone-wild-inside-telerik-sharepoint-and-asp-net/

Logging

Too much Logging#1: Top Problem: log4j.callAppenders

#2: Most of logging done from fillDetail method

#3: Doing “DEBUG” log output: Is this necessary?

Exceptions vs. Log Messages

http://apmblog.compuware.com/2014/04/01/dont-trust-your-log-files-how-and-why-to-monitor-all-exceptions/

Exception Logging Overload

Tomcat logAbandoned=true flag causes many exceptions objects to be created

http://apmblog.compuware.com/2012/08/01/top-performance-mistakes-when-moving-from-test-to-production-excessive-logging/

Exception Performance Overhead

http://apmblog.compuware.com/2014/04/01/dont-trust-your-log-files-how-and-why-to-monitor-all-exceptions/

Memory

Oracle JDBC Driver BugEach of the 10 JVMs per Host consumes up to 4.1GB until they crash (41GB per Host)

It is a reoccurring pattern over months

http://apmblog.compuware.com/2014/01/07/hunting-a-oracle-jdbc-memory-leak-crashing-an-80jvm-websphere-cluster/

No Object Cleanup Code!

http://apmblog.compuware.com/2014/02/26/memory-leaks-load-balancing-and-deployment-settings-testing-lessons-learned-from-the-ecommerce-industry/

Bad Coding

Slow Custom RegEx

http://apmblog.compuware.com/2014/10/16/15-minutes-spent-optimizing-performance-save-millions-lost-revenue/

Slow Content Rendering

Rendering Methods take very long!

Intializing GlyphLayout takes very long

These slow rendering methods are called very frequently!

Synchronization

http://apmblog.compuware.com/2013/09/24/100-performance-overhead-by-websphere-activity-log-when-dev-is-not-aware-of-settings-in-production/

15 Minute Sanity Check – Live Demo

• Dynatrace

Summary – Backend Best Practices• Additional Blog Posts

– http://apmblog.compuware.com/2013/04/10/top-8-application-performance-landmines/

– http://apmblog.compuware.com/2010/06/15/top-10-performance-problems-taken-from-zappos-monster-and-co/

• Online Java Enterprise Performance Book: http://javabook.compuware.com

• Key Takeaways– Educate Developers

– Understand Frameworks you are using

Hands-On

Deployment

Common Deployment Mistakes• Missing Resource Files

– Many HTTP 4xx– Many HTTP 3xx Redirects -> overhead!

• Missing Configuration Files– Web Server Access Rules -> Leads to HTTP 4xx– Web Server -> App Server: Connection & Thread Pools

• Bad Modules or Configuration Problems– Leading to bad requests and overhead– Rewrite and Redirect Modules: Long chains of redirects

• 3rd Party: – CDN Configuration Issues leads to outdated content or HTTP 4xx– Slow 3rd Party calls impact performance

• Delivery Problems– Web Site Up – but not available to the outside world

Missing Resource Files

http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/

Wrong Access Right Configuration

http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/

Bad Connection Pool Configuration

http://apmblog.compuware.com/2014/02/04/when-it-really-is-the-database-to-blame-for-bad-performance-a-story-about-slow-statements-and-resulting-connection-pool-issues/

Bad Web Server Modules

http://apmblog.compuware.com/2012/08/07/top-performance-mistakes-when-moving-from-test-to-production-deployment-mistakes/

CDN Configuration Problems

http://apmblog.compuware.com/2014/01/31/technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/

Monitor your CDNs

http://apmblog.compuware.com/2014/03/20/when-cdns-and-ssl-bring-down-your-site-lessons-learned-from-doritos-and-esurance-during-the-super-bowl/

Monitor your other 3rd Parties

http://apmblog.compuware.com/2011/11/21/ecommerce-business-impact-of-3rd-party-address-validation-service/

www.outageanalyzer.com

15 Minute Sanity Check – Live Demo

• Dynatrace Application Monitoring– http://www.dynatrace.com/en/products/application-

monitoring.html

• Dynatrace Synthetic Monitoring– http://www.dynatrace.com/en/products/synthetic-

monitoring.html

• Outage Analyzer– www.outageanalyzer.com

Tooling Support

• http://bit.ly/atd2014challenge

• http://www.outageanalyzer.com

• http://blog.dynatrace.com

• http://de.slideshare.net/grabnerandi

• @grabnerandi

Hands-On