the 80/20 rule: top purepath problem patterns of 2015! · the 80/20 rule: top purepath problem...
TRANSCRIPT
1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Andreas Grabner, Performance Advocate
The 80/20 Rule: Top PurePathProblem Patterns of 2015!
2 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Where do your
Stories come
from?
3 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
#1: Real Life & Real User Stories
4 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
#2: http://bit.ly/sharepurepath
5 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
#3: http://bit.ly/onlineperfclinic
20%
80%
8 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE8
Frontend PerformanceWe are getting FATer!
11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Make F12 or Browser Agent your friend!
12 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Online Tools for you!
Dynatrace Performance Center
www.dynatrace.com
PageSpeed Insights
developers.google.com/speed
13 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE13
Frontend AvailabilityBack to Basics Please!
17 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Online Services for you: Is it down right now?
18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Online Services for you: Outage Analyzer
19 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE19
Backend PerformanceThe Usual Suspects
• Symptoms
• HTML takes between 60 and 120s to render
• High GC Time
• Developer Assumptions
• Bad GC Tuning
• Probably bad Database Performance as rendering was simple
• Result: 2 Years of Finger pointing between Dev and DBA
Project: Online Room Reservation System
Developers built own monitoring
void roomreservationReport(int officeId)
{
long startTime = System.currentTimeMillis();
Object data = loadDataForOffice(officeId);
long dataLoadTime = System.currentTimeMillis() - startTime;
generateReport(data, officeId);
}
Result:
Avg. Data Load Time: 45s!
DB Tool says:
Avg. SQL Query: <1ms!
#1: Loading too much data24889! Calls to the
Database API!
High CPU and High Memory Usage to keep all
data in Memory
#2: On individual connections 12444!individual
connections
Classical N+1 Query Problem
Individual SQL really <1ms
#3: Putting all data in temp Hashtable
Lots of time spent in
Hashtable.get
Called from their Entity Objects
• … you know what code is doing
• Challenge the developers
• Explore Tools that “might seem” out of your league!
• Built-In Database Analysis Tools
• “Logging” options of Frameworks such as Hibernate, …
• JMX, Perf Counters, … of your Application Servers
• Performance Tracing Tools: Dynatrace, NewRelic, AppDynamics, …
Lessons Learned – Don’t Assume …
26 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE26
Backend PerformancePools and Queues
Online Banking: Slow Balance Check
1.69m (=101s!) To Check Balance!
87% spent in IIS 600! SQL Executions
#1 Time really spent in IIS?
Tip: Elapsed Time tells us WHEN a Method was executed!
Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET
Tip: Thread# gives us insight on Thread Queues / Switches
#2 What about these SQL Executions?
Finding: EVERY SQL statement is executed on
ITS OWN Connection!
Tip: Look at “GetConnection”
#2 SQL Executions! continued …
#1: Same SQL is executed 67! times
#2: NO PREPARATIONbecause everything
executed on new Connection
Lessons Learned!
ASP.NET Worker Thread Pool Sizing!
DB Connection PoolsMore Efficient SQL
32 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE32
Backend Performance(Micro)Services
33 @Dynatrace
26.7sExecution Time 33! Calls to the
same Web Service
171! SQL Queries through LINQ by this Web Service – request
similar data for each call
Architecture Violation: Direct access to DB instead from frontend logic
34 @Dynatrace
21671! Calls to Oracle
3136! Calls to H2 mostly executed on async background
threads
33! Different connections used
DB Exceptions on both Databases
DB Exceptions on both Databases
40! internal Web Service Calls that
do all these DB Updates
35 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE35
Tips & Tricks
36 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Web Server Health
Do we have enough WORKER THREADS on the
Web Server?
Sanity Check on BANDWIDTH requirements per Request
Can we handle the LOAD and BANDWIDTH requirements?
37 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: App Server Health over TimeHow is the LOAD DISTRIBUTION
on that JVM/CLR?
Any MEMORY ISSUES?
Do we have enough WORKER THREADS?
Do we have enough CPU?
38 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Host Health over Time
Do we have ENOUGH RESOURCES on these hosts?
Which processes are consuming these
resources? Need to RE-DEPLOY processes?
39 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Layer Breakdown over Time
With increasing load: Which LAYER doesn’t SCALE?
40 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Exceptions and Log Messages
How are # of EXCEPTIONSevolving over time?
How many SEVERE LOG messages to we write in relation to Exceptions?
41 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Failed Transactions
Are more TRANSACTIONS FAILING (HTTP 5xx, 4xx, …)
under heavier load?
42 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Database Activity
Do we see increased in AVG # of SQL Executions over Time?
Do TOTAL # of SQL Executions increase with load? Shouldn’t
it flatten due to CACHES?
43 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Database History Dashboard
How many SQL Statements are PREPARED?
What’s the overall Execution Time of different SQL Types (SELECT, INSERT, DELETE, …)
44 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: DB Connection Pool UtilizationDo we have enough DB
CONNECTIONS per pool?
45 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Group-By + Show All Options
46 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Tip: Advanced Filtering
47 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Memory, Thread, CPU Diagnostics
48 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
Server Timeline Dashlet (Community Supported)
49 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015
• http://bit.ly/onlineperfclinic
• http://bit.ly/dttutorials
• http://bit.ly/sharepurepath
• http://answers.dynatrace.com
• http://apmu.dynatrace.com
• Follow me: @grabnerandi
More Resources
50 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE