all the leaves aren’t brown - nyougnyoug.org/presentations/2013/winter/ezell-nyoug-2013...all the...
TRANSCRIPT
Chuck Ezell
Senior Applications Tuner, datAvail
All The Leaves Aren’t Brown
Many Ways to Profile Your Application Code
12/19/2013 2www.datavail.com
• Value in Profiling: the When, What & Why
• Profiling & Profilers: the right tool for the right job
• Server & Client Profiling Tools
• Summary
Agenda
12/19/2013 3www.datavail.com
Value in Profiling
• 80% of unplanned outages are due to ill-planned changes made by operations or developers
• 60% of availability and performance errors are the result of mis-configurations
• Looking Ahead: through 2015 80% of outages impacting mission-critical services will be caused by people and process issues. More that 50% will be caused by change/configuration/release integration and hand-off issues.
• 80% of incidents are caused by changes made to the IT environment including application code
Source: Causes of Failure in Web Applications, Soila Pertet and Priya Narasimhan December 2005, Parallel Data Laboratory Carnegie Mellon University, Pittsburgh, PA 15213-3890
12/19/2013 4www.datavail.com
Tools Compared
• How hard is it to Setup and Manage?
• Is it instrumented, a sampler or something else?
• What will it profile? Resources, Threading, Memory, Transactional Views, Single focused or Multi-faceted?
• What is the output? Will it keep history? How long?
• Is it useful as a proactive or reactive solution?
• Does it provide any analysis?
• Is it expensive?
Questions We Have to Ask:
12/19/2013 5www.datavail.com
Common Java Problems
1. Frequent or Long Garbage Collection App is spending too much time in GC
Causes periods of extreme slowness across entire app
2. Application runs out of Heap Memory (Memory Leak) App runs out of memory
Causes the application to crash
3. Thread Contention Thread locks on a resource for too long.
Causes app to slow or lock and can impact entire app
4. Resource Contention (Pooled Resources) Not enough instances of a pooled resource (e.g. DB connections)
Causes extreme slowness in app, can be wide spread or specific to certain transactions.
12/19/2013 6www.datavail.com
Profiling Tools
Just to name a few…
jstat/jstack/jmap
12/19/2013 7www.datavail.com
Server Profiling Tools: jstack, jmap jstat
• Jstat, jstack & jmap– Requires java 1.6 and up
– Only runs in the terminal
– Probably already installed
– Easy to use
– Does not retain any history unless you redirect it to a file
– No analysis provided just raw data
– Provides running threads & memory footprint
– Can show locked threads
– Does not require a kill -9 to the JVM for a dump
– Can be great in a pinch
Setup: None
jstat/jstack/jmap
12/19/2013 8www.datavail.com
Server Profiling Tools: jstack, jmap jstat
You will need java 1.6 or higher to utilize these tools fully.
To check your java version just enter “java -version” at the terminal.
12/19/2013 9www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Check your running JVMs by doing a quick “ps –ef | grep –Xmx”
12/19/2013 10www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Using “jps” is much easier.
Using the -l argument will give you the full package name. With this you can grep out JVMs.
12/19/2013 11www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Pass process id to jstackUsing the –l argument will give you locked threads
jstack -l 3439
12/19/2013 12www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Running a “ps -eL” and grep for the JVM process id
You get the lightweight process ids
What about long running threads or thread contention?
12/19/2013 13www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Convert the binary lightweight thread id to hex. Then grep the output of jstack.
You can find the lightweight process id for the thread in the thread stack after converting it into hex
12/19/2013 14www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Passing the same JVM process id to jmap will give you the memory footprintfor this point and time.
jmap -heap 3439
But how would we monitor the Garbage Collection (ongoing memory growth/shrink) ?
12/19/2013 15www.datavail.com
Server Profiling Tools: jstack, jmap jstat
Passing the same JVM process id to jstat will give you the garbage collection
jstat –gc 22174
jstat –gcoldcapacity –t 22174
12/19/2013 16www.datavail.com
Server Profiling Tools: jstack, jmap jstat
jstat –gcnew -t 22174 1000 10
1. Not a great proactive solution.2. Great tools for reactive fire fighting.3. Doesn’t maintain history or reporting.4. Output is essentially on screen and snapshot in time.5. Will help answer questions and identify real problems now.6. No Application Context & is not correlated with End User.
12/19/2013 17www.datavail.com
Server Profiling Tools: Visual VM
• VisualVM / Oracle Mission Control– Both are free.
– Profiler much like the depricated jConsole
– Collects classes, threads, GC, CPU and more…
– Capture Heap Dumps & Browse them
– Also built into NetBeans IDE (my IDE of choice)
– Has a plug-in architecture
Setup: Easy
12/19/2013 18www.datavail.com
Server Profiling Tools: VisualVM
You’ll need JMX enabled
JMX = Java Management Extensions
• Open a terminal and do a quick “ps –ef | grep jmxremote”
• Look for the JVM Process ID• Then look for the port it’s using for the JVM you want to monitor• Might or might not need authentication
12/19/2013 19www.datavail.com
Server Profiling Tools: VisualVM
12/19/2013 20www.datavail.com
Server Profiling Tools: VisualVM
12/19/2013 21www.datavail.com
Server Profiling Tools: VisualVM
JMX Console & JVM Browser
12/19/2013 22www.datavail.com
Server Profiling Tools: VisualVM
Flight Recorder
12/19/2013 23www.datavail.com
Server Profiling Tools: VisualVM
Flight Recorder
12/19/2013 24www.datavail.com
Server Profiling Tools: VisualVM
Flight Recorder
1. Not a great proactive solution.2. Great visual tool for reactive fire fighting.3. Maintains history in the form of independent snapshots.4. Output is essentially on screen but can be called back up with
the saved snapshots.5. Will help answer questions and identify real problems now.6. Is not correlated with End User Experience.7. No transactional views. Hard to determine application context.8. Does require JMXREMOTE settings to be added to JVMs. In
addition to security & ports to attach remotely.
12/19/2013 25www.datavail.com
Server Profiling Tools: Splunk
• Splunk– There are often multiple JVMs and Apache instances running– Often times we miss issues in the logs because there’s too much to grep,
wouldn’t know what to look for or don’t have the time– Any log or flat file data can be forwarded– Monitor code base for changes, network logs for drops, client machines– The best results come from the best input so collect everything– Create monitoring alerts, dashboards, configure transaction monitoring– Plugin architecture for many different tools (BigIP, Hadoop, MySQL, etc…)– Limited by time & creativity
Splunk Forwarders On Each Server or write directly to server syslog.
Data from the log is collected and forwarded on to Splunk where it’s indexed and acted on.
Setup: Moderate to Easy
12/19/2013 26www.datavail.com
Server Profiling Tools: Splunk
• sql_plan_timestamp="09-30-2013 02:51:01", database=OADEV, plan_hash_value=1649080325, sql_id=696c56fyx1rwn, operation="INDEX", options=FULL SCAN, object_name=GL_SETS_OF_BOOKS_U2, object_owner=GL, optimizer=, cost=1, cardinality=1, cpu_cost=7521, io_cost=1, time=1, filter_predicates="HOI2"."ORG_INFORMATION1"=TO_CHAR("GSOB"."SET_OF_BOOKS_ID")"
• sql_plan_timestamp="09-30-2013 02:51:01", database=OADEV, plan_hash_value=2300147657, sql_id=36dx1vwfqmn2j, operation="TABLE ACCESS", options=FULL, object_name=FND_PRODUCT_GROUPS, object_owner=APPLSYS, optimizer=, cost=2, cardinality=1, cpu_cost=7271, io_cost=2, time=1, filter_predicates="sql_plan_timestamp="09-30-2013 02:51:01", database=OADEV, plan_hash_value=171240807, sql_id=7m21y56326kha, operation="INDEX", options=FULL SCAN, object_name=GL_SETS_OF_BOOKS_U2, object_owner=GL, optimizer=, cost=1, cardinality=1, cpu_cost=7521, io_cost=1, time=1, filter_predicates="HOI2"."ORG_INFORMATION1"=TO_CHAR("GSOB"."SET_OF_BOOKS_ID")"
SELECT 'sql_plan_timestamp=""'|| to_char(timestamp,'mm-dd-yyyy hh24:mi:ss') || '", database='||b.name||
', plan_hash_value='||plan_hash_value||', sql_id='||sql_id||', operation="'||operation||
'", options='||options||', object_name='||object_name||', object_owner='||object_owner||
', optimizer='||optimizer||', cost='||cost||', cardinality='||cardinality||', cpu_cost='||cpu_cost||
', io_cost='||io_cost||', time='||time||', filter_predicates="'||filter_predicates||'"' as sql_plan_info
FROM V$SQL_PLAN a, v$database b
where (options like '%SKIP%' OR options like '%FULL%' OR options like '%PUSH%')--OR options like '%RANGE%')
and object_owner != 'SYS'
and timestamp > (sysdate - 1)
order by timestamp asc;
We start with a basic SQL Script. And format the output into key/value pairs.
Then add that SQL to a shell script and set cron to run it and produce the output.
12/19/2013 27www.datavail.com
Server Profiling Tools: Splunk
12/19/2013 28www.datavail.com
Server Profiling Tools: Splunk
Client Tier
Capture Opportunities
• Web Browser traffic
• Oracle Forms running in client side JVM
• Windows Event Logs
• CPU Load, Memory, Network Stats
• Active Directory Monitoring
• FND Diagnostics from Oracle Forms
12/19/2013 29www.datavail.com
Server Profiling Tools: Splunk
Application Tier
Capture Opportunities
• Oracle JVM – OACore
• JVM Threads
• JSP Request/Response
• Apache Requests (Access/Error)
• Forms Processes
• Linux/AIX Server Stats
ApplicationServers
Concurrent processing server
12/19/2013 30www.datavail.com
Server Profiling Tools: Splunk
Database Tier
Capture Opportunities
• Oracle Alert Logs
• User Session Data
• Concurrent Requests
• SQL IDs, SQL Profiles & Outlines
• SMON/PMON
• DBWR/LGWR
• Hot Database Objects
12/19/2013 31www.datavail.com
Server Profiling Tools: Splunk
12/19/2013 32www.datavail.com
Server Profiling Tools: Splunk
12/19/2013 33www.datavail.com
Server Profiling Tools: Splunk
12/19/2013 34www.datavail.com
Server Profiling Tools: Splunk
12/19/2013 35www.datavail.com
Server Profiling Tools: jmap/jstack
jstack & jmap can be setup to forward data to splunk….
12/19/2013 36www.datavail.com
Server Profiling Tools: jmap/jstack
1. Not out of the box solution – requires some planning and skill.2. Can be a great proactive solution.3. Great visual tool for reactive AND proactive fire fighting.4. Maintains history in the form of indexed data.5. Output can be anything: alerts, notifications, pdf reports, charts,
dashboards, html pages, etc…6. Will help answer questions and identify real problems now AND
provide data to research patterns over time.7. Does require setup and server to collect and maintain indexes.8. Can provide the greatest flexibility for building what you want if
you have vision, time, skill & creativity.9. It is priced reasonably well – based on the data indexed per day.
12/19/2013 37www.datavail.com
Server Profiling Tools: HP Diagnostics
• HP Diagnostics– Integrates will with LoadRunner
– Full Application/Server Diagnostics
– Java Profilers can be configured to save results
– Will capture details for analysis later
– Is a bit confusing to use
– Older architecture utilizing some client tools still
– Profilers Can be server intensive
– Instrumented
Setup: Involved
12/19/2013 38www.datavail.com
Server Profiling Tools: HP Diagnostics
12/19/2013 39www.datavail.com
Server Profiling Tools: HP Diagnostics
12/19/2013 40www.datavail.com
Server Profiling Tools: HP Diagnostics
1. Can be a great proactive solution.2. Great visual tool for reactive and proactive fire fighting.3. Maintains history in the form of snapshots and data.4. Output is saved snapshots and can be recalled from history.5. Will help answer questions and identify real problems now as
well as help identify patterns.6. Each java probe can be intensive (with more probes, etc…)7. You don’t know what you don’t know.8. Is not correlated with End User Experience.9. No transactional views. Hard to determine application context.10. Can be costly to purchase.
12/19/2013 41www.datavail.com
Server Profiling Tools: HP Diagnostics
• AppDynamics– Captures Full Transaction
– Provides Monitoring History
– Full Profiler down to threads and exception stacks
– Auto Detects other devices and services such as other DBs
– Employs powerful and efficient Java Byte Code Injection
– Full Splunk Cross Integration
– Fast Root Cause Analysis
– Alert monitoring & configurable SLAs
– Scheduled & Automatic Diagnostic Sessions
– Modern architecture (just plain pretty)
– Very low server resource consumption (Netflix runs thousands)
Setup: Easy
12/19/2013 42www.datavail.com
Server Profiling Tools: AppDynamicsDashboard – Runtime application architecture discovery, modeling and display
Number 2 on Gartner's 5 Dimensions of APM chart.
12/19/2013 43www.datavail.com
Server Profiling Tools: AppDynamics
Not instrumented – Auto detects all connections through Java Byte Code.
12/19/2013 44www.datavail.com
Server Profiling Tools: AppDynamics
User-defined transaction profiling. This provides application context that is intuitive.
Number 3 on Gartner's 5 Dimensions of APM chart.
12/19/2013 45www.datavail.com
Server Profiling Tools: AppDynamics
12/19/2013 46www.datavail.com
Server Profiling Tools: AppDynamics
Component deep-dive monitoring in application context. - full call stacks and find bottlenecks.
Number 4 on Gartner's 5 Dimensions of APM chart.
12/19/2013 47www.datavail.com
Server Profiling Tools: AppDynamics
12/19/2013 48www.datavail.com
Server Profiling Tools: AppDynamics
12/19/2013 49www.datavail.com
Server Profiling Tools: AppDynamics
12/19/2013 50www.datavail.com
Server Profiling Tools: AppDynamics
12/19/2013 51www.datavail.com
Server Profiling Tools: AppDynamics
Health Rule Notifications – JVM, SQL, Transaction Slowness, etc…
12/19/2013 52www.datavail.com
Server Profiling Tools: AppDynamics
12/19/2013 53www.datavail.com
Server Profiling Tools: AppDynamics
Monitor JVM Heap Segments and Ratios
12/19/2013 54www.datavail.com
Server Profiling Tools: AppDynamics
Number 1 on Gartner's 5 Dimensions of APM chart.
End-user experience monitoring
12/19/2013 55www.datavail.com
Server Profiling Tools: HP Diagnostics
1. Super Easy to configure and get started.2. Great visual tool for reactive and proactive fire fighting.3. Maintains full history in the form of diagnostic snapshots and
data.4. Output can be alerts, notifications, pdf reports, dashboards,
etc.5. Will help answer questions and identify real problems now as
well as help identify patterns.6. Very little overhead (often less than 2%).7. You don’t have to know what you don’t know. Plug and play.8. Can be costly to purchase if you’re running multiple JVMs.9. But there is a free version for single JVM setups.
12/19/2013 56www.datavail.com
Client Profiling Tools: Fiddler
• Fiddler (HTTP Request Profiler)
– Easy setup just download and launch
– Create, save & recreate http requests
– Investigate & Debug http header problems
– Capture or Create form submits and re-run them anytime
Setup: Easy
12/19/2013 57www.datavail.com
Client Profiling Tools: Fiddler
Review http request statistics
12/19/2013 58www.datavail.com
Client Profiling Tools: Fiddler
Capture and Manipulate http requests for testing purposes
12/19/2013 59www.datavail.com
Client Profiling Tools: Fiddler
Recreate http POST data requests for testing
12/19/2013 60www.datavail.com
Client Profiling Tools: Fiddler
Simulate connection speeds, disable cache, measure response time
12/19/2013 61www.datavail.com
Tools Compared
Threads MemoryTransaction
ViewsMaintains
HistoryProactive
MonitoringAnalysis Setup
Splunk If Logged If Logged If Logged YES YES No Moderate
HP Diagnostics
YES YES No YES YES Sort Of Involved
AppDynamics YES YES YES YES YES YES Easy
jstack/jmap YES YES NoRedirect
To Flat FileNo No Easy
VisualVM YES YES NoLimitedExports
YES Sort Of Easy
VisualStudio YES YES NoLimitedExports
No YES Easy
Fiddler No No YESLimitedExports
No No Easy
Firebug No YES YESLimitedExports
No No Easy
12/19/2013 62www.datavail.com
Questions?
Chuck [email protected]
478-714-1615Senior Applications Tuner, datAvail
• Profiling & Profilers: the right tool for the right job
• Server Profiling Tools
• Client Profiling Tools
• The Heat of the Moment: differentiating profiling tasks