it802 bruning

Download It802 bruning

Post on 06-May-2015

2.611 views

Category:

Documents

2 download

Embed Size (px)

TRANSCRIPT

  • 1.Using DTrace to InstrumentYour System Max Bruning,Training Director, Joyent Wednesday, March 26, 14

2. Topics Covered DTrace What it does. Uses How Instruments uses it Examples Wednesday, March 26, 14 3. Tools Used in This Talk DTrace Allows runtime observability into entire software stack Instruments performance, analysis, and testing tool for dynamically tracing and proling OS X and iOS code Many of the instruments tools use DTrace (some use llvm) Wednesday, March 26, 14 4. What is DTrace? Tool that allows one to dynamically instrument code from application level and into the kernel. Can be used safely on production systems. Uses: Performance Analysis Debugging Code coverage Find out what is happening in your software Available on OS X, SmartOS, illumos, *BSD, Solaris and derivatives. Useful with C, Objective-C, C++, as well as higher level languages (java, perl, node, python, etc.). Wednesday, March 26, 14 5. Terminology Probe - An instrumentation point in the code Dynamic and Static probes are provided, and new ones can be added A probe is specied by a 4-tuple: provider:module:function:probename{action} Action - Executed when a probe res Predicate - Optional boolean to determine whether or not to execute the action Example: syscall::read:entry/pid == 713/{trace();} Wednesday, March 26, 14 6. Some Simple Examples Show system calls made by a running process # dtrace -n syscall:::entry/pid==26442/{} dtrace: description 'syscall:::entry' matched 234 probes CPU ID FUNCTION:NAME 1 10157 write:entry 1 10155 read:entry 4 10155 read:entry ... # dtrace -n 'syscall:::entry/execname == "Safari"/{@[probefunc]=count();}' dtrace: description 'syscall:::entry' matched 456 probes (^C) ... mmap 183 psynch_mutexdrop 470 psynch_mutexwait 470 kevent64 6149 workq_kernreturn 6326 Count system calls made by a running safari process Wednesday, March 26, 14 7. An Example Measuring System Call Latencysystime.d #!/usr/sbin/dtrace -s #pragma D option quiet syscall:::entry /execname == "node"/ { self->ts = timestamp; } syscall:::return /self->ts/ { @[probefunc] = quantize(timestamp - self->ts); self->ts = 0; } END { ! printa("SYSCALL NSECS # OF OCCURANCESn%s%@lxn", @); } Wednesday, March 26, 14 8. An Example Measuring System Call Latency (Continued)# ./systime.d ... SYSCALL NSECS # OF OCCURANCES read value ------------- Distribution ------------- count 1024 | 0 2048 |@@@@@@@@@@@@@ 3 4096 |@@@@@@@@@ 2 8192 |@@@@@@@@@ 2 16384 |@@@@ 1 32768 | 0 ... 33554432 | 0 67108864 | 0 134217728 |@@@@ 1 268435456 | 0 ... Wednesday, March 26, 14 9. DTrace Providers for OS X # dtrace -l | grep -v PROVIDER | awk '{print $2}' | sort -u | sed -e 's/[0-9].*//' | uniq | paste - - - - | awk '{ printf "%20-s %20-s %20-s %20-sn", $1, $2, $3, $4 }' AppContainer AppSandbox Cocoa_Autorelease Cocoa_Layout CoreData CoreImage CoreText HALC_IO HALS_Engine HALS_IO HALS_System JavaScriptCore MetadataFramework MetadataImporter MobileDevice NSApplication NSTrackingArea NSXPCConnection NSXPCListener NSXPCProxy PDEProbes PMTiming PackageKit PrintCore QLSeamlessClosing QLSeamlessOpening QLThumbnail QuartzComposer QuickTimeX RawCamera RemoteViewServices SoftwareUpdate Suggestions afc cache codesign dtrace fbt garbage_collection io ip ldap_rb_stats lockstat mach_trap magmalloc mds node objc_runtime odprobes opencl_api opencl_cpu perl pid plockstat proc profile sched sdt security_debug security_exception security_machserver securityd syscall syspolicy tcp vminfo Providers not in grey are OS X specic (?) More providers may exist, and more can be added Wednesday, March 26, 14 10. Instruments Performance, analysis, and testing tool for dynamically tracing and proling OS X and iOS code Many of the instruments tools use DTrace (some use llvm) Instruments provides a GUI for implementing your own instruments using DTrace Wednesday, March 26, 14 11. Instruments and DTrace Instruments using Dtrace Sudden termination, Carbon Events, Core Data Saves/ Fetches/Faults, Cocoa Events/Layout, File Locks/Attributes, File Activity, Directory I/O, Garbage Collection, Reads/ Writes, Shared Memory, Process, Custom Instruments Instruments that Could be implemented using DTrace Time Proler, Network Activity Monitor, Sampler, Memory/ Disk/CPU/Activity Monitor,Thread States,VM Operations, System Calls, Scheduling Wednesday, March 26, 14 12. Example Instrument - File Activity Wednesday, March 26, 14 13. Example Instrument - File Activity (Continued) Edit as needed and save or cancel. Wednesday, March 26, 14 14. Example Instrument - File Activity (Continued) Wednesday, March 26, 14 15. Example Instrument - DTrace Script$ cat fileact.d #!/bin/bash # # To run this script, please type the following command while in the same # directory (or specify the full path rather than './'): # # sudo ./fileact.d -o dtrace_output.txt # # After the output file has been generated (here, it's dtrace_output.txt but you # may call it whatever you like), call up the trace document where # this script was exported from and choose the "DTrace Data Import..." option # from the "File" pulldown menu to import that data. # TOOLCHAINS=default /usr/sbin/dtrace $* -C -Z -s /dev/stdin prof1.out ^C bash-3.2$ ls -l prof1.out -rw-r--r-- 1 max staff 33694762 Mar 21 11:20 prof1.out bash-3.2$ less prof1.out Timestamp Depth Process Hot Frame Caller 2014 Mar 21 11:20:04 6 dtrace ffffff80290d41b2 ffffff80291f3ae1 mach_kernel`0xffffff80290d4060+0x152 mach_kernel`VNOP_IOCTL+0x150 mach_kernel`0xffffff80291f39b0+0x131 mach_kernel`fo_ioctl+0x43 mach_kernel`ioctl+0x466 mach_kernel`unix_syscall64+0x1f3 mach_kernel`hndl_unix_scall64+0x16 2014 Mar 21 11:20:04 1 kernel_task ffffff80290dd76e 0 mach_kernel`machine_idle+0x1de mach_kernel`call_continuation+0x17 ... Wednesday, March 26, 14 20. DTrace Scripts Available on OS X$ man -k dtrace bitesize.d(1m) - analyse disk I/O size by process. Uses DTrace cpuwalk.d(1m) - Measure which CPUs a process runs on. Uses DTrace creatbyproc.d(1m) - snoop creat()s by process name. Uses DTrace dappprof(1m) - profile user and lib function usage. Uses DTrace dapptrace(1m) - trace user and library function usage. Uses DTrace diskhits(1m) - disk access by file offset. Uses DTrace dispqlen.d(1m) - dispatcher queue length by CPU. Uses DTrace dtrace(1) - generic front-end to the DTrace facility dtruss(1m) - process syscall details. Uses DTrace errinfo(1m) - print errno for syscall fails. Uses DTrace execsnoop(1m) - snoop new process execution. Uses DTrace fddist(1m) - file descriptor usage distributions. Uses DTrace filebyproc.d(1m) - snoop opens by process name. Uses DTrace hotspot.d(1m) - print disk event by location. Uses DTrace ... 42 scripts available, and more can be found at http://www.dtracebook.com/index.php/Main_Page Wednesday, March 26, 14 21. Example - dtruss Trace system calls $ sudo dtruss -n Mail PID/THRD SYSCALL(args) = return 3113/0x47af: select(0x60, 0x61000020C600, 0x61000020C680, 0x0, 0x0) = 1 0 3113/0x47af: recvfrom(0x22, 0x1132ADAA0, 0x100) = 1 0 3113/0x1a3059: kevent64(0x3, 0x11623DE18, 0x1) = 1 0 3113/0x46b0: workq_kernreturn(0x20, 0x0, 0x1) = 0 0 3113/0x46b0: kevent64(0x3, 0x0, 0x0) = 1 0 3113/0x179187: fcntl(0x8, 0x8, 0x117BB8900) = 0 0 3113/0x179187: fcntl(0x8, 0x8, 0x117BB8920) = 0 0 3113/0x179187: fcntl(0x8, 0x8, 0x117BB8A60) = 0 0 3113/0x179187: pread(0x6, "r0", 0x1000, 0xC8C0000) = 4096 0 3113/0x179187: workq_kernreturn(0x20, 0x0, 0x1) = 0 0 3113/0x179187: pwrite(0x7, "0", 0x1018, 0x3DFC90) = 4120 0 3113/0x179187: pwrite(0x7, "0", 0x1018, 0x3E0CA8) = 4120 0 3113/0x1a28d7: open("/Users/max/Library/Mail/V2/IMAP-max@joyent.com@imap.gmail.com/ [Gmail].mbox/All Mail.mbox/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/Data/8/7/1/Messages/ 178177.emlx0", 0x8000, 0x0) = 46 0 ... Wednesday, March 26, 14 22. Example - iosnoop Trace Disk I/O Events $ sudo iosnoop -a STRTIME DEVICE MAJ MIN UID PID D BLOCK SIZE PATHNAME ARGS 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761912 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761952 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761976 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762576 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762840 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132763224 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132763320 8192 ??/Messages/177915.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132764944 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132765360 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132765472 12288 ??/Messages/177917.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767000 8192 ??/Messages/177918.emlx Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767152 16384 ??/Messages/177924.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762608 12288 ??/Messages/177913.emlx Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132764920 16384 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767016 8192 ??/Messages/177919.emlx.tmp Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132762896 12288 ??/Messages/177914.emlx Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132765384 4096 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132766960 20480 ??/MailData/Envelope Index-wal Mail0 2014 Mar 21 15:33:00 ?? 1 2 0 128 R 24160560 4096 ??/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/ reverseDirectoryStore mds_stores0 ... Wednesday, March 26, 14 23. Example - opensnoop Trace open(2) calls on les $ sudo opensnoop -a TIME STRTIME UID PID FD ERR PATH ARGS 276685505483 2014 Mar 21 15:39:59 0 128 5 0 . mds_stores0 276685505676 2014 Mar 21 15:39:59 0 128 5 0 . mds_stores0 276688258585 2014 Mar 21 15:40:02 502 17327 10 0 /Users/max/

Recommended

View more >