/

Full-stackIntrospectionCrashCourse

Operations

Why listen to Theo?

• I wrote a book(okay it has nothing to do with performance)

• I lead the performance and scalabilitypractice at OmniTI

• It’s my job to “un”break things... yes really.

• I build high-performance software.

• Why not?

• Listening never hurt, you can judge worth on your own.

Full-stack?

• How do you see the whole stack, all at the same time?

• DTrace

• What’s the “whole” stack? LAMP?

• Linux... err, not yet. (Mac OS X, FreeBSD, Solaris)

• Apache... yup.

• MySQL? Sure, but PostgreSQL is cooler.

• PHP... and Perl... and Java... and Ruby... and...

• anything with compiled with frames: C, C++, etc.

What is DTrace?

• DTrace is a comprehensive dynamic tracing frameworkcreated by Sun Microsystems

• DTrace allow you to write scripts that can consume output from various traceable points in a running system:

• kernel function boundaries,

• syscalls, I/O, network, mib, nfs, scheduler, vm, etc.

• arbitrary points in user-land code (entry/return/op codes)

• user-space tracepoints

• all at the same time

What is DTrace really?

• It is an Oracle of knowledge. A peephole into the “real world.”

• The quality of the answers it provides are limited bythe quality of the questions it is asked.

• DTrace can be a bit frustrating:

• Syntax is simple: Awk/C

• Concept is simple: watch stuff.

• Implementation is safe: designed “production safe”

• Learning to ask good questions is hard.

DTrace is a rabbit hole

Think Scientifically

• Use the scientific method (no, I’m not rehashing lower education)

• Be rigorous and analytical:

• Use your gut to define the hypothesis.

• Use repeatable tests to gather evidence to prove or disprove(PLEASE: don’t change more than one thing at a time)

• Use experience to draw conclusions.

• The proof of th' pudding's seen i' the eating.

Three Aproaches

• SAFEST:

• Start closest to the user; repeat the problem; track it down.

• SORCERY:

• Start at the problem; solve it; prove you were right.

• Requires hypothesizing that the problem is what it is.

• EXPERIENCED:

• Start in the middle; look for symptoms; form a targeted hypothesis

• Apply Sorcery

DTrace only works for real

• Huh?

• Contrived problems have contrived solutions.

• DTrace is input driven:

• garbage in, garbage out

• (contrived in, contrived out)

• Retracing the steps of a previous introspection exercise is boring

• I’ll do my best to make it interesting... maybe you can help.

DTrace incantations

Clatto

Verata

N...

Necktie...

Nickel...

It's an "N" word,it's definitely an "N" word!

Thanks!

• Sun & DTrace(esp. Bryan Cantrill, Adam Leventhal and Mike Shapiro)

• Apple & FreeBSD for porting DTrace

• Paul Fox, for serious effort toward porting it to Linux:http://www.crisp.demon.co.uk/blog/ftp://crisp.dynalias.com/pub/release/website/dtrace

• OmniTIFor being an awesome place to work...We’re hiring.