Overview of DynTG

Adam Leko

UPC Group

HCS Research Laboratory

University of Florida

Color encoding key:

Blue: Information

Red: Negative note

Green: Positive note

2

Basic Information Name: DynTG Developer: Center for Applied Scientific Computing,

Lawrence Livermore National Laboratory Current Version:

(Unreleased) Website:

http://www.llnl.gov/CASC/tool_gear/ Contact:

Martin Schulz (schulzm@llnl.gov) John May (johnmay@llnl.gov) John Gyllenhaal (gyllen@llnl.gov)

3

Overview of DynTG Not yet publicly available yet

A release should be available “in the next few weeks” as of 5/26/2005 according to developers

Still nothing available Information in this overview based on [1]

DynTG brings together two existing tools: DPCL and Tool Gear Lets the user instrument their program by viewing source

code and choosing instrumentation points at runtime Uses Tool Gear to provide source browser and metric

display to user Uses DPCL to perform dynamic instrumentation at runtime

4

Related Project: DPCL DPCL – Dynamic Probe Class Library

[2] Based on DynInst from Paradyn Meant to provide performance tool

developers with a library that handles instrumenting distributed programs during runtime

Also has built-in security features Architecture of DPCL shown right

Analysis tool uses DPCL “super daemon” to establish secure connection to DPCL daemons on each host

DPCL daemons (one per user per host) use DynInst to perform instrumentation of running programs and handle sending data back to the analysis tool

Currently only works on Linux and AIX platforms

5

Related Project: Tool Gear Tool Gear – “Software infrastructure for developing performance analysis and debugging

tools for large scale parallel programs” [3] Basic architecture shown below

Tool Gear provides a “database” by which Performance tools give information to the tool gear viewers Viewers pass asynchronous events (such as instrumentation requests) to a tool

Architecture handles both post-mortem and runtime analysis But Tool Gear is much more useful for tools that perform online analysis (and hence DynTG)

However, only a few viewers that display metrics inline with source code are currently available Tool Gear is used in mpiP’s mpipview (see mpiP evaluation)

6

Related Project: Tool Gear (2) A good description of the purpose of Tool Gear

(from [1], pg. 5):

“Source code viewers are complex and difficult to implement, yet required by a large number of tools. The Tool Gear project was born based on this observation and provides a versatile framework for the development of interactive tools. As its core it provides a scalable source viewer, which can display both the source code itself as well as multiple performance metrics on a per source line basis. The performance data itself is stored inside Tool Gear using a database. Data can directly be added to this database from a collector using the Tool Gear communication library.”

7

DynTG Architecture DynTG merges DPCL and ToolGear’s architectures together All analysis, instrumentation, and display are performed while a program is running

8

DynTG Modules DynTG can load “modules” (shared objects) at startup that report

information back to the main viewer Similar to Dynaprof’s loadable probe interface (see Dynaprof review)

Each probe split into two pieces Probe module

Contains the DPCL data probes that should be inserted when the user requests a function to be instrumented

Should contain code that sends data to collector modules via DPCL’s Ais_send routine

Collector module Receives data from the DPCL daemons, analyzes it, and gives it to the Tool Gear

viewer

According to paper [1], much low-level functionality is already provided by DynTG “allows even unexperienced [sic] users not familiar with DPCL or Tool Gear

to develop new probe/collector modules”

9

Built-In Modules

Counter Counts how often a program executes a specific location

Timer Keeps track of how much time is spent at a location using

the UNIX gettimeofday call PMAPI

Provides direct access to IBM’s hardware counters Users specify which hardware counters they are interested

and specify priorities for each Module maps user’s request to available hardware

resources

10

DynTG GUI User interacts with Tool Gear

GUI to select what parts of their code they want instrumented (see right)

GUI loads modules and inserts probes as previously discussed

It is not clear from the paper how the tool handles starting programs

11

DynTG GUI (2)

12

DynTG Overhead In paper, authors found runtime overhead of DynTG to be

between 1.94% and 12.4% ([1], page 6) Used two applications from the ASC Purple Benchmark Codes

SMG2000 (60x60x60) sPPM (64x64x64)

They inserted single instrumentation action into a central location (executed around 4000 times per second)

Based on our own experience with DynInst, overhead is very low for everything but very lightweight functions However, overhead can easily grow to several hundred percent

or more for applications with functions that execute > 105 times per second

Normally these types of functions should be inlined, so not a major concern

13

References

[1] M. Schulz, J. May, J. Gyllenhaal. “DynTG: A Tool for

Interactive, Dynamic Instrumentation,” ICSS 2005,

Atlanta, GA, May 22-25, 2005.

[2] L. DeRose, T. Hoover Jr. “The Dynamic Probe Class Library – An Infrastructure for Developing

Instrumentation for Performance Tools,” IPDPS 2001, April 2001.

[3] Toolgear website: http://www.llnl.gov/CASC/tool_gear/