mpi tuning with the intel® trace analyzer and collector cluster software and technologies software...
TRANSCRIPT
MPI Tuning with the Intel® Trace Analyzer
and Collector
Cluster Software and Technologies
Software & Solutions Group
2
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Agenda
Introduction
Basic Usage
Trace Collector in depth (lab exercises)
Trace Analyzer in depth (lab exercises)
3
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Introduction – What is Tracing?
• Record program execution• Program events such as function enter/exit, communication
• 1:1 protocol of the actual program execution• Sampling gathers statistical information
• Accurate data
• Easily get loads of data
4
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Trace Universe
Intel® TraceCollector
Application
TracefileIntel® Trace
Analyzer
5
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Components and Interaction
TracesSTF
Intel® Trace Collector Lib
API
Intel® Trace Collector Lib
itcpin instrument
Executable
Application
Compiler
Linker
Instrumented
Executable
pre-load
6
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Supported Programming Models
MPI: Message Passing
Languages: C/C++, Java*, Fortran
Multithreading: explicit threads - implicit threads (OpenMP)
Hybrid Models: MPI + OpenMP
7
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Supported Environments
Intel® Xeon® Processor
Intel 64 Bit Xeon Processor
Intel Itanium® Processor
Intel MPI, MPICH, SGI* MPT
Linux (RHEL, RHAS, SLES)
SGI* Altix*
Microsoft* Windows*
8
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Introduction:Intel® Trace Collector
Key features
Usage
9
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Trace Universe – Intel® Trace Collector
Intel® TraceCollector
Application
TracefileIntel® Trace
Analyzer
10
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Intel® Trace Collector - Overview
Event based approach
Low impact on application performance
Provides API to instrument user code
Trace optimized program runs
Analyzes communication layer (default)
11
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Event based approach
Event = time stamp + thread ID + description
Function entry/exit
Messages
Collective operations
Counter samples
12
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Strengths of Event-based Tracing
Predict detailed program behavior
Record exact sequence of program states – keep timing consistent
Collect information about exchange of messages: at what times and in which order
An event-based approach is able to detect temporal dependencies!
13
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Key Features
Catch all MPI events
Strong configuration mechanism• Filters, settings, features
Automatic source-code references
Instrumentation• Rich API• Binary instrumentation (itcpin)• Compiler based (-tcollect)
Fail-safe version
Comparison Feature
MPI Correctness Checking
14
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
How to use Intel® Trace Collector
Step 1: Generate instrumented binary re–link your application:• Using Intel MPI:mpicc –trace tst.o –o tst.inst
• Using any other supported MPI:mpicc –shlib tst.o $(LFLAGS) -lVT –lmpi –ldwarf –lelf –lnsl –lm –
o tst.inst
Step 2: Produce tracefile
run the instrumented binary for a representative amount of time (reduce initialization influences) on representative data (no corner cases)
Alternative 1: Instrument binary
itcpin –-run –- tst
Alternative 2: Preload the Trace Collector library
mpiexec –trace –n 2 ./tst
15
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Activity 1 - MPI Tracing with Trace Collector
Build two simple “hello world” parallel applications
Create and view the program traces
16
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Activity 2 – Trace File Generation
Objective: How to generate a tracefile.
This directory contains cpi.c which is a simple example program.
Re-link (or compile and link) with Intel® Trace Collector library.
Run again – check for console message on 'writing tracefile'.
Open tracefile with Intel® Trace Analyzer.
[traceanalyzer cpi_itc.stf]
17
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Introduction:Intel® Trace Analyzer in depth
Key features
Usage
18
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Trace Universe - Intel® Trace Analyzer
Intel® TraceCollector
Application
TracefileIntel® Trace
Analyzer
19
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Intel® Trace Analyzer - Overview
Enables the user to quickly focus at the appropriate level of detail Offline trace analysis (postmortem)
Use of hierarchical techniques to address scalability in function, time and processor–space
High–performance graphics, excellent zooming and filtering
20
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Views and Charts - I
What’s a Chart?
What Charts are provided?
What’s a View?
What’s the purpose of a View?
21
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Chart
A Chart is a numerical or graphical diagram
Chart
22
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Timelines: Event Timeline
Get impression of program structure
Display functions, messages and collective operations for each process/thread along time-axis
Retrieval of detailed event information
23
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Timelines: Qualitative Timeline
Find patterns and irregularities
Display attributes of functions, messages or collective operations as they occur for any process/thread
Retrieval of detailed event information
24
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Timelines: Quantitative Timeline
Get impression on parallelism and load balance
Show for every function how many threads/processes are currently executing it
25
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Profiles: Flat Function Profile
Statistics about functions
26
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Profiles: Call-Tree and Call-Graph
Function statistics including calling hierarchy
• Tree: call-stack
• Graph: calling dependencies
27
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Communication Profiles
Statistics about point-to-point or collective communication
Generic matrix supports grouping by several attributes in each dimensionSender, Receiver, Data volume per msg, Tag, Communicator, Type
Available attributesCount, Bytes transferred, Time, Transfer rate
28
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
View
Helps navigating through the trace data and keep orientation
Every View can contain several Charts
A View on a file is defined by a triplet of• time-span• set of threads• set of functions
All Charts follow changes to View (e.g. zooming)
Timelines are correctly aligned along time
29
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
View - zooming
30
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Flexibility of Views
Several Views can be opened (on the same or on different files)
Location, orientation and size of charts can easily be changed
Entire Views can and individual charts can be cloned and closed
Individual charts can be cloned in own View
31
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Activity 7 – Getting Started with Trace Analyzer
In this activity, you will use the Trace Analyzer tool to recreate a particular view.
32
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Activity 8 – Getting Started with Trace Analyzer
In this activity, you will use the Trace Analyzer tool to recreate a “tiled” view.
33
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Activity 9 – More Charts with Trace Analyzer
In this activity, you will use the Trace Analyzer to recreate more charts. This will familiarize you with many of the charts and features of the Trace Analyzer.
34
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Charting and Viewing
Start with
traces/ztomo/out_altixtest_ori.2.stf
Use Chart menu to open
• EventTimeline
• Quantitative Timeline
• Function Profile (Flat Profile)
• Function Profile (CallGraph)
35
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 4 - Results
36
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Charting and Viewing
Start with
traces/ztomo/out_altixtest_ori.2.stf
Use Chart menu to open• EventTimeline (zoom to very end (left))• Collective Operation Profile
Use Layout menu to get time line to left
Use Context menu in CollOpChart• Colums to show Communicator• Rows to show Processes
37
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 4b - Results
38
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Grouping and Aggregation
Allow analysis on different levels of detail by aggregating data upon group-definitions
Functions and threads can be grouped hierarchically• Function Groups and Thread Groups
Arbitrary nesting is supported• Functions/threads on the same level as groups• User can define his/her own groups
Aggregation is part of View-definition• All charts in a View adapt to requested grouping
• All charts support aggregation
39
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Aggregation Example
40
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 10 – Trace Analyzer Function Aggregation
In this activity, you will use the Trace Analyzer tool to aggregate functions to determine how much effect the library zlib has on overall performance.
41
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Tagging & Filtering
Help concentrating on relevant parts
Avoid getting lost in huge amounts of trace data
Define a set of interesting data • E.g. all occurrences of function x
• E.g. all messages with tag y on communicator z
Combine several filters:Intersection, Union, Complement
Apply it • Tagging: Highlight messages
• Filtering: Suppress all non-matching events
42
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Tagging Example
43
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Filtering Example
44
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 11 – Trace Analyzer Tagging
In this activity, you will use the tagging feature of Trace Analyzer and highlight a function to determine which is the most expensive caller to the function. You will also use tagging to look for an instrumented function “f” from an earlier Trace Collector exercise.
45
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 12 – Trace Analyzer Filtering
In this activity, you will use the Trace Analyzer tool to aggregate functions to determine how much effect the library zlib has on overall performance.
46
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Comparison Feature
Provides side-by-side comparison of any two trace files and/or regions
Ability to couple zoom and navigation keys
Addition of several comparison-specific charts, such as
• Function Profile
• Message Profile
• Collective Operations Profile
47
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Comparison Feature Optimized/non-optimized Poisson code
MPI reduced to 43% by optimization
48
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Comparison Feature cont’d
In comparison view, all charts like
• Function ProfileIn function profile, several settings are possible, e.g. the ‚B/A‘ setting
which shows factors of the second (B) as compared to the first (A) trace file
• Message Profile
• Collective Operations Profile
compare the two comparands.
49
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Understanding your code
Parallel Poisson Solver
Example of intuitive parallelization with disadvantageous communication pattern
50
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Partial Trace of Poisson Solver
51
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Understanding the problem
Blocking border exchange
Pn has blocks until communication between Pn+1 and Pn+2 was completed
Solution: Non blocking communication
52
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Non-Blocking Version
53
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Detecting Load Imbalance
Mandelbrot set
Example of intuitive parallelization with huge load imbalance
54
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Mandelbrot Example Trace
55
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Understanding Load Imbalance
Static partitioning assigns contiguous blocks of rows• Workload changes gradually with row-index!
What is the/a solution?• Use modulo operator• Use dynamic master/slave approach
56
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Understanding Load Imbalance
57
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 13 – Trace Analyzer Load Balance
In this activity, you will use the Function Profile Load balance feature of Trace Analyzer to see how well an application's calculations are spread across processors.
58
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Supplemental material
59
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 14 – Using Trace Collector to Generate Text-Based Statistics Files
In this activity, you will relink with the Trace Collector Library and set up a configuration file that causes the trace files produced to be readable by text based tools such as vi and awk, while keeping the size of trace files produced to a minimum.
Objective: How to generate a tracefile with only statistic information
Generate file my_vt_config
Set environment variable and build example
Run example and look at stats-file cpi_itc.prot
60
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 15 – Using STFTOOL to Consolidate stf Files for Analysis on Other Systems
In this activity, you will use the stftool to consolidate stf files for analysis on other systems. You will use this tool to create a single file containing all trace information. You will then use the tool to expand the single file back into the suite of related stf in another directory.
Objective: Learn about stftool, use single trace file to simplify transfer of storage
Note: Can degrade performance of ITA
Note: ITC config allows creation of single-STF
61
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Backup
62
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Instrumenting Functions
Generic API to instrument functions
Hierarchical grouping (classes)
Insert probes to
• declare function
• enter/leave the function
63
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
ITC API: Instrument a function
Allocate a handle for a class name (optionally)int VT_classdef( const char * classname,
int * classhandle )
classname: name of the class
returns handle (int) referencing this class
Allocate a handle for a stateint VT_funcdef( const char * symname,
int classhandle, int * statehandle )
symname: name of the symbol
classhandle: handle of the class (see above)
returns handle (int) referencing this function
64
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 3 – Instrument a Function: Trace Collector API
Add lines to the cpi.c source file to instrument a particular function in cpi.c.
Edit cpi.c to instrument function 'f' as of group 'Application'• Use VT_classdef() and VT_funcdef() to declare class/function• Use VT_enter() and VT_leave(), to record function begin and end
Objective: Use of Intel® Trace Collector API to instrument a function
Once instrumented and when the “Application Group” is ungrouped, the function can be seen in various charts in Trace Analyzer
Compile-link-run and view tracefile in Intel® Trace Analyzer• Find events 'f' in the timeline, in the function profile and in the call tree
display
65
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Solution – Use of API
#include <VT.h>[...]int application_class; double f( double a ){ static int f_state = 0; if( f_state == 0 ) { VT_funcdef( "f", application_class, &f_state ); } VT_enter( f_state, VT_NOSCL ); [...] VT_leave( VT_NOSCL ); return (result);}int main( int argc, char *argv[]) [...] MPI_Init(&argc,&argv); VT_classdef( "Application", &application_class );
66
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 4 – Instrument a Function Trace Collector API, including Source & Line Numbers
Add additional lines to the cpi.c source file to instrument a particular function in cpi.c so.
Once the code is instrumented, the resulting trace enaables us to drill down to the “f” function source and see line number information
67
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 5 – Instrument a Function Trace Collector API for for Selective Tracing
Add lines to the cpi.c source file to selectively turn tracing on and off, portions where tracing is performed in an application
68
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.
Lab 6 – Using Trace Collector API to Count Events
In this activity, you will add more lines of code to the cpi.c source file, re-compile and relink with the Trace Collector Library. The changes made will allow us to count events as we define them in the application.
In the example used, we define a counter, f_result, to count updates to the variable named 'result'
69
Copyright © 2008, Intel Corporation. All rights reserved.
MPI Tuning with Intel® Trace Analyzer & Intel® Trace Collector
Intel and the Intel logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States or other countries. *Other brands and names are the property of their respective owners.