Linux shared library profiler implementation

Edward Ryabikov

Mark Zaslavskiy

Kirill Krinkin

Open Source & Linux Lab

13th FRUCT Conference, Petrozavodsk, April 24, 2013

Agenda • Project motivation and goals;

• Linux linking mechanisms and tools: o ELF format fundamentals;

o L

• “Non-invasive” shared objects profiling;

• ELFPERF design o Modules;

o Call redirection;

o Wrappers;

o Call tooling;

• Conclusions

2/ 29

Motivation • Popular Linux Profilers (gprof, gcov, GPT, Valgrind)

have restrictions:

o Need to recompile with special options

(gprof, gcov)

o Need to relink program with 3rd-party libraries

(GPT)

o Need to use special environment for profiling

(Valgrind)

o Very long execution;

o Need to use super-user rights

3/ 29

Project goals • System-wide ELF executables performance analysis

• Requirements: o Profile function calls without recompilation and relinking with 3rd-party

libraries;

o Profiling only given set of C/C++ functions from shared libraries in Linux

o Profiling both dynamically linked and dynamically loaded functions

o Profiling without creating of special environment

• Statistics: o number of calls

o Time inside functions

• Perform profiling on x86/x64 platforms

4/ 29

Linux Loader and Linker • Glibc:

o /lib/ld-linux.so

o /lib/libdl.so

• ELF (Executable Linux File)

o Format interpreter

o Symbols

o Dynamic loading calls

5/ 29

Dynamic Linking

anylib.so

anyfunc1()

anyfunc2()

anyfuncn()

…

exe

int main()

{

…

anyfunc1();

…

return 0;

}

ld-linux.so

_dl_fixup

_dl_profile_fixup

_dl_call_pltexit

LD_LIBRARY_PATH

1

2

3

6/ 29

ELF Interpretation

init and shared lib segments

.text segment (r/o)

.data segment (initialized r/w)

.bss segment (uninitialized r/w)

Process image ELF header

Program header table (required for executables)

.text section

.data section

.bss section

.symtab

.rel.text

.dynamic

.debug

Section header table (required for relocatables)

Executable object file

Virtual Address

0x080483e0

0x08048494

0x0804a010

0x0804a3b0

7/ 29

Dependencies • Linking dependency – one or more shared objects

which are explicitly referenced.

• Runtime linker:

o locates and loads the associated shared objects.

o Inspects dependencies of shared objects;

o Continue process to the deep;

Note: if symbols are referenced several times, only first

reference will be used.

8/ 29

Runtime linker (LD)

• LD (dynamic link editor) is a system “service” for

resolving dependencies and caching shared

objects;

• Responsibilities: o Loading shared objects

o Dependency management

o Caching

o [symbol] -> address translation

o Fixing/creating relocation tables

• Corresponded tools:

o ldd – prints dependency tree

o objdump – inspections for ELF binaries

o objcopy – manipulations with ELF binaries 9/ 29

Dynamic loading

• .dynamic segment contains a pointer to string table

• LD

o creates a scope list (set of libraries to be loaded)

o maps the library to the process address space.

o creates linked list of symbols

o Bind the symbols to function addresses

10/ 29

Symbol Table Structure

typedef struct

{ Elf32_Word st_name; /* Symbol name (string tbl index) */

Elf32_Addr st_value; /* Symbol value */

Elf32_Word st_size; /* Symbol size */

unsigned char st_info; /* Symbol type and binding */

unsigned char st_other; /* Symbol visibility */

Elf32_Section st_shndx; /* Section index */

} Elf32_Sym;

11/ 29

Offset tables in ELF

• Dynamic linker primarily uses two processor-specific

tables: o Global Offset Table (GOT)

o Procedure Linkage Table (PLT).

• Dynamic linkers support PIC Code through the GOT

in each shared library.

12/ 29

Tags in .dynamic

• In the .dynamic section items have next tags:

o DT_STRTAB – address of the string table.

o DT_SYMTAB -- address of the symbol table.

o DT_NEEDED -- string table offset of a null-terminated string,

giving the name of a needed library. The offset is an index

into the table recorded in the DT_STRTAB entry.

o DT_HASH: This element holds the address of the symbol

hash table which refers to the symbol table referenced by

the DT_SYMTAB element.

13/ 29

Relocation Process

• Loading dependencies

• Do relocations for each object

• Connecting symbolic references with symbolic

definitions.

• Ex: when a program calls a function, the associated

call instruction must transfer control to the proper

destination address at execution. Relocatable files

must have information how to modify their section

contents.

14/ 29

Dynamic Loading

libdl.so

dlopen

dlsym

dlclose

dlopen

dlsym

dlerror

anylib.so

anyfunc1()

anyfunc2()

anyfuncn()

…

?

ld-linux.so

exe

int main()

{

void* p =

dlopen(“anylib.so”, RTLD_LAZY);

…

void(*f)() =

dlsym(p, “anyfunc1”);

…

f();

…

dlclose(p);

…

}

LD

_LIB

RARY_PATH

1

2

3

15/ 29

“Noninvasive” profiling

o Profiler can not be implemented in the program code

o Profiling should be performed at well-defined points of

function calls

o Profiling process should not corrupt the algorithm of

profiled application

o Profiling process should use minimum amount of system

resources

o Results of profiling should be as accurate as possible

16/ 29

Possible implementations

• Infiltration into the symbol relocation process

• Modification of Linux dynamic linker

(ld-linux.so)

• Modification of dynamic loading library (libdl.so)

17/ 29

Components of ElfPerf

• Shared library libelfperf.so o Call redirection and function wrapping mechanisms

o Collecting of calls statistics

o Memory management

• Modified dynamic linker (ld-linux.so) o Uses libelfperf.so for profiling of dynamically linked functions

o Displays the results of profiling

• Modified dynamic loading library (libdl.so) o Uses libelfperf.so for profiling of dynamically loaded functions

18/ 29

Call Redirection • Calls redirection mechanism (Redirector) is a set of

machine codes for the next assembly instructions:

• All they do is:

o Save address of profiled function in program stack

o Jump to wrapper-function

push $fcnPtr

jmp $wrapper_addr

19/ 29

Working scheme of Redirectors

push $fcnPtr

jmp $wrapper_addr

Redirector

…

any_function

push %ebp

mov %esp,%ebp

fcnPtr

leave

ret

…

void(*f)() = dlsym(p, “any_function”);

…

f();

…

Code 1

2

3

4

Wrapper wrapper_addr

...

call preProfile

...

ret

wrapper_rp:

...

call postProfile

...

5

20/ 29

Features of Redirectors

• Each redirector is created individually for each

profiled function

• Redirectors are placed into data segment of

process virtual memory

• The operating system allows to mark these memory

areas as executable

21/ 29

Functions Wrapping mechanism

• Function Wrapping mechanism (or Wrapper) is a

function that does next things:

o Takes control from redirector

o Performs pre-profile operations

o Performs replacement of return address

o Performs jump into profiled function

o Again takes control after the work of profiled

function

o Performs post-profile operations

o Returns to caller

22/ 29

Wrappers idea

Wrapper Params

Return value

Function Start time

End time

jmp

ret

Function address

Return address

Context

23/ 29

Wrappers at work

Function

leave

retl

...

push %ebp

mov %esp,%ebp

void wrapper()

{

// push %ebp

// movl %esp, %ebp

asm volatile (

"popl %ebp"

"pushal"

"pushl 32(%esp)"

"pushl 40(%esp)"

"call preProfile"

"addl $8, %esp"

"movl $wrapper_rp, 36(%esp)"

"popal"

"retl"

);

asm volatile (

"wrapper_rp:"

"pushl $0"

"pushal"

"call postProfile"

"movl %eax, 32(%esp)"

"popal"

"retl"

);

}

Wrapper

Context

Stack Base Higher Memory

Addresses …

fcn param #n

…

fcn param #0

ret address

fcn address

old %EBP

%ESP

%EBP %EAX, %EBX, %ECX, %EDX, %ESI, %EDI

fcn address

ret address

ret address

fcn address

start time

wrapper_rp

old %EBP %EBP

$0

%EAX, %EBX, %ECX, %EDX, %ESI, %EDI

end time

ret address

Registers

old %EAX

%EBX old

%ECX old

%EDX old

%EDI old

%ESI old

new

new

new

new

new

new

rval

oldr

oldr

rval

oldr

oldr

raddr

new

new

new

new

new

24/ 29

Features

• Wrapper doesn’t corrupt stack content

• Wrapper exists in a single copy for all functions in

each profiler implementation (x86 or x64)

• Saving/Restoring of registers’ state allows to escape

of uncontrollable changes in the program state

• Such implementation of wrapper mechanism allows

to profile wide set of C/C++ functions

25/ 29

Components interaction

elfperf-ld-linux.so

_dl_fixup

_dl_profile_fixup

_dl_call_pltexit

LD_PRELOAD elfperf-libdl.so

dlclose

dlopen

dlsym

dlerror

libelfperf.so

Wrapper

Redirectors

Statistics

LD

_LIB

RARY_PATH

Memory management

Shared memory

ElfPerf Storage

Function Infos

Function Statistics

26/ 29

Conclusion

• «Light» profiler based on «patched» ld-linux.so and

libdl.so

• Support of profiling for C/C++ functions from shared

libraries

(including libs compiled with –fomit-frame-pointer

flag)

• Collecting of information about number and total

duration of function calls

• Support of both x86 and x64 platforms

27/ 29

Links

• Project repository: https://github.com/OSLL/elfperf

• Full description of project (RUS): http://redmine.osll.spb.ru/projects/epat/wiki/_Architecture_description_

28/ 29

Questions & Answers

Edward Ryabikov, edward.ryabikov@gmail.com

Mark Zaslavskiy, mark.zaslavskiy@gmail.com

Kirill Krinkin kirill.krinkin@fruct.org

Open Source & Linux Lab, http://osll.ru

29/ 29