docker 활용법: dumpdocker

49
25 September 2014 Docker, 기발한 활용법 삼성SDS Perfect ICT 연구회 정영훈 수석, 박재화 수석 dumpdocker - 자동 덤프 분석

Upload: jaehwa-park

Post on 24-May-2015

3.824 views

Category:

Technology


0 download

DESCRIPTION

Presentation @ RED HAT FORUM SEOUL 2014

TRANSCRIPT

Page 1: Docker 활용법: dumpdocker

25 September 2014

Docker, 그 기발한 활용법

삼성SDS Perfect ICT 연구회

정영훈 수석, 박재화 수석

dumpdocker - 자동 덤프 분석 툴

Page 2: Docker 활용법: dumpdocker

Perfect ICT 연구회 삼성SDS의 기술 연구 모임

-세계 최고 수준의 장애 분석 전문가 양성을 목표로 함

-오픈 소스의 데이터센터 내 활용에 대한 연구 수행

-ICT 인프라 문제 해결 방법론 연구 수행

Perfect RCA + OPEN ICT = Perfect ICT

Page 3: Docker 활용법: dumpdocker

목차

• dumpdocker 탄생 비화: 속 터지는 덤프 분석

• 덤프 분석에 docker 를 활용해 보자

• First Pass 덤프 자동 분석

• dumpdocker 프로젝트 로드맵

Page 4: Docker 활용법: dumpdocker

dumpdocker 탄생 비화:

속 터지는 덤프 분석

Page 5: Docker 활용법: dumpdocker

리눅스 서버와 오픈 소스 사용의 급격한 증가

Page 6: Docker 활용법: dumpdocker

오픈 소스 운영 환경의 특징

다양한 하드웨어, 리눅스 배포판 유사한 기능의 많은 소프트웨어

Page 7: Docker 활용법: dumpdocker

KVM Crash 덤프 분석 사례 KVM crash 발생, 덤프 분석 요청 CRASH

조금만 기다려 주세요. D+05

헉 덤프 사이즈가 60GB 가 넘는 군요. 분석하는데 시간이 많이 들 듯. D+04

좀 더 기다려 주세요. D+06

조금만 더 기다려 주세요. D+07

NOT YET!!! D+10

NOT YET!!! size 가 너무 커서 분석에 시간이 걸려요. D+14

NOT YET. SORRY.... D+20

일단 완전치는 않으나 분석 결과 임. (gdb) info stack

#0 0x00007fd427c03445 in ?? () from gdbenv/lib/x86_64-linux-gnu/libc.so.6

#1 0x00007fd42ebd3970 in ?? ()

#2 0x202730203d3d2074 in ?? ()

#3 0x0a2e64656c696166 in ?? ()

#4 0x0000000000000000 in ?? ()

D+25

로그 좀 수집해 주세요...? D+30

Workaround 제공 D+32

Thank you ?

Page 8: Docker 활용법: dumpdocker

무엇이 문제인가?

분석 환경 구축이 어렵다. 덤프 분석에 대해 운영자도 (기본 정도는) 알면 좋다. First Pass 분석 수준

(gdb) bt

#0 0x00007fd427c03445 in ?? ()

#1 0x00007fd427c06bab in ?? ()

#2 0x00007fd42bbc48c3 in ?? ()

#3 0x00007fd427d4717d in ?? ()

#4 0x00007fd42bb0d070 in ?? () at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/memory.c:316

...

(gdb) info shared

From To Syms Read Shared Object Library

0x00007fd42b4ea1e0 0x00007fd42b4ed6b8 Yes (*) ./libs/lib/x86_64-linux-gnu/librt.so.1

0x00007fd42b20c990 0x00007fd42b279728 Yes (*) ./libs/lib/x86_64-linux-gnu/libglib-2.0.so.0

...

0x00007fd428e9d640 0x00007fd428f1c3f8 Yes (*) ./libs/usr/lib/x86_64-linux-gnu/libgnutls.so.26

0x00007fd428bf7f40 0x00007fd428c46e98 Yes (*) ./libs/usr/lib/x86_64-linux-gnu/libSDL-1.2.so.0

No /usr/lib/x86_64-linux-gnu/libX11.so.6

0x00007fd4286b85a0 0x00007fd4286b8754 Yes (*) ./libs/lib/x86_64-linux-gnu/libaio.so.1

...

좀 SMART 하게 이 문제들을 해결 할 수 없을까?

Page 9: Docker 활용법: dumpdocker

libraries

/usr/bin/kvm

gdb

dumpdb

덤프 분석에

를 활용해보자!

Page 10: Docker 활용법: dumpdocker
Page 11: Docker 활용법: dumpdocker

dumpdocker 아키텍처

GitHub docker Repository

docker-1 분석 환경 구축 자동 덤프 분석

docker-2 덤프 KDB

Search engine

Dump Knowledge DB

First Pass 분석보고서

알려진 문제 검색

dumpdocker

Page 12: Docker 활용법: dumpdocker

docker is an open-source engine that

automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.

Page 13: Docker 활용법: dumpdocker

Server

Host OS

Hypervisor

Guest OS Guest OS

Bins/Libs Bins/Libs

App A App B Virtual Machine

가상화된 어플리케이션에는

어플리케이션 자체와 필요한

바이너리/라이브러리 뿐만 아니라

운영체제 전체가 포함됨

Page 14: Docker 활용법: dumpdocker

Server

Host OS

Docker Engine

Bins/Libs Bins/Libs

App A App B

Docker

도커 엔진 컨테이너는

어플리케이션과 꼭 필요한

바이너리/라이브러리만 갖춤

운영체제 상의 사용자 공간에

격리된 프로세스로 구동되기

때문에 가상 머신의 이점과 함께

이동성이 훨씬 뛰어남

Page 15: Docker 활용법: dumpdocker

docker 퀵인트로: 이미지 다운로드

[root@rhel7sds ~/docker] docker pull rhel7

Pulling repository rhel7

e1f5733f050b: Pulling image (latest) from rhel7, endpoint:

http://registry.access.redhat.com/v1/

e1f5733f050b: Download complete

Page 16: Docker 활용법: dumpdocker

docker 퀵인트로: 이미지 리스트

[root@rhel7sds ~/docker] docker images

REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

fedora rawhide 58faa899733f 6 days ago 371.7 MB

fedora 20 ea7d6801c538 6 days ago 373.7 MB

fedora heisenbug ea7d6801c538 6 days ago 373.7 MB

fedora latest ea7d6801c538 6 days ago 373.7 MB

fedora 21 58faa899733f 6 days ago 371.7 MB

redhat/rhel latest 463ff6be4238 3 weeks ago 140.2 MB

rhel7 0 463ff6be4238 3 weeks ago 140.2 MB

rhel7 0-21 463ff6be4238 3 weeks ago 140.2 MB

rhel7 latest 463ff6be4238 3 weeks ago 140.2 MB

redhat/rhel7 0 463ff6be4238 3 weeks ago 140.2 MB

redhat/rhel7 0-21 463ff6be4238 3 weeks ago 140.2 MB

redhat/rhel7 latest 463ff6be4238 3 weeks ago 140.2 MB

Page 17: Docker 활용법: dumpdocker

docker 퀵인트로: 실행

[root@rhel7sds ~/docker] cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.0 (Maipo)

[root@rhel7sds ~/docker] docker run -i -t fedora /bin/bash

bash-4.2# cat /etc/redhat-release

Fedora release 20 (Heisenbug)

bash-4.2# exit

[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash

bash-4.3# cat /etc/redhat-release

Fedora release 21 (Twenty One)

bash-4.3# exit

Page 18: Docker 활용법: dumpdocker

docker 퀵인트로: 실행

[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash

bash-4.3# cat /etc/redhat-release

Fedora release 21 (Twenty One)

bash-4.3#

[root@rhel7sds ~] docker ps -l

CONTAINER ID IMAGE COMMAND CREATED

STATUS PORTS NAMES

4043a778e43b fedora:21 /bin/bash 30 seconds ago

Up 29 seconds evil_davinci4

[root@rhel7sds ~] ps aux

root 3642 0.0 0.3 229340 5044 pts/13 Sl+ 13:27 0:00 docker run -i

-t fedora:21 /bin/bash

root 3650 0.0 0.1 18056 1832 pts/1 Ss+ 13:27 0:00 /bin/bash

Page 19: Docker 활용법: dumpdocker

docker 퀵인트로: 실행

[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash

bash-4.3# cat /etc/redhat-release

Fedora release 21 (Twenty One)

bash-4.3# ls

bin dev home lib64 mnt proc run selinux sys usr

boot etc lib media opt root sbin srv tmp var

bash-4.3# gdb

bash: gdb: command not found

bash-4.3# yum install gdb

...

bash-4.3# gdb

GNU gdb (GDB) Fedora 7.8-20.fc21

Copyright (C) 2012 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

For help, type “help”.

Type “apropos word” to search for commands related to “word”.

(gdb) q

bash-4.3#

[root@rhel7sds ~] docker ps -l

CONTAINER ID IMAGE COMMAND CREATED STATUS

PORTS NAMES

4043a778e43b fedora:21 /bin/bash 10 minutes ago Up 10 minutes

evil_davinci4

Page 20: Docker 활용법: dumpdocker

docker 퀵인트로: 실행

bash-4.3# exit

exit

[root@rhel7sds ~/docker] docker run -i -t fedora:21 /bin/bash

bash-4.3# gdb

bash: gdb: command not found

[root@rhel7sds ~] docker ps -l

CONTAINER ID IMAGE COMMAND CREATED

STATUS PORTS NAMES

4d9c95ab95c6 fedora:21 /bin/bash 6 seconds ago

Up 5 seconds desperate_pare2

Page 21: Docker 활용법: dumpdocker

docker 퀵인트로: 활용 사례

참고: http://blog.flux7.com/blogs/docker/8-ways-to-use-docker-in-the-real-world

Page 22: Docker 활용법: dumpdocker

docker 기반 덤프 분석 환경?

Page 23: Docker 활용법: dumpdocker

/

var usr tmp sbin root proc lib mnt home etc dev boot bin

share sbin local lib include bin

sbin src lib bin

log tmp spool

Page 24: Docker 활용법: dumpdocker

Server

Host OS

Docker Engine

Bins/Libs

App B

덤프가 발생한 서버의

어플리케이션과

바이너리 및 라이브러리로

docker 이미지 생성

Page 25: Docker 활용법: dumpdocker

dumpdocker

Page 26: Docker 활용법: dumpdocker

root@dumpenv:/dump# ls

core.2073 libs libs.tar

root@dumpenv:/dump# gdb ./libs/usr/sbin/libvirtd ./core.2073

GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7

...

Reading symbols from ./libs/usr/sbin/libvirtd...(no debugging symbols found)...done.

warning: exec file is newer than core file.

[New LWP 2074]

[New LWP 2075]

[New LWP 2076]

[New LWP 2077]

[New LWP 2078]

[New LWP 2079]

[New LWP 2080]

[New LWP 2081]

[New LWP 2082]

[New LWP 2083]

[New LWP 2073]

warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address (wrong

library or version mismatch?)

warning: Could not load shared library symbols for 65 libraries, e.g. /usr/lib64/libvirt-qemu.so.0.

Use the "info sharedlibrary" command to see the complete listing.

Do you need "set solib-search-path" or "set sysroot"?

Core was generated by `libvirtd --daemon'.

#0 0x00007f79d41105bc in ?? ()

(gdb)

덤프 분석: w/o dumpdocker

Page 27: Docker 활용법: dumpdocker

덤프 분석: w/o dumpdocker (gdb) info shared

From To Syms Read Shared Object Library

No /usr/lib64/libvirt-qemu.so.0

No /usr/lib64/libvirt.so.0

No /lib64/libblkid.so.1

No /usr/lib64/libnetcf.so.1

No /lib64/libudev.so.0

No /usr/lib64/libpciaccess.so.0

No /usr/lib64/libpcap.so.1

No /lib64/libcap-ng.so.0

No /usr/lib64/libyajl.so.1

No /lib64/libnl.so.1

No /lib64/libaudit.so.1

No /lib64/libdevmapper.so.1.02

No /usr/lib64/libxml2.so.2

No /usr/lib64/libnuma.so.1

No /usr/lib64/libavahi-common.so.3

No /usr/lib64/libavahi-client.so.3

...

...

(gdb)

(gdb) bt

#0 0x00007f79d41105bc in ?? ()

#1 0x0000000000000000 in ?? ()

Page 28: Docker 활용법: dumpdocker

덤프 분석: w/o dumpdocker (gdb) set sysroot /dump/libs

Reading symbols from /dump/libs/usr/lib64/libvirt-qemu.so.0...(no debugging symbols found)...done.

Loaded symbols for /dump/libs/usr/lib64/libvirt-qemu.so.0

...

(gdb) info shared

From To Syms Read Shared Object Library

0x00007f79d73c1a30 0x00007f79d73c20a8 Yes (*) /dump/libs/usr/lib64/libvirt-qemu.so.0

0x00007f79d6fc7b20 0x00007f79d7129148 Yes (*) /dump/libs/usr/lib64/libvirt.so.0

...

No /lib64/ld-linux-x86-64.so.2

0x00007f79d27f0e80 0x00007f79d281b618 Yes (*) /dump/libs/lib64/libsepol.so.1

...

No /usr/lib64/sasl2/libanonymous.so

No /usr/lib64/sasl2/libsasldb.so

No /lib64/libdb-4.7.so

No /usr/lib64/sasl2/libcrammd5.so

No /usr/lib64/sasl2/libdigestmd5.so

No /lib64/libnss_files.so.2

(*): Shared library is missing debugging information.

(gdb) bt

#0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 ()

from /dump/libs/lib64/libpthread.so.0

#1 0x00007f79d6fe58c6 in virCondWait () from /dump/libs/usr/lib64/libvirt.so.0

#2 0x00007f79d6fe5e93 in ?? ()

#3 0x00007f79d6fe56e9 in ?? ()

#4 0x00007f79d410c9d1 in ?? ()

#5 0x00007f79d3a52b5d in ?? ()

Page 29: Docker 활용법: dumpdocker

덤프 분석: w/ dumpdocker

[root@ubuntu1204 ~/dumpdocker]# dumpdocker /usr/sbin/libvirtd ./core.2073

tar: Removing leading `/' from member names

/sbin/

/sbin/consoletype

/sbin/lsinitrd

/sbin/blkdeactivate

/sbin/partx

/sbin/ldconfig

/sbin/pvck

...

[root@ubuntu1204 ~/dumpdocker]#

root@dumpenv:~# scp root@ubuntu1204:/ubuntu1204.libvirtd.201409121615.tar .

root@ubuntu1204's password:

ubuntu1204.libvirtd.201409121615.tar 0% 0 0.0KB/s --:-- ETA

ubuntu1204.libvirtd.201409121615.tar 0% 16MB 15.8MB/s 01:29 ETA

ubuntu1204.libvirtd.201409121615.tar 100% 97MB 10.1MB/s 02:21

root@dumpenv:~# cat ubuntu1204.libvirtd.201409121615.tar | docker import - ubuntu1204

4537fca38755100fbc0e2a96168f397df1d6f76f3829506a5af0da8a61e6a3fa

덤프 발생 서버:

덤프 분석 서버:

Page 30: Docker 활용법: dumpdocker

덤프 분석: w/ dumpdocker root@dumpenv:~# docker run -i -v /dump:/var/crash -t ubuntu1204 /bin/bash

bash-4.1# mkdir /tmp

bash-4.1# cd /var/crash

bash-4.1# ls

core.2073 libs libs.tar

bash-4.1# gdb /usr/sbin/libvirtd ./core.2073

GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7

Reading symbols from /usr/sbin/libvirtd... (no debugging symbols found)...done.

[New Thread 2074]

...

[New Thread 2073]

Loaded symbols for /lib64/libnss_files.so.2Core was generated by `libvirtd --daemon'.

#0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 ()

from /lib64/libpthread.so.0

(gdb) info shared

From To Syms Read Shared Object Library

0x00007f79d73c1a30 0x00007f79d73c20a8 Yes (*) /usr/lib64/libvirt-qemu.so.0

0x00007f79d6fc7b20 0x00007f79d7129148 Yes (*) /usr/lib64/libvirt.so.0

...

0x00007f79c200ea00 0x00007f79c2016698 Yes (*) /usr/lib64/sasl2/libdigestmd5.so

0x00007f79c14001f0 0x00007f79c1408648 Yes (*) /lib64/libnss_files.so.2

(*): Shared library is missing debugging information.

(gdb) bt

#0 0x00007f79d41105bc in pthread_cond_wait@@GLIBC_2.3.2 ()

from /lib64/libpthread.so.0

#1 0x00007f79d6fe58c6 in virCondWait () from /usr/lib64/libvirt.so.0

#2 0x00007f79d6fe5e93 in ?? () from /usr/lib64/libvirt.so.0

#3 0x00007f79d6fe56e9 in ?? () from /usr/lib64/libvirt.so.0

#4 0x00007f79d410c9d1 in start_thread () from /lib64/libpthread.so.0

#5 0x00007f79d3a52b5d in clone () from /lib64/libc.so.6

Page 31: Docker 활용법: dumpdocker

TODO

[dumpdocker]

- docker 이미지 생성 자동화 스크립트

- First Pass 덤프 분석 리포트 자동 생성

- 덤프 DB docker 생성

- 덤프 분석 사례 수집

Page 32: Docker 활용법: dumpdocker

운영 서버와 동일한 환경으로 분석 가능

고객 시스템 구성 정보 제외로 보안 이슈 제거

배포본 독립적인 분석/테스트 환경 구축

신속한 장애 분석 환경 구축

오픈소스 기반의 경제적 분석 환경

dumpdocker 덤프 분석 환경

Page 33: Docker 활용법: dumpdocker

First Pass 덤프 자동 분석

살살해~

Page 34: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

12가지 항목에 대해 자동으로 분석하여 결과 출력

===============================

First Pass Dump Analysis Report

===============================

version 1.0

Fri Sep 12 08:30:18 UTC 2014

Exec file name : /usr/bin/kvm

Core file name : CoreDump

Table of contents

=================

1. General information

2. Environment varialbes

3. Stacktrace

4. The failed frame

5. The information of the failed frame

6. Source code information

7. Assembly code

8. Register information

9. Virtual address space

10. Thread information

11. Shared library information

12. Full stacktrace

Page 35: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

일반 정보, Signal, 환경 변수

1. General information

======================

Core was generated by `/usr/bin/kvm -name instance-00018bf4 -S -M pc-1.2 -cpu

SandyBridge,+pdpe1gb,+os'.

Program terminated with signal 6, Aborted.

2. Environment variables

========================

/usr/bin/kvm

-name

instance-00018bf4

-S

-M

pc-1.2

-cpu

SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes6

4,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme

-enable-kvm

-m

64225

-smp

16,sockets=16,cores=1,threads=1

...

Page 36: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

완벽한 stacktrace!! dumpdocker 환경 복제 덕분

NOTE를 참조해서 고객께 친절한 설명 가능

3. Stacktrace

=============

#0 0x00007fd427c03445 in raise () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007fd427c06bab in abort () from /lib/x86_64-linux-gnu/libc.so.6

#2 0x00007fd427bfc10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#3 0x00007fd427bfc1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6

#4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-

swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932

#5 kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-

1.2.0+noroms/kvm-all.c:925

#6 0x00007fd42bb0b465 in kvm_irqchip_send_msi (s=0x7fd42e6dfa90, msg=...) at /build/qemu-kvm-

swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:1113

#7 0x00007fd42baed55b in kvm_apic_mem_write (opaque=<optimized out>, addr=24576,

data=<optimized out>, size=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-

1.2.0+noroms/hw/i386/../kvm/apic.c:161

...

NOTE:

##################################################################

The process called abort().

You should focus on the frame before calling abort().

And it is good to check the application's log file.

##################################################################

Page 37: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

문제 발생한 frame 에서 좀 더 자세한 정보 모으기

4. The failed frame #4

======================

#4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-

swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932

932 }

5. The information of the failed frame #4

=========================================

Stack frame at 0x7fff368b0780:

rip = 0x7fd42bb0ad88 in kvm_irqchip_commit_routes (/build/qemu-kvm-swljsd/qemu-kvm-

1.2.0+noroms/kvm-all.c:932); saved rip 0x7fd42bb0b465

inlined into frame 5, caller of frame at 0x7fff368b0770

source language c.

Arglist at unknown address.

Locals at unknown address, Previous frame's sp is 0x7fff368b0770

Saved registers:

rbx at 0x7fff368b0748, rbp at 0x7fff368b0750, r12 at 0x7fff368b0758, r13 at 0x7fff368b0760,

rip at 0x7fff368b0768

Page 38: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

소스 코드 경로 지정 시, 친절한 listing 까지

6. Source code information

==========================

Current source file is /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c

Compilation directory is /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/build/x86_64-softmmu

Located in /dump/src/qemu-kvm-1.2.0+noroms/kvm-all.c

Contains 2045 lines.

Source language is c.

Compiled with DWARF 2 debugging format.

Does not include preprocessor macro info.

Line 932 of "/build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c" starts at address

0x7fd42bb0ad63 <kvm_irqchip_commit_routes+67> and ends at 0x7fd42bb0ad67

<kvm_irqchip_commit_routes+71>.

927 int ret;

928

929 s->irq_routes->flags = 0;

930 ret = kvm_vm_ioctl(s, KVM_SET_GSI_ROUTING, s->irq_routes);

931 assert(ret == 0);

932 }

933

934 static void kvm_add_routing_entry(KVMState *s,

935 struct kvm_irq_routing_entry *entry)

936 {

Page 39: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

어셈블리 코드 확인, 해설

7. Assembly code

================

=> 0x4005e5 <convertint+28>: mov (%rax),%eax

0x4005e7 <convertint+30>: mov %eax,-0x4(%rbp)

0x4005ea <convertint+33>: mov -0x4(%rbp),%eax

0x4005ed <convertint+36>: pop %rbp

0x4005ee <convertint+37>: retq

0x4005ef: nop

0x4005f0 <__libc_csu_init>: mov %rbp,-0x28(%rsp)

0x4005f5 <__libc_csu_init+5>: mov %r12,-0x20(%rsp)

0x4005fa <__libc_csu_init+10>: lea 0x200823(%rip),%rbp # 0x600e24

0x400601 <__libc_csu_init+17>: lea 0x20081c(%rip),%r12 # 0x600e24

0x400608 <__libc_csu_init+24>: mov %r13,-0x18(%rsp)

0x40060d <__libc_csu_init+29>: mov %r14,-0x10(%rsp)

0x400612 <__libc_csu_init+34>: mov %r15,-0x8(%rsp)

Failed Instruction=> 0x4005e5 <convertint+28>: mov (%rax),%eax

NOTE:

##################################################################

It tried to load from the memory, the address saved in rax

and copy to the register, eax. It failed because the content

in rax register is not a valid address.

If you need to check assembly code, trace the rax not eax.

##################################################################

Page 40: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

레지스터 정보

8. Register information

=======================

rax 0x0 0

rbx 0xffffffea 4294967274

rcx 0xffffffffffffffff -1

rdx 0x6 6

rsi 0x6064 24676

rdi 0x6064 24676

rbp 0x7fd42ebd3770 0x7fd42ebd3770

rsp 0x7fff368b0770 0x7fff368b0770

r8 0xfefefefefefefeff -72340172838076673

r9 0x0 0

r10 0x8 8

r11 0x202 514

r12 0x7fd42e6e0c28 140549288758312

r13 0x3f0 1008

r14 0x7fd42bb0d070 140549242802288

r15 0x7fff368b0888 140734108469384

rip 0x7fd42bb0ad88 0x7fd42bb0ad88

eflags 0x202 [ IF ]

cs 0x33 51

ss 0x2b 43

ds 0x0 0

es 0x0 0

fs 0x0 0

gs 0x0 0

Page 41: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

Virtual address space 사용 현황

9. Virtual address space

========================

0x00007fc308000000 - 0x00007fc308021000 is load1 : 135168 bytes

0x00007fc308021000 - 0x00007fc308021000 is load2 : 0 bytes

0x00007fc30c000000 - 0x00007fc30c021000 is load3 : 135168 bytes

0x00007fc30c021000 - 0x00007fc30c021000 is load4 : 0 bytes

0x00007fc310000000 - 0x00007fc310021000 is load5 : 135168 bytes

0x00007fc310021000 - 0x00007fc310021000 is load6 : 0 bytes

0x00007fc314000000 - 0x00007fc314021000 is load7 : 135168 bytes

....

0x00007fd42be47000 - 0x00007fd42bf16000 is load691 : 847872 bytes

0x00007fd42bf16000 - 0x00007fd42bf46000 is load692 : 196608 bytes

0x00007fd42bf46000 - 0x00007fd42c7d2000 is load693 : 8962048 bytes

0x00007fd42e6bd000 - 0x00007fd42e853000 is load694 : 1662976 bytes

0x00007fd42e853000 - 0x00007fd42e863000 is load695 : 65536 bytes

0x00007fd42e863000 - 0x00007fd42e8a1000 is load696 : 253952 bytes

0x00007fd42e8a1000 - 0x00007fd42e8b1000 is load697 : 65536 bytes

0x00007fd42e8b1000 - 0x00007fd436958000 is load698 : 134901760 bytes

0x00007fff36890000 - 0x00007fff368b2000 is load699 : 139264 bytes

0x00007fff369f3000 - 0x00007fff369f4000 is load700 : 4096 bytes

0xffffffffff600000 - 0xffffffffff600000 is load701 : 0 bytes

==========================================================================

TOTAL : 68041953280 bytes

64889 MB

Page 42: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

Thread 정보

10. Thread information

======================

Id Target Id Frame

56 Thread 0x7fc35cff9700 (LWP 27483) 0x00007fd427cb8384 in pwritev64 () from /lib/x86_64-linux-

gnu/libc.so.6

55 Thread 0x7fc35e7fc700 (LWP 57393) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()

from /lib/x86_64-linux-gnu/libpthread.so.0

54 Thread 0x7fc37e7fc700 (LWP 57404) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()

from /lib/x86_64-linux-gnu/libpthread.so.0

53 Thread 0x7fc412bf2700 (LWP 57395) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()

from /lib/x86_64-linux-gnu/libpthread.so.0

52 Thread 0x7fc410be7700 (LWP 28584) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()

from /lib/x86_64-linux-gnu/libpthread.so.0

51 Thread 0x7fd3d6cdc700 (LWP 28585) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()

from /lib/x86_64-linux-gnu/libpthread.so.0

...

3 Thread 0x7fc4133f3700 (LWP 65017) 0x00007fd427f960fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()

from /lib/x86_64-linux-gnu/libpthread.so.0

2 Thread 0x7fd3dce4d700 (LWP 24695) 0x00007fd427f95d84 in pthread_cond_wait@@GLIBC_2.3.2 () from

/lib/x86_64-linux-gnu/libpthread.so.0

* 1 Thread 0x7fd42b8e67c0 (LWP 24676) 0x00007fd427c03445 in raise () from /lib/x86_64-linux-

gnu/libc.so.6

Number of threads: 56

Page 43: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

Shared object 현황

11. Shared library information

==============================

From To Syms Read Shared Object Library

0x00007fd42b4ea1e0 0x00007fd42b4ed6b8 Yes (*) /lib/x86_64-linux-gnu/librt.so.1

0x00007fd42b20c990 0x00007fd42b279728 Yes (*) /lib/x86_64-linux-gnu/libglib-2.0.so.0

0x00007fd42aff0e20 0x00007fd42aff16a8 Yes (*) /lib/x86_64-linux-gnu/libutil.so.1

0x00007fd42addbe90 0x00007fd42ade9de8 Yes (*) /usr/lib/librbd.so.1

0x00007fd42aa394a0 0x00007fd42ab6f188 Yes (*) /usr/lib/librados.so.2

...

0x00007fd3de494740 0x00007fd3de57d008 Yes (*) /lib/x86_64-linux-gnu/libcrypto.so.1.0.0

0x00007fd3fc0e5f00 0x00007fd3fc0e7dc8 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libplain.so

0x00007fd3de22ffc0 0x00007fd3de232458 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libcrammd5.so

0x00007fd3dc4407c0 0x00007fd3dc449058 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libdigestmd5.so

0x00007fd3dc23a200 0x00007fd3dc23cf78 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so

0x00007fd3d6cfbca0 0x00007fd3d6e162d8 Yes (*) /usr/lib/x86_64-linux-gnu/libdb-5.1.so

0x00007fd3dc034ed0 0x00007fd3dc036cc8 Yes (*) /usr/lib/x86_64-linux-gnu/sasl2/liblogin.so

(*): Shared library is missing debugging information.

Page 44: Docker 활용법: dumpdocker

First Pass 자동 덤프 분석보고서

Full stacktrace

12. Full stacktrace

===================

#0 0x00007fd427c03445 in raise () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#1 0x00007fd427c06bab in abort () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#2 0x00007fd427bfc10e in ?? () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#3 0x00007fd427bfc1b2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#4 0x00007fd42bb0ad88 in kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-

swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:932

No locals.

#5 kvm_irqchip_commit_routes (s=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/kvm-

all.c:925

ret = -22

#6 0x00007fd42bb0b465 in kvm_irqchip_send_msi (s=0x7fd42e6dfa90, msg=...) at /build/qemu-kvm-

swljsd/qemu-kvm-1.2.0+noroms/kvm-all.c:1113

virq = 1008

msi = {address_lo = 0, address_hi = 0, data = 0, flags = 0, pad = '\000' <repeats 15 times>}

route = 0x7fd42ebd3770

__PRETTY_FUNCTION__ = "kvm_irqchip_send_msi"

#7 0x00007fd42baed55b in kvm_apic_mem_write (opaque=<optimized out>, addr=24576, data=<optimized out>,

size=<optimized out>) at /build/qemu-kvm-swljsd/qemu-kvm-1.2.0+noroms/hw/i386/../kvm/apic.c:161

msg = {address = 24576, data = 16594}

ret = <optimized out>

Page 45: Docker 활용법: dumpdocker

libraries

/usr/bin/kvm

gdb

dumpdb

dumpdocker 프로젝트 로드맵

Page 46: Docker 활용법: dumpdocker

dumpdocker 아키텍처

GitHub docker Repository

docker-1 분석 환경 구축 자동 덤프 분석

docker-2 덤프 KDB

Search engine

Dump Knowledge DB

First Pass 분석보고서

알려진 문제 검색

dumpdocker

Page 47: Docker 활용법: dumpdocker

dumpdocker 로드맵

Dump KDB docker

docker 이미지 자동 생성

많은 dump 분석으로 풍부한 KDB 콘텐츠 확보

First pass 자동 덤프 분석

(完)

(完)

Search engine

Page 48: Docker 활용법: dumpdocker

오픈 소스 덤프 분석 함께 합시다.

정영훈 Firsts Pass Dump Analysis

[email protected]

박재화 dumpdocker core

[email protected]

이종화 dumpdocker (python) [email protected]

- dumpdocker image + core dump - E-mail 연락 - Free 분석, 정보 공유 - https://github.com/pjhwa/dumpdocker

with YOU

Page 49: Docker 활용법: dumpdocker

THANK YOU