LCU14 BURLINGAME

Rémi Duraffort, LCU14

LCU14-211: LAVA Use Cases: SW testing & reproducing a LAVA failures locally using CARE

● Rémi Duraffort (aka ivoire)● STMicroelectronics engineer

● Porting v8 js engine on sh4 processors● PRoot co-developer

● Assignee working on LAVA● Open Source dev

● VLC media player (core developer)● dvblast, PRoot, NFS-Ganesha, ...

● … and ..● French :)

Who am I?

Testing PRoot within LAVAhttp://proot.me

Please, pronounced it “P-Root”...

● A syscall instrumentation engine:● alter syscalls arguments● change syscalls return values● replace and chain syscalls

● Based on PTrace● without any privileges● all linux ≥ 2.6.0

● In user space, we have:● mount --bind● chroot● binfmt_misc● ...

PRoot

Linux kernel

Application

PTrace

PRoot

open(‘/etc/hosts’) open(‘/tmp/hosts.local’)

● mount --bind:● proot -b /tmp/hosts.local:/etc/hosts cat /etc/hosts● proot -b ./local/gcc/install/bin:/usr/bin/ gcc

● chroot:● proot -R /tmp/ubuntu-14.04-x86_64 /bin/bash● proot -R /tmp/gentoo-x86 /bin/bash

● binfmt_misc:● proot -q qemu-aarch64 -R /tmp/debian-sid-aarch64 /usr/bin/gcc

●

Use cases (and demo)

execve(“gcc”, …) execve(“qemu-aarch64”, “gcc”, …)

● Kernel compatibility:● Allow to run new rootfs on old kernels● Emulate some kernel features (openat, fstatat, ...)

● proot -k 4.0 uname

● fake id● Fake some syscalls return values

● proot -0 id● proot -0 -R /tmp/ubuntu-14.04-x86 apt-get install vlc

● all together:● proot -0 -q qemu-arm -R /tmp/debian-sid-armhf apt-get install vlc

Use cases: having fun

● Tests requirements● Multi distro● Multi arch

● x86, x86-64, arm, aarch64, sh4● Multi testsuites

● coreutils, Perl, LTP, gdb, … (soon)● on distro kernel (unlike OBS, chroot)● as root and normal user

● Using LAVA:● KVM, qemu-system-aarch64, ST boards (soon)● results:

● http://armv7.com/dashboard/image-charts/PRoot● https://staging.validation.linaro.org/dashboard/image-charts/PRoot

Testing PRoot is challenging

LAVA: PRoot tests

proot-make-check-user.yaml

LAVA: PRoot jobs

Comprehensible Archiver for Reproducible Execution (aka CARE)

http://reproducible.io

● Just a PRoot extension:● Monitor filesystem accesses● Archive all accessed files before usage (/lib/ld-linux.so, /usr/lib/*, /etc/vim/vimrc, …)

● Archive = minimal root filesystem● Use the archive to re-execute the application

● Same filesystem environment● Same env variables● not replaying the syscalls!! (was often asked)

● A Visual Demo Is Worth a Thousand Words...

CARE

● Sometimes jobs are failing (really? a bug?)

● How to reproduce it?

● What do you need● the same hardware (or a good/same simulator)● the same software environment● the same env variables

● CARE can help for the software part● For the hardware, ask Linaro :)

Reproducing LAVA executions

● Inside LAVA● care -o build-failure.bin make -j4

● At home (on a board)● extract

● ./build-failure.bin (or “care -x build-failure.bin”)● Re-execute

● ./build-failure/re-execute.sh● ./build-failure/re-execute.sh make -j1 V=1● ./build-failure/re-execute.sh gcc -Wall -o foo foo.c

● At home (without a board (really, you should ask Linaro!))● PROOT=proot-x86 ./build-failure/re-execute.sh -q qemu-arm make -j1 V=1

Reproducing LAVA executions

Questions?

More about Linaro Connect: connect.linaro.org Linaro members: www.linaro.org/membersMore about Linaro: www.linaro.org/about/