1

Automated test of linux kernel byusing ktest

Version1.1.3Satoru Takeuchi

satoru.takeuchi@gmail.com

2

Introduction

● Purpose– Learn how to test linux kernel

automatically by using ktest– It's not a ktest reference, but a quick

ktest learning guide● Target readers

– Linux kernel developers who havex86_64 linux system and the KVMguest system on it (*1)

*1) Although ktest itself can be used for many environments, this slide focuses on this specific combination for simplicity.

3

Table of contents1. What is ktest?2. System requirement of ktest3. System environment of this document4. Initial configuration5. How to test6. Conclusion7. References

4

1. What is ktest?● A kernel automated test tool which is distributed as

part of linux kernel– Author: Steven Rostedt

● It keeps working even if test kernel haspanic/hangup problems.

● This document covers the following features– Basic tests for a kernel

● build test→ install test→ boot test→ user defined test– Test each patch of a patchset– Bisect to find the problematic commit from a series of

commits

5

2. System requirement of ktest● The following two system(VM is OK)

– Host: To build test kernel and to watch theprogress of test

– Target: Run test with test kernel● Can be poweron/off from Host.● Can be connected via console from Host● Can be connected via ssh as root from Host

● Kernel source– It should be git repository to use some

advanced features

6

3. System environment of this document

● I don't explain how to setup this environment here.● If you try to do the subsequent steps, please replace each

expression with appropriate one depend on your environment.

Host Target● System: debian/jessy x86_64● hostname/user: host/sat● Linux kernel's git repository

under /home/sat/src/linux● “make mrproper” clean

● kernel config file for testkernel can be boot on Target.

● /home/sat/kernel.config

● System: debian/jessy orCentOS7 x86_64 guest in Host

● Hostname/domain name: target● Bootloader: grub2● “grub-reboot” works fine● Can be connected via “virsh

console” from Host● Can be connected via “ssh

root@target” withoutpassphrase

7

4. Initial configuration (1/2): Make workspace

$ mkdir -p /home/sat/ktest

$ cd /home/sat/ktest

$ mkdir output

$ cp /home/sat/src/linux/tools/testing/ktest/ktest.pl .

● Run the following commands at Host.

8

4. Initial Configuration (2/2): Prepare the grub2 entry

● Add the following entry for your test kernel in /boot/grub/grub.conf,the grub2's configuration file.

menuentry 'ktest' {

...

linux /boot/ktest-vmlinuz ...

...

initrd /boot/ktest-initramfs.img

}

9

5. How to test● Learn how to run the following tests one by one.

1. Build test.2. Install test.3. Boot test.4. User defined test.5. Test patchset.6. Bisect

10

5.1. Build test (1/2): Configuration● Create /home/sat/ktest/ktest.conf as follows.

All pathnames should be full path here.MACHINE = target # hostname of Target

LOG_FILE = /home/sat/ktest/ktest.log # The logfile of ktest

BUILD_DIR = /home/sat/src/linux # your kernel source path

OUTPUT_DIR = /home/sat/ktest/output # Output dir of kernel build

LOCALVERSION = -ktest # local version of the test kernel

BUILD_TARGET = arch/x86/boot/bzImage # The name of bzImage

TARGET_IMAGE = /boot/ktest-vmlinuz # Full path of vmlinuz image on Target

MIN_CONFIG = /home/sat/kernel.config # The base of .config

BUILD_TYPE= olddefconfig # configuration target of kernel build

BUILD_OPTIONS = -j8 # (optional) kernel build option

TEST_START

TEST_TYPE = build

11

5.1. Build test (2/2): Run&Verify● Run ktest as follows.

$ ./ktest.pl

● Then ktest builds your test kernel automatically.● If succeeds, test finished with the following log....

******************************************

*******************************************

KTEST RESULT: TEST 1 SUCCESS!!!! **

*******************************************

*******************************************

1 of 1 tests were successful

● If fails, diagnose the problem with the logfile.● Failed during build → Your kernel would have some failed-to-build bug.● Failed in other place → ktest.conf would be wrong.

12

5.2. Install test● Modify ktest.conf

+ SSH_USER = root

+ POST_INSTALL = ssh ${SSH_USER}@${MACHINE} /usr/sbin/mkinitramfs -o /boot/ktest-initramfs.img$KERNEL_VERSION # (optional) The code run after installation on Host.

- TEST_TYPE = build

+ TEST_TYPE = install

● If your system is CentOS7, replace POST_INSTALl lineas follows.

● Run ktest

● Then ktest builds and installs your test kernelautomatically.

● If fails, ktest.conf would be wrong.

$ ./ktest.pl

+ POST_INSTALL = ssh {SSH_USER}@${MACHINE} /usr/sbin/dracut -f ktest-initrdamfsimg $KERNEL_VERSION

13

5.3. Boot test (1/2): Configuration● Modify ktest

+ REBOOT_TYPE = grub2

+ GRUB_FILE = /boot/grub/grub.cfg

+ GRUB_MENU = ktest

+ GRUB_REBOOT = grub-reboot # The command to boot your test kernel only once on Target

+ POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE} # The commands fromHost to force reboot Target.

+ REBOOT = virsh reset ${MACHINE} # reboot command which ktest uses

+ REBOOT_ON_ERROR = 1# (optional) Set if you want to reboot Target on failed-to-boot.

+ CONSOLE = virsh console ${MACHINE} # Command to connect from Host to Target

+ CLOSE_CONSOLE_SIGNAL = KILL # Signal to kill the console process.

...

- TEST_TYPE = install

+ TEST_TYPE = boot

● Modify ktest.conf

● If your guest is CentOS7, replace GRUB_FILE and GRUB_REBOOTlines as follows+ GRUB_FILE = /boot/grub2/grub.cfg

+ GRUB_REBOOT = grub2-reboot

14

5.3. Boot test (2/2): Run&Verify

● Run ktest

● Then ktest builds, installs, boot your test kernel , andreboot with the original kernel automatically.

● If fails, your kernel has failed-to-boot bugs orktest.conf would be wrong.

$ ./ktest.pl

15

5.4. User defined test1. Make test. Please note that the test is run under Host.

2. Modify ktest.conf

3. Run ktest.

● Then ktest builds, installs, boot your test kernel, run the user defined test with thiskernel, reboot with the original kernel automatically.

● If failed…

● on user defined test → Your kernel or your test would have some bugs.

● on other place → ktest.conf would be wrong.

$ ./ktest.pl

- TEST_TYPE = boot

+ TEST_TYPE = test

+ TEST = /home/sat/ktest/hello.test

$ cat hello.test

ssh 'root@target echo hello ktest!'

16

5.5. Test patchset (1/4): Assumption

● You made the patchset, consists of the following ten patches, for 3.14-rc6.– 7 th patch has panic-on-boot bug (of course you don't know it beforehand).

● You'd like to run boot test for each patch before submitting it to LKML.

● $ git log --oneline -11

● 3d02e41 test commit 10

● f3c04a3 test commit 9

● 3fec210 test commit 8

● 0641787 test commit 7 # It has a critical bug!

● a172eb6 test commit 6

● 63d9f91 test commit 5

● d6edbe0 test commit 4

● a350e10 test commit 3

● 52a2a97 test commit 2

● 8644b5e test commit 1

● fa389e2 Linux 3.14-rc6

17

5.5. Test patchset (3/4): Verify● Ktest detects panic during the test of 7 th patch with the

following log, and finishes.

Processing commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b test commit 7...[ 0.024472] Kernel panic - not syncing: Attempted to kill the idle task!bug timed out after 1 secondsTest forced to stop after 60 seconds after failureCRITICAL FAILURE... failed - got a bug reportREBOOTINGssh root@debian sync ... FAILED!ssh root@debian reboot ... FAILED!virsh destroy debian; sleep 5; virsh start debian ... SUCCESS See /home/sat/ktest/ktest.log for more info.failed - got a bug report...

18

5.5. Test patchset (2/4): Configuration&Run

1. Modify ktest.conf

2. Run ktest

● Then ktest runs boot test for each patch with the folloing log.

- TEST_TYPE = test

- TEST = /home/sat/ktest/hello.test

+ TEST_TYPE = patchcheck

+ PATCHCHECK_TYPE = boot # Run boot test for each patch.

+ PATCHCHECK_START = 8644b5e # The first patch of the patchset

+ PATCHCHECK_END = 3d02e41 # The last patch of the patchset

$ ./ktest.pl

Processing commit 8644b5e0896b075c19f9d52aea1e9a22d7fdc89d test commit 1

…

Processing commit 52a2a973c29dce4d85e3c0d570f4351b248ab311 test commit 2

...

19

5.5. Test patchset (4/4): Further work

● Modify PATCHCHECK_TYPE option if you want to change the test foreach patch.

● The same value as TEST_TYPE option can be set.● build, install, boot, test (You should also set TEST option here)

● ex) Run your own test, hello.test, for each patches.

- PATCHCHECK_TYPE = boot

+ PATCHCHECK_TYPE = test

+ TEST = /home/sat/ktest/hello.test

20

5.6 Bisect (1/5): Assumption

● 3.14-rc6 boots without any problem.● 3.14-rc6 with the following ten commits get panic on boot.

● 7 th commit has panic-on-boot bug (of course you don't know it beforehand).● You want to find the problematic commit and report it to LKML.

● $ git log --oneline -11

● 3d02e41 test commit 10

● f3c04a3 test commit 9

● 3fec210 test commit 8

● 0641787 test commit 7 # It has a critical bug!

● a172eb6 test commit 6

● 63d9f91 test commit 5

● d6edbe0 test commit 4

● a350e10 test commit 3

● 52a2a97 test commit 2

● 8644b5e test commit 1

● fa389e2 Linux 3.14-rc6

21

5.6. bisect (2/5): Configuration

● Modify ktest.conf- TEST_TYPE = patchcheck

- PATCHCHECK_TYPE = boot

- PATCHCHECK_START = 8644b5e

- PATCHCHECK_END = 3d02e41

+ TEST_TYPE = bisect

+ BISECT_TYPE = boot

+ BISECT_GOOD = b28a960 # One before the first of suspect commits

+ BISECT_BAD = 3d02e41 # The last of suspect commits

22

5.6. bisect (3/5): Run

● Run ktest

● Then ktest bisects the suspect commits automatically.git bisect start ... SUCCESS

git bisect good fa389e220254c69ffae0d403eac4146171062d08 ... SUCCESS

git bisect bad 3d02e41fd18ddda74352beb35aa952266b34b92f ... Bisecting: 4 revisions left to test afterthis (roughly 2 steps)

…

git bisect good ... Bisecting: 2 revisions left to test after this (roughly 1 step)

…

git bisect bad ... Bisecting: 0 revisions left to test after this (roughly 0 steps)

...

●

$ ./ktest.pl

23

5.6. Bisect (4/5): Verify

● Finally ktest detects 7th commit from 3.14-rc6 is the bad one

git bisect good ... 0641787fc60d4926b6d286a84417e4b6bb5eca3b is the first bad commit

commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b

Author: Satoru Takeuchi <satoru.takeuchi@gmail.com>

Date: Sun Feb 16 22:48:02 2014 +0900

test commit 7

...

*******************************************

*******************************************

KTEST RESULT: TEST 1 SUCCESS!!!! **

*******************************************

*******************************************

...

24

5.6. Bisect (5/5): Further work

● Modify BISECT_TYPE option if you want to change the test foreach bisect.

● The same value as TEST_TYPE option can be set.● build, install, boot, test (you should also set TEST option here)

● ex) Run your own test, hello.test, for each bisect.

- PATCHCHECK_TYPE = boot

+ PATCHCHECK_TYPE = test

+ TEST = /home/sat/ktest/hello.test

25

6. Conclusion

● Now you can do the automated kernel test. Congratulations!● It can reduce plenty of your precious time.

● For more information, please read the references attached at thenext page.

● Feel free to ask me if you have any questions/comments.

26

7. References● The files under <kernel source>/testing/tools/ktest:

● ktest.pl: The ktest program● sample.conf: The document for all options.● examples/*: Sample configuration files for some Targets

● examples/kvm.conf is for the KVM Target● Ttest tutorial session by Steven Rostedt:

– http://video.linux.com/videos/automated-testing-with-ktestpl