DUSK: Develop at User level, inStall in Kernel

Alexey Smirnov and Tzi-cker ChiuehECSL Research Seminar

09/13/05

Outline

• Introduction• Related Work• Netfilter Overview• System Architecture• Evaluation• Conclusion

Introduction

• Kernel is often extended using kernel modules.

• Netfilter – hooks in network component.• Development at user-level is

convenient: gdb, ddd, etc.• Kernel modules have lower overhead.• DUSK achieves best of both worlds.

User-level Development of Kernel Extensions

• Provide kernel API at user-level. A lot of functions, changes from one version to another.

• Implement a new API that resembles kernel API. Not convenient for the programmer, requires code rewriting.

DUSK Overview

• DUSK is an extension to GCC that compiles the source code of a kernel module into a user-level program.

• It links several helper functions to the user-level module.

• Same helper functions for different versions of Linux kernel.

DUSK Framework

• User-level component: the kernel module compiled as a user level program.

• Kernel-level component: handles user-level module’s requests.

• The two components are connected using UDP sockets.

• Can run on different architectures: user-level module on x86 running in gdb and kernel-level module on MIPS wireless router.

Related Work

• VFS layer: UFS uses NFS loopback API. UpcallFS – a stackable file system.

• Device drivers: Userdev provides a user-level library.

• Network stack: icTCP exposes TCP stack parameters to user programs. Netfilter simulator runs Netfilter hooks at user level.

Netfilter Architecture• Netfilter adds a set of five hooks to

kernel network stack:

• A Netfilter module verdict: NF_ACCEPT, NF_DROP, NF_STOLEN, NF_QUEUE, NF_REPEAT.

Netfilter User Mode Modules• NF_QUEUE verdict sends sk_buff to

user space using a netlink socket. The queue handler is either standard or programmer-defined.

• A user module can only read sk_buff but cannot change kernel memory.

iptables

• iptables controls parameters of Linux firewall.

• iptables -A PREROUTING –p tcp --destination-port 80 –j QUEUE

• iptables –A POSTROUTING –p udp --source-port 5678 –j DROP

• Has string matching capabilities. Stateless IDS.

Kernel-level Programming at User Level

• User-level kernel modules need to:• Read/write kernel memory;• Call kernel functions;• Access global kernel variables, e. g.

current• Programmers can modify source

code of kernel module using these blocks to run it at user level.

Three Basic Blocks

• Kernel memory access: /dev/kmem• Kernel function calls: new system call

sys_exec(). System.map for address lookup.

• A new system call sys_kvar() for variable access.

• sys_exec() and sys_kvar() are implemented in DUSK kernel module.

• Kernel module and user module communicate using UDP sockets.

Using Basic Blocks to Run Kernel Module at User Level

• Kernel module source code:sk_buff->len• User-level code:

• Define a shadow variable u_sk_buff;• malloc() memory for it;• Use kmem_read() to copy sk_buff from kernel

memory to shadow variable;• Commit changes to u_sk_buff using

kmem_write().

• Our goal: automatic transformation.

DUSK Framework

DUSK Run-Time Support

• Kernel extension registration/de-registration. The address of a function is sent to the kernel.

• Processing kernel requests. An appropriate extension handles each request.

• Module initialization/cleanup. init_module() and cleanup_module() are called when user-level module starts and terminates.

Extension Management

nfho_post_routing.hook=khook_ip_input;nfho_post_routing.hooknum = NF_IP_POST_ROUTING;

nf_register_hook(&nfho_post_routing);

• Kernel-level DUSK module replaces khook_ip_input with a proxy kernel function and returns extension type and sequence number to the user program.

• DUSK clones a new process for each extension. They share same address space.

Blocking and Non-blocking Extensions• Extensions that block: the process

that called the extension blocks until the user-level module returns a verdict. Exampe: procfs, timer.

• Extensions that don’t block: the kernel callback returns as soon as a message is sent to the user space. Example: Netfilter.

Why Netfilter Is Non-blocking?• Hardware interrupt handler receives

network packets.• Runs with interrupts disabled. It saves

packets in a queue and invokes software interrupt, then returns.

• One software interrupt handler per processor. Hardware interrupts enabled.

• A packet is re-injected into the network stack when user-level processing is finished.

Extension Types

• DUSK supports Netfilter only.

• Timers, threads, procfs, ioctl, etc. are required for advanced modules.

DUSK Run-Time Configuration

Compile-time Support

• DUSK adds two functions: check() and commit().

• check() ensures that every buffer accessed using a pointer exists in kernel memory and in user-space.

• commit() updates the kernel version when a function is called; it refreshes the user version when the function returns.

Function check()

• check() is called for every array access, de-referenced pointer, or &.

• It maintains the kernel-to-user address mapping buffer.

• For a new user address DUSK calls kmalloc() and kmem_write()

• For a new kernel address DUSK calls malloc() and kmem_read().

• Addresses under PAGE_OFFSET are user addresses, above are kernel addresses.

Type Information

• Type information is gathered at compile-time. Type size, offsets of pointer fields are stored.

• Type size is used in check(). • commit() uses field information.

Kernel Function Calls

• A kernel function is replaced with:• void_kernel_func() for void functions;• int_kernel_func() for int functions;

• Function arguments are written into kernel memory when the function is called.

• Each pointer-type argument is wrapped into commit() call.

Function commit()

• commit() traverses data types recursively.

• Issue: array elements accessed using a pointer: *(a+1), *(a+2). If a is a function parameter then commit() ensures that all elements get synchronized.

• Issue: a union’s elements of different type have same offset. Types of the argument and that of the address mapping table are compared.

Function commit()

Function Prolog and Epilog

• All changes are committed when user-level module returns a verdict.

• DUSK adds a function epilog that commits the changes.

• depth variable counts the current nesting level.

• Incremented in the function prolog, decremented in the epilog.

check() and commit() Example• printk(“test”);• int_kernel_func(addr_of(“printk”), commit(check(“test”)));

Cross-Platform Debugging

• Linksys WRT54gs wireless router: 32 MB RAM, 200 MHz MIPS CPU. Runs Linux 2.4.20.

• Cannot run gcc, gdb. Supports Netfilter.

• printk() debugging possible. Logs are not stored after reboot.

X86 and MIPS Differences

• Data structures are different:

• PAGE_OFFSETs are different.• DUSK compiles module twice: with the cross-

compiler and with x86 compiler.• Addresses in data structures are changed when

data from kernel is received.

Evaluation

• nfsiff – FTP password sniffer• POSTROUTING: finds USER and PASS

commands;• PREROUTING: replies to special ICMP packets;

• lwfw – light-weight firewall• PREROUTING: drops certain packets (port,

interface).

• ipp2p – P2P traffic classifier• POSTROUTING: scans for P2P signatures.

nfsiff

• Sniffer on wireless router:

Compile-time Overhead

Saved Work

Conclusion

• DUSK allows to debug kernel modules at user level without changing the source code.

• Future work: add better debugger support, i.e. integrate DUSK with gdb and ddd.

• Support device driver development.