1

setuid Demystified-- Examining the API of Security Operation in

OS using Formal Models

Hao Chen, David Wagner

UC Berkeley

Drew Dean

SRI International

2

Objective

• Understand the semantics of security operation API in OS precisely

• Applications– Using these system calls properly in programs– Verifying their documentations– Detecting inconsistency in OS kernels– Building security properties and checking

them in programs automatically (e.g. by modelchecker)

3

What is setuid

• Access control in Unix is based the User ID model

• Each process has 3 user Ids:– Real uid (ruid)– Effective uid (euid)– Saved uid (suid)

• Uid-setting system calls– setuid() seteuid() setreuid() setresuid()

4

The setuid Mystery

• Uid-setting system calls are a semantic mess– Counter-intuitive semantics– Subtle differences among different calls– Incompatible semantics of the same call in

different Unix systems (e.g.: Linux, Solaris, FreeBSD)

– Incomplete, inaccurate, or even wrong documentation

• Reason: historical artifacts

5

Solution: Formal Model

• Use a formal model to describe the user ID model

• Build an FSA where– The states describe the user IDs of a process– The transitions describe the semantics of the

uid-setting system calls

6

Determine the States of the FSA

• Each state is a tuple (ruid, euid, suid)• The range of user ID values determines

the number of states• Example:

– A process switches between a privileged user ID and an unprivileged ID

– 2 user IDs: 0(root), x(non-root)– 8 states

7

Problem: Difficult to Determine Transitions

• Large number of transitions. E.g.– Range of user ID values: {0, x} where x!=0– Number of states: 8– Number of transitions per state:

• setuid(uid): 2 transitions• seteuid(euid): 2 transitions• setreuid(ruid, euid): 4 transitions• setresuid(ruid, euid, suid): 8 transitions

– Total transitions: 8*(2+2+4+8)=128

• A laborious, error-prone process

8

Determine Transitions Automatically by Simulation

• Idea: Exhaustively make all system calls at each stateFor each state s=(ruid, euid, suid) where ruid, euid, suid {0, uid1 , uid2 , …} For each system call c {setuid(e), seteuid(e), setreuid(r,e), setresuid(r,e,s)} { Make the system call c in the state s Observe the ensuing state s’ Add the transition }

'ss c

9

FSA for setuid() in FreeBSD

10

FSA for setuid() in Linux

11

FSA for setreuid() in Linux

12

FSA for setresuid() in Linux

13

Benefits

• Correctness: the FSA reflects what programs experience

• Efficiency: the automatic method is portable to– Different Unix systems– Different kernel versions

14

Application: Understanding the semantics of the system calls

• Find subtle semantic differences– Among different uid-setting system calls– Among the same system call on different Unix

systems

• Find surprising, counter-intuitive semantics

15

Application: Verifying Man Pages

• Incompete man page– Man page for setuid() in Linux fails to mention

capabilities which affect how setuid() behaves

• Wrong man pages– FreeBSD 4.4:Unprivileged users may change the ruid to the euid and vice versa

– Redhat Linux 7.2:The setgid function checks if the egid of the caller and if it is the superuser, …

16

Application: Detecting Inconsistency in OS Kernel

• Linux has fsuid– Used for filesystem permission checking– Normally follows euid

• Invariant in Linux 2.4.18 (kernel/sys.c)– fsuid is 0 only if at least one of ruid, euid, suid is 0

• Rationale– ensuring that an fsuid-unware cross-platform

application can automatically drop root privilege in fsuid by dropping it in ruid, euid, suid

17

Application: Detecting Inconsistency in OS Kernel (cont)

• A bug breaks the invariant:– The invariant is satisfied in setuid(), seteuid(),

setreuid()– But it is broken in setresuid()

• We found the bug using the simulator• The bug has been confirmed by Linus and

Alan and will be fixed using our patch.

18

Application: Checking Proper Usage of Syscalls in Programs

• Modelchecking security properties in programs– Model a program as a PDA– Intersect the PDA (program) with the FSA of uid-setting

system calls to get a new PDA– Check reachability of risky states in the new PDA

• Can answer questions like:– Can a uid-setting system call fail in this program?– Can this program fail to drop privilege?– Which part of this program run with privilege?

• Result: Found known bugs in sendmail 8.10.1 and 8.12.0

19

Conclusion

• Formal models are useful in– Understanding the APIs of security operations– Verifying their documentations– Detecting inconsistency in OS kernels– Checking proper usage of security-relevant

APIs in programs