1

CS4023 – Operating Systems(week 3)

Operating System Services and Structures

Dr. Atif AzadAtif.azad@ul.ie

2

Acknowledgement

• Significant material in this set of lectures has been borrowed from:– http://www.cs.berkeley.edu/~kubitron/courses/cs16

2/. Copyright © 2010 UCB

– http://www.cs.berkeley.edu/~kubitron/cs194-24/index_lectures.html. Copyright © 2013 UCB

– http://www.os-book.com . © 2014 Silberschatz et al.

– Dr Patrick Healy at CSIS Department, University of Limerick.

3

Review – Week 2• Storage Hierarchy• Protection

– Policy– Mechanism

• Mechanisms – Address Translation– Dual Mode Operation

• History of Operating Systems– History of choices – Batch processing vs Interactive programming

4

Address Translation• Address Space

– A group of memory addresses usable by something – Each program (process) and kernel has potentially

different address spaces.• Address Translation:

– Translate from Virtual Addresses (emitted by CPU) into Physical Addresses (of memory)

– Mapping often performed in Hardware by Memory Management Unit (MMU)

CPU MMU

VirtualAddresses

PhysicalAddresses

Example of Address Translation

Prog 1Virtual

AddressSpace 1

Prog 2Virtual

AddressSpace 2

CodeDataHeapStack

CodeDataHeapStack

Data 2

Stack 1

Heap 1

OS heap & Stacks

Code 1

Stack 2

Data 1

Heap 2

Code 2

OS code

OS dataTranslation Map 1 Translation Map 2

Physical Address Space

Q: Still vulnerable?

Dual Mode Operation• Hardware provides at least two modes:

– “Kernel” mode (or “supervisor” or “protected” or “previliged”)– “User” mode: Normal programs executed

• Some instructions/ops prohibited in user mode:– Example: cannot modify MMU functionality in user mode

• Attempt to modify Exception generated

• System boots into kernel mode; then, goes into user mode.• Transitions from user mode to kernel mode:

– System Calls, software Interrupts (traps or exceptions)

Section 1.5

7

Lecture Objectives

• To describe the services an operating system provides to users, processes, and other systems

• To discuss the various ways of structuring an operating system

8

What is OS: summary

– A program that acts as an intermediary between a user of a computer and the computer hardware

– A program that allocates computer resources such as CPU time, memory and I/O access to all the programs running, efficiently and fairly.

– A control program that supervises the execution of everything on the computer: they should execute correctly (as allowed by OS) and they should not affect other programs (fault isolation)

A View of Operating System Services

– A hierarchical system design– The OS space: kernel, services, system calls, and

interfaces.

Operating System Services: User View

• Operating systems provide an environment for execution of programs and services to programs and users

• Services from the User View:– User interface

• Varies between Command-Line (CLI), Graphics User Interface (GUI), Batch

– Program execution – load into memory, run it, and end: normally or abnormally (indicating error)

– I/O operations - I/O, with a file or an I/O device– File-system manipulation – For files & directories:, create &

delete, search, list file Information, permission management.– Communications – Processes may exchange information, on the

same computer or over a network• via shared memory or through message passing (packets moved by

the OS)

– Error detection – be aware of errors e.g.:• In the CPU and memory hardware, in I/O devices, in user program• OS must act to ensure correct and consistent computing• Debugging facilities can greatly enhance the user’s and programmer’s

abilities to efficiently use the system

Section 2.1

11

OS Services: Efficient Operation View– Resource allocation - to multiple users or multiple

jobs running concurrently:• Many types of resources - CPU cycles, main memory, file

storage, I/O devices.

– Accounting - To keep track of which users use how much and what kinds of computer resources

– Protection and security• Protection involves ensuring that all access to system

resources is controlled (see lecture in week 2)• Security of the system from outsiders requires user

authentication, extends to defending external I/O devices from invalid access attempts

User Operating System Interface - CLI

• CLI or command interpreter allows direct command entry– Can be in kernel, or as systems program– multiple flavors implemented – shells

• Bash, Korn Shell etc. echo $SHELL

– Primarily fetches a command from user and executes it– Both batch and interactive mode (e.g. in linux):

• java myclass &

– Sometimes commands built-in, sometimes just names of programs (linux commands)

• Which is better?• If the latter, adding new features doesn’t require shell modification

• Preferred Interface of power users or system administrators.

Section 2.2

Example: MS-DOS

• Single-tasking• Shell invoked when

system booted• Simple method to run

program– No process created

• Single memory space• Loads program into

memory, overwriting all but the kernel

• Program exit -> shell reloaded

At system startup running a program

Example: FreeBSD

• Unix variant• Multitasking• User login -> invoke user’s choice

of shell• Shell executes fork() system call

to create process– Executes exec() to load program

into process– Shell waits for process to terminate

or continues with user commands

• Process exits with:– code = 0 – no error – code > 0 – error code

User Operating System Interface - GUI

• User-friendly desktop metaphor interface– Invented at Xerox PARC

• Many systems now include both CLI and GUI interfaces– Microsoft Windows is GUI with CLI “command”

shell– Apple Mac OS X is “Aqua” GUI interface with

UNIX kernel underneath and shells available– Unix and Linux have CLI with optional GUI

interfaces (CDE, KDE, GNOME)

Resource Allocation – at the center of it all!• What do modern OSs do?

– Control access to resources!• Control of Resources

– Access/No Access/Partial Access

• Check every access to see if it is allowed

– Resource Multiplexing• When multiple valid requests occur at

same time – how to multiplex access?• What fraction of resource can requester

get?

– Performance Isolation• Can requests from one entity prevent

requests from another?

• What or Who is a requester???– Process? User? Think of this as a

“Principle”

Access Control and Multiplexing

IndependentRequesters

What is a Resource?• Processor, Memory, Cache

– Multiplex through: scheduling, Virtual memory– Abstraction: Process, Thread– Need Kernel Level to Multiplex?

• Need to Sandbox somehow• Kernel control of memory, prevent certain instructions

• Network– Multiplex through: Queues, Input Filters– Abstraction: Sockets API– Need Kernel Level to Multiplex?

• Not necessarily – New hardware has on-chip filters• Setup Cost, but not necessarily a per-packet cost• Is network really secure anyway? (Need Crypto!)

• Disk– Multiplex through: Buffer Cache– Abstraction: File System API– Need Kernel Level to Multiplex?

• Traditionally all access control through kernel• What about assigning unlimited access to partitions?

More Complex Resources: Operating System Services

• System Services are really complex resources– File system (Uses Disk Drive)

• File API: Create, Read, Write, Delete• Access Control: User, Group, World, Read/Write/Execute

– Windows System (Uses Graphics Card)• Windowing API: Write Text, Draw/Fill in figures• Access Control: Per Window (User created)

– Data Base (Uses Disk Drive or Memory or Network)• DB API: SQL Queries and Transactions• Access Control: Per user, Group, others

– Lock Service (Memory)• Lock API: Acquire (Read, Write), Release• Access Control: By group/per user

• Access controlled through syscall interface (Kernel Level)– Funnel all access through trusted (verified) API

• Kernel controls access to API, verifies identity• Service controls access to resources using identity

– Service decides multiplexing/isolation policies• Often based on first-come-first-serve!

19

Lab/Tutorials

• Changed Again!• University decides what labs stay• Announcement on the website

– Lab Tuesday at 1200 Wednesday 1700• Long exercises?

– Read them before coming to sessions.• Unsupervised Tutorial this week.

– Group Study– Submit solutions next week.

Traditional Approach to Handling Resource• Example: Send message from one processor to another

– Check Permissions, Format Message– Enforce forward progress, Handle interrupts– Prevent Denial Of Service (DOS) and/or Deadlock

• Traditional Approach: Use a system call+OS Service

SourceProgram

SysCall

MessageService

Device Driver

HardwareInterface

DestProgram

SysCall

MessageService

Device Driver

HardwareInterface

Network

Kernel (OS)Code

User-Level Code (With Libraries)

OS Service

Hardware

Alternative: Mechanisms (or even Policy!?) in HW

• Permit user-level code to send messages– Have hardware check permission and/or rate– Have hardware enforce format/consistency– Have hardware guarantee forward progress– Have Hardware deliver messages/interrupts to usercode

• OS sets registers to control behavior based on policy

SourceProgram

DestProgram

Network

User-Level Code (With Libraries)

HardwareHardwareInterface(User-level)Q

oS

Sp

ecs Hardware

Interface(User-level) Q

oS

Sp

ecs

OS Policy

System Calls: Details• Challenge: Interaction Despite Isolation

– How to isolate processes and their resources…• While still permitting them to request help from the kernel• Letting them interact with resources while maintaining usage policies such as security, QoS, etc

– Letting processes interact with one another in a controlled way• Through messages, shared memory, etc

• Enter the System Call interface– Layer between the hardware and user-space processes– Programming interface to the services provided by the OS

• Mostly accessed by programs via a high-level Application Program Interface (API) rather than directly– Get at system calls by linking with libraries in glibc

• Three most common APIs are:– Win32 API for Windows– POSIX API for POSIX-based systems (including virtually all versions of UNIX, Linux, and Mac

OS X)– Java API for the Java virtual machine (JVM)

Call toprintf()

Printf() in the C library

Write() system call

Example of System Call usage• System call sequence to copy the contents of one file to another

file:

• Many crossings of the User/Kernel boundary!– The cost of traversing this boundary can be high– Context Switching (later)

Example: Use strace to trace syscalls• prompt% strace wc production.log

execve("/usr/bin/wc", ["wc", "production.log"], [/* 52 vars */]) = 0brk(0) = 0x1987000mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff24b8f7000access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)open("/etc/ld.so.cache", O_RDONLY) = 3fstat(3, {st_mode=S_IFREG|0644, st_size=137151, ...}) = 0mmap(NULL, 137151, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff24b8d5000close(3) = 0open("/lib64/libc.so.6", O_RDONLY) = 3read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\355\241,0\0\0\0"..., 832) = 832fstat(3, {st_mode=S_IFREG|0755, st_size=1922112, ...}) = 0mmap(0x302ca00000, 3745960, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x302ca00000mprotect(0x302cb89000, 2097152, PROT_NONE) = 0mmap(0x302cd89000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x189000) = 0x302cd89000mmap(0x302cd8e000, 18600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x302cd8e000close(3) = 0mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff24b8d4000mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff24b8d3000mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff24b8d2000arch_prctl(ARCH_SET_FS, 0x7ff24b8d3700) = 0mprotect(0x302cd89000, 16384, PROT_READ) = 0mprotect(0x302c81f000, 4096, PROT_READ) = 0munmap(0x7ff24b8d5000, 137151) = 0brk(0) = 0x1987000brk(0x19a8000) = 0x19a8000open("/usr/lib/locale/locale-archive", O_RDONLY) = 3fstat(3, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff245a41000close(3) = 0stat("production.log", {st_mode=S_IFREG|0644, st_size=526550, ...}) = 0open("production.log", O_RDONLY) = 3read(3, "# Logfile created on Fri Dec 28 "..., 16384) = 16384open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 4fstat(4, {st_mode=S_IFREG|0644, st_size=26060, ...}) = 0mmap(NULL, 26060, PROT_READ, MAP_SHARED, 4, 0) = 0x7ff24b8f0000close(4) = 0read(3, "m: cannot remove `/tmp/fixrepo/g"..., 16384) = 16384read(3, "a36de93203e0b4972c1a3c81904e': P"..., 16384) = 16384read(3, "xrepo/git-tess/gitolite-admin/.g"..., 16384) = 16384

Many repetitions of these readsread(3, "ixrepo/git-tess/gitolite-admin\n "..., 16384) = 16384read(3, "ite/redmine/vendor/plugins/redmi"..., 16384) = 16384read(3, "ited with positive recursionChec"..., 16384) = 16384read(3, "ting changes to gitolite-admin r"..., 16384) = 2262read(3, "", 16384) = 0fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff24b8ef000write(1, " 4704 28993 526550 production."..., 36 4704 28993 526550 production.log) = 36close(3) = 0close(1) = 0munmap(0x7ff24b8ef000, 4096) = 0close(2) = 0exit_group(0)

Example of Standard API

System Call Implementation• Typically, a number associated with each system call

– System-call interface maintains a table indexed according to these numbers– The fact that the call is by “number”, is essential for security reasons!

• The system call interface invokes intended system call in OS kernel and returns status of the system call and any return values– Return value: often a long (integer)

• Return of zero is usually a sign of success, but not always• Return of -1 is almost always reflects an error

– On error – return code placed into global “errno” variable• Can translate into human-readable errors with the “perror()” call

• The caller need know nothing about how the system call is implemented– Just needs to obey API and understand what OS will do as a result call– Most details of OS interface hidden from programmer by API

• Managed by run-time support library (set of functions built into libraries included with compiler)

API – System Call – OS Relationship

Standard C Library Example

• C program invoking printf() library call, which calls write() system call

System Call Parameter Passing

• Often, more information is required than simply identity of desired system call– Exact type and amount of information vary according to OS and call

• Three general methods used to pass parameters to the OS– Simplest: pass the parameters in registers

• In some cases, may be more parameters than registers– Parameters stored in a block, or table, in memory, and address of

block passed as a parameter in a register • This approach taken by Linux and Solaris

– Parameters placed, or pushed, onto the stack by the program and popped off the stack by the operating system

– Block and stack methods do not limit the number or length of parameters being passed

Parameter Passing via Table

• Kernel must always verify parameters passed to it by the user– Are parameters in a reasonable range?– Are memory addresses actually owned by the calling user

(rather than bogus addresses)

Types of System Calls

• Process control– create process, terminate process– end, abort– load, execute– get process attributes, set process attributes– wait for time– wait event, signal event– allocate and free memory– Dump memory if error– Debugger for determining bugs, single step execution– Locks for managing access to shared data between

processes

Types of System Calls

• File management– create file, delete file– open, close file– read, write, reposition– get and set file attributes

• Device management– request device, release device– read, write, reposition– get device attributes, set device attributes– logically attach or detach devices

Types of System Calls (Cont.)

• Information maintenance– get time or date, set time or date– get system data, set system data– get and set process, file, or device attributes

• Communications– create, delete communication connection– send, receive messages if message passing model to

host name or process name• From client to server

– Shared-memory model create and gain access to memory regions

– transfer status information– attach and detach remote devices

Types of System Calls (Cont.)

• Protection– Control access to resources– Get and set permissions– Allow and deny user access

• List of Linux System Calls: – http://syscalls.kernelgrok.com

POSIX standard• Portable Operating System Interface for UNIX

• An attempt to standardize a “UNIXy” interface

• Conformance: IEEE POSIX 1003.1 and ISO/IEC 9945 – Latest version from 2008– Originally one document consisting of a core programming inteface – now 19 separate

docs– Many OSes provide “partial conformance” (including Linux)

• What does POSIX define?– POSIX.1: Core Services

• Process Creation and Control• Signals• Floating Point Exceptions, Segmentation/memory violations, illegal instructions, Bus Erors• Timers• File and Directory Operations• Pipes• C Library (Standard C)• I/O Port Interface and Control • Process Triggers

– POSIX.1b: Realtime Extensions– POSIX.2: Shell and Utilities

POSIX (cont)• Process Primitives:

– fork, execl, execlp, execv, execve, execvp, wit, waitpid– _exit, kill, sigxxx, alarm, pause, sleep….

• Example file access primitives:– opendir, readdir, rewinddir, closedir, chdir, getcwd, open, creat, umask, link, mkdir, unlink,

rmdir, rename, stat, fstat, access, fchmod, chown, utime, ftruncate,pathconf,fpathconf• I/O primitives:

– pipe, dup, dup2, close, read, write, fcntl, lseek, fsync • C-Language primitives:

– abort, exit, fclose, fdopen, fflush, fgetc, fgets, fileno, fopen, fprintf, fputc, fputs, fread, freopen, fscanf, fseek, ftell, fwrite, getc, getchar, gets, perror, printf, putc, putchar, puts, remove, rewind, scanf, setlocale, siglongjmp, sigsetjmp, tmpfile, tmpnam, tzset

• Synchronization:– sem_init, sem_destroy, sem_wait, sem_trywait, sem_post, pthread_mutex_init,

pthread_mutex_destroy, pthread_mutex_lock, pthread_mutex_trylock, pthread_mutex_unlock

• Memory Management– mmap, mprotect, msync, munmap

• How to get information on a system call?– Type “man callname”, i.e. “man open”– System calls are in section “2” of the man pages

System Programs

• System programs provide a convenient environment for program development and execution. – Can be interfaces to syscalls, or much more.

• They can be divided into:– File manipulation – Status information sometimes stored in a File modification– Programming language support– Program loading and execution– Communications– Background services– Application programs

• Most users’ view of the operation system is defined by system programs, not the actual system calls

Operating System Design and Implementation

• User goals and System goals– User goals – OS should be convenient to use, easy to learn, reliable,

safe, and fast– System goals – OS should be easy to design, implement, and

maintain, as well as flexible, reliable, error-free, and efficient• Design and Implementation of OS not “solvable”, but some

approaches work

• Internal structure of different Operating Systems can vary widely

• Important principle to separate– Policy: What will be done? – Mechanism: How to do it?

Implementation

• Much variation– Early OSes in assembly language– Then system programming languages like Algol, PL/1– Now C, C++

• Actually usually a mix of languages– Lowest levels in assembly– Main body in C– Systems programs in C, C++, scripting languages like PERL,

Python, shell scripts

• More high-level language easier to port to other hardware– But slower

• Emulation can allow an OS to run on non-native hardware

Operating System Structure

• General-purpose OS is very large program

• Various ways to structure ones– Simple structure – MS-DOS– More complex -- UNIX– Layered – an abstrcation– Microkernel -Mach

Simple Structure • All aspects of the OS linked together in one binary

– APIs not carefully designed (and/or lots of global variables)– Interfaces and levels of functionality not well separated– No address protection

• Example: MS-DOS– provide the most functionality

in the least space– Made sense in early days

of personal computers withlimited processors (e.g. 6502)

• Advantages?– Low memory footprint

• Disadvantages?– Very fragile, no enforcement

of structure/boundaries• What about Language enforcement? (Microsoft Singularity?)

Non Simple, Monolithic Structure: UNIX System Structure

• Two-Layered Structure: User vs Kernel– All code representing protection and management of resources placed in same

address space– Compromise of one component can compromise whole OS

• Clear division of labor?– The producer of the OS and the User of the OS

User Mode

Kernel Mode

Hardware

Applications

Standard Libs

Layered Structure• Operating system is divided many layers (levels)

– Each built on top of lower layers– Bottom layer (layer 0) is hardware– Highest layer (layer N) is the user interface

• Each layer uses functions (operations) and services of only lower-level layers– Advantage: modularity Easier debugging/Maintenance– Not always possible: Does process scheduler lie above or below

virtual memory layer?• Need to reschedule processor while waiting for paging• May need to page in information about tasks

• Important: Machine-dependent vs independent layers– Easier migration between platforms– Easier evolution of hardware platform– Good idea for you as well!

• Can utilize hardware enforcement– x86 processor: 4 “rings”– Call gates

Layered Operating System

Microkernel Structure

• Moves functionality from the kernel into “user” space– Small core OS running at kernel level– OS Services built from many independent user-level processes– Communication between modules with message passing

• Benefits:– Easier to extend a microkernel– Easier to port OS to new architectures– More reliable (less code is running in kernel mode)– Fault Isolation (parts of kernel protected from other parts)– More secure

• Detriments:– Performance overhead can be severe for naïve implementation

• Mach example of microkernel– Mac OS X kernel (Darwin) partly based on Mach

Figure ©WikipediaM

on

olith

icK

ern

el

Mic

rokern

el

Modules-based Structure

• Most modern operating systems implement modules– Uses object-oriented approach

• careful API design/Few if any global variables

• Each core component is separate– Each talks to the others over known interfaces– Each is loadable as needed within the kernel

• Overall, similar to layers but more flexible

Hybrid Systems

• Most modern operating systems are actually not one pure model– Hybrid combines multiple approaches to address

performance, security, usability needs– Linux and Solaris kernels in kernel address space, so

monolithic, plus modular for dynamic loading of functionality– Windows mostly monolithic, plus microkernel for different

subsystem personalities• Apple Mac OS X hybrid, layered, Aqua UI plus Cocoa

programming environment– Below is kernel consisting of Mach microkernel and BSD Unix

parts, plus I/O kit and dynamically loadable modules (called kernel extensions)

Mac OS X Structure

iOS

• Apple mobile OS for iPhone, iPad– Structured on Mac OS X, added functionality– Does not run OS X applications natively

• Also runs on different CPU architecture (ARM vs. Intel)

– Cocoa Touch Objective-C API for developing apps

– Media services layer for graphics, audio, video

– Core services provides cloud computing, databases

– Core operating system, based on Mac OS X kernel

Android

• Developed by Open Handset Alliance (mostly Google)– Open Source

• Similar stack to IOS• Based on Linux kernel but modified

– Provides process, memory, device-driver management– Adds power management

• Runtime environment includes core set of libraries and Dalvik virtual machine– Apps developed in Java plus Android API

• Java class files compiled to Java bytecode then translated to executable than runs in Dalvik VM

• Libraries include frameworks for web browser (webkit), database (SQLite), multimedia, smaller libc

Android Architecture