System Programming 1

Advanced Programming in the Unix Advanced Programming in the Unix EnvironmentEnvironment

Ch 3. File I/OCh 3. File I/O

System Programming 2

ContentsContents

Unix File I/O basic featuresUnix File I/O basic features Unix File I/O functionsUnix File I/O functions

– open, creat, close, lseek, read, writeopen, creat, close, lseek, read, write

File Sharing and Kernel Data StructuresFile Sharing and Kernel Data Structures Atomic operationsAtomic operations Other File I/O functionsOther File I/O functions

– dup, fnctl, ioctldup, fnctl, ioctl

System Programming 3

IntroductionIntroduction

Files in UnixFiles in Unix– Simple, unstructured sequences of bytes or characters.Simple, unstructured sequences of bytes or characters.

Unbuffered I/O Unbuffered I/O – Each read and write invokes a system call in the kernelEach read and write invokes a system call in the kernelRef) Standard I/O library - buffered I/ORef) Standard I/O library - buffered I/O

File DescriptorsFile Descriptors– All open files are referred to by All open files are referred to by file descriptors.file descriptors.– Kernel returns a file descriptors when open or create a file.Kernel returns a file descriptors when open or create a file.– Default file descriptors assigned by Unix ShellsDefault file descriptors assigned by Unix Shells

• 0 : standard input (STDIN_FILENO)0 : standard input (STDIN_FILENO)• 1 : standard output (STDOUT_FILENO) 1 : standard output (STDOUT_FILENO) • 2 : standard error (STDERR_FILENO) 2 : standard error (STDERR_FILENO)

System Programming 4

File I/O functionsFile I/O functions Open and Close functionsOpen and Close functions

– open, creat, closeopen, creat, close Position functionsPosition functions

– lseeklseek Read/Write functionsRead/Write functions

– read, writeread, write Duplicating functionsDuplicating functions

– dup, dup2dup, dup2 Property controlProperty control

– fcntlfcntl MiscMisc

– ioctlioctl

System Programming 5

Opening and Closing a FileOpening and Closing a File

int open(const char *int open(const char *pathnamepathname,int ,int oflagoflag,.../* mode_t ,.../* mode_t modemode */); */);returns : file descriptor if OK, -1 on errorreturns : file descriptor if OK, -1 on error

int creat(const char int creat(const char *pathname*pathname, mode_t , mode_t modemode););

returns : file descriptor opened for write-only if OK, -1 on errorreturns : file descriptor opened for write-only if OK, -1 on error

int close(int int close(int fildesfildes);); returns : 0 if OK, -1 on errorreturns : 0 if OK, -1 on error

open(): open(): – opens a file specified by opens a file specified by pathname pathname withwith oflag oflag andand mode mode

(file access permissions)(file access permissions)

– returns a file descriptor which is the returns a file descriptor which is the lowestlowest numbered numbered unused descriptorunused descriptor

System Programming 6

Opening and Closing a File (cont’d)Opening and Closing a File (cont’d)

– Oflag : Oflag : • required option flags: O_RDONLY, O_WRONLY, O_RDWRrequired option flags: O_RDONLY, O_WRONLY, O_RDWR• optional flags: O_APPEND, O_CREAT, ….., O_SYNCoptional flags: O_APPEND, O_CREAT, ….., O_SYNC• O_APPEND : Append to the end of file on each writeO_APPEND : Append to the end of file on each write• O_CREAT : create a file if not exists; needs the third argument O_CREAT : create a file if not exists; needs the third argument modemode

– If the file exists, no effect (w/o O_EXCL)If the file exists, no effect (w/o O_EXCL) • O_EXCL: Generate error if used with O_CREAT and the file existsO_EXCL: Generate error if used with O_CREAT and the file exists

– The test and creation of a file is an The test and creation of a file is an atomic operationatomic operation• O_TRUNC: if the file exists and opened for either write-only or read-O_TRUNC: if the file exists and opened for either write-only or read-

write, truncate its length to 0 write, truncate its length to 0• O_SYNC: Each write wait for physical I/O to completeO_SYNC: Each write wait for physical I/O to complete• O_NONBLOCK, O_NOCTTY, ..O_NONBLOCK, O_NOCTTY, ..

– ex) openex) open(“/home/kim/afile”,(“/home/kim/afile”, ( (O_RDWR | O_CREAT | O_TRUNC),O_RDWR | O_CREAT | O_TRUNC),0644)0644)

System Programming 7

Opening and Closing a File (cont’d)Opening and Closing a File (cont’d) creat() : creat() :

– equivalent to equivalent to open(pathname, open(pathname,

O_WRONLY | O_CREAT |O_TRUNCO_WRONLY | O_CREAT |O_TRUNC, mode), mode)

– file opened only for writingfile opened only for writing

** if we want to creat a file for writing and then reading it, use if we want to creat a file for writing and then reading it, use open(pathname, open(pathname, O_RDWR | O_CREAT |O_TRUNCO_RDWR | O_CREAT |O_TRUNC, mode), mode)

close() : close a fileclose() : close a file– When a processor terminates, all open files are automatically closedWhen a processor terminates, all open files are automatically closed

System Programming 8

Positioning a FilePositioning a File

Off_t lseek(int Off_t lseek(int filedesfiledes, off_t , off_t offsetoffset, int , int whencewhence););

returns : new file offset if OK, -1 on errorreturns : new file offset if OK, -1 on error

lseek sets the file offset.lseek sets the file offset.– whencewhence

• SEEK_SETSEEK_SET : the file offset is set from the : the file offset is set from the beginningbeginning of file of file• SEEK_CURSEEK_CUR : the file offset is set to : the file offset is set to offsetoffset plus the plus the currentcurrent file position file position• SEEK_ENDSEEK_END : the file offset is set to : the file offset is set to offsetoffset plus the plus the endend of file position.of file position.• offset offset can be positive or negativecan be positive or negative

– current file offset current file offset : measures the number of : measures the number of bytes bytes from the from the beginning of the file (nonnegative number)beginning of the file (nonnegative number)

– read()read() and and write()write() start at the current file offset and cause the start at the current file offset and cause the offset to be incrementedoffset to be incremented

– The offset is initialized to 0 when a file opened w/o The offset is initialized to 0 when a file opened w/o O_APPENDO_APPEND

System Programming 9

lseeklseek cont’d cont’d

– DOES NOTDOES NOT cause any I/O to take place; only record the offset cause any I/O to take place; only record the offset within the kernelwithin the kernel

– current_position = lseek(fd, 0, SEEK_CUR);current_position = lseek(fd, 0, SEEK_CUR);• to determine the current offsetto determine the current offset• to determine if the referenced file is capable of seeking (pipe or FIFO to determine if the referenced file is capable of seeking (pipe or FIFO

returns -1)returns -1)

– The file offset can be greater than the file’s current size and The file offset can be greater than the file’s current size and the next write will extend the file (creating a hole in the file)the next write will extend the file (creating a hole in the file)

System Programming 10

Reading and Writing a FileReading and Writing a File

ssize_t read(int ssize_t read(int filedesfiledes, void , void *buff*buff, size_t , size_t nbytesnbytes););

returns : number of bytes read, 0 if end of file, -1 on errorreturns : number of bytes read, 0 if end of file, -1 on errorssize_t write(int ssize_t write(int filedesfiledes, const void , const void *buff*buff, size_t , size_t nbytesnbytes););

returns : number of bytes written if OK, -1 on errorreturns : number of bytes written if OK, -1 on error

read() : read() : reads data up to nbytes from a file and store into the buffer.

– read() may not read nbytes because (of)• end of file is reached• reading from a terminal device, normally up to one line is read• reading from a network, buffering within the network• Some record-oriented devices (magnetic tape) return up to a single

record at a time

System Programming 11

Reading and Writing a File (cont’d)Reading and Writing a File (cont’d)

ssize_t read(int ssize_t read(int filedesfiledes, void , void *buff*buff, size_t , size_t nbytesnbytes););

returns : number of bytes read, 0 if end of file, -1 on errorreturns : number of bytes read, 0 if end of file, -1 on errorssize_t write(int ssize_t write(int filedesfiledes, const void , const void *buff*buff, size_t , size_t nbytesnbytes););

returns : number of bytes written if OK, -1 on errorreturns : number of bytes written if OK, -1 on error

write(): writes nbytes data from the buffer to a file • write may not write nbytes data due to write errors caused by full disk

or exceeding the file size limit

System Programming 12

File Sharing in UNIXFile Sharing in UNIX Kernel data structures for file sharing by Kernel data structures for file sharing by processesprocesses

– A File descriptor table within a process table entry (or per A File descriptor table within a process table entry (or per process) process)

• Each file descriptor with the Each file descriptor with the file descriptor flagsfile descriptor flags and and a pointer to a filea pointer to a file table entrytable entry

– A File table for all open files in the kernelA File table for all open files in the kernel• thethe file status flagsfile status flags for the file (read, write, append, sync …) for the file (read, write, append, sync …)• thethe current file offsetcurrent file offset• a pointer to the v-node table entry for the filea pointer to the v-node table entry for the file

– V-node table for open filesV-node table for open files• a v-node (i-node) structure for each open filea v-node (i-node) structure for each open file

– v-node : type of the file, pointers to file operation functions, i-nodev-node : type of the file, pointers to file operation functions, i-node

– i-node : owner of the file, i-node : owner of the file, the size of thethe size of the filefile, the device, pointers to data , the device, pointers to data block,.. etc.block,.. etc.

System Programming 13

Kernel data Structures for open filesKernel data Structures for open files

Process table entry

...

file status flag

current file offset

v-node ptr

file status flag

current file offset

v-node ptr

v-node

information

i-node

information

current file size

v-node

information

i-node

information

current file size

File table

V-node table

fd flag ptr fd 0 :

fd 1:

fd 2:

System Programming 14

...

fd flag ptr fd 0 :

fd 1:

fd 2:

Process table entry

...

fd flag ptr fd 0 :

fd 1:

fd 2:

Processes sharing a open fileProcesses sharing a open file

file status flag

current file offset

v-node ptr

file status flag

current file offset

v-node ptr

File table

v-node

information

i-node

information

current file size

V-node table

fdb = open(“/etc/passwd”, O_RDONLY);

proc A proc B

fda = open(“/etc/passwd”, O_RDONLY);

System Programming 15

What really happens?What really happens? After each write completeAfter each write complete

– the the current file offsetcurrent file offset incremented by the number of bytes incremented by the number of bytes written.written.

– if the current file offset > the current size, the current file size is if the current file offset > the current size, the current file size is set to the current file offsetset to the current file offset

Each write with O_APPEND Each write with O_APPEND – O_APPEND flag set in the file status flagsO_APPEND flag set in the file status flags– the current file offset is set to the current file size from i-node the current file offset is set to the current file size from i-node

before every writebefore every write– this forces every write to be appended to the current end of filethis forces every write to be appended to the current end of file

lseek() lseek() - only modifies the current file offset (no I/O)- only modifies the current file offset (no I/O) When lseek() positions a file to the end of fileWhen lseek() positions a file to the end of file

– the current file offset is set to the current file size (no I/O)the current file offset is set to the current file size (no I/O)

System Programming 16

Atomic OperationsAtomic Operations

An operation that cannot be separated, that is, indivisibleAn operation that cannot be separated, that is, indivisible What can happen if two processes perform the following What can happen if two processes perform the following

code on the same filecode on the same file– Appending to a FileAppending to a File

– Create a FileCreate a File

/* A old way without O_APPEND option */fd = open(“afile”, O_WRONLY);lseek(fd, 0L, SEEK_END); /* position to EOF */write(fd, buff, 100); /* and write */

/* A old way without O_CREAT and O_EXCL options */if ( (fd = open(“afile”, O_WRONLY)) < 0) if (errno == ENOENT) /* no such file or dir */

creat(“afile”, 0644); /* create a file even if already exists */

Atomic Operations (continued)Atomic Operations (continued)

New ways with optionsNew ways with options– Appending to a fileAppending to a file

– Create a fileCreate a file

System Programming 17

open("afile", O_WRONLY|O_CREAT, 0644);

fd = open(“afile”, O_WRONLY|O_APPEND);write(fd, buff, 100);

System Programming 18

Duplicating a File DescriptorDuplicating a File Descriptorint dup(int int dup(int filedesfiledes););

int dup2(int int dup2(int filedesfiledes, int , int filedes2filedes2))

Both return : new file descriptor if OK, -1 on errorBoth return : new file descriptor if OK, -1 on error

Duplicate an existing file descriptorDuplicate an existing file descriptor– dup() returns dup() returns the lowest numbered availablethe lowest numbered available file descriptor for file descriptor for

the new file descriptorthe new file descriptor• Ex) dup(5) copies file descriptor 5 into Ex) dup(5) copies file descriptor 5 into the lowest numbered availablethe lowest numbered available

file descriptor.file descriptor.

– dup2 copies dup2 copies filedesfiledes to to filedes2filedes2• dup2 closes filedes2 if already opened before the duplicationdup2 closes filedes2 if already opened before the duplication• dup2 is an dup2 is an atomic operationatomic operation

System Programming 19

file status flag

current file offset

v-node ptr

File table

Process table entry

...

fd flag ptr fd 0 :

fd 1:

fd 2:

Duplicating a File DescriptorDuplicating a File Descriptor

v-node

information

i-node

information

current file size

V-node table

/* connect stdout to a file */ fd=open(“file”,O_WRONLY);close(1); dup(fd); /* or */fd=open(“file”,O_WRONLY); dup2(fd,1)

/* I/O redirection: shell example */$ a.out > file