advanced unix
TRANSCRIPT
1
Complete Course in Unix
•Overview
•File System
•Shell Programming
•Process Management
•Inter Process Communication
•System Administration
•Utilities
2
Overview
3
Characteristics of Unix System
• Time-sharing
• Multi-user, multi-process system
• Written in high level language
• Hierarchical file system
• Consistent file formats
4
Architecture of Unix System
• System Kernel
• Utilities and application software
• File Structure
5
Kernel Functions
• Schedule Processes
• Keep track of files
• Control hardware devices
6
File System
7
File System
• Characteristics of file system
• Types of files
• File system layout
• File structure related system calls
8
File SystemCharacteristics of Unix File System
• Hierarchical Structure
• Consistent treatment of file data
• Ability to create and delete files
• Dynamic growth of files
• Treatment of peripheral devices as files
9
File SystemTypes of Files
• Regular files
• Directories
• Special files
– Pipes
– Named Pipes
– I/O devices
10
File SystemFile System Layout
• Boot block contains bootstrap code to initialize the operating system
• Super block describes the state of a file system
• Inode list is a list of inodes
• Data blocks contain file and administrative data
11
File structure related system calls
12
OPEN
#include <fcntl.h>
fd = open ( pathname, flags, modes)
pathname is the filename
flags indicate the type of open
modes give the file permission if it is being created
13
CREAT
Creates a new file with the indicated filename and access permission modes. If the file already exists, creat truncates the file.
fd = creat ( filename, mode)
char *filename;
int mode;
14
READ
Read returns the number of bytes it read into the buffer. It is generally count or a value less than count if the number of bytes left to be read are less than the count
number = read ( fd, buffer, count )
fd is the file descriptor
buffer is the address of the data structure
count is the no of bytes to be read
number is the no of bytes read
15
WRITE
Write writes count bytes of data from user address buffer to the file descriptor fd
number = write ( fd, buffer, count )
fd is the file descriptor
buffer is the address of the data structure
count is the no of bytes to be written
number is the no of bytes actually written
16
LSEEKChanges the position of read-write pointer
lseek ( fd, offset, reference )
int fd is the file descriptor
long offset is the byte offset
int reference indicates where the offset is to be considered from
reference values
0 set the pointer to offset bytes from beginning of file
1 increment the current value of pointer by offset
2 set the pointer to size of file plus offste bytes
17
CLOSE
Close closes a file descriptor obtained from a prior open, creat, dup, pipe or fcntl system call
close ( fd )
int fd;
18
STAT
Stat returns status information about the specified file
stat ( filename, statbuffer )
char *filename is the absolute file name
struct stat *statbuffer is the address of the data structure in the user process which will contain status information
19
FSTAT
Fstat returns the status information of an open file whose file descriptor is fd
fstat ( fd, statbuffer )
int fd is the file descriptor
struct stat *statbuffer is the address of the data structure in the user process which will contain status information
20
DUP
Dup duplicates the specified file descriptor, returning the lowest available file descriptor
newfd = dup ( fd )
int fd is the file descriptor
int newfd is the new file descriptor
21
LINK
Link gives another name filename2 to an existing file filename1
link ( filename1, filename2 )
char *filename1 is an existing file
char *filename2 is the new name which will also point to the source file name
22
ACCESSChecks if calling process has r,w or x permission for a file
access ( filename, access_mode )
char *filename is the name of the file
access_mode
04 read
02 write
01 read
00 existence
Return value
0 access allowed
-1 access not allowed
23
CHMODChmod changes the access permissions of the indicated file to the specified mode
chmod(filename, mode)
char *filename
Modes
4000 setuid bit
2000 setgid bit
1000 sticky bit
0400 read for owner
0040 read for group
0004 read for others
24
CHOWN
Chown changes the owner and the group of the indicated file to the specified owner and group IDs
chown(filename, owner, group)
char *filename name of the file
int owner, group
25
UMASK
Sets the file mode creation mask and return the old value. When creating a file, permissions are turned off if the corresponding bits in the mask are set
umask(mask)
int mask
26
IOCTL
Ioctl does device-specific operations on the open device whose file descriptor is fd.
ioctl ( fd, cmd, arg_pointer)
int fd;
int cmd is the device specific command
arg_pointer defines a parameter whose type depends on the command
27
SHELL PROGRAMMING
28
What is a Shell
• Standard command and programming language
• Interprets command lines
• Runs other programs
• Reads commands either from the prompt or from a file
29
Shell Types
• sh (bourne shell)
• csh
• ksh (korn shell)
• bash (bourne again shell)
30
Input/Output Redirection
• Create Files
• Append to files
• Use files as input to the shell
• Truncate files
31
Input/Output RedirectionRedirector Operators
< Open the following file as stdin
> Open the following file as stdout
>> Append to the following file
2>&1 Merge file descriptor 2 with 1
| Pipe stdout into stdin
32
File/Directory Commandscd Change directory
ls List directory’s contents
mkdir Create directory
rmdir Remove directory
cat Concatenate and write files’ contents
ln Create links to existing files
mv Move files
cp Copy files
rm Remove files
33
Selection Commandsawk Pattern scanning and processing language
cut Select columns
diff Compare files
grep Select lines or rows
sed Edit streams of data
head Select lines from top in a file
tail Select lines from bottom in a file
wc Count words, characters and lines in a file
34
Shell Variables• Special Variables
• Environment Variables
• Program Variables
35
Special Variables$# Number of positional arguments
$0 Program/command name invoked
$1,$2.. Positional parameters passed to the shell
$* Expands to “$1 $2...”
$@ Expands to “$1” “$2”..
$? Return code from the last executed command
$$ PID of the current shell
36
Environment VariablesHOME Pathname of the user’s login directory
PATH Shell’s search path for commands
PS1 Primary prompt
IFS Internal field separator
PWD Current working directory
37
Program VariablesNormal variables using assignment operator
e.g. age=12
38
QuotesSingle Quotes No evaluation whatsoever
Double Quotes No evaluation expect for $, \ and `
Backslash Character following backslash is protected from evaluation
39
Expr
• Evaluates arguments and returns 0 or 1
• Returns numeric value from an expression
E.g. if expr $a = $b
or if expr $str = “abc”
a=`expr $a + 1`
40
Foreground/Background Execution
• Background processing by placing an “&” at the end
• Background processes run effectively wrapped around by nohup
• loops run in background enclosed in parentheses
• fg command moves a background job into foreground
41
READ
• Main input mechanism for a shell program
• Gets a line from the standard input and can store the value in a variable
• Can read a file line by line
42
Control Structures
43
If ..then ..else
If <condition>
then
<command>
[elif <condition>
<command>
else
<command>]
fi
44
Case
case word in
pattern1) action1 ;;
pattern2|pattern3) action2 ;;
.......................................
.......................................
*) default action ;;
esac
45
While .. do
while <condition>
do
commands
done
46
test
Tests if an expression is true
Returns a 0 if true else returns a non-zero value
47
test
Integer operators used with test
•eq is equal to
•ne is not equal to
•lt is less than
•le less than or equal to
•gt greater than
•ge greater than or equal to
48
testSyntax
test <expression>
or
[ <expression> ] alternative syntax
49
testFile Operators
-d file exists and is a directory
-f file exists and is an ordinary file
-r file exists and is readable
-s file exists and has a non-zero size
-w file exists and is writable
-x file exists and is executable
50
Filename Wildcards
* Match 0 or more characters
? Match exactly one character
[1,2,3] Match any character in the square brackets
[a-z] Match any characters between a to z
[!0-9] Match any character not in the square brackets
+(abc) Match one or more instance of abc
*(abc) Match zero or more instances of abc
51
Functions
function identifier {list;}
or
identifier() {list;}
52
Command Execution
Grave accents or backquotes
`command`
Executing in a separate environment
(command)
Simple execution
{command}
53
Disk Usage Commands
df Gives the disk usage statistics of all the filesystems
du Summarizes disk usage
54
Setuid/Setgid Permissions
This powerful feature enables the effective userid/groupid of the calling process to be the same as the userid/groupid of the owner of the file.
Used most frequently when a script needs to be executed as root.
55
Setuid/Setgid Permissions
The setuid bit is turned on like this
chmod 4000 filename
and the permissions then look like this
-rws------
For setgid
chmod 2000 filename
----rws---
56
Sticky Bit
This is set to prevent files from being removed from a directory by arbitrary users.
chmod 1000 directoryname
and the permissions look like this
drwxr-xr-t
In this case, only the owner of the directory, owner of the file or the superuser can remove a file in the directory
57
Find
Finds files that match a given set of parameters
find -name filename Finds files with the “filename”
find -perm mode Find files with “given permissions”
find -type c File types
find -user name Belonging to given user
find -size n Of size n
find -atime n Access time n days ago
58
Find
Operators used with find
find -name filename -print Prints the filename
-exec
find -name filename -exec rm -f {} \; Executes command
59
File
Tells about the contents of the file
file filename
60
Copy Files
cp command
example
cp file1 file2
61
Copy Directories
cp -r|R command
Recursively copies the files under a directory and its subdirectories
tar command
Use tar cvf to create a tar of the subdirectories and files under a given directory
e.g. tar cvf “tarname” “path of files/subdirectories”
62
Copy Directories
Untar a tar file using
tar xvf tarname “untar location”
63
Used for sending mails
e.g.
cat msg | mail -s “subject” recipient
Note that the mail program expects its input from the stdin
64
Cut
Selects columns or fields
-c for col
-f for field
-d for delimiter
65
Sort
Sort divides each lines into fields at whitespace (blanks or tabs) and sorts the lines by fields from left to right
sort +n -m
sorts the nth field, stops sorting at the mth field
66
Process Status
Produces a report summarizing execution statistics for the current processes.
E.g.
ps -ef gives general stats
ps -fu username gives stats about all the processes running under the user username
67
Time
Used to time a command
e.g.
time progname
68
Uptime
Used to get a rough estimate of the system load
e.g.
Uptime
The result looks like this
12:40pm up 4 day(s), 23:59, 1 user, load average: 0.00, 0.00, 0.01
69
Cron
Does periodic execution of jobs
Listing cron entries
crontab -l
Editing cron entries
crontab -e
e.g 20 10 * * * /$HOME/myscript
70
Process Related System Calls
71
EXEC
Execve executes the program file filename, overlaying the address space of the executing process.Argv is an array of character string parameters to the execed program and envp is an array of character strings that are the environment of the new process
execve(filename, argv, envp)
char *filename
char *argv[]
char *envp[]
72
FORK
Fork creates a new process. The child process is a logical copy of the parent process, except that the parent’s return value from the fork is the process ID of the child, and the child’s return value is 0
73
WAIT
Wait causes the process to sleep until it discovers a child process that had exited or a process asleep in trace mode. If wait_stat is not 0, it points to an address that contains status information on return from the call.
wait(wait_stat)
int *wait_stat
74
EXIT
Exit causes the calling process to terminate, reporting the 8 low-order bits of status to its waiting parent. The kernel may call exit internally, in response to certain signals
exit(status)
int status
75
Some other System Calls
• getuid()• geteuid()• getgid()• getegid()• getpid()• getppid()
• Returns the real user ID• Returns the effective user
ID• Returns the real group ID• Returns the effective group
ID• Returns the process ID• Returns the parent process
ID
76
SIGNAL
Software Interrupts
77
Classification of Signals
• Termination of a process
• Process induced exceptions
• Unrecoverable conditions during system calls
• Unexpected error condition during a system call
• Tracing execution of a process
78
Handling Signals
• Exit on receipt of a signal
• Ignore the signal
• Execute a particular function on receipt of a signal
79
Signal Syntax
#include <signal.h>
signal(sig,function)
int sig;
void (*func)()
Signal allows the calling process to control signal processing.
80
Signal SyntaxSome values of sig
SIGHUP hangup
SIGINT interrupt
SIGQUIT quit
SIGKILL kill
SIGFPE floating point exception
SIGBUS bus error
SIGPIPE write on a pipe with no reader
SIGTERM software termination
81
Signal Syntax
The function is interpreted as
SIG_DFL default operation. Process terminates except for some
signals
SIG_IGN ignore the signal.kernel calls a function in the process with signal number as the argument.
SIGKILL can’t be ignored
82
KillKill sends the signal sig to the process identified by pid
kill(pid,sig)
int pid,sig
pid > 0 send signal to process whose PID is pid
pid 0 send signal to processes whose process group ID is pid of sender
pid -1 if effective UID of sender is of superuser send signal to all processes otherwise to processes whose real UID equals effective UID of the sender
83
Inter Process Communication
84
Types of IPC
• Pipes
• Named Pipes
• Messages
• Shared Memory
• Semaphores
• Sockets
85
Pipes
• Allow data transfer in a FIFO manner
• Allow synchronization of process execution
• Use file system for data storage
• Can be created using “pipe()” system call
• Pipe manipulation is done using regular system calls for files e.g. read(), write()
• Only related processes can access a pipe
86
PipesSyntax
pipe (file_descriptors)
where file_descriptors is a pointer to an array of 2 integers which are the file descriptors for reading and writing the pipe
87
Named Pipes
• Similar to pipes
• Have directory entries
• “open” system call used to open named pipes
• “mknod” system call to create named pipes
• Unrelated processes can also communicate using named pipes
88
MESSAGES
• Allow processes to send data streams to arbitrary processes
• Four system calls cover all the operations
– msgget
– msgsend
– msgrec
– msgctl
89
msgget
Returns the message queue identifier assocaited with the key
msgget(key, flag)
key is the name of the message queue
flag is the modes which can be passed
90
msgsndmsgsnd (id, message, size, flag)
Sends message to the queue associated with the message queue identifier specified by id
message is of the form
struct msgbuf
{
long mtype;
char mtext[];
}
Flags can be
IPC_NOWAIT- Return -1 if the message cant be stored
MSG_NOERROR- Message bigger than size is shortened with no error returned
91
msgrcvmsgsnd (id, message, size, type, flag)
Reads message from the queue associated with the message queue identifier specified by id
message is of the form
struct msgbuf
{
long mtype;
char mtext[];
}
92
msgctlMsgctl ( id, command, buffer )
Provides a variety of message control functions as specified by command
Commands can be
IPC_STAT store information about queue in the buffer
IPC_SET Change the information in the buffer
IPC_RMID remove the message queue identifier specified by id
93
SHARED MEMORY
• Allows processes to share part of their virtual address space
• System calls similar to the ones in messages
94
shmget
int shmget ( key, size, flag )
key is the name of the shared memorysize is the size of shared memory segment in bytesflag is a combination of
IPC_CREATIPC_EXCLmode
95
shmat
Inserts a shared memory segment into a process address spacechar *shmat ( id, at_address, flag )
id is the shared memory segment identifiersize is the address in process address space where the shared memory segment has to be attachedflag is a combination of
SHM_RDONLY read only0 read,write
96
shmdt
Removes a shared memory segment into a process address space
int shmdt ( dt_address )
dt_address is the address in the process address space of a shared memory segment to be detached
97
shmctl
shmctl ( id, cmd, buf)
Provides shared memory control operations as specified by thecmd
cmd IPC_STATIPC_SET
IPC_RMIDSHM_LOCKSHM_UNLOCK
98
SEMAPHORE
• Non-negative integer count used to coordinate access to the resources
• Initial semaphore count set to the number of free resources
• Count decremented or incremented by the threads or processes as and when they acquire or free resources.
• Threads block at count zero till it becomes positive
99
semget
semget ( key, num, flag )
Returns or creates the semaphore identifier associated with the key
key is the name of the semaphore set
number defines the number of semaphores in the set
flag is a combination of
IPC_CREAT
IPC_EXCL
100
semopsemop ( id, operations, number )
Performs operations on the semaphore
id is the semaphore identifier
operations is the array of semaphore operation structure
operation consists of
- semaphore number
-operation
-flags
number is the number of entries in operations
101
semctl
semctl ( id, semnum, cmd, ... )
Performs operations on the semaphore as specified by the cmd
id is the semaphore identifier
Fourth argument is optional, depending upon the operation requested.
If required, then it’s a buffer.
cmd GETVAL, SETVAL, GETPID, GETNCNT, GETZCNT
102
SOCKET
Creates an endpoint for communication and returns a descriptor
int socket ( domain, type, protocol )
domain specifies communication domain
type type of communication over socket (virtual circuit or datagram)
protocol protocol to control the communication
103
Other Socket System Calls
104
Close
Closes the communication endpoint
close ( sd )
sd is the socket descriptor
105
Bind
Assigns a name to an unnamed socket
int bind ( sd, sockname, namelength)
sd is the socket descriptor
sockname is the name of the socket
namelength is the length of the name
106
Connect
Makes a connection to an existing socket
int connect ( sd, sockaddr, namelength)
sd is the socket descriptor
sockaddr is the name of the target socket
107
Accept
Receives incoming requests for a connection
accept ( fd, addr, addrlen )
addr user data array which kernel fills with the address of the connecting client
108
send
Transmits data over a connected socket
send ( sd, msg, length, flag)
msg pointer to data being sent
length length of the data
109
recv
Receives data over a connected socket
recv ( sd, buf, length, flag)
buf pointer to data being sent
length length of the data
110
shutdown
Closes a socket connection
shutdown ( sd, mode )
mode specifies which side no longer permits data transmission
111
System Administration
• File System Administration
• Machine Management
• System Tuning and Performance Management
• System Accounting
• User Management
• Backup/Restore Activities
• Setting up Peripheral Devices
• Routine Maintenance
112
File System Administration
• Creation and Maintenance of File Systems
• File System consistency checks
• Monitoring of disk space
• Routine Maintenance such as removing inactive files
113
File System Administration Commands
• TASK
• Check File System
• Display disk usage
• List files by size
• Identify FileSystem type
• Create File System
• Mount File System
• Unmount File System
• SHELL COMMAND
• fsck
• df
• du
• fstyp
• mkfs
• mount
• umount
114
Machine Management Commands
• TASK
• Set boot defaults
• Change to firmware
• Power Off
• Reboot
• Display Configuration
• Display System
• SHELL COMMAND
• fltboot
• shutdown
• shutdown
• shutdown
• prtconf
• uname
115
System States
• System State
• 0
• s
• 2
• 3
• 5
• 6
• q
• Description
• Powerdown State
• Single user
• Multiuser
• Start Remote File Share
• Switch to firmware
• Halt & Reboot OS
• Reexamine (/etc/inittab)
116
Changing System StatesSwitching from multiuser to single user
SHUTDOWN -I S
INIT S
Switching to any state
INIT <System State>
Switching to multiuser
INIT 2
Turning system off
SHUTDOWN -I 0
117
User and Group Management
• Assigning and Maintaining logins and passwords
• Organization of system resources to suit particular needs of the users
• Communicating with users
118
User Management Commands
• Task
• Add User
• Add Group
• Change password
• Listing users and groups
• Modify user attributes
• Modify group attributes
• Remove user
• Remove group
• Command
• useradd
• groupadd
• passwd
• logins, listusers
• usermod
• groupmod
• userdel
• groupdel
119
Communicating with Users
• Keeping users informed about system status and servicing details
• Using “Wall” to send messages to the users
• Using utilities like “mail”
• Message of the day facility using “/etc/motd”
120
SYSTEM UTILITIES
121
MAKE
Provides a method for maintaining an up-to-date version of programs that consist of a number of files generated in a variety of ways
122
MAKEOperations
• Find the target in the description file
• Ensure that all the dependencies (files) of the target exist and are up to date
• Create the target in case any of the dependencies have been modified
123
MAKEMakeFile
Comments All characters after a ‘#’
Continuation lines Lines ending in ‘\’
Macro Definition Identifier followed by ‘=‘ sign
124
MAKEExample MakeFile “makedemo”
#makedemo
prog : x.o y.o z.o
cc x.o y.o z.o -o prog
x.o : x.c defs.h
cc x.c
y.o : y.c defs.h
cc y.c
z.o : z.c defs.h
cc z.c
125
MAKEMacros
Macro Definition
name = value
Example
OBJECTS = x.o y.o z.o
prog : $(OBJECTS)
cc $(OBJECTS) -o prog
126
SYSTEM UTILITIES
127
Grep
Searches the named files or the standard input and prints each line that contains an instance of the pattern
128
GrepUsage
grep [options] pattern filenames
Options
-n prints the line numbers
-v inverts the sense of the test
-i case insensitive search
129
GrepExample
Locate variable in C source code
grep variable *.[ch]
See if “abc” is logged in
who | grep abc
Filenames that don’t have “temp” in their names
ls | grep -v temp
List the subdirectories in the current directory
ls -ltr | grep ‘^d’
130
AWK
Scans a set of input lines, searching for lines that match any set of patterns and take a specified action for each pattern on each line that matches the pattern
131
AWKUsage
awk ‘pattern-action statements’ optional list of input files
e.g. awk ‘{print $1, $2}’ file1 file2
OR
awk -f programfile optional list of input files
where programfile is the file where the pattern-action statements are stored
132
AWKFields
Record is the sequence of characters separated by a newline character
Fields in a record are separated by sequence of blanks or tabs
$0 refers to the entire record
$1 refers to the first field
$2 refers to the second field
133
AWKPrinting
{ print }
prints all the records
{print $1 $3}
prints the first and third fields of each of the input line
134
AWKBuiltIn Variables
ARGC number of command line arguments
ARGV array of command line arguments
FILENAME name of current input file
FNR record number in current file
NR number of current record
NF number of fields in the record
OFS Output field separator
ORS Output record separator
135
AWKExamples
Print last field of each input line
{ print $NF }
Print total number of input lines
END {print NR}
Print input lines with more than 4 fields
NF > 4
Print the total no of lines that match string “abc”
/abc/ {nlines++} END {print nlines}
136
AWKBuiltIn Arithmetic Functions
cos(x)
log(x)
exp(x)
rand(x)
sin(x)
sqrt(x)
srand(x)
137
AWKBuiltIn String Functions
index(s,t) returns position of t in s
length(s) returns length of s
split(s,a) splits into array a on FS
sub(s,r) substitutes s for first r
substr(s,p) returns suffix of string starting at position p
138
AWKControl Flow Statements
if (expression) stat1 else stat2
while (expression) stat
for (expr1;expr2;expr3) stat
do stat while (expr)
break
continue
next
exit(expr)
return(expr)
139
SEDStream Editor
•Reads one line at a time from input files
•Applies the commands from a list of commands one at a time, in order, to each line
•writes to standard output
140
SEDUsage
sed ‘list of ed commands’ filenames...
Example
sed ‘s/UNIX/UNIX(TM)/g’ filenames
replaces UNIX by UNIX(TM) in all the file occurences and writes the result to standard output
141
SEDCommands
a\ text Appends text on the output before reading the next input line
b label Branch to the : command bearing the label
c\ text change lines to the following text
d delete line; read next input line
p print
r read
w write
q quit