Shell Programming & Scripting Languages

Dr.K.Sasidhar

Shell Programming & Scripting LanguagesCourse Objectives

This course explains UNIX Operating System in thorough with

its commands, editor, file systems along with applications.

Implementation of Shell scripts with Bash shell programming

using functions, Process communication, Administration.

Scripting languages extends the functionality of existing

software. This was addressed by the Python language with its

features and programming with object oriented concepts.

This course also explains about the designing of integrated web

Applications and Data base interfaces.

Course Outcomes

Identify the basic commands as well as different utilities of the UNIX & analyze why to use unix.

Understand the commands, functions and Practice with real time applications

Learn about shell and how to write, debug & execute shell script along with different commands.

Explain file handling and directory handling using different system calls.

Discuss the fundamentals of the Process as well as Signal functions & controlling of process using signals.

Learning the Python language programming with its features Design the web applications through python Implementation of object oriented programming through python.

Prerequisites

Dos commands C programming

UNIT – I Contents Introduction to Unix:- Architecture of Unix,

Features of Unix , Unix utilities – process utilities, disk utilities, networking commands, text processing utilities and backup utilities.

Introduction to UNIX file system, vi editor, file handling utilities, security by file permissions.

Software Collection of Programs 2 types

SystemSoftware

ApplicationSoftware

Software

Application software Easy-to-use programs designed to perform specific

tasks System software

Programs that support the execution and development of other programs

Two major types Operating systems Translation systems (compilers & linkers)

Software

Computer Software Relationships

UserInterface

Basic Input and Output Services (BIOS)• needed for a computer to boot up

User Interface Operating System

User Interface Application Programs

Computer Hardware

Controls and manages the computing resources Examples

Windows, Unix, MSDOS, Important services that an operating system provides:

Security: prevent unauthorized users from accessing the system

Commands to manipulate the file system Input and output on a variety of devices Window management

Operating System

Operating SystemAn Operating System is a system software that provides interface between the user and computer hardware. Operating System controls (manages) hardware and software.provides support for peripherals such as keyboard, mouse, screen, disk drives, … software applications use the OS to communicate with

peripherals. The OS typically manages (starts, stops, pauses, etc)

applications.

What is UNIX

• UNIX is a Multi-user and Multi-tasking operating system.

• That is it is designed to be used by many people at the same time (multi-user) and for multiple tasks at a time.

• Runs on a variety of processors• It provides a number of facilities:

– management of hardware resources– directory and file system– loading / execution / suspension of programs

12

UNIX versionsM

ultic

s

Unic

s

UNIX

V1

UNIX

V3

UNIX

V4

UNIX

V7

1965 1969 1970 1973 1973 1979BSD U

NIX

AT&T UNIX

BSD

4.1

BSD

4.2

BSD

4.4

1981 1983 1993

System

IIISys

tem V

1982 1984

1998 – X/OPEN & IEEE started work to merge two standards2001 – Single UNIX Specification Version 3 (SUSV3) Also known as IEEE1003.1:20012002 – ISO standardized SUSV3 and IEEE1003.1:2001

Multiple StandardsAT&T – System V Interface DefinitionX/OPEN – X/OPEN Portability Guide (XPG)IEEE – Portable Operating System Interface for Computing Environments (POSIX)

UNIX History

The UNIX operating system was developed at AT & T Bell Labs in late 1960s.

It originally began as a one man project led by Ken Thompson of Bell Labs, and has since grown to become the most widely used operating system.

In the time since UNIX was first developed, it has gone through many different generations and even mutations.

Some differ substantially from the original version, like Berkeley Software Distribution (BSD) or Linux.

Why to Use UNIX?

• Multi-tasking / multi-user

• Networking capability

• Distributed Processing

• graphical (with command line)

• Easy to program

• portable (PCs, mainframes, super-computers)

15

Features of UNIX Operating System Multi-user & Multi-tasking – UNIX allows multiple users to log onto the

system, and have each run multiple tasks. This is standard for most modern OS.

Large Number of Applications – There are an enormous amount of applications available for UNIX. They range from commercial applications such as CAD, Maya, WordPerfect, to many free applications.

Free Applications and Open source Operating System - of all of the applications available under UNIX, many of them are free. Most of the development that we do in programming courses is done under the Linux OS.

Less Resource Intensive - UNIX installations tend to be much less demanding on system resources. In many cases, the old family computer that can barely run Windows is more than sufficient to run the latest version of Linux.

Internet Development - Much of the backbone of the Internet is run by UNIX servers. Many of the more general web servers run UNIX with the Apache web server - another free application.

Structure of the UNIX system

Applications

Shell

\Kernel (OS)

Hardware

There are many standard applications:

• file system commands

• text editors

• compilers

• text processing

UNIX System Structure ….. The Kernel - handles memory management, input and output

requests, and program scheduling. The kernel is the heart of OS. It provides the basic software connection to the hardware.

The Shell and Graphical User Interfaces (GUIs) - UNIX shell provides a “command line” interface which allows the user to type in commands. These commands are translated by the shell into something the kernel can comprehend, and then executed by the kernel.

Primary command-line shells are: Bourne shell, Bash Shell, korn Shell, C Shell

The Built-in System Utilities - Programs that allow a user to perform tasks which involve complex actions. Utilities provide user interface functions that are basic to an operating system.

Examples: Commands to see the contents of a directory, move & copy files, remove files, etc...

Application Software & Utilities – The programs that are bundled with the OS distribution, or available separately. These can range from additional or different versions of basic utilities, to full scale commercial applications.

UNIX System Structure …..

Flavors of Unix

There are many versions of Unix: SCO UNIX

SysV (from AT&T) BSD (from Berkeley) Solaris (Sun) IRIX (SGI) AIX (IBM) LINUX (free software)

File System is a group of files The relevant information arranged in a logical manner. Different Operating systems have different file systems. In Unix, everything will be treated as a file. Whether it is application or

device or utility or data or directory It is similar to tree data structure. The files will be denoted with absolute or relative path

Ex: /home/usr2/file1 absolute path Relative path is nothing but pwd (present working directory) cd ram relative path To check the block size the command is $cmchk

The UNIX File System

The UNIX File System

• A simplified UNIX directory/file system:

/ Root

bindev lib

User

home/usr tmp. . .

. .

. User2

User1 User3

22

cat date

etc

• /bin: contains executable files for most of the unix commands.

• /dev: contain files that control various input & output devices.

• /etc: contains system administration files, such as password file.

• /lib: contains all the library functions in binary form.• /usr: contains several directories each associated with a

particular user.• /tmp: contain the temporary files created by unix or by

any user.• /etc: contains configuration files of the system.

23

• File: It is a container for storing information.

• A file is of the following types.

• Text file or Ordinary file : contains printable characters.

Ex: program source code, documents, scripts

• Binary file: contains both printable & non printable characters. Ex: gif, jpeg, executable etc..

• Directory file: It maintains some details of the files & subdirectories that it contains.

24

Files

• Every directory entry contains 2 components.• 1.file name.• 2. Unique identification number for the file or

directory.

25

Files The file systems in UNIX maintains with four blocks

Boot Block Bootstrap loader program Super Block state of the file system– how large it

is, etc.. Inode Table Data Block contains actual file contents. An

allocated block can belong to only one file in the file system.

Inode Table The information related to all the files is stored in an Inode Table on the

disk. For each file there is an inode entry in the table. Each entry is made up of 64 bytes and contains relevant details for that

file. The details are:

Owner of the file Group to which the owner belongs Type of the file File access permissions Date and time of last access Date and time of last modification Number of links to the file Size of the file Addresses of the blocks where the file is physically present

How Does Unix Access Files? A file is identified by Unix by a unique “ Inode number”

associated with it. Can display the inode number with a file by using the

command ls-i Ex: $ls – i reports

reports 12324 Here 12324 is inode number. The inode no is nothing but the index into the inode table

where the information about the file is stored. Among all the slots the inode table slot number 12324 contains

information about the file reports

Storage of Files Each inode entry in the inode table specifies

completely where the contents of the files are stored on the disk.

These addresses may be numbered 0 through 12. The addresses 0 to 9 point to 1kb blocks on disk. Ex: a file of 3kb may have its entries as shown in the

following diagram.

owner GroupFile Type

PermissionsAccess Time

Modification Time Inode Modi. Time File size 0123456

12

497052313401

4970

5231

3401

Creating files cat

Used to create a fileTo append the filesTo display the file content.

$cat > test Type the content of the file and press Ctrl d to

stop / save the file. $cat test

Displays the content of file test. $cat sample1 sample2 >> newsample

newsample file will be created with the contents of sample and sample2

Creating files …

If new sample already contains something it would be overwritten.

If you want it should remain intact and content of sample1 and sample2 should get append to it then use

“ append output redirection operator” >> Ex: $ cat sample1 sample2 >> newsample

Cp (Copying Files)• To create an exact copy of a file you can use the “cp” command.

Syntax: cp [-option] source destinationEg: cp file1 file2Here the contents of file1 is copied to file2. if file2 is not existed

then it will be created.

Eg: cp file1 file2 dirfile1 file2 contents are copied to the directory named as dir.

• To copy files from or to different directories is by specifying their names with the path.

• $cp /usr/aa/chapter1 /usr/aa/newbook/chap1

33

File handling utilities

Cp turns to interactive when –i option is used &

destination file also exists.

$cp -i file1 file2overwrite file2 (yes/no)?

Y at this prompt overwrites the file.

34

Copying Files

• mv (Moving and Renaming Files)

Used to rename the files / directories.

$ mv test sample

Here test is renamed as sample, provided sample directory name is not already existing.

• Moving a file implies removing it from its current location and copying it at a new location.

35

mv command

Listing Files and Directories

$ls lists files and directoriesls – a hidden file. Displays the hidden files also.$ ls p* lists the files starts with p$ls ?ain lists the files whose names match with ain

from second character onwardsls [a-m][c-z][4-9]?? this will list all 5 character

filenames in the current directory whose first character is in the range a to m,

second character in the range c to z, the third character is in the range 4 to 9, where as the fourth and fifth character are any valid characters.

ls - l ls –l

long listing Out put: total 22 no. of disk blocks occupied -rwxr-x—x 1 user1 group 02 jan 16 10:15 ecm32 drwxr-xr-x 1 user1 group 04 jan 16 12:23 ecm31

File types ( meaning of first character)

File Type Meaning

- Ordinary file

d Directory file

c Character special file

b Block special file

l Symbolic link file

s Semaphore

p Named pipe

M Shared memory file

file command Recognizes several types of files. Works with wild cards also $ file *

• Used to create links (both soft & hard links).• It creates the alias & increase the link count by one.• $ln file1 file2• Hard link means the link will be established with the

data on the disk• Softlink is nothing but symbolic link that refers to

existed filename. ( it refers to fine name itself.)

40

ln command

Hard Link hard links : Refer to the specific location of physical data.

Soft Link symbolic links: Refer to a symbolic path indicating the abstract location of another file

wc command Counts the number of lines, words and

characters in the specified file or files.

-l counts and display no. of lines in the file -w counts and display no. of words in the file -c counts and display no. of bytes in the file. -m displays the no. of characters in the file. -L displays the length of the longest line in

the file.

$ wc –lc file1 file2

• Rm (Deleting Files)– To delete or remove a file, you use the “rm”

command. – Ex 1: $rm mylisting will delete “mylisting”. -f deletes the file forcibly –i option removes the files interactively. Ex 2: $rm –i file1 With –r option recursively removes directories. $rm –r dir1

44

• rmdir (remove directories )• Removes empty directories.• $rmdir book removes directory named book if it is empty.• $rmdir dbs doc dmc Removes the directories dbs doc dmc.

45

• find: Searches the files in a directory hierarchy • It recursively examines a directory tree to look for

matching file and then takes some action on the selected files.

Syntax:• find path_list selection_criteria action• To locate all files named a. out use$find / -name a. out –print• ‘/’ indicates search should start from root directory.

46

• To locate all c files in current directory $find . -name “*.c” –print• To find all files begin with an uppercase letter use $find . –name ‘[A-Z]*’ –print Find operators:Find uses 3 operators!,-a ,-o

47

Security by file permissionsUnix follows a 3-tiered file protection system.There are 3 categories- owner ,group ,others.

owner’s permission group permission other’s permission r w x r - x r - -Each category contains read ,write ,execute permissions .

rwx->presence of all permissions.

r-x->absence of write permission

r-- -> absence of write ,execute permission

48

Security by file permissions• Chmod: change file permission mode

chmod sets a file’s permission (read, write and execute) mode for all three categories of users (owner, group and others)

49

Syntax

chmod category operation permission file(s)

The command contains three components:

• category of user (owner, group or others)

• operation to be performed (assign or remove a permission)

• Permission type (read, write or execute)

50

• Abbreviations used by chmodCategory

u-user g-group o-others

operation +-assign permission -remove permission=-assigns absolute permission a-all

permissionsr-read permission w-write permission

x-execute permission

51

octal methodPermission weight

read(r) 4 write (w) 2 execute (e) 1 Ex: $chmod 700 myfile

chown This command changes file owner and / or

group ownership for each given file.

Syntax: chown [OPTION]... [OWNER][:[GROUP]] FILE Ex1: chown ramu file1 Ex2: chown ram /path/to/file1 /path/to/file2

/path/to/file3 or $ chown ram /path/to/{file1, file2, file3}

Disk related Commands

$df: disk filesystem Displays free as well as used disk space for all the file systems on the

computer. i.e. The information of device name, total blocks, total disk space, used

disk space, available disk space and mount points on a file system.

Filesystem 1K-blocks Used Available Use% Mounted on

/dev/cciss/c0d0p2 78361192 23185840 51130588 32% / /dev/cciss/c0d0p5 24797380 22273432 1243972 95%

/home Ex: $df

/ (/dev/root): 12970 blocks 27857 i-nodes The root file systems is /dev/root.df reports the number of free disk

blocks and free inodes for this file system. If we want a more detailed information about disk usage we have to

use: $df -ivt

disk utilitiesulimit

user limit and it contains a value which signifies the largest file that can be created by the user in the file system.

Ex: $ulimit 2097152 the user cannot create a file whose size

is bigger than 2097152 bytes, or 2048 KB. To reduce this value:$ulimit 1 Allocates 512 bytes of space. After the user logout the

value will change

General Commands

cal calendar $cal 2 2009

banner prints a message in large letters which

looks like a banner. $ banner ECM III Year

file commands …rm used to remove a file. If you have write

permission to the file, then only you can remove the file.

mkdir creates a directory.$mkdir ecm$mkdir –p works/bpb/unix/book$mkdir dbs doc dmcCreates three directories

pwd: present working directory

sort used to sort the contents of a file. Can merge multiple sorted files and store the

result in the specified output file. $ sort myfile To sort the contents of several files at one shot: $sort –o result file1 file2 file3 To merge already sorted files sort –m file1 file2

Text Processing Utilities

Sort options-b Ignores leading spaces and tabs-c Checks if files are already sorted. If already

sorted then it do nothing-d Sorts in directory order-f Ignore cases-m Merges files that have already been sorted.-n Sorts in numeric order-ofile Sorts output file.-r Reverses sort-tc Separates fields with character-u Unique output

cut command It cuts or picks up a given number of

character or fields from the specified file. In a database, to see only some fields( say 2nd,

7th fields) then use cut command. $cut –f 2,7 emp To see from 2nd field upto 7th field. $cut –f 2-7 emp

head, tail – Other text processing Utilities To view the beginning or end of the lines of a

file use head and tail commands.

$head -5 myfile$pg +10 -15 -p “page no. %d” –s myfile

Starts displaying of myfile contents, 15 lines at a time from 10th line onwards. At the end of each page the page no will be displayed.

diff command Compares two files, directories and reports all

the differences between two files.

diff file1 file2

63

more• more displays files on a page-by-page basis• Format: more [options] filename

• Options (selected):• -n : number of lines to show at a time

• Commands to use while in more:• space : advance a page at a time• return : advance a line at a time• q : quit more

more examplemore +10 -15 -s –d myfile yourfile -s option squeeze multiple blank lines in a file

to a single blank line. -d option changes the normal -- more –

prompt displayed at the end of each page to explanatory ‘[ hit space to continue, delete to abort]’

65

uniq

• uniq removes duplicate adjacent lines from a file• To ensure that all of the data is unique you should sort the file first

(recall that sort also can produce unique values with the –u option)

• Format: uniq [options] filename

• Options (selected):

-c : count the instances

-d : print only the duplicates (once)

-u : print only the unique lines

66

uniq Examples

uniq –u abc only prints out the unique lines in file abc

uniq –c abc only prints out the count of unique lines assuming

duplicates are next to each other

Calculations in Unix

$bc 10/2*2 quit$bc for(i=1;i<=5;i++) I 1

2345

Process utilities

• Ps (process status)• Display some process attributes.• Ex: $ps

PID TTY TIME CMD

1078 pts/2 0:00 bash

• Ps presents a snapshot of the process table.

68

Process utilities

• –f option displays a full listing that includes the

PPID.

• –u option followed by user-id displays the processes

owned by the user-id.

• –e option displays the system processes.

• -a option displays all processes for all the users.

• -t option displays the process launched at a particular

terminal 69

nohup command The processes we have executed should not

die even after we log out, for this purpose we have to use nohup command.

nohup stands for no hangups. Ex: $ nohup sort emp.dat > output.emp If we don’t mention the output file then the

sorted output will be stored in the file nohup.out

Killing a process

In case of system hanging, we have to kill the running process.

To kill the process, we must know the process-id using ps command.

Ex: kill 9876

Changing Process Priorities The priority of a process is decided by a

number associated with a number. This number is called nice value of the process.

The nice value of a process can range between 0 to 39.

20 is the default value of the process. nice cat employee.dat nice value increases to 30 $nice -15 cat employee.dat

at command, batch command Used to execute the unix commands at a

future date and time. $at 16:30 echo “ it’s 4:30 PM! Backup your files and

logout” ctrl + dbatch command:The system decides when to execute a

sequence of commands. i.e. whenever it is free, the system will execute these commands.

crontab command More powerful than at command. Carry out a

submitted job everyday for years together.

• Who: know the users

• Displays the users currently logged in the system.

• $who

• Whoami: Show you the owner of this account

• $whoami

• W: Tell you who is logging in and doing what!

• $w

75

• Finger: Displays the information about the users. Who are connected and who can receive the messages

• $finger user

Find out the personal information of a user

• $finger name

Try to find the person’s info. by his/her name

• finger email-address

Try to find the person’s info across the network76

finger options -i displays the idle time of the system

Ex: $ finger –iLogin tty when idleRamu *tty01 Wen Jan27 8 Minutes 14

seconds

* means message is set to -n

Back Up commands ufsdump

1. Used for complete file system backup .2. It copies every thing from regular files in a file system to special character and block device files.2. It can work on mounted or unmounted file systems.

tar:1. Used for single or multiple files backup .2. Can’t backup special character & block device files ( 0 byte files ).3. Works only on mounted file system.

cpio:1. Used for single or multiple files backup .2. Can backup special character & block device files .3. Works only on mounted file system.4. Need a list of files to be backed up .5. Preserve hard links and time stamps of the files .

Compress and Un-Compress files compress -v file_name

gzip filename uncompress file_name.Z

orgunzip filename

Disk utilities

• Du: disk usage

• Du command estimate the file space usage on the disk.

• It produces a list containing the usage of each subdirectory of its argument and finally produces a summary.

• $du /home/usr1

80

Disk utilities

• Df: displays the amount of free space available on the disk. The output displays for each file system separately.

• $df

• Mount:

• Used to mount the file systems.

• Takes 2 arguments-device name ,mount point.

81

Disk Utilities

• Mount uses an option to specify the type of file system.

• To mount a file system on the /oracle directory on Linux system use

$mount –t ext2 /dev/hda3 /oracle

$mount –t iso9660 /dev/cdrom /mnt /cdrom

$mount –t vfat /dev/hda1 /msdos

$mount –t msdos /dev/fd0 /floppy82

Disk utilities

• Umount: unmounting file systems• Unmounting is achieved with the umount command.

which requires either file system name or the mount point as argument.

• $umount /oracle• $umount /dev/hda3• Unmounting a file system is not possible if the file is

opened.

83

• ulimit: user limit

• It contains a value which signifies the largest file that can be created by the user in the file system.

• When used by itself it displays the current setting.

• $ulimit

unlimited

User can also set the ulimit value by using

$ulimit 10

84

unmask:

When u create files and directories, the default permissions that are assigned to them depend on the system’s default setting. Actually this default is transformed

By subtracting the user mask from it to remove one or more permissions. This value is evaluated by umask without arguments.

$umask

02285

Networking commands

• ftp: file transfer protocol

• ftp is used to transfer files. It can be used

with host name.

$ftp Saturn

Connected to Saturn

220 Saturn ftp server

Name (Saturn: summit ): Henry

Password: ******86

• To quit ftp use close and then bye or quit.

• ftp>close

221 good bye

• ftp>bye

• Transferring files:

can be of 2 types.

• Uploading( put & mput):

• To upload the web pages & graphic files to website.• The put command sends a single file to the remote machine.

87

• ftp>binary 200 type set to I• ftp>put penguin. gif To copy multiple files use mput.• ftp>mput t*.sql Downloading files: get & mget• To download the files from remote machine use get &

mget.• ftp>get ls-lR.gz• ftp>_

88

Networking commands

telnet: Remote login

If u have an account on the host in a local network (or on internet ),u can use this with the host name or the ip address as argument.

$telnet SaturnTrying to 192.168.0.1…Connected to Saturn

89

• Login:----

• Password:-----

• quit telnet by using exit command.

• telnet prompt:

When telnet used without Ip address the system displays a telnet> prompt . U can invoke a login session from here with open.

telnet> open 192.168.0.8

Trying to 192.168.0.8…

Connected to 192.168.0.8

90

Networking commands

• rlogin: remote login without password• rlogin is the Berkley's implementation of the remote

login facility.• U can log on to ur own identical remote account

without using either the user name or password.• $rlogin Jupiter• Last login :….• rlogin is terminated with ctrl+d or exit or logout.

91

vi Editor

• vi is a full screen text editor

• It was created by Bill Joy.

• Bram Moolenaor improved it and called it vim (vi improved).

• Invoking vi:

• $Vi file name

92

vi Editor

We notice a tilde (~) on each line following the cursor. A tilde represents an unused line. If a line does not begin with a tilde and appears to be blank, there is a space, tab, newline, or some other nonviewable character present.

vi editor Command Description

vi filename Creates a new file if it already does not exist, otherwise opens existing file. vi -R filename Opens an existing file in read only mode.view filename Opens an existing file in read only mode.

Modes of operation

• Vi has 3 mode of operation.

1.Command mode: In this mode all the keys pressed by the user are interpreted as commands. It may perform some actions like move cursor, save, delete text, quit vi, etc.

2. Input/Insert mode: used for inserting text.

– start by typing i; finish with Esc

95

Modes of operation

• Ex mode or last line mode:• Used for giving commands at command line. • The bottom line of vi is called the command

line.

96

Basic Cursor Movements

j down onek up oneh move cursor one place to leftl right onew move forward one wordb back one word

97

Finishing a vi Session• Get to command mode (press ESc)

ZZ save changes to the file and quit(no RETURN)

:q! quit without saving(press RETURN)

:wq! Saves the file & quit.

98

Inserting Text

• Move to insertion point

• Switch to input mode: i

• Start typing; BACKSPACE or DELETE for deletion

• ESC finish; back in command mode

No RURN

99

Deletion

• Must be in command mode.

x Delete character that cursor is on. dd Delete current line. D Delete from cursor position to

end of line u Undo last command

100

Screen Commands

• Ctrl-f To move screen forward• Ctrl-b To move one screen backwards.• Ctrl-u Moves the cursor half a screen forward.• Ctrl- d Moves the cursor half screen

backward.• Ctrl-r or Ctrl-l To redraw the original screen.• Ctrl-g Displays the status on the status line. name of file, current line number