Introduction to shell scripting

Andreas Buzh Skaubuzh@usit.uio.no

Research Computing ServicesSlides: http://folk.uio.no/buzh/bash

RCS Course Week Fall 2015

Outline

◮ What is shell scripting ?◮ How to run the scripts◮ Variables◮ Control flow (tests, if-then-else-fi, loops)

Poll

◮ Who has already◮ used a shell?

Poll

◮ Who has already◮ used a shell?◮ written a shell script?

Poll

◮ Who has already◮ used a shell?◮ written a shell script?◮ written a script (R, MATLAB, . . . )?

Poll

◮ Who has already◮ used a shell?◮ written a shell script?◮ written a script (R, MATLAB, . . . )?◮ written a program (C/C++, Fortran, Java, . . . )?

What is a shell?

◮ The shell is a computer program that presents you with acommand line interface.

◮ You type in a command or set of commands, hit enter andthe shell does what you ask

◮ A supplement or alternative to a Graphical User Interface(GUI)

⇒ The shell is made for human interaction with computers

What is a script?

◮ A script is a list of commands that the shell will execute inorder.

◮ It’s kept in a text file, rather than being a single line on thecommand prompt

◮ Think of it like the script to a movie or play. You are adirector giving a script to the actor.

◮ The actor reads, interprets and executes the scriptaccording to the director’s instructions.

⇒ The shell does the same with your scripts.

Why shell scripting?

◮ A way to store, edit & execute shell commands◮ store ⇒ avoid repetitive work◮ adapt ⇒ easily change commands◮ logic ⇒ gather information and act accordingly◮ automate ⇒ computers are here to work for you

⇒ Simplifies the repetition of long & complex commandsequences.

What is shell scripting used for? 1

◮ A wrapper for a program◮ SLURM job scripts are shell scripts◮ Start a program with the same options over and over, easily◮ Prepare job input data, run job, check results, copy results⇒ Save time by automating

◮ To manage research environment◮ Create data/jobs layout (e.g., for parameter studies)◮ Perform post-processing of results◮ Document exactly what you did⇒ May help in reproducing results if necessary.

1on Abel

Example script: simple.sh

◮ use any (ASCII) text editor: vim, emacs, nano, gedit, . . .

#!/bin/bash# An example script for the RCS course week

WHO="All"

if [ $# -gt 0 ]then

WHO="$1"fi

DATE=$(date)echo "Welcome to the tutorial, $WHO"echo "$DATE"

Simple script: How to run it

> bash simple.sh

Welcome ’All’ to RCS Course WeekOct 27 12:50:23 CET 2015

> bash simple.sh Abel

Welcome ’Abel’ to RCS Course WeekOct 27 12:50:23 CET 2015

◮ Run script without bash ... ?

> ./simple.sh

-bash: simple.sh: Permission denied

◮ Check

> ls -l simple.sh-rw-r--r-- 1 buzh users 117 Oct 22 18:04 simple.sh

◮ Fix

> chmod u+x simple.sh> ls -l simple.sh-rwxr--r-- 1 buzh users 117 Oct 22 18:04 simple.sh

Invocation – Put it in the PATH◮ How about saving two more keystrokes?

> simple.sh

-bash: simple.sh: command not found

◮ Check

> which simple.sh/usr/bin/which: no simple.sh in (/hpc/bin:...

which searches the directories in the environment variable PATH

> PATH=$PATH:$HOME/scripts> simple.shWelcome ’All’ to RCS Course WeekOct 27 12:50:23 CET 2015

◮ Alternative: use absolute path, i.e., /path/simple.sh

Hashbang 2

#!/bin/bash

◮ 1st line in each shell script◮ Lets the OS know which interpreter to use for a script◮ Not mandatory, but recommended◮ Different shells have different syntax (sh, ksh, tcsh, zsh etc)◮ Also works for some other languages, like Perl, PHP etc

2# is the hash, and ! is the bang. Remnants from the printing industry

Comments

◮ Anything preceded by a # (hash) is ignored by the shell◮ We call this a ”comment”◮ The hashbang is a special comment◮ If you wish to disable part of your script temporarily, just

comment it out

# echo "Velkommen til ITF kurs-uke!echo "Welcome to the RCS course week!"

Arguments – Welcome ’everybody’

◮ Passing arguments

> arguments.sh everybodyWelcome ’everybody’ to RCS Course Week

◮ Simplified script

echo "Welcome ’$1’ to RCS Course Week"

◮ $1 is a positional parameter◮ positional parameters are set when the shell is invoked◮ set with everybody in the invocation above

Arguments – Welcome everybody

◮ Passing multiple arguments

> argumentsX.sh Alice BobWelcome ’Alice’ (1) to DRC Course WeekWelcome ’Bob’ (2) to DRC Course WeekWelcome ’’ (3) to DRC Course Week

◮ Enhanced script

#!/bin/bashecho "Welcome ’$1’ (1) to RCS Course Week"echo "Welcome ’$2’ (2) to RCS Course Week"echo "Welcome ’$3’ (3) to RCS Course Week"

◮ $n is a positional parameter◮ unused positional parameters are empty

Variables

◮ AssignmentVARIABLENAME="any string you like"

◮ VARIABLENAME is of characters a-z , A-Z , 0-9 , _, . . .

◮ No space between VARIABLENAME and = and value◮ Quoting (" or ’ ) needed if value contains spaces

◮ Usage$VARIABLENAME

◮ no need to declare before use (default: empty string)◮ Before the shell executes a line, it will expand

$VARIABLENAMEto the actual value contained within.

◮ Note the difference (with/out $)◮ Untyped (no difference between numbers, text etc)

Variables – Examples

◮ Script

#!/bin/bashABEL=AbelABLE="Abel is able"echo "Welcome ’$ABEL’ (ABEL) to DRC Course Week"echo "Welcome ’$ABLE’ (ABLE) to DRC Course Week"echo "Welcome ’$UNSET’ (UNSET) to DRC Course Week"

◮ Output

Welcome ’Abel’ (ABEL) to DRC Course WeekWelcome ’Abel is able’ (ABLE) to DRC Course WeekWelcome ’’ (UNSET) to DRC Course Week

◮ Experiment with quoting◮ remove double quotes in assignment of ABLE◮ remove single and/or double quotes in echo for UNSET

Variables – Examples (2)

◮ Variables

mypath=/tmp/scripting_coursemyfilename=magic_script.shmyfile=$mypath/$myfilenamemytext="Welcome !"mynumber=99myarray=(apple banana strawberry)

◮ Use

mkdir -p $mypathecho "$mytext" > $myfileecho $mynumber >> $myfilemv $myfile $myfile.${myarray[2]}for ((i=0; i<=$mynumber; i++)); do echo $i; done

Status

◮ You should now understand the highlighted parts.

#!/bin/bash

WHO=All

if [ $# -gt 0 ]then

WHO=$1fi

DATE=$(date)echo "Welcome ’$WHO’ to RCS Course Weekecho "$DATE"

Variables – Command substitution

◮ The ”date” commands returns the date. We couldtimestamp our output like this:

#!/bin/bashecho -n "Work start at "; datesleep 3 # Placeholder for actual workecho -n "Work finished at "; date

Work start at Thu Apr 16 14:28:33 CEST 2015Work finished at Thu Apr 16 14:28:36 CEST 2015

◮ But what if we wanted to save the date ouput for later use?

Variables – Command substitution(2)

◮ SyntaxVARIABLENAME=$(command with arguments)

◮ Variable is substituted with the command’s output.

#!/bin/bashSTART=$(date)sleep 3 # Placeholder for actual workSTOP=$(date)echo "Work started at $START"echo "Work finished at $STOP"

Work started at Thu Apr 16 14:48:49 CEST 2015Work ended at Thu Apr 16 14:48:52 CEST 2015

◮ Put any command output into a variable!

Control flow

◮ Automate decision making◮ Adapt to different conditions◮ Scripts are more than a list of sequential commands

If-then-else – Syntax

if conditionthen

some_commandselse

some_other_commandsfi

◮ else branch is optional◮ condition can be any command!

A quick note about exit codes

◮ All commands in Linux/unix return an exit code◮ A command that finishes succesfully returns 0◮ Any non-zero exit code indicates failure◮ The exit code of the previous command is stored in the

special variable $?

⇒ ”if” responds to the exit code of the condition

If-then-else – Example

◮ Example

#!/bin/bashCARROTS=10HORSES=8if test $CARROTS -ge $HORSESthenecho "There are enough carrots"

elseecho "Warning: Not enough carrots!"SADHORSES=$(expr $HORSES - $CARROTS)echo $SADHORSES will not get a carrot.

fi

There are enough carrots

test – It’s a command called test

◮ For details see man test

◮ Examples

expression evaluates to true if

EXP1 -a EXP2 both EXP1/2 are trueEXP1 -o EXP2 either EXP1/2 is true

INT1 -eq INT2 integers are equalINT1 -ge INT2 INT1 ≥ INT2

-e FILE FILE exists◮ Square brackets in bash is an alias for ’test’◮ Can improve readability

if [ $CARROTS -ge $HORSES ]

Subshells

◮ Multiple commands can be grouped together withinparenthesis.

if ( command; othercommand && ... )thenecho "Success!"

fi

◮ Run any number of commands, but make sure youunderstand the final exit status

Subshells - Final exit status

if (ls /TMP; ls /tmp)thenecho "Success!"

fi

◮ The last command returns 0, so it’s a success, despite thefirst command failing

if (ls /TMP && ls /tmp)thenecho "Success!"

fi

◮ This would NOT return ”Success!”

Subshells - Using the output

echo $(date)

DATE=$(date)echo $DATE

◮ Preceding a (subshell) with a $ will return the output of theexecuted command

Loops

◮ Loops can do the same thing over and over◮ Two main types of loops: ”for” and ”while”

(for) – loop syntax

for VAR in LISTdo

work goes heredone

◮ VARis set to the first item of the list, then the work is done.(no $)

◮ LIST can be given like ”apple cherry kiwi” or as output of aprogram. E.g. $(cat fruits.txt)

◮ Once the work is done for the first item, VARis set to thenext item in LIST and so on, until LIST is empty

Loops – Example

for NAME in Alice Bob Cindy Dave Erica Frankdoecho "Hello $NAME"

done

for NAME in $( cat names.txt)doecho "Hello $NAME"

done

(While) – loops

#!/bin/bash

i=1while (( i <= 10 ))doecho "I can count to $i"i=$(($i+1))

done

◮ While loops start only if the expression is true, but continueuntil it becomes false

Become a shell scripting user/expert

◮ Read man pages: man bash , man test

◮ Advanced Bash-Scripting Guide:http://www.tldp.org/LDP/abs/html/

◮ Learn to use cmdline tools like grep, awk, sed, ...◮ Study examples◮ Online search◮ Most important! Practice, practice, practice.

Happy scripting

◮ Course by Andreas Buzh Skau◮ buzh@usit.uio.no◮ Slides: http://folk.uio.no/buzh/bash/

Tools, tips and tricks◮ Debug output

[buzh@stridselg ˜]$ cat debug.sh#!/bin/bashcd /tmpif [ -f test ]; then

ls -l testelseecho Nopefi[buzh@stridselg ˜]$ bash debug.shNope[buzh@stridselg ˜]$ bash -x debug.sh+ cd /tmp+ ’[’ -f test ’]’+ echo NopeNope

◮ Common pitfall: Invisible characters

[buzh@stridselg test]$ cat invisible.shone=1two=2echo one plus two equals $(( $one + $two ))[buzh@stridselg test]$ cat -v invisible.shone=1two=2echoM-BM- one plus two equals $(( $one + $two ))

◮ This script would give an error: ”command not found”◮ The reason is that the character between ”echo” and ”one”

is not a space, even though it looks like it◮ This particular one comes from accidental Alt-Space

instead of Space◮ ’cat -v’ displays non-printing characters in files

◮ Search text with ”grep”

$ cat debug.sh | grep testecho test$ grep test debug.shecho test$ grep -c test debug.sh1

◮ ’grep’ is purpose built to search text. Very powerful!◮ ’man grep’ has all the options, but can be daunting.

◮ Display part of a string with ’cut’

$ grep nobody /etc/passwdnobody:x:99:99:Nobody:/:/sbin/nologinnfsnobody:x:65534:65534:Anonymous NFS User:/var/lib$ grep nobody /etc/passwd | cut -d: -f5NobodyAnonymous NFS User

◮ -d declares the delimiter, -f the field◮ If you have advanced needs, look at ’awk’ instead

◮ Change text with sed

$ grep nobody /etc/passwdnobody:x:99:99:Nobody:/:/sbin/nologinnfsnobody:x:65534:65534:Anonymous NFS User:/var/lib$ grep nobody /etc/passwd | sed ’s/x/y/g’nobody:y:99:99:Nobody:/:/sbin/nologinnfsnobody:y:65534:65534:Anonymous NFS User:/var/lib

◮ The command above is ”swap x with y for all lines”◮ Forward slashes delimit the components of the command◮ Powerful: Whole books have been written about it

The end!