week 9 - nov 7, 2005 1 week 9 agenda i/o redirection i/o redirection pipe pipe tee tee
TRANSCRIPT
Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005 11
Week 9 AgendaWeek 9 Agenda I/O redirectionI/O redirection pipepipe teetee
22 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirection and PipingRedirection and Piping
Most “processes” in UNIX:Most “processes” in UNIX:
send their output to the terminal screensend their output to the terminal screen
get their input from the keyboardget their input from the keyboard
send any error messages to the terminalsend any error messages to the terminal
33 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirection and PipingRedirection and Piping
Most “processes” in UNIX:Most “processes” in UNIX:
send their output to the terminal screensend their output to the terminal screen
get their input from the keyboardget their input from the keyboard
send any error messages to the terminalsend any error messages to the terminalDo you
remember what a “process” is?
44 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
A process is the execution of a A process is the execution of a command by the UNIX system.command by the UNIX system.UNIX assigns a unique process UNIX assigns a unique process identification (PID) number at the identification (PID) number at the creation of each process. creation of each process. As long as a process is in existence, it As long as a process is in existence, it keeps the same PID number. keeps the same PID number.
55 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
RedirectionRedirection
Most “processes” in UNIX:Most “processes” in UNIX:
send their to thesend their to theoutputoutput terminal screenterminal screen
inputinput keyboardkeyboardget their from theget their from the
send any to thesend any to theerror messageserror messages
terminal screenterminal screen
66 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
RedirectionRedirectionUNIX allows us to UNIX allows us to
““redirect” redirect” where where
processes send their outputprocesses send their output
wherewhere
processes get their input processes get their input
and and
where where
processes send any error messages.processes send any error messages.
77 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
88 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
I/O Channel 0 - standard input (stdin)
I/O Channel 1 - standard output (stdout)
I/O Channel 2 - standard error (stderr)
These “channels” are called file descriptors
99 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
stdinstdout
stderr
1010 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
I/O Channel 0 – stdin -default is the terminal keyboard
I/O Channel 1- stdout - default is the terminal screen
I/O Channel 2- stderr - default is the terminal screen
These “channels” are called file descriptors
Let’s learn to redirect
”stdout” first
1111 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Standard Input, Standard Input, Standard Output, Standard Output, Standard Error.....Standard Error.....
not a piece of not a piece of hardwarehardware
not a keyboard, a not a keyboard, a screen or a card in screen or a card in your computeryour computer
Actually “virtual Actually “virtual files”...opened by the files”...opened by the kernel for every kernel for every command command
RememberRemember
1212 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the output Redirecting the output of a command to a fileof a command to a file
Use the Use the greater thangreater than symbol symbol >>
this redirects the “standard output” or this redirects the “standard output” or “channel 1” of a command “channel 1” of a command
this is the same as this is the same as 1>1> but you don’t but you don’t need to type the “1” - it is need to type the “1” - it is “understood”.“understood”.
1313 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the output of a Redirecting the output of a command to a filecommand to a file
““The Rules”The Rules”
a command always comes before the > symbola command always comes before the > symbol
a filename always comes after the > symbola filename always comes after the > symbol
command > file
always a command before the >
always a file name after the >
You can think of “>” symbol as being an arrow You can think of “>” symbol as being an arrow pointing from a command to a filepointing from a command to a file
1414 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the output of a Redirecting the output of a command to a filecommand to a file
The output of this command is displayed on the screen by
default
Nothing is displayed on the
screen – the output of the date command is
redirected to the file “today”
Issue the command w >users
1515 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Now, look at the contents of the file named users
Redirecting the output of a Redirecting the output of a command to a filecommand to a file
1616 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the output Redirecting the output of a command to a fileof a command to a file
Use the double Use the double greater thangreater than symbol symbol >>>>
this appends the “standard output” or this appends the “standard output” or “channel 1” of a command to a file“channel 1” of a command to a file
You can think of the symbol as a double You can think of the symbol as a double arrow from the command to the file, arrow from the command to the file, meaning 'add the output from the meaning 'add the output from the command to the END of the file'.command to the END of the file'.
1717 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
We can also redirect the We can also redirect the outputoutput
of one commandof one command to be to be inputinput
to another commandto another command
Redirection and PipingRedirection and Piping
1818 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Piping - redirecting the output Piping - redirecting the output from a command to a commandfrom a command to a command
Use the pipe symbol to “connect” Use the pipe symbol to “connect” commandscommandsThe “pipe” character is a broken vertical line The “pipe” character is a broken vertical line on the keyboardon the keyboardMost fonts display it as a single line that Most fonts display it as a single line that looks like this: “ | ”looks like this: “ | ”Data goes in the start of a pipe and flows Data goes in the start of a pipe and flows out the endout the end
command1 | command2
1919 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Using pipesUsing pipes
Q. How many files are in my current directory?
ls | wc -w
2020 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Using pipesUsing pipes
ls | wc -w
The output of the ls
command is a list of file
names
The wc command takes the output of the ls command as
its input and displays a count of
the words
2121 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Using PipesUsing Pipes
ls | wc –w > file_count
The wc command sends its output to a file called
file_count
The spaces before and after
a pipe are optional
(you will find your commands easier to read if you use them)
The spaces before and after a file redirection symbol (>, >>) are also optional
2222 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Using PipesUsing Pipes
UNIX commands were designed to do UNIX commands were designed to do “one thing very well”“one thing very well”We use a series of pipes between We use a series of pipes between simple UNIX commands to create simple UNIX commands to create complex commands linescomplex commands linesMuch of the power and flexibility of Much of the power and flexibility of the UNIX command line is built upon the UNIX command line is built upon “pipelines” of commands! “pipelines” of commands!
2323 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Using PipesUsing Pipes
Command aCommand a | | Command bCommand b | | Command cCommand c
The stdout of command ' The stdout of command ' Command aCommand a' is ' is connected to the stdin of 'connected to the stdin of ' Command Command b', b',
and the stdout of ' and the stdout of ' Command bCommand b' is ' is connected to the stdin of ' connected to the stdin of ' Command cCommand c'.'.
2424 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
“Always-Remember-and-Never-Forget”...
RULE #1 – Use file redirection symbols such
as > and >> before a RULE #2 – Use a pipe before a
filenamecommand
2525 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
“Always-Remember-and-Never-Forget”...
RULE #1 ...
RULE #2 ...
2626 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
“Always-Remember-and-Never-Forget”...
RULE #3
Don’t use pipes when you don’t need them!
example:
cat users |more
more users
WRONG!
CORRECT!
2727 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
“Always-Remember-and-Never-Forget”...
RULE #3
Don’t use pipes when you don’t need them!
example:
cat users |head -5
head -5 users
WRONG!
CORRECT!
2828 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Skill Testing Question #1Skill Testing Question #1
On the command line what comes On the command line what comes AFTER a > or >> symbol?AFTER a > or >> symbol?
2929 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Skill Testing Question #2Skill Testing Question #2
On the command line what comes AFTER a On the command line what comes AFTER a | symbol?| symbol?
3030 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Skill Testing Question #3Skill Testing Question #3
When should you NOT use file redirection When should you NOT use file redirection OR piping?OR piping?
3131 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
I/O Channel 0 – stdin -default is the terminal keyboard
I/O Channel 1- stdout - default is the terminal screen
I/O Channel 2- stderr - default is the terminal screen
We’ve done this!
Now let’s do this!
Redirection and Piping
3232 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the input of a command Redirecting the input of a command
by default commands get their “input” by default commands get their “input” from the terminal keyboardfrom the terminal keyboard
you can redirect “stdin” so that it you can redirect “stdin” so that it comes from a file or from another comes from a file or from another commandcommand
3333 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the input of a command Redirecting the input of a command
Use the Use the less thanless than symbol symbol <<
this redirects the “standard input” or “channel 0” this redirects the “standard input” or “channel 0” of a command to come from a file instead of the of a command to come from a file instead of the keyboardkeyboard
this is the same as this is the same as 0<0< but you don’t need to type but you don’t need to type the “0” - it is “understood”.the “0” - it is “understood”.
You can also think of the 'You can also think of the '<<' symbol as being an ' symbol as being an arrow pointing from the file to the command, arrow pointing from the file to the command, meaning that the information will flow from the meaning that the information will flow from the file into the input of the command.file into the input of the command.
3434 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the input of a commandRedirecting the input of a command
Examples:Examples:
cat < tempcat < temp
cat < temp > temp2cat < temp > temp2
mail –s “week 9 Notes” mail –s “week 9 Notes” [email protected]@senecac.on.ca < week9.html < week9.html
3535 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Redirecting the input of a commandRedirecting the input of a command
Are these the same?Are these the same?
cat < tempcat < tempcat tempcat temp
Are these the same?Are these the same?
cat < temp >temp2cat < temp >temp2 cp temp temp2cp temp temp2
and and
and and
3636 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
The < symbol redirects the input of The < symbol redirects the input of a command to come from a file a command to come from a file
Q. What if you want the input of a command Q. What if you want the input of a command to come from another command?to come from another command?
A. When you place a command after a pipe you are redirecting its input to come from a command instead of from the keyboard
3737 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
I/O Channel 0 – stdin -default is the terminal keyboard
I/O Channel 1- stdout - default is the terminal screen
I/O Channel 2- stderr - default is the terminal screen
We’ve done this!
We’ve done this!
Redirection and Piping
Let’s do this!
3838 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Error RedirectionError RedirectionBy default, stderr goes to the terminal screenBy default, stderr goes to the terminal screen
The “standard error” of a command can be The “standard error” of a command can be redirected to a file using the redirected to a file using the 2>2> symbol symbol
Source: Ling ZhuSource: Ling Zhu
3939 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Error RedirectionError Redirection
cp file1 file2 2cp file1 file2 2>> error.logerror.log
This would This would overwrite overwrite the file error.logthe file error.log
cp file1 file2 2cp file1 file2 2>>>>error.logerror.log
This would This would appendappend the error message of the cp the error message of the cp command to the file error.logcommand to the file error.log
Source: Ling ZhuSource: Ling Zhu
Warning:There is no space in between
4040 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
A Special Place of UNIX/LinuxA Special Place of UNIX/Linux
/dev/null/dev/null Can be called bit bucketCan be called bit bucket Can be called black holeCan be called black hole
Source: Ling ZhuSource: Ling Zhu
4141 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Use > or >> to redirect output to a Use > or >> to redirect output to a filefile
Use a | to redirect output to a Use a | to redirect output to a commandcommand
Q. What if you want to do both?Q. What if you want to do both?(this is on the test!)(this is on the test!)
A. The tee command
4242 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
The tee command duplicates the The tee command duplicates the standard output of a command standard output of a command
you can then redirect one “copy” of the you can then redirect one “copy” of the output to a file and one “copy” of the output to a file and one “copy” of the output to a commandoutput to a command
example:example:
ls –l | tee listing | morels –l | tee listing | more
the filename you are sending one copy of the output of the ls –l
command to
the command you are sending one copy of the output of the ls –l
command to
4343 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
The tee command duplicates the The tee command duplicates the standard output of a command standard output of a command
Example:Example:
ls –l | tee listing | morels –l | tee listing | more
if this file exists it will be overwritten
To append to an existing fileuse tee –a filename
ls –l | tee –a listing | more
4444 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Never use > within the tee Never use > within the tee command!command!
ls –l | tee > listings | morels –l | tee > listings | more
or or
ls –l | tee >> listings | morels –l | tee >> listings | moreWRONG!!!!
4545 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Let’s review some commands then Let’s review some commands then practice “redirection” with them...practice “redirection” with them...
1.1. sortsort
2.2. grepgrep
4646 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Some Special symbols when Some Special symbols when you you grepgrep a pattern a pattern
^̂ - beginning of a line - beginning of a line $$ - end of a line - end of a line
4747 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
sortsort and and grep grepbelong to a group of UNIX utilities belong to a group of UNIX utilities
called “filters”called “filters”
Filters are utilities that:Filters are utilities that:
1.1. read from stdinread from stdin
2.2. write to stdoutwrite to stdout
3.3. do not change the contents of the original do not change the contents of the original filefile
4848 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
How do you learn a new How do you learn a new command?command?
1.1. Learn its purposeLearn its purpose
2.2. Learn its “syntax” including common Learn its “syntax” including common optionsoptions
3.3. Learn where it gets input and where it Learn where it gets input and where it sends its outputsends its output
4949 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Other UNIX utilities that are Other UNIX utilities that are “filters”“filters”
headhead
tailtail
wcwc
teetee
5050 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Filters can be used between pipes Filters can be used between pipes in a complex command linein a complex command line
Example:Example:
grep happy diary >childhoodgrep happy diary >childhood
sort mymarks | tee sortedmarks | moresort mymarks | tee sortedmarks | more
5151 Week 9 - Nov 7, 2005Week 9 - Nov 7, 2005
Commands that are Commands that are not not filters include:filters include:
cd cd
chmodchmod
cpcp
rmrm
datedate
echoecho
ls ls
mvmv
pwdpwd
who....who....
Warning!Warning!
You can not put these
commands between pipes!
Moral of the Story:
ALWAYS be careful about “what” you
put in a pipe!