linux intermediate its research computing center c. d. poon, ph.d. email: [email protected]
TRANSCRIPT
Linux IntermediateLinux Intermediate
ITS Research Computing CenterC. D. Poon, Ph.D.
Email: [email protected]
its.unc.edu 2
Point web browser to http://its.unc.edu/Research
Click on “Training” on the left column
Click on “ITS Research Computing Training Presentations”
Click on “Linux Intermediate – Commands, Tips and Tricks”
Class MaterialClass Material
its.unc.edu 3
Linux Command Category
Stdout/Stdin/Stderr, Pipe and Redirection, Wildcards
Linux Command Review
Break
Tips and Tricks
Conclusion
Question and Exercise
OutlineOutline
its.unc.edu 5
Communication
ssh scp File/Directory Management
cat cd chmod cp ln ls mkdir more less mv pwd dirs rm
head tail wc Comparisons
diff
Linux Command Category
Linux Command Category
its.unc.edu 6
Searching
grep find locate
Archiving
compress uncompress gzip gunzip zcat tar
Text Processing
cut paste sort sed awk uniq
Linux Command Category Cont’d Linux Command
Category Cont’d
its.unc.edu 7
System Status
chgrp chown date df du env who w uptime
Miscellaneous
bc cal clear man
Linux Command Category Cont’d Linux Command
Category Cont’d
its.unc.edu 8
Stdout/Stdin/StderrPipe and Redirection
Wildcards
Stdout/Stdin/StderrPipe and Redirection
Wildcards
its.unc.edu 9
Output from commands • usually written to the screen
• referred to as standard output (stdout)
Input for commands• usually come from the keyboard (if no
arguments are given
• referred to as standard input (stdin)
Error messages from processes• usually written to the screen
• referred to as standard error (stderr)
stdout stdin stderr stdout stdin stderr
its.unc.edu 10
Pipe (|): stdout of one command to stdin of another command
Output Redirection (>): stdout of a command to a file
Output Appending (>>): stdout of a command appending to a file
Input Redirection (<): stdin of a command from a file
Pipe and Redirection Pipe and Redirection
its.unc.edu 12
Multiple filenames can be specified using special pattern-matching characters. The rules are:
• ‘?’ matches any single character in that position in the filename
• ‘*’ matches zero or more characters in the filename.
• ‘[…]’ Characters enclosed in square brackets match any name that has one of those characters in that position
Note that the UNIX shell performs these expansions before the command is executed.
Wildcards Wildcards
its.unc.edu 14
Log on to remote machine from Linux
ssh killdevil.unc.edu –l cdpoon
ssh killdevil
ssh –X kure.unc.edu
ssh –Y kure.unc.edu
ssh ssh
its.unc.edu 15
Using ssh, login to killdevil.unc.edu
To start ssh using SecureCRT in Windows, do the following.• Start -> Programs -> Remote Services ->
SecureCRT
• Click the Quick Connect icon at the top.
• Hostname: killdevil.unc.edu
• Login with your ONYEN and password
ssh using SecureCRTin Windows
ssh using SecureCRTin Windows
its.unc.edu 16
Copy files and directories to and from remote computers
scp file1 killdevil.unc.edu:/nas02/home/cdpoon/.
scp zircon.its.unc.edu:/home/cdpoon/file2 .
scp –r dir1 killdevil.unc.edu:/netscr/cdpoon/.
scp –r killdevil.unc.edu:/netscr/cdpoon/dir2 dir3
scp kure:/netscr/cdpoon/f killdevil:/nas02/home/cdpoon/.
scp scp
its.unc.edu 17
Read one or more files and print the on stdout
cat file1cat file1 file2 file3 > file_all
cat file4 >> file_allAppend file4 to file_all
cat > file5 Create file at stdin, end with EOF (cntl-d normally, use “stty –a” to find out)
cat > file6 << STOP
Create file at stdin, end with STOP
cat cat
its.unc.edu 18
Change directory, build-in shell command
cd /nas02/home/c/d/cdpoon
cd ../../ Change directory to 2 levels up
cd .. Change directory to 1 level up
cd ~ Change directory to Home
cd Change directory to Home
cd – Change to previous directory
cd cd
its.unc.edu 19
Change the access mode of one or more files
chmod u+x file1
chmod go-w file2
chmod u=rwx, g=rx, o=x file3
chmod 751 file3 Same as above, 7=rwx, 5=rx, 1=x
chmod =r file4
chmod 444 file4 Same as above, 4=r, 2=w, 1=x
chmod chmod
its.unc.edu 20
Copy a file/dir to another file/dir
cp file1 file2 Copy to the same directory and change filename
cp file1 ../dir/file2 Copy to different directory and change filename
cp file1 ../dir/. Keep the same filename
cp –r dir1 dir2 Copy directory recursively
cp –r dir1 new_dir/dir2Copy directory recursively to another
directory
cp –p file3 file4 Preserve the modification time and permission modes
cp cp
its.unc.edu 21
Create links for file/dir and allow them to be accessed by different names
ln file1 file2 Hard link for file
ln dir1 dir2 Hard link not allowed for directory
ln –s dir1 dir2 Soft link for directory, dir2 -> dir1
ln –s file3 file4 Soft link, file4 -> file3
ln –s dir/file5 file6 Soft link, file6 -> dir/file5
ln ln
its.unc.edu 22
List all files and directories in the current directory
ls
ls –a List files/directories starting with “.” too
ls –l Long listing
ls –lh List file sizes in human readable format and long list format
ls –F Flag filenames by appending / to directories, * to executables files,
and @ to symbolic links
ls ls
its.unc.edu 23
Create one of more directories
mkdir dir1
mkdir –p dir1/dir2/dir3
Create intervening parent directories if they don’t exist
Same as mkdir dir1; cd dir1; mkdir dir2; cd dir2; mkdir dir3; cd ../../
mkdir mkdir
its.unc.edu 24
Display files on a terminal, one screen at a time
more file1 Hit space bar for another page, q to quit
more –d file2 Display the prompt “Press space to continue, ‘q’ to quit
more –c file3 Page through the file by clearing each window instead of
scrolling
more more
its.unc.edu 25
Works like “more” but allows backward and forward movement
less file1 Hit space bar for another page, q to quit
Hit b to scroll backward one window
Hit /pattern to highlight “pattern” in the text
Hit Return to scroll one line at a time
less less
its.unc.edu 26
Move files and directories within the same machine and/or rename them
mv file1 dir1/file1 Move file1 to dir1, Same as mv file1 dir1/
mv file3 file4 Rename file3 to file4
mv dir1 dir2 Rename directory dir1 to dir2
mv dir3 dir4/dir5/dir6
Rename directory dir3 to dir6 and move to dir4/dir5 directory
mv mv
its.unc.edu 27
Print the full pathname of the current directory
pwd
dirs C shell and bash shell built-in command, works like “pwd”
dirs –l Print working directory in long listing
pwd dirs pwd dirs
its.unc.edu 28
Delete one or more files and directories
Delete empty directory with “rmdir”
rm file1
rm file* Remove all files with filename starting as “file”
rm –i file* Prompt for y (remove the file) or n (do not remove the file)
rm –r dir1 Delete directory “dir1” and its content
rm rm
its.unc.edu 29
Print first/last few lines of one or more files
head file1 Print the first 10 lines of file “file1”
head –n100 file2 Print the first 100 lines of file “file2”
tail file* Print the last 10 lines of files with filename starting as “file”
tail –f file3 Print the last 10 lines of file “file3” and follow file as it grows
head tail head tail
its.unc.edu 30
Print a character, word, and line count for files
wc –c file1 Print character count for file “file1”
wc –l file2 Print line count for file “file2”
wc –w file3 Print word count for file “file3”
wc wc
its.unc.edu 31
Report lines that differ between file1 and file2, with file1 text flagged by < and file2 by >
diff file1 file2 Show difference between file1 and file2
diff diff
its.unc.edu 32
Search for lines that match a regular expression
grep abc file1
Print line(s) in file “file1” with “abc”
grep –i abc file2
Print line(s) in file “file2” with “abc” ignoring uppercase and lowercase distinctions
grep grep
its.unc.edu 33
Find particular groups of files
find . –name temp Find file named “temp” in current directory
find /etc –name ‘rc*’ Find file(s) in /etc directory with name
starting with “rc”
find /usr/share/man –type d –name ‘man*’
Find directories in /usr/share/man with name starting with “man”
find find
its.unc.edu 34
Find files with matching pattern in database prepared by updatedb, Database needed to be updated daily
locate which Find files named with pattern “which” in the OS
locate –c which Count number of files named with pattern “which” in the OS
locate –i which Find files named with pattern “which” in the OS ignoring case distinctions
locate locate
its.unc.edu 35
Reduce or expand the size of one or more files using adaptive Lempel-Ziv coding
Use uncompress to expand data
compress file1 Reduce the size of file1 and create new file named file1.Z
compress –f file2 Force to reduce the size of file2 and create new file named file2.Z
uncompress file3.Z Expand file3.Z and restore file3
compress uncompress
compress uncompress
its.unc.edu 36
Reduce or expand the size of one or more files using Lempel-Ziv coding (LZ77)
Use gunzip to expand data
gzip file1 Reduce the size of file1 and create new file named file1.gz
gzip –f file2 Force to reduce the size of file2 and create new file named file2.gz
gunzip file3.gz Expand file3.gz and restore file3
gzip gunzip gzip gunzip
its.unc.edu 37
Expand the size of one or more files created by compress or gunzip
List file contents to stdout without deleting the .Z or .gz file
zcat file1.Z Expand file1.Z and list the content of file1 in stdout
zcat file2.gz Expand file2.gz and list the content of file2 in stdout
zcat zcat
its.unc.edu 38
Archive files and directories
Create a single file with extension .tar
tar –cvf file123.tar file1 file2 file3Create archive file named file123.tar in verbose mode
with contents, file1, file2, and file3
tar –xvf file123.tar
Expand file123.tar in verbose mode and generate the original files and directories back
tar tar
its.unc.edu 39
Remove sections from each line of files
cut –d: -f1,5 /etc/passwdUse field delimiter “:” to locate fields 1 and 5 from
file /etc/passwd to extract usernames and real names
cut –c4 file1Take character 4 out from each line of file1 and
display in stdout
cut cut
its.unc.edu 40
Merge lines of files
$ cat file1
1
2
$ cat file2
a
b
c
paste paste
$ paste file1 file2
1 a
2 b
c
$ paste –s file1 file21 2a b c
its.unc.edu 41
Sort lines of text files
sort –fd file1
Alphabetize lines (-d) in file1 and ignore lower and upper cases (-f)
sort –t: -k3 -n /etc/passwdTake column 3 of file /etc/passwd separated by “:” and sort in
arithmetic order
sort sort
its.unc.edu 42
Edit one or more files without user interaction using stream editor
sed s/xx/yy/g file1
Substitude all occurrences of “xx” in file1 with “yy” and display on stdout
sed /abc/d file1 Delete all lines containing “abc” in file1
sed /BEGIN/,/END/s/abc/123/g file1
Substitute “XYZ” on lines between BEGIN and END with “xyz” in file1
sed sed
its.unc.edu 43
Process files by pattern-matchingawk –F: ‘{print $1}’ /etc/passwd
Extract the 1st field separated by “:” in /etc/passwd and print to stdout
awk ‘/abcde/’ file1Print all lines containing “abcde” in file1
awk ‘/xyz/{++i}; END{print i}’ file2Find pattern “xyz” in file2 and count the number
awk ‘length <= 1’ file3Display lines in file3 with only 1 or no character
awk awk
its.unc.edu 44
Report or omit repeated lines
uniq file1Filter adjacent matching lines from file named file1 , writing to stdout
uniq –c file1Prefix lines by the number of occurrences from file named file1
uniq uniq
its.unc.edu 45
Change the group ownership of one or more files or directories
chgrp employee file1Change group ownership to “employee” for file “file1”
chgrp –R student dir1Change group ownership to “student” for directory “dir1” including
subdirectories recursively
chgrp chgrp
its.unc.edu 46
Change the ownership of one or more files or directories
chown employee file1Change ownership to “employee” for file “file1”
chown –R student dir1Change ownership to “student” for directory “dir1” including
subdirectories recursively
chown chown
its.unc.edu 47
Print the current date and time in certain format
Set the current date and time
datePrint the current date and time
date +%DPrint the current date and time in mm/dd/yy format
date 1201160108Set the current date and time to Dec 01 4:01pm 2008
date –d friShow the date of the coming Friday
date date
its.unc.edu 48
Report the number of used and free disk block on all mounted file systems
dfPrint used and free disk block on all mounted file system
df -hPrint used and free disk block in human readable format
df -k
Print used and free disk block in kb
df df
its.unc.edu 49
Print disk usage of directories and its subdirectories
du dir1 Print disk usage in kilobyte of directory “dir1”
du –-block-size=1M dir2Print disk usage in megabyte of directory “dir2”
du –hs dir3 Print summarized disk usage in human-readable format of directory “dir3”
du du
its.unc.edu 50
Display the current environment variables or set new values
env Display all of the current environment variables
env env
its.unc.edu 51
Display information about the current status of the system
who Display the names of users currently logged in to the system
who –b Report information about the last reboot
who am I Print the username of the invoking user
who who
its.unc.edu 52
Print summaries of system usage, currently logged-in users, and what they are doing
w Print summaries of system usage, currently logged-in users
w –s Display in short form
w w
its.unc.edu 53
Print the current time, amount of time logged in, and the system load averages
uptime Print a one line display of the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, 15 minutes
uptime uptime
its.unc.edu 54
Interactively perform arbitrary-precision arithmetic or convert numbers from one base to another, type “quit” to exit
bc Invoke bc
1+2 Evaluate an addition
5*6/7 Evaluate a multiplication and division
ibase=8 Change to octal input
20 Evaluate this octal number
16 Output decimal value
ibase=10 Change back to decimal input
bc bc
its.unc.edu 55
Print calendar of a month or all months in a year
cal Print calendar of the current month
cal 2 2009 Print calendar of February 2009
cal 2009 Print calendar of all months in 2009
cal -3 Display previous/current/next months
cal cal
its.unc.edu 56
Clear the terminal display and have the prompt locate at the top of the terminal window
clear Clean up the current terminal display
clear clear
its.unc.edu 57
Display information from the online reference manuals
man man Display the manual for the command “man”
man –k link compile Display commands related to linking and compiling using a keyword search
man man
its.unc.edu 60
Show files changed on a certain date in all directories
ls –l * | grep ‘Sep 26’
Show long listing of file(s) modified on Sep 26
ls –lt * | grep ‘Dec 18’ | awk ‘{print $9}’
Show only the filename(s) of file(s) modifed on Dec 18
Tips and Tricks #1 Tips and Tricks #1
its.unc.edu 61
Sort files and directories from smallest to biggest or the other way around
du –k –s * | sort –n
Sort files and directories from smallest to biggest
du –ks * | sort –nr
Sort files and directories from biggest to smallest
Tips and Tricks #2 Tips and Tricks #2
its.unc.edu 62
Change timestamp of a file
touch file1
If file “file1” does not exist, create it, if it does, change the
timestamp of it
touch –t 200902111200 file2
Change the time stamp of file “file2” to 2/11/2009 12:00
Tips and Tricks #3 Tips and Tricks #3
its.unc.edu 63
Find out what is using memory
ps –ely | awk ‘{print $8,$13}’ | sort –k1 –nr | more
Tips and Tricks #4 Tips and Tricks #4
its.unc.edu 64
Remove the content of a file without eliminating it
cat /dev/null > file1
Tips and Tricks #5 Tips and Tricks #5
its.unc.edu 65
Backup selective files in a directory
ls –a > backup.filelist
Create a file list
vi backup.filelist
Adjust file “backup.filelist” to leave only filenames of the files to be backup
tar –cvf archive.tar `cat backup.filelist`
Create tar archive “archive.tar”, use backtics in the “cat” command
Tips and Tricks #6 Tips and Tricks #6
its.unc.edu 66
Get screen shots
xwd –out screen_shot.wd
Invoke X utility “xwd”, click on a window to save the image as “screen_shot.wd”
display screen_shot.wd
Use ImageMagick command “display” to view the image “screen_shot.wd”
Right click on the mouse to bring up menu, select “Save” to save the image to other formats, such as jpg.
Tips and Tricks #7 Tips and Tricks #7
its.unc.edu 67
Sleep for 5 minutes, then pop up a message “Wake Up”
(sleep 300; xmessage –near Wake Up) &
Tips and Tricks #8 Tips and Tricks #8
its.unc.edu 68
Count number of lines in a file
cat /etc/passwd > temp; cat temp | wc –l; rm temp
wc –l /etc/passwd
Tips and Tricks #9 Tips and Tricks #9
its.unc.edu 69
Create gzipped tar archive for some files in a directory
find . –name ‘*.txt’ | tar –c –T - | gzip > a.tar.gz
find . –name ‘*.txt’ | tar –cz –T - -f a.tar.gz
Tips and Tricks #10 Tips and Tricks #10
its.unc.edu 70
Find name and version of Linux distribution, obtain kernel level
uname -a
head –n1 /etc/issue
Tips and Tricks #11 Tips and Tricks #11
its.unc.edu 71
Show system last reboot
last reboot | head –n1
who -b
Tips and Tricks #12 Tips and Tricks #12
its.unc.edu 72
Combine multiple text files into a single file
cat file1 file2 file3 > file123
cat file1 file2 file3 >> old_file
cat `find . –name ‘*.out’` > file.all.out
Tips and Tricks #13 Tips and Tricks #13
its.unc.edu 73
Create man page in pdf format
man –t man | ps2pdf - > man.pdf
acroread man.pdf
Tips and Tricks #14 Tips and Tricks #14
its.unc.edu 74
Remove empty line(s) from a text file
awk ‘NF>0’ < file.txt
Print out the line(s) if the number of fields (NF) in a line in file
“file.txt” is greater than zero
awk ‘NF>0’ < file.txt > new_file.txt
Write out the line(s) to file “new_file.txt if the number of fields (NF)
in a line in file “file.txt” is greater than zero
Tips and Tricks #15 Tips and Tricks #15
its.unc.edu 76
Many ways to do a certain thing
Unlimited possibilities to combine commands with |, >, <, and >>
Even more powerful to put commands in shell script
Slightly different commands in different Linux distributions
Emphasized in System V, different in BSD
Conclusion Conclusion