Download - Essential Unix
-
8/6/2019 Essential Unix
1/81
Essential UnixTips, Tricks, and Hints Every DBA Should Know
-
8/6/2019 Essential Unix
2/81
2
Topics of Discussion
1. Basic Concepts
2.
Fundamental UnixCommands
3. Working with Multiple Servers
4. Scripts
-
8/6/2019 Essential Unix
3/81
3
1. Basic Concepts
Terminology
Case Sensitivity
Unix Building Blocks Wildcards
-
8/6/2019 Essential Unix
4/81
4
Terminology
Basic Concepts
Unix Utilities
Unix Kernel
Unix Shell
-
8/6/2019 Essential Unix
5/81
5
Unix Utilities
Basic Concepts - Terminology
The collection of various programs that individuals execute fromthe command prompt to perform a variety of tasks
Examples:
listing contents of a directory (ls) Sorting (sort)
Displaying contents of a file (cat, more, head, tail)
Editing files (vi)
Printing (lp)
Searching files for a keyword (grep)
Changing directories (cd)
-
8/6/2019 Essential Unix
6/81
6
Unix Kernel
Basic Concepts - Terminology
A program that starts running when the system is booted and runscontinuously until the system is shutdown.
Functions of the kernel include file management, disk I/O, processscheduling and management, memory management, systemaccounting, and interrupt and error handling.
-
8/6/2019 Essential Unix
7/81
7
Unix Shell
Basic Concepts - Terminology
The component that interacts with the user
A program that is executed on behalf of the user upon logon to thesystem and presents the command prompt (typically a $) to the
user Reads input typed into the command prompt, hands off
information to the kernel, and delivers results back to the usersscreen
Environmental settings adjusted via variables
i.e. ORACLE_HOME=/u01/app/oracle
-
8/6/2019 Essential Unix
8/81
8
Unix Shell (continued)
Basic Concepts - Terminology
env Command to view environment variables
See man page for environ for more variables
Environment variables can be set at login time via .profile a textfile in home directory
$ cat .profile
export ORACLE_HOME=/u01/app/oracle/product/10202
export ORACLE_SID=orclexport LD_LIBRARY_PATH=/u01/app/oracle/product/10202/lib
-
8/6/2019 Essential Unix
9/81
9
Unix Building Blocks
Basic Concepts
Pipes
Standard Output
Standard Error Standard Input
-
8/6/2019 Essential Unix
10/81
10
Pipes
Basic Concepts Unix Building Blocks
Mechanism that allows the output from one command to be fed asthe input to another
Vertical bar | character is the pipe operator
Example:$ cat oratab | cut f1 d: | sort
dca
drm
dse
dta
-
8/6/2019 Essential Unix
11/81
11
Standard Output
Basic Concepts Unix Building Blocks
a.k.a Standard Out, stdout, filehandle 1
Default location is terminal screen
Redirect stdout to a new file with the redirect operator > Append to an existing file with the concatenation operator >>
Redirect stdout to the input of another command with the pipeoperator |
-
8/6/2019 Essential Unix
12/81
12
Standard Output
Basic Concepts Unix Building Blocks
Examples:
ls l *log
date > rmlogfiles.txt
echo removing log files >>logfile.txt
ls l *.log >>logfile.txt
rm *.log
cat oratab | cut f1 d: | sort 1> sids.txt
-
8/6/2019 Essential Unix
13/81
13
Standard Error
Basic Concepts Unix Building Blocks
a.k.a stderr, filehandle 2
Default location is terminal screen
Redirect stderr to a new file with the redirect operator 2> Append to an existing file with the concatenation operator 2>>
Redirect stderr to the same location as stdout using 2>&1
Pipes typically not used with stderr
-
8/6/2019 Essential Unix
14/81
14
Standard Error
Basic Concepts Unix Building Blocks
Example:
$ cat sids.txt bogusfile 1>stdout.txt 2>stderr.txt
$ cat stdout.txt
dca
drm
dse
dta
$ cat stderr.txtcat: cannot open bogusfile
-
8/6/2019 Essential Unix
15/81
15
Standard Input
Basic Concepts Unix Building Blocks
a.k.a Standard In, stdin, filehandle 0
Default location is keyboard input
Redirect stdin from a new file with the redirect operator
-
8/6/2019 Essential Unix
16/81
16
Wildcards
Basic Concepts
A means of matching patterns; usually in file names or whensearching contents of a text file
Sometimes referred to as regular expressions
Examples:
*An asterisk matches any number of letters or digits
? Question mark matches a single letter or digit
[ ] Square brackets matches range of letters or digits
-
8/6/2019 Essential Unix
17/81
17
Wildcard Examples (cont)
Basic Concepts
list all temp tablespace data files that contain two alphanumeric characters
after the word temp in the name. i.e. temp01.dbf, tempAB.dbf
ls temp??.dbf
list all files that start with a letter followed by a digit followed by any
combination of letters or characters and ends with .log; i.e. a2test.log
ls [a-z][0-9]*.log
-
8/6/2019 Essential Unix
18/81
18
Topics of Discussion
1. Basic Concepts
2. Fundamental Unix Commands
3. Working with Multiple Servers
4. Scripts
-
8/6/2019 Essential Unix
19/81
19
2. Fundamental Unix Commands
Commands That Provide Information
Commands That Do Something
-
8/6/2019 Essential Unix
20/81
2
0
Commands that Provide Information
Fundamental Unix Commands
Online Documentation (man command)
Examining Contents of Files
File and Directory Information
System Information
-
8/6/2019 Essential Unix
21/81
2
1
man command
Fundamental Unix Commands that provide information
Syntax: man command_name or man k command_name
-k option used for keyword searches Optional arguments specified within [ ]
Multiple arguments followed by
Read entire man page towards the end is list of bugs, relatedcommands, related files, exit codes, or other information
-
8/6/2019 Essential Unix
22/81
22
Examining Contents of Files
Fundamental Unix Commands that provide information
head
tail
more
wc
view
grep
-
8/6/2019 Essential Unix
23/81
2
3
head
Fundamental Unix Commands that provide informationExamining Contents of Files
Displays only first few lines of files. Useful in scripts to extractspecified number of lines
Default is 10 lines
Syntax: head [-number] [file] Examples:
head listener.log
head 24 listener1.log listener2.log
X=`cat output.txt | head -1`
-
8/6/2019 Essential Unix
24/81
2
4
tail
Fundamental Unix Commands that provide informationExamining Contents of Files
Syntax: tail [+-number] [-f] [file]
+number start relative to beginning of file
-number start relative to end of file
-f keeps file open and continuously updates; cancel with
Examples:
tail -f listener.log
cat output.txt | tail +30
tail 24 alert_orcl.log
-
8/6/2019 Essential Unix
25/81
25
more
Fundamental Unix Commands that provide informationExamining Contents of Files
Displays text file one page at a time
Use space bar to advance
Type q to quit
Type h for help on more options
Syntax: more [file]
Examples:
more listener.log
cat listener.log | more
-
8/6/2019 Essential Unix
26/81
26
wc
Fundamental Unix Commands that provide informationExamining Contents of Files
Word Count Counts lines, words, and chars
Syntax: wc [ -lwc ] file
Default is all
Example: (shows how many trace files)
ls l *.trc | wc -l
-
8/6/2019 Essential Unix
27/81
27
view
Fundamental Unix Commands that provide informationExamining Contents of Files
Read-only version of vi editor
Same as vi R command
Syntax: view file
Type :q to exit
-
8/6/2019 Essential Unix
28/81
28
grep
Fundamental Unix Commands that provide informationExamining Contents of Files
Global Regular Expression Print
Finds patterns within a file
Syntax: grep [-cilnv] pattern file
-c returns count of lines matching pattern
-i ignores case
-l only prints names of files with matches
-n precede each matching line with its line #
-v prints all lines except those matching pattern
-
8/6/2019 Essential Unix
29/81
29
grep - Examples
Fundamental Unix Commands that provide informationExamining Contents of Files
grep c ORA-1650 alert_orcl.log
1
grep i ^ora-1650 alert_orcl.log
ORA-1650: unable to extend rollback segment R01 by 128 intablespace RBS
grep l ORA alert*.log
alert_orcl.log
alert_orcl2.logalert_orcl.old.log
ps ef | grep v oracle
Shows all non-oracle owned processes
-
8/6/2019 Essential Unix
30/81
30
File and Directory Information
Fundamental Unix Commands that provide information
ls
find
du
fuser
-
8/6/2019 Essential Unix
31/81
31
ls - List directory contents
Fundamental Unix Commands that provide informationFile and Directory Information
Syntax: ls [-aAdlqrR1] [file]
-a lists all files including those beginning with .
-A same as a minus directories . and ..
-d if argument is a directory, only list its name
-l long format including owner, group, size, time
-q prints non-printable characters as a ?
-r reverse order sort
-R recursively list subdirectories and their contents-1 prints only one entry per line
-
8/6/2019 Essential Unix
32/81
32
find
Fundamental Unix Commands that provide informationFile and Directory Information
Recursively descends directory tree to find files
Syntax: find path options
path directory to begin searching
Options:
-print prints file names found
-ls prints long listing of file names found
-mtime n prints file names modified n days ago
-mtime +n prints file names modified >n days ago-mtime n prints file names modified
-
8/6/2019 Essential Unix
33/81
33
find
Fundamental Unix Commands that provide informationFile and Directory Information
Options (continued)
-name pattern prints file names matching pattern
-perm mode prints file names matching permissions-size +nc prints file names >n bytes
-size nc prints file names
-
8/6/2019 Essential Unix
34/81
34
find Examples:
Fundamental Unix Commands that provide informationFile and Directory Information
Show files greater than 2 GB
find /u02/oradata size +200000000c ls
Show files less than 2 GB
find /u02/oradata size -200000000c ls
Find files modified in the last 24 hours
find /u01/app/oracle type f mtime 1 -ls
Remove trace files older than 30 daysfind /u01/app/oracle/admin name *.trc mtime +30
\
exec rm {} \;
-
8/6/2019 Essential Unix
35/81
35
find Examples (continued)
Fundamental Unix Commands that provide informationFile and Directory Information
Find scripts where new users are created
find $ORACLE_HOME type f name *.sql \
exec grep il identified by {} \;
Find scripts that might have passwords
find $ORACLE_HOME type f -exec grep il sqlplus \;
Find files owned by oracle that are world readable
find / -user oracle perm o+r ls Find files owned by oracle that are world writeable
find / -user oracle perm o+w ls
-
8/6/2019 Essential Unix
36/81
36
find Examples (continued)
Fundamental Unix Commands that provide informationFile and Directory Information
Find linked files
find $ORACLE_HOME type l -ls
Compress export files that are larger than 1MB and older than 30 days
find $ORACLE_HOME name *.dmp size +1048576c \
-mtime +30 exec compress {} \;
-
8/6/2019 Essential Unix
37/81
37
du Disk Usage
Fundamental Unix Commands that provide informationFile and Directory Information
Syntax: du [-sk] [file]
Options:
-s sum of all files and subdirectories
-k results in K bytes rather than blocks
Example:
cd /u02/oradata
du sk * | sort -nr
-
8/6/2019 Essential Unix
38/81
38
fuser
Fundamental Unix Commands that provide informationFile and Directory Information
Identifies processes using a file
Use just before deleting a file
Syntax: fuser [-u] [file]
-u shows username too
Example: (note, 9i and above use and datafiles)
fuser u user_data.dbf
user_data.dbf: 708o(oracle)
ps ef| grep 708oracle 708 1 0 Jul 26 ? 35:11 ora_smon_drm
-
8/6/2019 Essential Unix
39/81
39
System Information
Fundamental Unix Commands that provide information
ps
crontab
w
who
ipcs
-
8/6/2019 Essential Unix
40/81
40
ps List info about processes
Fundamental Unix Commands that provide informationSystem Information
Syntax: ps [-ef] [-o options]
Options*:
-e list info about every process
-f full listing
-o info on specific attributes like mem, cpu
*lots more options available, consult man page
-
8/6/2019 Essential Unix
41/81
41
ps Examples
Fundamental Unix Commands that provide informationSystem Information
Show %CPU, Userid, ProcessID, and command
ps eo pcpu,user,pid,comm
Show Memory, %Memory, Userid, Process IDand command
ps eo vsz,pmem,user,pid,comm | sort nr | head -5
-
8/6/2019 Essential Unix
42/81
42
crontab Scheduled Job execution
Fundamental Unix Commands that provide informationSystem Information
To list crontab file
crontab l To edit crontab file
crontab e
See man page for formatMust have environment variable EDITOR set,
typically, vi
-
8/6/2019 Essential Unix
43/81
43
System uptime
Fundamental Unix Commands that provide informationSystem Information
Four similar commands:
uptime shows how long system is up w shows uptime and load average
who b shows date/time of last reboot
whodo l same as w command
-
8/6/2019 Essential Unix
44/81
44
ipcs inter process communication status
Fundamental Unix Commands that provide informationSystem Information
Displays info on shared memory segments and
semaphores Syntax: ipcs a
Difficult to determine which instance segmentsbelong to
See also, Oracles sysresv command
-
8/6/2019 Essential Unix
45/81
45
Commands that Do Something
ipcrm
[un]compress
touch script
split
mknod
nohup
*these commands discussed in white paper
Fundamental Unix Commands
tar*
ln*
vi*
kill*
umask*
dos2unix / unix2dos*
rm*
sort*
uniq*
cut*
-
8/6/2019 Essential Unix
46/81
46
ipcrm Removes ipc resources
Syntax: ipcrm [-m shmid] [-s semid]
Options:-m shared memory segment to remove
-s semaphore id to remove
Note, see also Oracles sysresv command
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
47/81
47
compress / uncompress Compresses or uncompresses a file
Syntax:
compress fileuncompress file
Compressed files have .Z naming extension
Example:compress myexport.dmp
ls
myexport.dmp.Z
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
48/81
48
touch Changes a files access and modification time
Creates an empty file if name does not exist
Useful in scripts to create flag files
Syntax:
touch filename
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
49/81
49
script Records an interactive terminal session
Useful for documenting series of typed
commands Use to stop recording
Syntax:
script [-a] scriptfilename-a appends to scriptfilename
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
50/81
50
split
Breaks up a large file into pieces
Syntax:
split -linecount filenamesplit b nm filename
-linecount subfiles of linecount lines
-b nm subfiles of n MB in size
Examples:split 1000 fullexport.dmp
split b 50m coldbackup.tar
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
51/81
51
mknod
Make a special file
Typically used to create a pipe
Syntax:mknod filename p
Example:
mknod /tmp/mypipe puncompress /tmp/mypipe&
imp file=/tmp/mypipe
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
52/81
52
nohup No hang up Used when running a process in the
background to prevent it from dying if the
parent is killed Output sent to file nohup.out unless redirected
Example:
nohup ten_hour_imp.ksh &exit # log off box and go home.
Fundamental Unix Commands that do Something
-
8/6/2019 Essential Unix
53/81
53
Topics of Discussion
1. Basic Concepts
2. Fundamental Unix Commands
3. Working with Multiple Servers
4. Scripts
-
8/6/2019 Essential Unix
54/81
54
3. Working with Multiple Servers
Old school vs SecureSchool
ssh
scp
rsync
-
8/6/2019 Essential Unix
55/81
55
ssh Secure Shell
De-facto standard for remote access client
Offers encryption for remote login Prevents clear text passwords on network
Use in place of telnet, rlogin, rsh, ftp, rdist
Working With Multiple Servers
-
8/6/2019 Essential Unix
56/81
56
Remote commandsWorking With Multiple Servers
Non-secure
(old school)
Secure
(preferred)
rsh sshtelnet ssh
ftp sftp
rcp scprlogin Slogin
Rdist rsync
-
8/6/2019 Essential Unix
57/81
57
scp Remote CopyAllows files to be copied to other servers
without id/password
Especially useful prior to 8i when copying archlogs to standby servers
Syntax: scp [host:]srcfile [host:]destfile
Examples:scp oraprod01:/var/opt/oracle/oratab ./oratab.1 # gets
file
scp oratab.1 oraprod02:/tmp/oratab.2 # puts
file
Working With Multiple Servers
-
8/6/2019 Essential Unix
58/81
58
ssh secure remote Shell Connects to remote host and executes the
specified command without id/password
Useful for running command across multiplehosts
Syntax:
ssh host command
Example:ssh oraprod01 cat /var/opt/oracle/oratab
Working With Multiple Servers
-
8/6/2019 Essential Unix
59/81
59
rsync Remote Distribution Open source utility for keeping files in sync
across multiple servers
Secure alternative to rdist
Useful for disaster recovery scenarios
Useful for porting scripts, config files, etc.
Typically installed by system administrators
Many options; man page is worth while
More info google, wiki, etc
Working With Multiple Servers
-
8/6/2019 Essential Unix
60/81
60
Topics of Discussion
1. Basic Concepts
2
. Fundamental Unix Commands
3. Working with Multiple Servers
4. Scripts
-
8/6/2019 Essential Unix
61/81
61
4. Putting It All Together With Scripts
Scripting Techniques
Sample Scripts
-
8/6/2019 Essential Unix
62/81
62
Scripting Techniques
Use Command Line Arguments
Use Variables Use `Backticks`
Derive Values, Dont Hardcode
Validate Use of Script
Putting It All TogetherWith Scripts
-
8/6/2019 Essential Unix
63/81
63
Use Command Line Arguments
Write Scripts with flexibility in mind
Pass values in via command line, donthardcode
Shell built in variables:
$# - number of arguments on command line
$0 the name of the shell script$1 the first argument passed in
$2 - $n the 2nd through nth arguments
Putting It All TogetherWith Scripts Scripting Techniques
-
8/6/2019 Essential Unix
64/81
64
Use Variables
Using variables makes codeeasier to maintain
Avoid hard coding values!
Putting It All TogetherWith Scripts Scripting Techniques
-
8/6/2019 Essential Unix
65/81
65
Use `Backticks` Backticks (tilde ~ key) allow the output from a
shell command to be assigned to a variable
For Example:NOW=`date`
MYDIR=`pwd`
NUM9iSIDS=`grep c 9.2 $ORATAB`
Putting It All TogetherWith Scripts Scripting Techniques
-
8/6/2019 Essential Unix
66/81
66
Derive Values, Dont Hardcode
Instead of hard coding values, derive themfrom config files
Most common is ORACLE_HOME, can bederived given the SID (passed in?)
Example:
ORACLE_HOME=`grep ^$SID: $ORATAB | cut d: -f2`
Note, $ORATAB can usually be derived based onoutput from uname command:
i.e. Linux=/etc/oratab, Solaris=/var/opt/oracle/oratab
Putting It All TogetherWith Scripts Scripting Techniques
-
8/6/2019 Essential Unix
67/81
67
Validate Use of Script If a script should only be executed by
the oracle user, then validate it!
if [ `whoami` != oracle ]; then
echo error. Must be oracle
exit 1
fi
Putting It All TogetherWith Scripts Scripting Techniques
-
8/6/2019 Essential Unix
68/81
68
Validate Use of Script
If a script should (or should not) be executedon a given host, then validate it!
THISHOST=`uname a | awk {print $1}`
if [ $THISHOST = $PRODHOST ]; then
echo Danger! Dont run inproduction
exit 1
fi
Putting It All TogetherWith Scripts Scripting Techniques
P tti It All T th With S i t
-
8/6/2019 Essential Unix
69/81
69
Sample Scripts
Putting It All TogetherWith Scripts
resourcepig
dbsize
dbcount runsql
Finding default passwords (white paper only)
Finding users passwords (white paper only)
P tti It All T th With S i t
-
8/6/2019 Essential Unix
70/81
70
Resource pig
Putting It All TogetherWith Scripts
$ cat pig
# pig -- list top 5 cpu & mem users
echo "%CPU USERID PID COMMAND"
ps -eo pcpu,user,pid,comm | sort -nr | head -5
echo "MEM %MEM USERID PID COMMAND"
ps -eo vsz,pmem,user,pid,comm |sort -nr |head -5
P tti It All T th With S i t
-
8/6/2019 Essential Unix
71/81
71
dbsizePutting It All TogetherWith Scripts
#!/opt/bin/perl
#
# assumes OFA directory layout
$argc = @ARGV;if( $argc != 1 )
{
print STDERR "$0: Wrong number of arguments.\n";
print STDERR "Usage: $0 sid\n";
exit 1;}
$sid=$ARGV[0];
$sid =~ tr/A-Z/a-z/;
P tting It All Together With Scripts
-
8/6/2019 Essential Unix
72/81
72
dbsize (continued)Putting It All TogetherWith Scripts
@Lines=`du -sk /u*/oradata/$sid /u01/app/oracle/admin/$sid`;
$total = 0;
foreach $line (@Lines) {($size) = split('\s',$line);
$total = $total + $size;
}
$total = $total / 1024;
print "$total MB\n";
Putting It All Together With Scripts
-
8/6/2019 Essential Unix
73/81
73
dbcountPutting It All TogetherWith Scripts
#!/bin/ksh
# dbcount - show entries in oratabs across all servers
cat /u01/app/oracle/share/etc/dbhosts | while read LINE
do
case $LINE in\#*) ;; # ignore comment-line
*) HOST=`echo $LINE | awk '{print $1}' -`
echo "********* ${HOST} **************
# note n option redirects input to /dev/null
ssh -n $HOST cat /var/opt/oracle/oratab |
grep -v "^#" | grep -v "^*"
;;
esac
done
Putting It All Together With Scripts
-
8/6/2019 Essential Unix
74/81
74
runsqlPutting It All TogetherWith Scripts
#!/opt/bin/perl
# loops through the oratab file and for each instance,
# logs on as ops$oracle via sqlplus and executes runsql.sql
# with the output being sent to runsql.txt.
open (oratab,"
-
8/6/2019 Essential Unix
75/81
75
runsql (continued)Putting It All TogetherWith Scripts
($sid,$home,$yn) = split ":"; # parse out the fields
if ( $yn eq 'Y' ) {
$ENV{"ORACLE_SID"} = $sid;
$ENV{"ORACLE_HOME"} = $home;
$ENV{"PATH"} = $home."/bin:/bin:/usr/bin";
$ENV{"LD_LIBRARY_PATH"} = $home."/lib:/usr/openwin/lib";
print STDERR "processing $sid\n";
system("sqlplus -silent \/ \@runsql.sql >>runsql.txt");}
} # while oratab
close (oratab);
F th R di
-
8/6/2019 Essential Unix
76/81
76
Further Reading
Books
Journals
Web www.unixguide.net/unixguide.sh
tml
F th R di B k
-
8/6/2019 Essential Unix
77/81
77
Further Reading Books
Just about anything from O'Reilly &Associates (www.ora.com) is good.
Personal favorites: Practical Unix & Internet Security, 3rd Edition
Unix for Oracle DBAs, Pocket Reference
Mastering Regular Expressions, 2nd Edition
Oracle Scripts
Perl for Oracle DBAs Oracle and Open Source
sed & awk, 2nd Edition
Learning the vi Editor, 6th Edition
Learning Perl, 3rd Edition
-
8/6/2019 Essential Unix
78/81
78
Further Reading Journals
http://dir.yahoo.com/Computers_and_Internet/s
oftware/operating_systems/unix/magazines/
-
8/6/2019 Essential Unix
79/81
79
Further Reading Web
http://www.unixguide.net/unixguide.shtml (A personal favorite. Greatreference)
http://www.unixguide.net
http://isc.faqs.org/faqs/unix-faq/faq/
http://www.ugu.com/ http://www.perl.org
http://www.ee.surrey.ac.uk/Teaching/Unix/
http://www.sans.org/
http://www.cert.org
http://www.orafaq.org http://www.sun.com/bigadmin/
-
8/6/2019 Essential Unix
80/81
80
Q & A
-
8/6/2019 Essential Unix
81/81
Thank you!
Essential Unix for DBAs
Kristopher Cook