System Administration IbFile Systems File Maintenance

File Systems

File systems

•Directory structure with its own root•Disk can be divided into multiple file

systems▫Separates data▫Corruption of one file system will not affect

another▫Each file system can be backed up

independently

File Systems•Components

▫boot block boot program partition table

▫superblock global info on file system OS keeps a copy in memory

▫inode blocks keeps track of every disk block

▫data blocks all the data

Unix/Linux File Systems• Types

▫s5▫ufs▫ext2, ext3 and ext4

Current linux systems▫ iso9660 or hsfs

cd rom file system▫msdos or pcfs

original floppy for DOS▫swap▫bfs▫proc or procfs

File Systems• Mounting

▫ A directory in one filesystem which contains another file system is a mount point

▫ Classical mount points /dev/fd

floppy disk fd0

▫ first floppy drive /dev/hda

first ide drive hda1

▫ a partition in hda /dev/hdb

second ide drive /dev/sda

first scsi drive also used for pata/sata drives

/dev/ttyS0 a serial port

File systems

•mount points may be different on various distros▫My Debian at home and the one at work

have different dev names for the same function

File Systems

•Interesting note▫Can mount multiple drive blocks to make

one larger file system Combine partitions, drives LVM

Logical Volume Manager

File Systems

•Mounting▫assign file blocks to a directory name

•Unmounting▫removes the mount

•fstab▫lists all available disks and partitions

where they should be mounted and how▫can be used by mount

mounting• mount [opts] devname dirname

▫must know device name▫dirname must exist

it should be empty permissions set to at least 755

777 works but is less secure some may be predefined

• Notes:▫mount

If no options it lists all current mounts▫fdisk –l

will list available hard drives

mount example> mount/dev/hda1 on / type ext3 (rw,errors=remount-ro)tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)proc on /proc type proc (rw,noexec,nosuid,nodev)sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)procbususb on /proc/bus/usb type usbfs (rw)udev on /dev type tmpfs (rw,mode=0755)tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)nfsd on /proc/fs/nfsd type nfsd (rw)/dev/sda1 on /media/EGMONITOR type vfat (rw,nosuid,nodev,uhelper=hal,shortname=lower,uid=1000)

fdisk –l example

• Notes:▫ hda is an ide drive with 3 partitions▫ sda is a USB drive currently plugged in

> sudo fdisk -l

Disk /dev/hda: 40.0 GB, 40020664320 bytes255 heads, 63 sectors/track, 4865 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk identifier: 0x93dd93dd

Device Boot Start End Blocks Id System/dev/hda1 * 1 4660 37431418+ 83 Linux/dev/hda2 4661 4865 1646662+ 5 Extended/dev/hda5 4661 4865 1646631 82 Linux swap / Solaris

Disk /dev/sda: 1018 MB, 1018167296 bytes2 heads, 63 sectors/track, 15782 cylindersUnits = cylinders of 126 * 512 = 64512 bytesDisk identifier: 0x017c9ef7

Device Boot Start End Blocks Id System/dev/sda1 * 1 15783 994288 6 FAT16>

unmounting

•umount dirname▫can also use devname▫unmount before removing device

prevents loss or corruption of data▫note the name is umount, not unmount

fstab

•File of default mount points•Gets mounted during the booting process

▫Notes: mount –a

will mount the fstab entries mount /dirname

will mount to the device listed in fstab umount –a

will try to unmount the entries in fstab if device is busy will get an error cannot unmount a directory you are in

fstab• Sample fstab:

• Columns:▫ 1: device name▫ 2: mount point▫ 3: file system type▫ 4: mount options▫ 5: enable backing up (dump)▫ 6: fsck order

> cat /etc/fstab# /etc/fstab: static file system information.## <file system> <mount point> <type> <options> <dump> <pass>proc /proc proc defaults 0 0/dev/hda1 / ext3 defaults,errors=remount-ro 0 1/dev/hda5 none swap sw 0 0/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0/dev/hdd /media/cdrom1 udf,iso9660 user,noauto 0 0/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0>

File System Checking• superblock must be kept in sync between the

version on the disk and the one in memory▫update does a sync every 30 seconds▫ if there is a powerfailure before a change can be

synced there can be discrepancies Examples:

two or more inodes claiming same disk block block marked free, but not listed in superblock used block marked free mismatch in sizes file not having a directory entry

• fsck can check for and fix these problems

Files system checking

•fsck▫five sequential phases

each feeding the next phase▫if there is an error

fsck becomes interactive user answers questions to fix

Manageing Disk Space

Managing Disk space

•Fact: disks fill up!▫Need to monitor and maintain usage

•Two good base tools:▫df▫du

Managing Disk space

•df▫reports free space on disk▫-h option makes it human readable

> df -hFilesystem Size Used Avail Use% Mounted on/dev/sda1 323M 160M 147M 53% /tmpfs 2.0G 0 2.0G 0% /lib/init/rwudev 2.0G 236K 2.0G 1% /devtmpfs 2.0G 0 2.0G 0% /dev/shm/dev/sda9 440G 2.9G 415G 1% /home/dev/sda8 368M 11M 339M 4% /tmp/dev/sda5 8.3G 4.2G 3.7G 54% /usr/dev/sda6 2.8G 1.1G 1.6G 41% /var/dev/sdb1 114G 1.9G 106G 2% /opt/dev/sde1 16G 7.2G 8.1G 48% /media/PENDRIVE

Managing Disk space

•du▫reports disk usage▫Useful options:

-s: summary -h: human readable

> sudo du -sh /home/*2.7G /home/ajkombol8.0K /home/ftp16K /home/lost+found1.2M /home/tkombol

Managing Disk space

•du with the –sh option for the whole drive

# sudo du -sh /*1.0K /ajkubertooth5.5M /bin15M /boot236K /dev7.6M /etc2.7G /home0 /initrd.img111M /lib2.4M /lib320 /lib6412K /lost+found7.1G /media2.0K /mnt1.6G /opt0 /proc2.5M /root3.6M /sbin1.0K /selinux1.0K /srv0 /sys755K /tmp4.1G /usr1004M /var0 /vmlinuz

Managing Disk space•find: The adminstartor's friend•Some Options:

▫Can find files larger than a certain size Syntax:

find startdir -size n[cwbkMG]▫Can find unaccessed or unmodified files by age

-atime: time since last accessed -mtime: time since last modified

Syntax:▫find startdir –atime n

▫Where n is the number of 24hr periods

Managing Disk space• Find all users on size example ( >20Mb):

• Find files on access time (500 days):

• Note: for ls –lu, the u returns last accessed date instead of modified date

# find /home -size +20M –print/home/ajkombol/Downloads/netbeans-7.1.2-ml-php-linux.sh/home/ajkombol/Downloads/metasploit-latest-linux-x64-installer.run/home/ajkombol/Downloads/netbeans-7.1.2-ml-linux.sh/home/ajkombol/.cache/chromium/Cache/data_3/home/ajkombol/EYC Movie/EYC Movie.iso/home/ajkombol/eyc/eyc.iso/home/ajkombol/Music/11 Epica/01 The Divine Conspiracy/13 The Divine Conspiracy.m4a/home/ajkombol/.mozilla/firefox/r797acrr.default/urlclassifier3.sqlite

# sudo find ~ -atime +500/home/ajkombol/.bzf/cache/t9de3f8cab7b93d8aabfd653089a2fbde.bwc# cd .bzf/cache/# ls -lutotal 20-rw-r--r-- 1 ajkombol ajkombol 6012 Oct 12 18:04 BZFS0026_1-Servers.bzs-rw-r--r-- 1 ajkombol ajkombol 194 Oct 12 18:04 CACHEDIR.TAG-rw-r--r-- 1 ajkombol ajkombol 102 Oct 12 18:05 CacheIndex.txt-rw-r--r-- 1 ajkombol ajkombol 1876 Dec 31 1969 t9de3f8cab7b93d8aabfd653089a2fbde.bwc

Tar, Compress, Gzip, Zip

tar, compress, and gzip

•Historically tar and compress were used to package and compress a set of files▫Back when tape was a popular backup

medium•gzip is the modern equivalent of compress

tar, compress, and gzip

•tar – tape, archive, restore▫General form:

tar mainOption otherOptions otherOptions vary by mainOption mainOptions:

c – Create r – Replace t – Table of contents u – Update x – extract or restore

tar, compress, and gziptar c [ bBeEfFhiklnopPqvwX [ 0-7 ] ] [ block ] [ tarfile ] [ exclude-file ] {-I include-file | -C directory | file | file }c - Create. Writing begins at the beginning of the tarfile, instead of at the end.

tar r [ bBeEfFhiklnqvw [ 0-7 ] ] [ block ] {-I include-file | -C directory | file | file }r - Replace. The named files are written at the end of the tarfile. A file created with extended headers must be updated with extended headers (see E flag under Function Modifiers). A file created without extended headers cannot be modified with extended headers.

tar t [ BefFhiklnqvX [ 0-7 ] ] [ tarfile ] [ exclude-file ] {-I include-file | file } ...t - Table of Contents. The names of the specified files are listed each time they occur in the tar file. If no file argument is given, the names of all files in the tarfile are listed. With the v function modifier, additional information for the specified files is displayed.

tar u [ bBeEfFhiklnqvw [ 0-7 ] ] [ block ] [ tarfile ] file ...u - Update. The named file s are written at the end of the tarfile if they are not already in the tar file, or if they have been modified since last written to that tarfile. An update can be rather slow. A tarfile created on a 5.x system cannot be updated on a 4.x system. A file created with extended headers must be updated with extended headers (see E flag under Function Modifiers). A file created without extended headers cannot be modified with extended headers.

tar, compress, and gziptar x [ BefFhiklmnopqvwX [ 0-7 ] ] [ tarfile ] [ exclude-file ] [ file ... ]

x - Extract or restore. The named file s are extracted from the tarfile and written to the directory specified in the tarfile, relative to the current directory. Use the relative path names of files and directories to be extracted. If a named file matches a directory whose contents has been writ ten to the tarfile, this directory is recursively extracted. The owner, modification time, and mode are restored (if possible); otherwise, to restore owner, you must be the super-user. Character special and block-special devices (created by mknod(1M)) can only be extracted by the super user. If no file argument is given, the entire content of the tarfile is extracted. If the tar- file contains several files with the same name, each file is written to the appropriate directory, overwriting the previous one. Filename substitu tion wildcards cannot be used for extracting files from the archive; rather, use a command of the form:

tar xvf... /dev/rmt/0 `tar tf... /dev/rmt/0 | grep 'pattern' `

When extracting tapes created with the r or u functions, directory modification times may not be set correctly. These same functions cannot be used with many tape drives due to tape drive limitations such as the absence of backspace or append capabilities.

When using the r, u, or x functions or the X function modifier, the named files must match exactly the corresponding files in the tarfile. For example, to extract ./thisfile, you must specify ./thisfile, and not thisfile. The t function displays how each file was archived.

tar, compress, and gzip• Typical use:

▫tar –cvf archive.tar file1 file2 file.ext Where:

-cvf: (c)reate an archive, (v)erbose, (f)ilename archive.tar is the archive created file1, file2 and file.ext are packed into archive.tar

▫tar –xvf archive.tar Where:

-xvf: e(x)tract, (v)erbose, (f)ilename

▫tar –tvf archive.tar Where the –t shows the contents of the tar

table of contents

> lsbackupWaittoread.sh file1.bak mv.sh xfile3.bakbigfile1 file3 myfirst.sh xfile5bigfile2 file3.bak myscp.sh xfile5.bakbigfile3 filelist noread xfilelistbigfile4 filelist.save strlen.sh> tar -cvf bigfile.tar bigfile1 bigfile2 bigfile3 bigfile4bigfile1bigfile2bigfile3bigfile4> rm bigfile1 bigfile2 bigfile3 bigfile4> lsbackupWaittoread.sh file3.bak myfirst.sh xfile3.bakbigfile.tar filelist myscp.sh xfile5blurb.txt filelist.save noread xfile5.bak> tar -xvf bigfile.tarbigfile1bigfile2bigfile3bigfile4> lsbackup.sh bigfile.tar listwhenreadable.sh xfile1.bakbackupWaittoread.sh file1 logicalop.sh xfile3bigfile1 file1.bak mv.sh xfile3.bakbigfile2 file3 myfirst.sh xfile5bigfile3 file3.bak myscp.sh xfile5.bakbigfile4 filelist noread xfilelist>

tar - example

tar, compress, and gzip• compress

▫"Obsolete" command Not found on many Linux systems

▫Compacts a file so that it is smaller▫When compressing a file it will be replaced with a file

with the extension .z keeping all the same ownership modes

▫Syntax: compress [-c] [-f] [-v] filenames

• uncompress▫Uncompressed compressed files.▫Syntax:

uncompress [-c] [-f] [-v ] [file]

tar, compress, and gzip• gzip

▫ replacement for compress▫ Syntax:

gzip filename compresses filename replaces filename with filename.gz

Notes: -c leaves original file -r recursively compresses all files if filename is a directory

• gunzip▫ uncompresses▫ Syntax:

gunzip filename.gz replaces the filename.gz with filename

▫ same as gzip –d

# ls -l bi*-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile4-rw-r--r-- 1 tkombol tkombol 10240 2013-04-09 15:42 bigfile.tar# gzip bigfile.tar# ls -l bi*-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile4-rw-r--r-- 1 tkombol tkombol 208 2013-04-09 15:42 bigfile.tar.gz# gunzip bigfile.tar.gz# ls -l bi*-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile4-rw-r--r-- 1 tkombol tkombol 10240 2013-04-09 15:42 bigfile.tar# tar -tvf bigfile.tar-rwxr-xr-x tkombol/tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x tkombol/tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x tkombol/tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x tkombol/tkombol 73 2013-03-28 18:04 bigfile4#

gzip example

zip• zip and unzip

▫ PKZIP and PKUNZIP in the Unix/Linux world• zip

▫ Does both combining and compressing in one program▫ -r

recursive will pack and compress the subdirectories also

• unzip▫ non-interactive restore if restored file not present▫ if file exists unzip asks permission to restore

[y]es, [n]o, [A]ll, [N]one, [r]ename options▫ -v

view the contents of the zip file

# ls -l big*-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile4# zip bigfile.zip big* adding: bigfile1 (deflated 75%) adding: bigfile2 (deflated 68%) adding: bigfile3 (deflated 68%) adding: bigfile4 (deflated 68%)# ls -l big*-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile4-rw-r--r-- 1 tkombol tkombol 613 2013-04-10 10:35 bigfile.zip# rm bigfile[0-9]# ls -l big*-rw-r--r-- 1 tkombol tkombol 613 2013-04-10 10:35 bigfile.zip# unzip bigfile.zipArchive: bigfile.zip inflating: bigfile1 inflating: bigfile2 inflating: bigfile3 inflating: bigfile4# ls -l big*-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile1-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile2-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile3-rwxr-xr-x 1 tkombol tkombol 73 2013-03-28 18:04 bigfile4-rw-r--r-- 1 tkombol tkombol 613 2013-04-10 10:35 bigfile.zip#

# cd ; zip -r tkombol.zip . > zip.log zip warning: name not matched: .kde/tmp-SIS-43SPP11-PC zip warning: name not matched: .kde/socket-SIS-43SPP11-PCzip warning: Permission denied zip warning: could not open for reading: .Xauthority.backup# ls -ltotal 150368-rw-r--r-- 1 tkombol tkombol 12 2013-04-04 12:32 ajktestdrwxr-xr-x 4 tkombol tkombol 4096 2013-04-10 10:41 Desktopdrwxr-xr-x 11 tkombol tkombol 4096 2009-11-19 16:53 grades-rw-r--r-- 1 tkombol tkombol 153626580 2013-04-10 10:48 tkombol.zip-rw-r--r-- 1 tkombol tkombol 172323 2013-04-10 10:48 zip.log#

• Example:▫ zip one's home directory and log the results

• Next pages: ▫ partial listing of zip.log

2140 lines long▫ partial view of unzip –v tkombol.zip

long

# cat zip.log adding: .bash_profile (deflated 39%) adding: .gnome2/ (stored 0%)... adding: .gconf/apps/evince/ (stored 0%)... adding: .mozilla/firefox/jlammzq3.default/Cache/4A74584Bd01 (deflated 0%)... adding: .openoffice.org/3/user/config/standard.sob (deflated 2%)... adding: .openoffice.org2/user/registry/data/org/openoffice/Office/Recovery.xcu (deflated 70%)... adding: .config/totem/state.ini (deflated 20%) adding: Desktop/ (stored 0%) adding: Desktop/my-ls.txt (deflated 62%)... adding: Desktop/bashscripts/ (stored 0%) adding: Desktop/bashscripts/awk1.awk (deflated 32%)... adding: Desktop/bashscripts/xfile1 (stored 0%) adding: Desktop/bashscripts/bigfile2 (deflated 68%) adding: Desktop/a.htm~ (deflated 71%) adding: .dmrc (stored 0%)... adding: .thumbnails/ (stored 0%) adding: .thumbnails/normal/ (stored 0%) adding: .thumbnails/normal/7cc9e66b999b867f73edf672dcd4eef7.png (deflated 0%)... adding: .thumbnails/normal/afb4bf3db3cb1f1004599f801ee4af7b.png (deflated 1%) adding: .fontconfig/ (stored 0%) adding: .fontconfig/e13b20fdb08344e0e664864cc2ede53d-x86.cache-2 (deflated 83%)... adding: .fontconfig/7b4a97c10f6c0166998ddfa1cf7392fb-x86.cache-2 (deflated 79%) adding: .lesshst (deflated 17%) adding: .gconfd/ (stored 0%)... adding: grades/ (stored 0%)... adding: grades/apache_pb22_ani.gif (deflated 17%) adding: grades/_private/ (stored 0%) adding: .java/ (stored 0%) adding: .bash_history (deflated 70%)#

# unzip -v tkombol.zip > view.txt# ls -ltotal 150448-rw-r--r-- 1 tkombol tkombol 12 2013-04-04 12:32 ajktestdrwxr-xr-x 4 tkombol tkombol 4096 2013-04-10 10:41 Desktopdrwxr-xr-x 11 tkombol tkombol 4096 2009-11-19 16:53 grades-rw-r--r-- 1 tkombol tkombol 153626580 2013-04-10 10:48 tkombol.zip-rw-r--r-- 1 tkombol tkombol 245847 2013-04-10 11:10 view.txt-rw-r--r-- 1 tkombol tkombol 1481 2013-04-10 11:03 zip.log-rw-r--r-- 1 tkombol tkombol 2534 2013-04-10 11:00 zip.log~# lsajktest Desktop grades tkombol.zip view.txt view.txt~ zip.log zip.log~# cat view.txtArchive: tkombol.zip Length Method Size Ratio Date Time CRC-32 Name-------- ------ ------- ----- ---- ---- ------ ---- 414 Defl:N 254 39% 06-07-07 09:25 2f4166e5 .bash_profile 0 Stored 0 0% 04-04-13 16:03 00000000 .gnome2/... 0 Stored 0 0% 06-16-08 10:27 00000000 .gnome/gnome-vfs/ 51 Defl:N 44 14% 03-28-13 18:03 287c14a5 .gnome/gnome-vfs/.trash_entry_cache 5922 Defl:N 1770 70% 04-08-13 11:41 c33fdb16 .bash_history-------- ------- --- -------230245501 153152546 34% 2139 files#