course overview and introduction to the unix shell209/lectures/w01/w01_furkan.pdf · overview 1...

Course Overview and Introduction to the UNIX ShellCSC209: Software Tools and Systems Programming (Winter 2019)

Furkan Alaca & Paul Vrbik

University of Toronto Mississauga

https://mcs.utm.utoronto.ca/~209/

Week 1

Alaca & Vrbik (UTM) 1-Shell Week 1 1 / 38

https://mcs.utm.utoronto.ca/~209/

Overview

1 Course Objectives & Outcomes

2 Course Evaluation Scheme and Workflow

3 Academic Integrity and Effective Ways of Getting Help

4 Preparing for CSC209

5 What is UNIX?

6 Using a UNIX Shell to Run Programs and Access the File System

Acknowledgement: These slides are derived, in part, from content written byAndrew Petersen and Dan Zingaro.


What is this Course About?

Using a UNIX shellThe shell is an interface for interacting with the operating system(OS)It is programmable

Programming tools: Develop more efficiently

Debugging tools (gdb, valgrind)Build automation tools (make)Version control (git)

Systems programming: Interacting with the OS

The file systemSystem callsProcessesCommunication

The C programming language will be our tool of choice

Other systems-level languages exist, e.g., Go


Course Outcomes

After taking this course, you should be able to...

Diagram and explain the memory usage of an arbitrary C program

Read the man page of an unfamiliar system call or library functionand be able to understand and use it

Write command-line tools in adherence to the UNIX philosophy,which leverage system calls, inter-process communication, thestandard library (libc), and the network stack

Effectively use standard UNIX development tools (e.g., gdb, make,git, valgrind, gdb, bash) and command-line utilities (e.g., cat,less, grep)


Evaluation Scheme

5%: Weekly lecture prep (due 10pm on Sundays)

10%: Weekly tutorial exercises (due 10pm on Fridays)

5%: Assignment 1 (C Basics and Pointers)

10%: Assignment 2 (System Calls and Dynamic Memory)

10%: Assignment 3 (Processes)

10%: Assignment 4 (Communication)

10%: Midterm Exam (Fri. Feb. 15, 5pm)

40%: Final Exam (Min. 40% grade required to pass the course)


A Typical Week in CSC209

Prepare for class

Watch videos on PCRS to familiarize yourself with conceptsSolve simple exercises (worth marks)URL: https://pcrs.teach.cs.toronto.edu/UTM209-2019-01/

Attend class to consolidate and deepen your knowledge

Having prepared in advance, you can take better advantage of yourin-class hoursPractice by solving more advanced problemsAsk questions

Put your knowledge into action with assignments and weekly tutorials


https://pcrs.teach.cs.toronto.edu/UTM209-2019-01/

Assignment and Tutorial Submission

Assignments will be submitted over git

Repositories will be managed with MarkUs:https://mcsmark.utm.utoronto.ca/csc209s19/

Each assignment or tutorial will have its own directory in yourrepositoryStarter code will be pushed to your repositoriesMarks will be made available through the MarkUs web interface

Marking will be automated

Be careful with required file names, directory structure, and outputformatEnsure that your code compiles without warnings or errors, and can berun without crashing the lab PCs


https://mcsmark.utm.utoronto.ca/csc209s19/

Assignment and Tutorial Submission (2)

Code submissions that do not compilewill receive a grade of zero


Assignment Resubmission

To encourage you to learn by looking at the test output and fixing theerrors, we will allow resubmissions

We will send you feedback in the form of test results soon after thesubmission deadline

The feedback is not your mark: Some tests are worth more than othersSome tests might not even be worth marksDo your best to fix the errors that we’ve identified

You will have 48 hours after the feedback is sent out to correct theerrors and resubmit

Your assignment will receive the higher of:

The grade on the original submissionThe grade on the resubmission, with a 10% deduction


Academic Integrity

The work you submit must be your own, done without participationby others. It is an academic offence to hand in anything writtenby someone else without acknowledgement.

You are not helping your friend when you give them a copy of yourassignment

You are hurting your friend when you ask them to give you a copy oftheir assignment

Things that you shouldn’t do:

Looking at another student’s assignmentUsing code that you haven’t written, without attributionAsking someone else to write your codeDiscussing the assignment implementation with other students


On the Other Hand...

Give a man a fish and feed him for a day. Teach a man to fishand you feed him for a lifetime. (Ancient Chinese proverb)

You are helping your friend when you:

Explain and/or clarify important conceptsReview/modify/practice exercises from PCRS, lectures, and previousweeks’ tutorials togetherHelp each other understand error messages, man pagesYou are encouraged to do the above, by, e.g., participating on thediscussion board (https://mcs.utm.utoronto.ca/forum/), formingan FSG (https://www.utm.utoronto.ca/asc/facilitated-study-groups-fsgs)

Your TAs are there to help you

Same guidelines apply: Don’t ask them “How do I do the assignment?”TAs won’t give you the solution, but they can help you understandtools/concepts that you need for completing the assignment, commonmistakes, helpful techniques for debugging your code, etc.


https://mcs.utm.utoronto.ca/forum/

https://www.utm.utoronto.ca/asc/facilitated-study-groups-fsgs

https://www.utm.utoronto.ca/asc/facilitated-study-groups-fsgs

Linux

This course assumes some familiarity with Linux

Get accustomed to using a text editor on Linux

Familiarize yourself with basic shell commands/utilities

Learn to ssh into the lab machines to work remotely

Install a Linux virtual machine on your PC (even better: dual-boot)

This is how cool you will feel after taking CSC209.


C Programming

This isn’t a C programming course

We will use C, and cover some key concepts in C

But after taking this course, you should just as easily be able to learnanother language (like Go) and accomplish the same tasks (refer backto Course Objectives & Outcomes slides)

You have already learned at least two programming languages:Python (CSC108 and 148) and Java (CSC207)

Java syntax is heavily influenced by C

We are expecting you to quickly pick up C programming on your own


Preview of What’s Next

This week: UNIX shell and compiling C programs

Tutorial 1 is due on Friday (Jan. 11)

Next week: Arrays and pointers

Lecture prep is due on Sunday (Jan. 13)Tutorial will be due next Friday (Jan. 18)

Assignment 1 will be posted next week


The UNIX Philosophy

A simple summary of the UNIX philosophy (from A Quarter-Century ofUnix by Peter H. Salus, 1994):

Write programs that do one thing and do it well.

Write programs to work together.

Expect that the output from your program will be used as input foranother (e.g., by piping)Don’t require interactive input

Write programs that handle text streams, because that is a universalinterface.

More on Wikipedia:https://en.wikipedia.org/wiki/Unix_philosophy


https://en.wikipedia.org/wiki/Unix_philosophy

UNIX vs. Linux vs. UNIX-like

UNIX is a proprietary OS developed by AT&T in 1969

UNIX quickly gave rise to both free and commercial imitations suchas BSD, Linux, Solaris...

The macOS kernel is a derivative of BSD

When we say UNIX we generally refer to these “UNIX-like” OSs,which are often referred to colloquially as “*nix”

We will use Linux in this course

Linux is the most widely-used OS in the UNIX-like family: It runs on allsorts of devices, e.g., PCs, smartphones, printers, security cameras,wireless routers...

More on Wikipedia:https://en.wikipedia.org/wiki/Unix_philosophy


https://en.wikipedia.org/wiki/Unix_philosophy

The UNIX Timeline

System III & V family

BSD (Berkeley Software Distribution)

FreeBSD

NetBSD

OpenBSD

SunOS

Solaris

NextStep

Xenix OS

GNU

Linux

CommercialUNIX

HP-UX

AIX

UnixWare

IRIX

BSD family

1970 1980 1990 2000 Time

Microsoft/SCO

Richard Stallman

Darwin

GNU/Hurd

Linus Torvalds

Andrew S. Tanenbaum

Minix

3.3

6.5.30

4.1.4

Research UNIX 10.5

Bell Labs: Ken Thompson,Dennis Ritchie, et al.

Bill Joy

AT&T

IBM

SGI

Univel/SCO

2010

4.4

6.1

11.0

macOS 10.12

7.1

11.3

7.2

11i v 3

4.11

3.4

16.4

Sun/Oracle

Apple

Theo de Raadt

0.9

Matthew Dillon

DragonFly BSD 4.8

Timeline of UNIX-like operating systems.Alaca & Vrbik (UTM) 1-Shell Week 1 17 / 38

Examples of Common UNIX Tools

cd

mkdir

ls

cp, mv, rm

touch

who

diff

head, tail

cat

sort

grep

wc

tr

(The commands in

this column are for

text filtering)

stdin (standard input)

stdout (standard output)

stderr (standard error)

pipes

(All of the above in red have a man page: See next slide)


How to Learn Linux

Use it.

Don’t worry about memorizing stuff

Jump in and start working on your task(s) at hand, and look thingsup as needed

Excellent resources:Man pages: type man name-of-command to learn how a commandworks (Pro-tip: sometimes it’s easier to jump straight to the examples)Man pages also contain more general topics/tutorials, e.g., try man

intro or man gittutorial

Arch wiki: https://wiki.archlinux.org/ (very useful even forusers of other Linux flavours such as Ubuntu)Our ultimate goal is to teach you how to learn on your own—getting tothat stage will be the biggest achievement of your 4-year Universityeducation!

Common tasks will quickly start to become familiar


https://wiki.archlinux.org/

How to Learn Linux (2)

How you will feel when you find an answer in the man pages.


How to Learn Linux (2)

Even the man command has its own man page: Check man man.


The Shell: Commands

$ gcc -o hello hello.c

The $ is a prompt, i.e., prompting the user for input (commands)

We are using the bash shell, but there are others too

A command can be built-in to the shell (e.g., cd, ls, type)

See man builtins

A command can also be the name of an executable file

Can provide the path to the executable fileIf no path is provided, the shell searches for the executable file in alldirectories listed in the PATH environment variable


The Shell: Running a Program


$ ./hello

The first line compiles the C program hello.c into an executable filehello

The second line loads the hello program into memory and hands itto the OS to run the program

Recall from CSC207: Java programs are compiled into bytecode; Cprograms are compiled into native code


Memory Model

Memory is dividedinto segments

The program itselfis loaded into thebottom segments:

Read/writedataRead-only codeand data


Wait a Second...


$ ./hello

What’s with the ./? Why put it in front of hello but not gcc?

This is a good opportunity to discuss the importance of askingquestions if you spot something unfamiliarIn this course, a single missing or extra character can make a bigdifference (remember this when we introduce you to our good friends*, **, and & from the C language)

Can’t say “it’s just one character” if that character holds importantconceptual meaning, compared to perhaps a simple ; which justterminates a statement

The current directory is typically not in the PATH, so executing such acommand is done by inputing ./command, where . represents thecurrent directory

Type echo $PATH to output your PATH variable


The UNIX File System Hierarchy

/ "root"

"essential user command binaries"

bashcatchmodcpdateechogrepgunzipgziphostnamekilllesslnlsmkdirmoremountmvnanoopenpingpspwdrmshsutartouchumountuname

/bin/dev"device filesincl. /dev/null"

/home"user homedirectories"

/proc"process & kernelinformation files"

/lib"libraries &kernel modules"

/mnt"mount files fortemporary filesystems"

/usr"read-only user applicationsupport data & binaries"

"standard includefiles for 'C' code"

"obj, bin, libfiles for coding& packages"

/usr/bin"most usercommands"

/usr/include

/usr/lib

/usr/local"local software"

/usr/local/bin/usr/local/lib/usr/local/man/usr/local/sbin/usr/local/share

/usr/share"static data sharableaccross all architectures"

/usr/share/man"manual pages"

/etc"configuration files for the system"

crontabcupsfontsfstabhost.confhostnamehostshosts.allowhosts.denyinitinit.dissuemachine-idmtabmtools.confnanorcnetworkspasswdprofileprotocolsresolv.confrpcsecurettyservicesshellstimezone

/var"variable data files"

/var/cache"application cache data"

"data modified asprogrammes run"

/var/lib

"lock files to track resources in use"

/var/lock

/var/log"log files"

/var/spool"tasks waiting tobe processed"

/var/spool/cron/var/spool/cups/var/spool/mail

/var/opt"variable data for installed packages"

/var/tmp"temporary files saved between reboots"

/sbin"essential systembinaries"

fdiskfsckgettyhaltifconfiginitmkfsmkswaprebootroute

/opt"optional softwareapplications"

/root"home dir. forthe root user"


File System Hierarchy

/usr/bin/bash

The above is the path to the executable file bash (our shell)

The leading / represents the root directory

usr is a subdirectory of /

bin is a subdirectory of usr

bash is a file located in bin

A directory is a file that contains directory entries

Each directory entry maps a file name to an inode

An inode is a data structure that contains information about a file,such as its physical address on the disk

Stay tuned for CSC369


Directory Entries and inodes

12345 afile

sizeowner UID, GIDaccess timemodified timecreation timelink and block countspermissions

direct pointersto file blocks

single indirect pointer

double indirect pointer

triple indirect pointer

pointers to next fileblocks

12345Inode

Directory Entry


File System Hierarchy on the Lab PCs

On most UNIX systems, a user bob’s home directory is /home/bob

On the lab PCs, it is /student/bob

Your home directory is located on a networked file system that ismounted to the local directory /student on every lab PCType df to see a list of devices, network locations, etc. that aremounted to directories in your file system

Your home directory is also represented by the shortcut ~

cd ~ changes your current working directory to your home directoryType pwd to see what your current working directory is


Files in UNIX

“Everything is a file” is a defining feature of UNIX

It even has its own Wikipedia page:https://en.wikipedia.org/wiki/Everything_is_a_file

UNIX provides a file interface (i.e., open, read, write, and close) forall Input/Output (I/O)

Regular filesDirectoriesSpecial files (e.g., /dev/null, /dev/urandom)Devices (e.g., keyboard, mouse, printer, video output)

Turn your volume up and try running cat /dev/urandom | padsp

tee /dev/audio > /dev/null

Pipes for inter-process communication (can be named or unnamed*)Network sockets (sometimes)


https://en.wikipedia.org/wiki/Everything_is_a_file

Redirecting Output

By default, programs read from stdin (default: keyboard), writeresults to stdout (default: screen), and write errors to stderr

(default: screen)

Use > to redirect stdout, and 2> to redirect stderr

> overwrites the output file; use >> to append

1 and 2 are the file descriptors of stdout and stderr, respectively

Example:

ls >myfiles.txt: List the files in the current directory, saving theoutput in myfiles.txt (sample output for ls coming up on next slide)

More: https://www.tutorialspoint.com/unix/unix-io-

redirections.htm


https://www.tutorialspoint.com/unix/unix-io-redirections.htm

https://www.tutorialspoint.com/unix/unix-io-redirections.htm

Pipes and Process Substitution

Pipes transfer output from one command as input to anothercommand, e.g.,

ls | wc -l: Counts and outputs the number of lines of text that weregenerated by the ls command

You can also feed the output of one or more processes to stdin ofanother process with <; this is called process substitution

Try: ls >out.txt, followed by wc -l <out.txt

Now, try: wc -l <(ls) (and observe how the output differs)Whereas a pipe redirects stdin, process substitution saves the outputof a command into a temporary file, and passes the descriptor to theother process

The round brackets in the last command above tell the shell toevaluate the enclosed commandMore here: https://unix.stackexchange.com/questions/17107/

process-substitution-and-pipe


https://unix.stackexchange.com/questions/17107/process-substitution-and-pipe

https://unix.stackexchange.com/questions/17107/process-substitution-and-pipe

Directories and Links

$ ls

file1 file2 file3 test1 test2

$ ls -a

. .. file1 file2 file3 .hidden test1 test2

$ ls -la

total 16

drwxr -xr -x 4 bob staff 4096 Jan 6 20:18 .

drwxr -xr -x 3 bob staff 4096 Jan 6 20:18 ..

-rw-r--r-- 1 bob staff 0 Jan 6 20:16 file1

-rw-r--r-- 1 bob staff 0 Jan 6 20:17 file2

lrwxrwxrwx 1 bob staff 5 Jan 6 20:17 file3 -> file2

-rw-r--r-- 1 bob staff 0 Jan 6 20:18 .hidden

drwxr -xr -x 2 bob staff 4096 Jan 6 20:16 test1

drwxr -xr -x 2 bob staff 8192 Jan 6 20:16 test2

$


Directories and Links (2)

In the ls -la output from the previous slide:First column is a permission string, e.g., drwxr-xr-x

First character indicates whether it is a directory (d), link (l), or aregular file (-)A symbolic link is a file that contains a reference to another fileA hard link is a regular file that references an inode (doesn’t actuallyappear to ls as a “link”—having multiple hard links to an inodeessentially means referencing the same inode by multiple file names)

The rwx flags represent the read, write, and execute privileges on thefile possessed by the user (file owner), group, and other usersFor a directory, r allows you to list its contents (ls), w allows you tocreate and delete directory entries in it, and x allows you to enter thedirectory (cd)


Directories and Links (3)

In the ls -la output from the previous slide (continued):

Second column is the number of hard links to the inode referenced bythe file

Note that . and .. are hard links that are present in every directoryIf you rm a file name with only one hard link, and if the file is notcurrently opened by any running process, the file is deleted

Otherwise, only the specified file name is deleted, and the hard linkcount is decremented

Third and fourth columns are the user (owner) and group of the file

Fifth column is the size (in bytes) of the file

Sixth column is the last modified date of the file

Last column is the file name

Running ls -ali would also show you the inode number (everyinode has a unique numeric ID) referenced by each file


Changing File Permissions

If you are the file owner (or root), you can change a file’spermissions with chmod

Example: chmod 754 file.txt changes the permissions offile.txt to grant:

7: rwx to the user (owner)5: rx to the group4: r to all other users

Octal notation: For each subject (user, group, other), add up thevalues for r (4), w (2), and x (1)

Other ways of using chmod:

chmod u+rwx

chmod go-x

Exercise: See man chmod to find out what the above commands do


Globbing

Glob patterns are strings that expand to match multiple file names(similar, but simpler, than regular expressions)

Check man 7 glob (this displays the contents glob(7), i.e., the glob

entry from section 7 of the man-pages)

A ? matches any single character

A * matches any string, including the empty string

An expression [list of characters] matches a single characterinside the list, e.g., [abc], [a-z], [ax-z]

Usage examples:

rm *.log: Remove all files ending in .log

ls *.pdf: List all files in the current directory ending in .pdf


Job Control

A job is a program that the user has executed

When a program is running, hit ctrl+z to suspend it

jobs gives you the list of jobs, each with an associated job number

fg num brings job number num to the foreground and resumesexecution

Foreground job has control of the terminal, e.g., all user input iscaptured by that job

bg num resumes execution of job number num in the background

Background job does not control the terminal, but will be stopped ifthe terminal is closed

kill %num kills job number num

Any command can be started in the background by appending & tothe command (e.g., sleep 10&)


course overview and introduction to the unix shell209/lectures/w01/w01_furkan.pdf · overview 1...

Documents