Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016 Creating directoriesandnavigating

through thefilestructure

Part3

1

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

mkdir Make directory:createsanewdirectory

rmdir Removesanemptydirectory

pwd Displaystheabsolutepathofthecurrentworkingdirectory

cd Changedirectory:allows movingfromonedirectorytoanother

ls Listsadirectorycontent

2

Commandsformanipulatingdirectories

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• ls liststhecontentofthecurrentdirectorybydefault• Syntax:ls [OPTION][dirname]

Someusefuloptions:• -l:showssizes,modifieddateandtime,fileorfolder

nameandowneroffileandpermissions

• - a:Listallfilesincludinghiddenfilestartingwith‘.’• -lh:showssizesineasierreadableformat• -R:recursivelylistssub-directories• -lS:sortingbyfilesizes

3

ls command

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• mkdir:makesadirectory• Syntax:mkdir dirname [path]

• mkdir dirname:wouldcreateadirectorywiththespecifieddirname

• Thenewcreateddirectorywillbecreatedinyourcurrentworkingdirectory

• Ifyouwanttocreateitelsewhere,youhavetospecifythepath:mkdir dirname path

4

Create directories

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• Norealdistinctionbetweenthenamesofordinaryfilesandthenamesofdirectoryfiles.

• Notwofilesinthesamedirectorycanhavethesamename.

• Filesindifferentdirectoriescanhavethesamename.

• Linuxiscase-sensitive:Sanger,sangerandSANGERaredifferentandwouldrepresentthreedistinctfiles.

• Inmostcases,fileextensionsareoptional(.txt,.exe,etc.)

5

WhatyoushouldknowaboutfilenamesinLinux

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• cd:changetheworkingdirectory

• Commandsyntax:cd <path>

• Thepathnameofthedirectoryyouwanttomovetoshouldbespecified

• Youcanspecifyeithertheabsolutepathortherelativepath

6

Moveinthefilessystem

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

/

home

Watson

INDA2016Currentworking

directory

• MovetoWatsondirectory

1. cd/home/Watson2. cd../..3. cd..+cd..

• MovetoGenomics directory

1. cd/home/Watson/INDA2016/Genomics

2. cd../Genomics3. cd..+cdGenomicsLinuxGenomics

7

Moveinthefilessystem:example

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

command [–options] [arguments]Examples:

ls –lh /home/Watson/INDA2016pwdmkdir Test1

8

Commandsbasicsyntax

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• mancommandname

9

Howtogethelpforacommandfromtheterminal?

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016 Creating newfilesandusing text

editors

Part4

10

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• nano: a simpleandeasy-to-usetexteditor• IsinstalledbydefaultinUbuntuandmanyotherLinuxdistros

• It’saWYSIWYGeditor:“whatyouseeiswhatyouget.”Whatyoutypedirectlygoesintothetextinput

• vim,emacs,gedit,Geany:excellentprogramsbutdorequiresomelearning

11

Texteditors

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

nameoftheprogramandversionnumber thenameofthefileyou

areediting

shortcuts

12

Getstartedwithnano

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• Tosearchforatextstring,hitCtrl+W,andenteryoursearchterm

• Thissearchcanthenbecancelledmid-executionbyhittingCtrl+C withoutdestroyingyourbuffer

• Ctrl+X:finishtypingandcloseanopenfile

Remember:nano pathname

- Opensthefileifit’sexistingalready,youcanmodifyandsavechanges

- Createsanewfileinthespecifiedpathifitdoesnotexist

13

Somenano shortcuts

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

Let’splaywithnano

• Createexamplefile undersession1

• Open theterminal• Log in(/home/Username)• Typepwd tocheck• MovetotheSession1directoryunder theLinuxdirectory

cdSession1?cd./INDA2016/Linux/Session1

Username

Coursera

Genomics

/

home

INDA2016

Linux

Session1

Session3Session2

14

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016 Manipulating

filesanddirectories

Part5

15

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

16

• Displayingwholecontentofafileorpartsofit(default+options)

• cat:viewthecontentofashortfilesyntax cat <filename>

more:viewthecontentofalongfileandnavigatethroughitsyntax more <filename>

• less:viewthecontentofalongfile,byportionssyntax less <filename>

• head:viewthefirstlinesofalongfilesyntax head <filename>

• tail:viewthelastlinesofalongfilesyntax tail <filename>

Basicviewingfilecommands

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

17

• head commanddisplaysatextfilecontent,bydefault:10firstlinesatatime

• Syntax:head <options> <filename>

• tail commanddisplaysatextfilecontent,bydefault:10lastlinesatatime

• Syntax:tail <options> <filename>

Viewfilecontents:headandtailcommands

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• cp:copyfilesanddirectoriessyntax cp <pathfrom><pathto>

• mv:moveorrenamefilesanddirectoriessyntax mv <pathfrom><pathto>

• rm:removefilesanddirectoriessyntax rm pathname

18

Copy,moveandremove

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

• Simplestform:cp file1file2

è Copythecontentsoffile1intofile2.Iffile2doesnotexist,itiscreated.Otherwise,file2issilentlyoverwrittenwiththecontentsoffile1.

• cp filenamedirpath

èMakeacopyofthefile(ordirectory)intothespecifieddestinationdirectory

19

Copyingcommand

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

Themvcommandmovesorrenamesfilesanddirectoriesdependingonhowitisused• Torenameafile:

mv filename1filename2Iffile2exists,itscontentsaresilentlyreplacedwiththecontentsoffile1.Toavoidoverwriting,usetheinteractivemode:

mv-i filename1filename2

• Tomoveafile(oradirectory)toanotherdirectory:mv filedirpath

• Tomovedifferentfiles(oradirectory)toanotherdirectory:mv file1file2file3dirpath

• Tomovedirectorytoanotherdirectory:mv dir1dir2

Ifdir2doesnotexist,thendir1isrenameddir2.Ifdir2exists,thedirectorydir1ismovedwithindirectorydir2

20

Copyingcommand:mv

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

Therm commanddeletes filesanddirectories

Toremove afile:rm filename

Toremove manyfiles:rm filename1filename2

Addtheinteractivemodetopromptuserbeforedeletingwith–irm -i filename1filename2

Deletedirectorieswithalltheircontentsrm -r dir1dir2

21

Thermcommand

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016 Extracting informationfrom files

Part6

22

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

23

• wc printsnewline,word,andbytecountsforeachfile

• syntax:wc <options> <filename>• Someusefuloptions:• -c:printthebytecounts• -m:printthecharactercounts• -l:printthenewlinecounts• Formoreinfoaboutthedifferentcommandsuseman commandname

Somestatisctics aboutyourfilecontent:wc command

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

24

• sort:reorderthecontentofafile“alphabetically”syntax: sort <filename>uniq:removesduplicatedlinessyntax: uniq <filename>join:comparethecontentsof2files,outputsthecommonentriessyntax: join <filename1><filename2>

• diff:comparethecontentsof2files,outputsthedifferencessyntax: diff <filename1><filename2>

Basicscommandstomanipulatefiles

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

25

• sort outputsasortedorderofthefilecontentbasedonaspecifiedsortkey(default:takesentireinput)

• Syntax:sort <options><filename>• Defaultfieldseparator:Blank• Sortedfilesareusedasaninputforseveralothercommandssosortisoftenusedincombinationtoothercommands

• For<options>see man

Sortingdata

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

26

• sort examples

w Sortalphabetically(defaultoption):sort <filename>w Sortnumerically:sort-n<filename>w Sortonaspecificcolumn(n°4):sort–k4<filename>w Sortbasedonatabseparator:sort-t$'\t’<filename>w ...

Sortexamples

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

27

• grep:tosearchfortheoccurrenceofaspecificpattern(regularexpressionusingthewildcards…)inafileSyntax:grep <pattern><filename>cut:isusedtoextractspecificfieldsfromafile

Syntax:cut <options><filename>

Extractingdatafromfiles

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

28

• grep (“globalregularexpressionprofile”)isusedtosearchfortheoccurrenceofaspecificpattern(regularexpression…)inafile

• Grep outputthewholelinecontainingthatpattern• Syntax:grep <options><pattern><filename>• For<options>see man

Example:Extract lines containing thepatternxxxfrom afile:

grep xxx <filename>Extract lines that donotcontain patternxxxfrom afile:

grep –vxxx <filename>

grep command

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

29

• uniq outputsafilewithnoduplicatedlines• Uniq requiresasortedfileasaninput• Syntax:uniq <options><sorted_filename>• For<options>see man• Usefuloptionis-ctooutputeachlinewithitsnumberofrepeats

uniqcommand

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

30

• join isusedtocompare2inputfilesbasedontheentriesinacommonfield(called“joinfield”)andoutputsamergedfile

• joinrequiressortedfilesasaninput• Lineswithidentitical“joinfield”willbepresentonlyonceintheoutput

• Structure:join <options><filename1><filename2>

• For<options>see man

Joincommand

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

31

• diff isusedtocompare2inputfilesanddisplaysthedifferententries

• Canbeusedtohighlightdifferencesbetween2versionsofthesamefile

• Defaultoutput:common lines notshowed,onlydifferent lines areindicated andshowswhat hasbeenadded (a),deleted (d)orchanged (c)

• Structure:diff <options><filename1><filename2>• For<options>see man

diffcommand

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

32

• grep:tosearchfortheoccurrenceofaspecificpattern(regularexpressionusingthewildcards…)inafileSyntax:grep <pattern><filename>cut:isusedtoextractspecificfieldsfromafile

Syntax:cut <options><filename>

Extractingdatafromfiles

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

33

• grep (“globalregularexpressionprofile”)isusedtosearchfortheoccurrenceofaspecificpattern(regularexpression…)inafile

• Grep outputthewholelinecontainingthatpattern• Syntax:grep <options><pattern><filename>• For<options>see man

Example:Extract lines containing thepatternxxxfrom afile:

grep xxx <filename>Extract lines that donotcontain patternxxxfrom afile:

grep –vxxx <filename>

grep command

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

34

• grep example

Let’sconsiderafilenamed“ghandi.txt”$ cat ghandi.txtThedifferencebetweenwhatwedoandwhatwearecapableofdoingwouldsufficetosolvemostoftheworld'sproblems

$ grep what ghandi.txtThedifferencebetweenwhat wedoandwhat wearecapableofdoing

$ grep-vwhat ghandi.txtwouldsufficetosolvemostoftheworld'sproblems

grep command(2)

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

35

• cut isusedtoextractspecificfieldsfromafile• Structure:cut <options><filename>• For<options>see man• Importantoptionsare

w -d(fielddelimiter)w -f(fieldspecifier)

Example:extractfields2and3fromafilehaving‘space’asaseparatorcut-d’‘-f2,3<filename>

cut command

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

Outputsredirectionandcombiningdifferent commands

Part7

36

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

37

• Bydefault,thestandardoutputofany commandwillappear totheterminalscreen.

• Redirectionoftheoutputresult toafileis possible.• Thisis particularly useful forbig files• Syntax:command optionsfilename.in > filename.out

Commands outputs

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

38

• Ifthefileexists,theresultwill be redirected toit$ cat ghandi.txtThedifference between what we doandwhat we arecapableofdoingwould suffice tosolvemost oftheworld's problems$ cut -d’‘-f2,3ghandi.txtdifference betweenwhat wesuffice toofthe

$ cut -d’‘-f2,3ghandi.txt > ghandi.txt.out$ cat ghandi.txt.outdifference betweenwhat wesuffice toofthe

• Ifthefiledoes notexist,it will beautomatically created andtheresultredirected toit.

Outputsredirection

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

39

• The standard output of any command will be oneunique output

• As seen previously, this output can be printed in thescreen or redirected to a file

• However, the output result of a command can also beredirected to another command

• This is particularly useful when several operations areneeded for a file, with no need to store theintermediate outputs

Commands combination

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

40

• Combiningseveralcommandsisdonethankstotheuseofa“|”character

• Structure:command1 options1filename1.in |command2 options2> filename.out

• Thiscanbedoneforasmanycommandsasneeded

Commands combination:example

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016Wildcards

41

• Sincetheshellusesfilenamessomuch,itprovidesspecialcharacterstohelprapidlyspecifyinggroupsoffilenames

• Agroupofspecialcharacters arecalledwildcardsallowselectingfilenamesbasedonpatternofcharacters

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016Wildcards

42

Wildcard Meaning* Matches anycharacters

^ Matchesanysetofcharacters startingbythecharacterfollowing thesign

? Matchesanysingle character

[!characters] Matchesanycharacterthatisnotamemberofthesetcharacters

[characters] Matchesanycharacterthatisamemberofthesetcharacters.ThesetofcharactersmayalsobeexpressedasaPOSIXcharacterclass suchasoneofthefollowing:[:alnum:]Alphanumericcharacters[:alpha:]Alphabeticcharacters[:digit:]Numerals[:upper:]Uppercasealphabeticcharacters[:lower:] Lowercasealphabeticcharacters

Am

elG

houi

la &

Em

naA

chou

riC

3BI H

and

s-on

NG

S co

urse

–IP

P -2

1stN

ov 2

016

Wildcards:examples

43

Wildcard Meaninga* Any filename starting with a

* Allpossible filenames

^a Anyfilenamestartingbya

A*.fasta AllfilenamesthatbeginwithAandendwith.fasta

????.vcf Anyfilenamesthatcontainexactly4charactersandendwith.vcf

[abc]* Anyfilenamethatbeginswith"a"or"b"or"c"followedbyanyothercharacters

[[:upper:]]* Anyfilenamethatbeginswithanuppercaseletter.Thisisanexampleofacharacterclass