mastering modern linuxbmansoori.ir/book/mastering modern linux, 2nd edition.pdf · 2018-10-10 ·...
TRANSCRIPT
CRCPressTaylor&FrancisGroup6000BrokenSoundParkwayNW,Suite300BocaRaton,FL33487-2742
©2018byTaylor&FrancisGroup,LLCCRCPressisanimprintofTaylor&FrancisGroup,anInformabusiness
NoclaimtooriginalU.S.Governmentworks
Printedonacid-freepaperVersionDate:20180420
InternationalStandardBookNumber-13:978-0-8153-8098-6(Paperback)InternationalStandardBookNumber-13:978-0-8153-8111-2(Hardback)
Thisbookcontains informationobtained fromauthentic andhighly regarded sources.Reasonable effortshave been made to publish reliable data and information, but the author and publisher cannot assumeresponsibilityforthevalidityofallmaterialsortheconsequencesoftheiruse.Theauthorsandpublishershaveattemptedtotracethecopyrightholdersofallmaterialreproducedinthispublicationandapologizetocopyrightholdersifpermissiontopublishinthisformhasnotbeenobtained.Ifanycopyrightmaterialhasnotbeenacknowledgedpleasewriteandletusknowsowemayrectifyinanyfuturereprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,transmitted,orutilizedinanyformbyanyelectronic,mechanical,orothermeans,nowknownorhereafterinvented, includingphotocopying,microfilming,andrecording,or inany informationstorageor retrievalsystem,withoutwrittenpermissionfromthepublishers.
For permission to photocopy or use material electronically from this work, please accesswww.copyright.com(http://www.copyright.com/) or contact theCopyrightClearanceCenter, Inc. (CCC),222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization thatprovides licenses and registration for a variety of users. For organizations that have been granted aphotocopylicensebytheCCC,aseparatesystemofpaymenthasbeenarranged.
TrademarkNotice:Productorcorporatenamesmaybetrademarksorregisteredtrademarks,andareusedonlyforidentificationandexplanationwithoutintenttoinfringe.
LibraryofCongressCataloging-in-PublicationDataNames:Wang,PaulS.,author.Title:MasteringmodernLinux/PaulS.Wang.Othertitles:MasteringLinuxDescription:Secondedition.|BocaRaton:Taylor&Francis,CRCPress,2018.|Revisededitionof:MasteringLinux/PaulS.Wang.2011.|Includesbibliographicalreferencesandindex.Identifiers:LCCN2018008944|ISBN9780815380986(pbk.:alk.paper)|ISBN9780815381112(hardback)Subjects:LCSH:Linux.|Operatingsystems(Computers)Classification:LCCQA76.774.L46W362018|DDC005.4/46--dc23LCrecordavailableathttps://lccn.loc.gov/2018008944
VisittheTaylor&FrancisWebsiteat
http://www.taylorandfrancis.com
andtheCRCPressWebsiteathttp://www.crcpress.com
CHAPTER1■1.11.2
1.3
1.41.5
1.6
1.71.81.91.10
1.111.121.13
1.141.15
ContentsPreface
Introduction
ALinuxPrimerWhatIsanOperatingSystem?GettingStarted:LoginandLogout
DesktopLoginStartingaTerminalWindowRemoteLogin
UnderstandingtheShellEnteringCommandsTryingaFewCommandsCorrectingTypingMistakesAbortingaCommand
ExerciseAUsingFilesandDirectories
CurrentWorkingDirectoryandFilenamesHandlingFilesandDirectoriesStandardPersonalDirectories
ProtectingFiles:AccessControlTheSuperUserExaminingthePermissionSettingsSettingPermissions
ExerciseBTextEditingGettingHard/SavedCopiesCommunicatingwithOthers
Who’sWhoontheSystem:fingerEmail
BrowsingtheWebExerciseCCreatingandRunningYourOwnProgram
CompilingExerciseDConsultingLinuxDocumentation
1.161.171.18
CHAPTER2■2.12.22.32.42.5
2.6
2.7
2.8
2.92.102.11
CHAPTER3■3.13.23.3
ExerciseERoundingUpUsefulCommandsSummary
TheDesktopEnvironmentDesktopOverviewDesktopComponentsTheGNOME3DesktopUnderstandingGUIWorkingwiththeDesktop
SessionControlandSystemSettingsLaunchingApplicationProgramsManagingFilesMultipleWorkspacesDesktopAppearance
WindowsTheXWindowSystemWindowManagerWindowInformation
TheFileBrowserNavigatingtheFileTreeOpeningaFileorFolderFindingFilesManagingFilesandFoldersAccessControlforFilesandFoldersWritingCDsorDVDsChangingYourDefaultFileBrowser
TerminalWindowStartingaGNOMETerminalTerminalWindowandtheShellSelect,Copy,andPasteWebandEmailLinks
AccessingHelpandDocumentationSummaryExercises
InteractingwiththeBASHShellBashInteractingwithBashCommand-LineEditingandCommandCompletion
3.43.5
3.63.7
3.83.93.10
3.11
3.123.133.14
3.153.163.173.18
CHAPTER4■4.1
BashCommandExecutionBashInput/OutputRedirection
StandardInputandOutputI/ORedirectionPipes
BashJobControlBashShellExpansions
HistoryExpansionAliasExpansionBraceandTildeExpansionsVariableExpansionCommandExpansionProcessExpansionFilenameExpansion
BashBuilt-inCommandsShellVariablesEnvironmentofaProgram
CommandExecutionEnvironmentExamplesofBashUsage
CustomizedPromptRemovingFilesSafelyCopy,Paste,andI/ORedirectionDisplayingManualPagesSettingUpYourPersonalWebFolder
DefaultFilePermissionsShellStartupandInitializationShellSpecialCharactersandQuoting
QuotinginBashSimpleFunctionsForMoreInformationSummaryExercises
PuttingCommandsandApplicationstoUseUsefulGUIApps
WordProcessingDocumentFormattingandTypesettingDrawingandDiagrammingRasterGraphicsandImageProcessing
4.2
4.34.4
4.54.64.7
4.84.94.10
CHAPTER5■5.15.25.35.45.55.65.7
5.85.95.105.115.125.13
FileUploadandDownloadPasswordManagerCloudStorage3DModelingand3DPrintingMathematicalCalculations
CommandsandFiltersLeadingandTrailingLines:headandtailCharacterTranslation:trTabExpansionFoldingTextLinesCalendarRemindersbyEmailSortingTextLines
ThegrepCommandRegularExpressions
QuotinginSearchPatternsPatternsforgrepAStreamEditor:sedBuildingPipelines
AddressProcessingForMoreInformationSummaryExercises
WritingBASHScriptsInvokingShellScriptsAFirstShellScriptShellScriptExecutionPositionalParametersTheforCommandTheifCommandTestExpressionsandExitStatus
ExitStatusTestExpressions
TheshiftCommandThecaseCommandThewhileanduntilCommandsNumericalExpressionsThebreakandcontinueCommandsFileQueries
5.145.155.165.175.18
5.195.20
5.215.22
5.235.245.255.26
CHAPTER6■6.16.2
6.3
6.4
VariablesArraysVariableModifiersTheHereDocumentMoreonFunctions
FunctionArgumentsReturnValueofaFunction
RedefiningBashBuilt-inFunctionsExampleBashScripts
Example:RemovingUnwantedFilesExample:ConditionalCopyExample:TotalFileSizesExample:SecureFileTransferExample:ResizingPictures
DebuggingShellScriptsErrorandInterruptHandling
InterruptHandlingThePerlandPHPAlternativesForMoreInformationSummaryExercises
TheFileSystemAFileLocationRoadMapFileTypes
OrdinaryFilesDirectoriesSpecialFilesLinksSymbolicLinks
MoreonFileAccessControlMeaningofPermissionsforaDirectory
FileStatusFileModeFileUseridandGroupidAccessControlEnforcementSetuidandSetgidModesEstablishingaGroupDACandMAC
6.5
6.66.76.86.96.106.11
6.126.136.146.156.16
CHAPTER7■7.17.2
7.37.4
7.57.6
7.77.87.97.10
7.11
7.12
FileSystemImplementationFilesystemOrganizationMountedFilesystemsFilesystemSuperBlockandBlockGroups
TheFilesystemTableCreatingSpecialFilesNetworkFilesystemSearchingtheFileTree:findThelocateCommandSaving,Compressing,andDistributingFiles
PackingFileswithsharFileSharingwithSambaMoreFile-RelatedCommandsForMoreInformationSummaryExercises
Networking,Internet,andtheWebNetworkingProtocolsTheInternet
NetworkAddressesPacketSwitchingClientandServer
TheDomainNameSystemNetworkinginNautilus
AccessingSambaSharedFilesNetworkingCommandsSSHwithX11Forwarding
NoPasswordssh,sftp,andscpRemoteFileSynchronization
CryptographySymmetricCryptosystemsPublic-KeyCryptographyandDigitalSignatureGNUPrivacyGuard
SettingUpGnuPGKeysncryption/DecryptionwithGnuPG
SecureEmailSecureEmailwithThunderbird
MessageDigests
7.13
7.147.157.167.177.18
7.19
7.20
7.217.227.237.247.25
CHAPTER8■8.1
8.2
8.38.48.58.6
8.7
8.8
SoftwareandMessageSigningTheWeb
HypertextMarkupLanguageURLsAccessingInformationontheWeb
HandlingDifferentContentTypesPuttingInformationontheWebWhatIsHTML?WebHostingDomainRegistration
AccessingDomainRegistrationDataTheDNS
DNSServersDNSResolvers
DynamicGenerationofWebPagesDynamicServerPages
HTTPBrieflyARealHTTPExperienceForMoreInformationSummaryExercises
BasicSystemAdministrationManagingUsers
SudoPackageManagement
SoftwareManagementTasksManagingProcessesNetworkConfigurationFirewallSettingsManagingFilesystemsandDisks
DiskPartitionsManagingFilesystemsAddingaNewDiskLVMFileStorageQuotas
FileandSystemBackupBackupwithDéjàDup
SystemBooting
8.9
8.108.118.12
CHAPTER9■9.19.29.39.49.59.6
9.7
9.8
9.9
9.109.11
9.129.139.149.159.16
9.17
SELinuxSELinuxStatusandEnforcingModesSecurityContextsMaintainingandManagingFileContexts
ForMoreInformationSummaryExercises
WebHosting:Apache,MySQL,andPHPWhatIsaWebServer?URLandURIRequestProcessingResponseandContentTypesTheApacheWebServerApacheonLinux
InstallingApachewithPackageManagementRunningtheApacheWebServerControllingtheApacheServer
ApacheRun-TimeConfigurationApacheConfigurationFileBasicsAboutConfigurationDirectivesLoadingModulesGlobalDirectivesContainerDirectives
AccessControlunderApacheWhatIsAccessControl?AccessControlbyHost
RequiringPasswordsSettingUpUserLoginunderApache
HowHTTPBasicAuthenticationWorksHowHTTPDigestAuthenticationWorks
Basicvs.DigestAuthenticationPasswordEncryptionAutomaticFileDeflationHTTPSandSSL/TLSHTTPSSupportManualInstallationofApache
ConfigureandCompileWhatIsPHP?
9.18
9.199.209.219.229.23
9.249.259.269.279.28
CHAPTER10■10.1
10.210.3
10.4
10.510.6
10.7
ThePHPModuleforApacheInstallingthePHPModule
TestingPHPPHPConfigurationPHPCommandLineScriptingDatabaseSupportfortheWebMySQL
Initializing,Starting,andStoppingMySQLMySQLRun-TimeConfigurationAdministeringMySQLResettingtheMySQLRootPassword
InstallingphpMyAdminInstallingMySQL/MariaDBForMoreInformationSummaryExercises
CProgramminginLinuxCommand-LineArguments
ExitStatusCompileandExecute
LinuxCommandArgumentConventionsTheGCCCompiler
ThegccCommandOptionsforgccTheCPreprocessorPreventingMultipleLoadingofHeaderFilesCompilationAssemblyLinkingandLoading
TheCLibraryI/OtoFilesFileUpdatingI/ORedirection
CreatingLibrariesandArchivesErrorHandlinginCPrograms
ErrorsfromSystemandLibraryCallsErrorIndicationsfromMathematicalFunctions
ErrorRecovery
10.8
10.910.1010.1110.12
CHAPTER11■11.111.211.3
11.4
11.5
11.6
11.711.811.9
11.10
11.1111.12
11.1311.1411.15
11.16
DebuggingwithGDBInteractiveDebuggingBasicgdbCommandsASampleDebuggingSessionwithgdb
ExaminingCoreDumpsForMoreInformationSummaryExercises
I/OandProcessControlSystemCallsSystem-LevelI/OI/ODescriptorsReadingandWritingI/ODescriptors
MovingtheCurrentPositionOperationsonFiles
CreatingandDeletingaFileLinkingandRenamingFilesAccessingFileStatusDeterminingAllowableFileAccess
OperationsonDirectoriesCreatingandRemovingaDirectory
DirectoryAccessCurrentWorkingDirectory
AnExample:ccpShell-LevelCommandsfromCProgramsProcessControl
VirtualAddressSpaceProcessLifeCycle
TheProcessTableThepsCommand
ProcessCreation:forkProgramExecution:execRoutines
Example:ASimpleShellSynchronizationofParentandChildProcessesProcessTerminationTheUserEnvironmentofaProcess
Example:CommandSearchInterruptsandSignals
BasicConcepts
11.1711.1811.19
CHAPTER12■12.112.2
12.312.412.5
12.6
12.712.812.9
12.10
12.1112.12
12.1312.1412.1512.1612.17
SendingSignalsSignalDeliveryandProcessingSignalTrapping
ForMoreInformationSummaryExercises
Inter-processandNetworkCommunicationOpeningaProcessforI/OIPCwithpipe
PipebetweenTwoCommandsConnectingaFileDescriptortoaFileStreamTwo-WayPipeConnectionsNetworkCommunication
ClientandServerSockets
CreatingSocketsSocketAddressLocalandInternetSocketAddresses
ATCPEchoClientUsingDatagramSocketsSocketI/OSystemCalls
ShuttingDownSocketsTCP-BasedServers
AcceptingaConnectionAnExampleTCP/IPServer
NetworkLibraryRoutinesDaemonProcesses
ProgrammingaDaemonInput/OutputMultiplexingTCPOut-of-BandDataForMoreInformationSummaryExercises
AppendicesOnline
WebsiteandExampleCodePackage
References
Preface
Linux,agreatsuccessstoryofopen-source,community-developedsoftware, isincreasingly used in Web and application servers, software developmentplatforms,personalworkstations,andresearchmachines.Inthepastfewyears,Linux has improved its user interface, addedmany useful and powerful apps,andgreatlyexpandeditshomeandbusinessuserbase.Incomputerscienceandengineeringdepartments,you’ll findLinuxsystems
inclassrooms,programminglabs,andcomputercenters—notjustbecauseLinuxisfreebutalsobecauseitoffersarichcomputingenvironmentforteachingandlearning.Fromitsbeginningin1991,andwithhelpfromtheGNUProject,Linuxhas
evolved quickly and has brought new powers and conveniences to users.CompetencyonLinuxwillbeimportantforanyseriouscomputerprofessional.ThisbookisarevisededitionofMasteringLinux(late2010),whichwasvery
well received and had the most wonderful review from ACM ComputingReviews:
“This authoritative and exceptionally well-constructed book has myhighest recommendation. It will repay careful and recursive study.—ComputingReviews,August2011”
Theneweditionhasanewtitle,MasteringModernLinux,yetretainedmuchofthegoodmaterialswhileupdatingthem,addingnewtopicsandremovingoldones.Thisbookprovidesacomprehensiveandup-to-dateguidetoLinuxconcepts,
usage, and programming. This text will help you master Linux with a well-selectedsetoftopics.Hands-onpracticeisencouraged;itistheonlywaytogainfamiliarity with an operating system. A primer gets you started quickly. Thechaptersleadyoufromuserinterfaces,commandsandfilters,Shellscripting,thefile system, networking, basic system administration, andWeb hosting, to C-levelprogrammingandkernelsystemcalls.Therearemanyexamplesandcompleteprogramsreadytodownloadandrun.
A summary and exercises of varyingdegrees of difficulty canbe found at the
end of each chapter. A companion website provides appendices, informationupdates,anexamplecodepackage,andotherresourcesforinstructorsaswellasstudents.Seepage353fordetails.
UserFriendlyandComprehensiveThereisbothbreadthanddepthinthisbook’spresentation.Chapter1containsaLinux primer to get the new user started as quickly as possible, withoutawkwardnessorconfusion.Beingable toplayandexperimentwith thesystemadds to the user’s interest andmotivation to learnmore.Once introduced andcomfortable,theuserisguidedthroughawell-selectedsetoftopicscoveringthetypeofdetailedmaterialappropriateforaone-semestercourseat theadvancedundergraduateorbeginninggraduatelevel.ThefirstpartofthetextbookcoversinteractiveuseofLinuxviatheGraphical
User Interface (GUI) and the Command-Line Interface (CLI), includingcomprehensive treatment of the Gnome desktop and the Bash Shell. Usingdifferentapps,commandsandfilters,buildingpipelines,andmatchingpatternswithregularexpressionsaremajorfocuses.Next come Bash scripting, file system structure, organization, and usage,
whichbringustoaboutthemiddleofthebook.The next chapters present networking, the Internet and the Web, data
encryption,andbasicsystemadmin,aswellasWebhosting.TheLinuxApacheMySQL/MariaDBPHP(LAMP)Webhostingcombinationispresentedindepth.SuchpracticalknowledgecanbevaluableformanyLinuxprogrammers.InChapters–12,attentionisthenturnedtoC-levelprogramming.Becausethe
LinuxkernelandmostofitsapplicationsareimplementedinC,itisconsideredthe native language of Linux. In-depth knowledge of Linux requiresunderstanding the Standard C Libraries and the system calls which form theinterface to the Linux kernel. Topics covered include the C compiler,preprocessor, debugger, I/O, file manipulation, process control, inter-processcommunication,andnetworking.Manycompleteexampleprograms,written inthestandardISOC99,areprovided.Appendicesarekeptonthebook’swebsite(mml.sofpower.com).Theysupply
usefulsupplementalinformationforstudents,includingtexteditingandhowtosetupLinuxlearningenvironmentsontheirownWindows®orMac®computers.
FlexibleUsageThisbookisforpeoplewhowishtolearnLinuxandtobecomegoodatusingitandwritingprogramsinit.ThebookdoesnotassumepriorknowledgeofLinux
orUNIX,buthasthedepthtosatisfyeventhosewithLinuxexperience.ComparedtootherLinuxbooks, this text isnota thickvolume.However, it
presentsmanytopicscomprehensivelyandindepth.Manyexamplesaregiventoillustrate concepts and usage. It is well-suited for a one-semester course. Aninstructor can cover all the chapters in sequence or choose among them,dependingontheclassbeingtaught.ForanIntroductiontoLinuxcourse,thechaptersonC-levelprogrammingand
perhapsonWebhostingcanbeskipped.Forasystemprogramming-orientedcourse,theLinuxprimer,interactiveuse
ofBash,andtheGNUdesktopmaterialcanbeomittedorassignedforreadingatthebeginningoftheclass.Thiswillprovidemoretimeforthehardcoretopicsonprogramming.ForanIntroductiontoOperatingSystemPrinciplescourse,thisbookisagood
supplement. Discussion of Linux subjects—the Shell, file system structure,concurrent process management, I/O structure, signals/interrupts, and inter-processcommunication—providesconcreteexamplesandadds to thestudents’understandingoftheabstractoperatingsystemprinciplesbeingstudied.Foraserver-sideWebprogrammingcourse,thecoverageofBash,filesystem,
InternetandtheWeb,andWebhostingcanmakethisbookagreatsupplementaltext.Forcoursesonnetworkprogramming,graphics,Cprogramming,distributed
computing,etc.,thebookcanbeavaluablesupplementaswell.ForthosewhouseLinuxinschooloratwork,thisbookenablesyoutoapply
the system’s capabilities more effectively, resulting in much increasedproductivity.Ready-to-useexamplesprovidemanyimmediatepracticalapplications.Goingbeyond, you can learnhow towrite programs at theShell and theC
levels.This ability enables you to build new capabilities and custom tools forapplicationsorR&D.
ExampleCodePackageThroughout the book, concepts and usages are thoroughly explained withexamples. Insteadofusingcontrivedexamples,however,everyefforthasbeenmade to give examples with practical value and to present them as completeprogramsreadytorunonyourLinuxsystem.Theseprogramsarecollectedinanexamplecodepackagereadytodownload
fromthecompanionwebsite(mml.sofpower.com).Seepage353forinstructionsondownloadingandunpackingtheexamplecodepackage.Thedescriptionfor
eachexampleprogramiscross-referencedtoitsfilelocationwithanotationsuchas(Ex:ex05/argCheck.sh).
EasyReferenceYou’ll find a smooth, readable style uncharacteristic of a book of this type.Nevertheless,itisunderstoodthatsuchbooksareusedasmuchforreferenceasfor concentrated study, especially once the reader gets going on the system.Therefore, information isorganizedandpresented inawaythatalsofacilitatesquickandeasyreference.Thereareampleresourcelistingsandappendices(onthewebsite)andathoroughandcomprehensiveindex.Thein-textexamplesarealsocross-referencedwithfilesintheexamplecodepackage.Thisbookwillbeavaluableaidforanyonewhousestools,accessestheInternet,orwritesprogramsunderLinux,evenoccasionally.
AcknowledgmentsIwould like to thank the editorsRandiCohen,PaulBoyd, andothers atCRCpressfor theirhelpandguidancethroughout thewritingandproductionof thissecondedition.Theirdedicationisindeedmuchappreciated.During the planning and writing of this book, several reviews have been
conducted.SpecialthanksgotoProfessorCharlesBorder,RochesterInstituteofTechnology,NY,whomadeexcellentsuggestionsforimprovingthemanuscriptandmakingthebookeasiertouseforstudentsandteachers,includingtheideatoprovidestudentswithinformationtoeasilysetupLinuxlearningenvironmentsontheirowncomputers.Finally, Iwant to expressmy sincere gratitude tomywife, Jennifer,whose
supportandencouragementhavebeensoimportanttomethroughtheyears.
PaulS.Wangwww.cs.kent.edu/pwang
Introduction
Eversinceitsintroductionintheearly1990s,Linuxhasevolved,improved,andsignificantly expanded its user base. It has become an important factor inmoderncomputing.Linux is a free and open-source operating system that works, in many
respects, just like UNIX. Linux became popular as a widely preferred serverplatformforWebhosting,cloudcomputing,andotherpurposes.However,withthe introductionof theGNOMEandKDEdesktopuser interfaceenvironments(and othermore recent ones), plusmany improvements and new applications,Linux has been gaining ground as a home/office system, as well as a moredominantforceintheservermarket.Becauseitisfreeandopensource,1Linuxisaveryattractiveteachingtoolfor
computer science and engineering departments. Also, because it is fast andreliable,businesses,suchasAmazon,GoogleandFacebook,oftenchooseLinuxto run their Web and application servers. Companies and developercommunities, in the United States and worldwide, contribute to kerneldevelopment,newproducts,personneltraining,andtechnicalsupportforLinux,whiletheoperatingsystemitselfremainsfree.Let’s takeabrief lookat thehistoryofLinux, itsversionsandfeatures,and
thetopicsinvolvedinlearninghowtouseLinux.
ABriefHistoryofLinuxThe beginning of Linux can be traced back to 1991 when Linus Torvalds, ayoung student at the University of Helsinki, Finland, began to create a newPOSIX2compliantkernelandanoperatingsystemmorepowerfulthanMINIX(MIni-uNIX).3Threeyearslater,version1.0oftheLinuxkernel,thecentralpartofthenewUNIX-likesystem,wasreleased.The GNU open-source software movement would also later make many
contributionstoLinux,asremarkeduponbyRichardStallman:
“When you are talking about Linux as aOS, you should refer to it asGNU/Linux.Linux is just thekernel.All the tools thatmakeLinuxanOShave been contributed by GNU movement and hence the name
GNU/Linux."
Linuxhascomealongwaysinceitsearlydays.Today,itisaprimeexampleofthesuccessofopen-source,community-developedsoftware.Linuxisusedonservers,desktopcomputers,laptops,andnetbooks.ThehighlysuccessfulmobilesystemAndroid(version1.02008)deriveditskerneldirectlyfromLinux.ThearticleFedoraproject leaderMatthewMillertalksworlddominationon
Linux’s25thbirthday(PCWorld08/25/2016)says:
“In many ways, we’ve actually reached the fabled ‘world domination’everyone joked about 20 years ago," says Miller. “Linux is the defaultoperatingsystemformostthings...AndroidputsLinuxat theheartof themost common consumer operating system in the world. Open source, tosomedegreeoranother,isnowthedefaultlicensingmodel."
LinuxVersionsUnlike proprietary operating systems, Linux is a combination of open-sourceprograms, including the Linux kernel, GNU tools, desktop managers, andinstallationandpackagemanagementsystems,plusmanyothersystem-,server-,anduser-level applications.Anyone can create different combinations of thesecomponents, perhaps also change or improve them, and create a Linuxdistribution with unique characteristics. Thus, it is not surprising that manycompanies and groups all over the world have been distributing somewhatdifferentversionsofLinuxreadytoinstallonyourcomputer.Linux systems are widely used by individuals, academic institutions,
corporations, and serviceproviders suchasWebhosts,datacenters, andcloudservers.WidelyusedLinuxversionsinclude
Ubuntu—“Ubuntu”means“humanity” inZulu.UbuntuLinuxstartedasaversionof thepopularDebianGNU/Linux.AllversionsofUbuntuLinuxarefree,andthereisnochargeformailingaCDtoyou.UbuntusupportstheGNOMEDesktop environment,while another version,Kubuntu, usestheKDEDesktop.Ubuntu iseasy to install andveryuser friendly,whichhasquicklymadeitthemostpopularversionofLinux.Ubuntuissponsoredby theU.K.-based Canonical Ltd., owned by SouthAfrican entrepreneurMarkShuttleworth.RedHatEnterpriseLinux—TheoriginalRedHatLinuxstartedin1994andwasdiscontinuedbyRedHat Inc. in2004.Thecompanynowfocuseson
Red Hat Enterprise Linux (RHEL) for business environments and onFedoraasacommunity-supportedsoftwareprojectforhome,personal,andeducationaluse.CentOS—RHELlargelyconsistsoffreeandopen-sourcesoftware,buttheexecutables are made available only to paying subscribers. CentOS(CommunityENTerpriseOperatingSystem)isacompletelyfreeversionofRHEL(minustheRedHatlogos)madeavailabletousersasnewversionsofRHELarereleased.Fedora—Fedora is a leading-edge Linux distribution where new featuresand improvements are tested before being included in RHEL/CentOS.Fedoramakes frequent softwareupdatesand is tightly integratedwith theGNOMEuserenvironment.openSUSE—ThisisamajorretailLinuxdistributionsupportedworldwidebyNovell (nowpartofMicroFocus).Novellacquired theSuSELinux(aGerman translation of the original Slackware Linux) in 2004. In thefollowingyear,NovelldecidedtomaketheSUSEProfessionalseriesmoreopen as a community-developed, open-source software and to rename itopenSUSE.Debian—DebianLinuxconsistsentirelyoffreeandopen-sourcesoftware.Its primary form,DebianGNU/Linux, is a popular and influential Linuxdistribution.Debianisknownforanabundanceofoptions.Recentreleasesincludeover26,000softwarepackagesforallmajorcomputerarchitectures.UbuntuisaderivativeofDebian.Mint—Linux Mint, a newcomer, is a reliable and popular desktopdistribution. It adopts a conservative approach to software updates and isbasedonDebianandUbuntu.Raspbian—Based on Debian, Raspbian is Linux optimized for theRaspberry Pi, a credit-card-sized computer for education as well aspracticaluses.
TherearemanykindsofLinuxdistributions fordesktopcomputers, servers,and mobile devices, as well as embedded systems. The Linux Standard Base(LSB)isaneffort,throughtheLinuxFoundation,tostandardizemanyaspectsofLinux distributions, and is a superset of the POSIX standards. Major Linuxdistributions follow LSB. This textbook addresses features common to mostLinuxsystemsandindicatesimportantdifferenceswhereappropriate.
TheUNIX/LinuxPhilosophy:SmallIsBeautifulThe UNIX philosophy influenced not just the original operating system
developed by Ken Thompson at Bell Labs (1969), but also the many UNIXclonesandUNIX-likesystemscreatedafterward.Taken together, theseUNIX-likesystemsaresomeoftheverybestoperatingsystemsdevelopedtodate.The generally agreed-upon central tenants of the UNIX Philosophy can be
listedas
Keepprogramssmall—Writeaprogramtodoonewell-definedtask;doitefficiently,anddoitwell.Avoid verbosity—Perform no unessential output from any programs; useshortnamesforcommandsandcommandoptions.Make programs modular—Build small, independent, and self-sufficientprogramparts,witheachservingaspecific function.Theseprogrampartscan be flexibly combined to form larger programs. This principle isreflectedinthesmallkernel(coreoftheoperatingsystem)cooperatingwithalargesetofsmallcommandswhichworkwelltogether.Compose programs through interfaces—Write programs that are easy tointerfacewithotherprograms.ThefamousUNIXpipe,whichinterfacestheoutput of a program to the input of another, is a primary exampleof thisphilosophy.
Keepingprograminput/output,configuration,anddocumentationinplaintext(characterstrings)asmuchaspossiblemakeselementsoftheoperatingsystemeasytointerface,read,understand,andimprove.LinuxsystemshavegenerallyadheredtotheseprinciplesofUNIX,buthave
alsointroducedrefinementsandimprovements.
LinuxFeaturesLinux incorporates all the outstandingUNIX core features and adds graphicaluser interface (GUI), software update and package management, securityimprovements, and many useful applications. Important features of Linuxinclude
Multi-user and multi-processing—The ability to allow multiple users tologinatthesametimeandtheabilitytorunmanyprogramsconcurrently.Graphicaluser interface—Offeringadesktop environmentwithwindows,icons, panels, and menus, making it easy to use point-and-click foroperations.Most Linux systems use theXWindow system and allow theuser to choose between two popular desktop environments,GNOME andKDE.
Package management—A systematic way to find, install, upgrade,configure, and remove themany software packages available.A packagecontains theexecutableprogramandmetadataspecifyingits title,version,purpose, author/vendor, dependencies (on other packages), etc. Packagesaremadeavailableinrepositoriesfordownloading.TheFedoraandtheRedHat familyLinux systems use thednf (dandifiedyum) tool and the rpmpackage format, while the Debian varieties use the apt (AdvancedPackagingTool)andthedebformat.Shells—AShellisacommand-lineinterface(CLI)totheoperatingsystem.It provides interactive processing and execution of user commands. Thestandard (default) Shell for Linux isBash (born-again Sh), but youmayeasilychoosetouseadifferentShell.Hierarchical file system—The entire file system is tree structured and isanchored at a singledirectory called the root.The rootdirectorycontainsfiles andotherdirectories that, in turn, containmore files anddirectories.Eachuserhasahomedirectoryforhis/herownfiles.Thefilesystemtreeisdivided into volumes, which can bemounted ordismounted by attachingthemtoanodeinthefiletree.Aphysicalstoragedevicecancontainoneorseveralfilesystemvolumes.Fileaccesscontrol—Eachfileinthefilesystemisprotectedbyasequenceofbitswhosevalueisspecifiedbytheownerofthefile.Accesstofilesiscontrolled by the operating system. System-wide access is granted to so-calledsuperusers,usually thesystemadministrators.To improvefileandsystem security, the Security-Enhanced Linux (SELinux) (kernel moduleavailableonmostmodernLinuxdistributions)canalsobeenabled.Compatiblefile,device,andinter-processI/O—I/OtophysicaldevicesandI/O to a file look the same to a user program. A user can redirect aprogram’sI/Osothatwithoutchangingtheprogramitself, inputoroutputcanbedirectedtoaterminalwindow,file,oreventoanotherprogram’sI/O.The ability to combine and connect existing programs in this pipelinefashionprovidesgreatpowerandflexibility.Concurrent processes—Following UNIX, Linux provides a set of ShellcommandsandC-languageAPIs to initiateandmanipulateasynchronous,concurrent processes. This allows a user tomaintain several jobs at onceand to switch between them. It is also critical for pipelining severalcommands(processes)together.ServingtheInternet—AsUNIX,Linuxsystemsprovidelocalandwideareanetworking through sockets that support the Internet Protocol (IPv4 andIPv6) and provides efficient network services. System admin can easily
manage,configureandstartorstopdifferentnetworkservices.Linuxalsoworks well with the Apache Web Server to provide Web hosting. As aresult,Linuxisverypopularasanetworkserverplatform.Utilities—The Linux architecture encourages building self-containedprograms to add new facilities. Linux systems come with many utilityprograms including text editors, document processors, email servers andclients,Webbrowsers,rasterandvectorimageeditors,scriptinglanguages,language compilers, file manipulation tools, databases, multimedia tools,GUI design and programming tools, software engineering tools, andnetworkingandothersystemfacilities.Theseutilitiesusuallycomein theformofaLinuxpackagewhichcanbedownloaded,installed,andmanagedeasilywithapackagemanager.
TheLinuxkernel, thecentralpartof theoperating systemwhichprovidesaprogramminginterfaceto thehardware, isrobustandhighlyefficient.Figure1showsanoverviewoftheLinuxsystemorganization.Whenstudyingthevarioustopicsinthistextbook,thisorganizationaldiagramhelpstotiethemtogether.
Figure1LinuxOrganization
TheLinuxEnvironmentLinuxisamulti-user, time-sharingsystemthatoffersbothaGUI(desktopandapplicationwindows)aswellasaCLI(theShells).Thedesktopisthefirstthingyouseeafter login.Thedesktopdisplaysoneormorepanelsat the topand/orbottom of your screen. A panel provides menus, launchers, and workspaceswitchers which perform various tasks. Icons on the screen provide access to
Computer,Services,FileSystem,andsoon.Application programs fall into two broad categories: GUI applications and
CLI applications. A GUI application displays its own graphical window withwhichtheusermayinteractviathemouseandthekeyboard.Incontrast,aCLIapplicationmustruninsidea terminalwindowand interactswith theuseronlythroughthekeyboard.Launchers in panels, in menus, or on the desktop make starting programs
easy. However, any program can be invoked by typing a command inside aterminalwindow.Youcancontrolandswitchamongmultiplewindowson thescreen.AcommandShellhelpsyoucontrolandmanagemultiplejobsinsideanysingleterminalwindow.Thefilesystemcontainspublicfilesandprogramsforallusers.Eachuseralso
hasapersonalfiledirectoryknownastheuser’shomedirectory.Accesstofilesanddirectoriesiscontrolledbythefileownerandgroupdesignations.Linuxallowsahighdegreeofcustomizationonaper-userbasis.TheShell,as
wellasimportantutilitiessuchastheXWindowSystemandtexteditors,refersto initialization andconfiguration files.You can tailor these files tomake theutilities run according to your needs and preferences. You can even chooseamongdifferentShells to serveasyourCLI.Documentation forLinuxand itsutilitiescanbeconvenientlyaccessedlocallyonyourcomputer,aswellasontheWeb.
LearningLinuxLinuxsystemsareusedwidelyincolleges,universities,andcompanies,bothasserversandasworkstationsincomputerlabs.ManyusershaveLinux/Windowsdualbootontheirpersonalmachines.KnowledgeofLinuxisimportantforbothlearningandemployment.Tomakeexperimentationeasier,astudentcansetupaprivateLinuxlearning
environmentonhis/herownWindows®orMac®computer.SpecificinstructionscanbefoundintheAppendix.This book covers a set of carefully selected topics that enable you to
understandoperatingsystemconcepts,touseLinuxeffectively,andtotakefulladvantageofyourLinuxcomputer.Thechaptersaresequencedinadrill-downprogressionstartingwithaprimer
to get you started quickly on Linux with hands-on learning and meaningfultasks.Next,wepresenttheLinuxGUIandthestandardLinuxCLI(theBashShell).
Thenwediscussusefulapps,commands, filters tobuildpipelines,andregular
123
4
expressions for pattern matching. All this paves the way for writing BashprogramscalledShellscripts.Diggingdeeper,wediscusshowtocontrol filesandfolders,andhowLinux
organizesandmanipulatesfilesinasetoffilesystemsthatisanimportantpartoftheLinuxkernel.Computers are rarely used in isolation, and, like other modern operating
systems,Linuxreliesheavilyonnetworkingformanyoperations.Withagoodfoundation from the earlier chapters, we discuss networking, Web, Internet,public-keyencryptionanddigitalsignature.Linuxsystemadministrationbecomesimportantafterausergetsfamiliarwith
the operating system. For people serious about a Linux-related career, systemadminknowledgeiscritical.WecoverthebasicsofLinuxsystemmanagementinChapter8.Attention then turns toC-levelprogramming,kernelsystemcalls,processes,
and inter-process communication. These topics shed light on the internals ofLinux and provide a deeper understanding of concepts and topics covered inearlier chapters. The material should prove especially important for CS/CEmajors.Thus, youwill find traditional aswell as contemporary topics important for
the modern Linux environment. The material in this book applies to mostpopular Linux systems. The knowledge gained will enable you to use anyversionofLinuxwithease.MajordifferencesamongLinuxversionsarenotedwhereappropriate.BecauseLinuxisbestlearnedthroughfrequentexperimentationandpractice,
webeginwithaprimerthatgetsthenewuserstartedquickly.Weofferexamplesand practical ways to use Linux throughout the book. Many examples areprovidedtoillustrateconceptsandtodemonstrateprogrammingtechniques.Thistextbook also contains an example code package 4 which provides completeprograms ready to download and run on your computer. The material ispresented for smooth reading as a textbook, but also for convenient referencelateron.
LinuxisdistributedundertheGNUGeneralPublicLicense.EEEComputerSocietystandardsforPortableOperatingSystemInterface.MINIX is the first open-source clone of UNIX for the IBM PCwritten by Professor Andrew S.
Tanenbaumin1987.Seepage353fordownloadinginstructions.
1.1
Chapter1
ALinuxPrimer
If you are serious about computing, safeguarding security, and understandinghowanoperatingsystemworks,Linuxis thesystemofchoice.TolearnLinuxyoumustuse it, and,of course, touse it youmust learn it.Suchaparadox israther common—you probably learned to drive a car this way. You just needsome basic help and pointers to get started. Here we present an overview ofbasics.Onceyouunderstandthematerialinthischapter,youwillbeabletousetheoperatingsystemtolearnmoreineachsuccessivechapter.Atfirst,youneedalearner’spermittodriveacar.Considerthischapteryourlearner’spermitforLinux;withalittlepracticeyouwillbeusingLinuxalmostrightaway.LearningLinuxinvolvesunderstandinghowtouseitfromtheuserlevel,how
toapplyitspowersandappseffectively,featuresmakingitsuchagoodserver,andhowtoprogramitfromthesystemlevel.Withthistextbook,you’llbegintomasterLinux.Thisprimerprovidesbasic informationandaselectionof topicsdesigned to
getyoustartedusingLinuxquickly.Asyou read thischapter, try thedifferentcommands and features as you come to them. In each case, we will provideenoughinformationtogetyouonthesystemandlearning.
WHATISANOPERATINGSYSTEM?Theoperatingsystemcontrolsacomputerandmakesitusable.Itbringslifetothe innate electronic hardware components and orchestrates all activities on acomputer. The same hardware under a different operating system is literally adifferentcomputer.The operating system provides service and control functions to users,
programs,files,operators,displaymonitors,printers,networkconnections,andeverythingelseonacomputersystem.Acomputeroperatingisoneofthemost
1.2
complicatedandsophisticatedobjectshumanseverbuilt.AmodernoperatingsystemlikeLinuxconsistsofthreemainparts:akernel,
interfacesforusers,programs,devicesandnetworks,andasetofcommandsandapps. The kernel deals with central functions, including concurrent programexecution, memory management, input/output (I/O), file services, networkingandsecurity.Commandsandappssupplyotheroperationssuchasfilemanagers,text editors, email processors, Web browsers, software package managers,audio/video and image processing tools, language compilers, and so on.Interfacesdefineandsupportcommunicationsamongallthecomponents.Forusers,Linuxprovideseasy-to-useGraphicalUserInterfaces(GUIs)inthe
form of desktop environments. Linux also provides efficient and effectiveCommand-LineInterfaces(CLIs)intheformofShells.
GETTINGSTARTED:LOGINANDLOGOUTTo access your Linux system, you must have a user account, identified by auserid and a password, that have been created by a system administrator. Atmost installations, youruseridwill beyour last nameor your first initials andlastname(oftenalllowercase).Your password is a safeguard against unauthorized use of your computer
account.Youneed to choose a passwordof at least eight or twelve characters(your localsystemmayenforceotherconventionsaswell, suchasaminimumlengthorthattherebeatleastonenumeralorsymbol).Passwordsmustbehardto guess. Correctly spelled words or names of relatives are bad choices. Asequence containing upper and lower case characters, digits, and symbols isusually better. Longer passwords can be a phrase. Since you are the only onewho knows your password, you must be careful with it. Forgetting yourpasswordmeansthesystemadministratormustcreateanewoneforyou.Givingyour password to thewrong person could have evenmore dire consequences;youcouldbeblamedforwhateverdamageiscaused,intentionallyorotherwise,bytheotherperson.Donottellorshowanybodyyourpassword.Keepitwrittendownsomewheresafe.Onceyouhaveauseridandpassword,youcanbeginyourLinuxsession.The
firststepistheloginprocedure.Loginprotectsthesystemagainstunauthorizeduse and authenticates the identity of the user. You can use Linux from theconsoleoracrossanetwork.
Figure1.1LinuxLoginScreen
DesktopLoginFindacomputerdisplayingtheLinuxdesktoploginscreen(Figure1.1).Thiscanbetheconsolewherethekeyboard,mouse,anddisplayaredirectlyconnectedtothe computer hardware running the Linux system. Or it can be a differentcomputer on the LAN (Local Area Network). Colleges, universities, andcompanies often run computer labs with Windows or Mac stations that canaccessLinuxserversanddisplaytheirdesktopscreens.Inanycase,enteryourcorrectpasswordcarefullyandprivately.Ifyouarea
newuserand,afterseveralcarefultries,youareunabletologin,itmaybethatthe system administrator has not yet established your userid on the computer.Wait a reasonable length of time and try again. If you still have a problem,contactyoursystemadministrator.Afterlogin,you’llseeyourdesktopdisplayed.Thedesktopenablestheuseof
full-GUI (Graphical User Interface) applications that allow point-and-clickoperationswiththemouse,touchpadortouchscreen.From the desktop, you can press thewindows or super key or click on the
starticontoshow/hideastartmenuorapplicationsmenudisplayingmanytasksyoucando.ThestarticonisusuallyaLinuxdistributionlogolocatedontheleftendofyourdesktopPanel(normallyahorizontalbaracrossthetoporbottomofyourscreen). InGNOME3 forexample, simplymoving themousequickly totheupperleftcornershows/hidestheActivityScreen(Figure1.2).
Figure1.2AGNOME3ActivityScreen
TologoutfromLinuxlookforalogouticononthedesktopPanel.Morewillbesaidaboutdesktopsin2.
StartingaTerminalWindowFrom the desktop, you can conveniently initiate many operations includingstartingaterminalwindow(Figure1.3)thatrunsaShell(Section1.3).TheShellprovides you with a command-line interface (CLI) where you can entercommandstoperformalmostanytaskonLinuxquicklyandefficiently.
Figure1.3ATerminalEmulationWindow
Tostartaterminalwindow,gotothestartmenuandclickontheSystemtools-> Terminal option or the Accessories- > terminal option, depending on yourLinuxanddesktop.Theterminalmaybegnome-terminal,konsole,oranotherdepending on your Desktop environment. The GNOME 3 Applications menu(Figure1.4) includesagnome terminal icon toconveniently launcha terminal
window.Aterminalwindowemulatesacharacter-basedcomputer terminalandallowsyoutouseLinuxthroughacommandinterpretercalledtheShell(Section1.3).Theterminalwindowallowsyoutochangeitsappearanceandfonttoyourownliking.Asitstarts,theShellalsopositionsyouatyourhomedirectory(Section1.5),
the file folder reserved for you as a user on Linux. The Shell indicates itsreadiness to takeyourcommandsbydisplayingapromptat thebeginningofaline.
Figure1.4GNOME3ApplicationsMenu
Whenyouarefinishedwithaterminalwindow,youmaycloseitbyexit(exitsfromShellandclosestheterminalwindow)logout(sameasexit)Thecharacterctrl+d(thecharacterd typedwhileholdingdownthectrlkey)
typedaloneonacommandlineoftencanbeusedinplaceoftheexitcommand.Exitwithctrl+disconvenientbutdangerous,becauseonetypingerrorcancloseyourterminalwindow.SeeChapter3forhowtodisableexitviactrl+d.Bytheway,weshallusethenotationctrl+Xto denote a control character, where X is some character. Note also that
althoughtheconvention is toshowanuppercasecharacter,youdonotneed toholddownshiftwhentypingacontrolcharacter.
RemoteLoginUniversities and other institutions often run large Linux servers for users toaccessthroughaLANoreventheInternet.YoucanuseTELNET,ormorelikelySSH(SecureShell),toaccessaLinuxsystemfromanothercomputer,whichcan
be a PC, anotherLinux system, or anyother platform.Figure1.5 shows SSHaccess, via the Putty tool (free and recommended), to a Linux hosttiger.cs.kent.edufromMSWindows®.OnLinux,theShell-levelcommandsshprovidesSSHandisusedtoaccessa
remoteLinuxserverfromaLinuxsystem.Forexample,[email protected]@tiger.cs.kent.edunetworkstothecomputertiger.cs.kent.edu(thedomainnameofthecomputer)
andattempts to log inwith theuseridpwang.Remote loginnormallysupportsonlyCLIaccess.The-X(capitalX)optionallowstheremotecomputertoopenthegraphicaldisplayonthelocalLinuxandthereforeenablesyoutoalsolaunchremote applications that require a GUI. Running GUI programs remotelyinvolvesmuchheaviernetworktrafficandcanbeslow.Without the -Xoptionyou’llbeable to runonlycommand-lineapplications
ontheremotecomputerwhichisusually theefficientandsensible thingtodo.WewillreturntoSSHinChapter7(Section7.6)wherenetworkingisdiscussed.Download,installation,andusageinformationforSSH/SFTPcanbefoundintheappendicesonthecompanionwebsite(mml.sofpower.com).SuccessfulremoteloginviaSSHresultsinyourSSHwindowbeingconnected
toaloginShellrunningontheremoteLinux.
Figure1.5SSHLoginviaPutty
After login,Linuxwill recordyour login inasystemlog,displayamessageshowingthetimeandplaceforyourlastlogin,andinitiateaShelltotakeyourcommands.Whenyousee theprompt,youare ready tobegincomputing.Afteryouare
1.3
done,youwillneedtologoutfromtheremoteLinux.Tologout,firstcloseanyprograms that youhavebeen running and then issue theShell-level commandexitorlogout.Itisagoodpracticetofirstcloseallrunningprogramsmanuallyinsteadofrelyingonthelogoutprocesstoclosethemforyou.
UNDERSTANDINGTHESHELLTheShell displays a prompt to signal that it is ready for your next command,which it then interprets and executes. On completion, the Shell re-signalsreadinessbydisplayinganotherprompt.ThereareseveralavailableShells:theoriginalShellwrittenbyS.R.Bourne
known as the Bourne Shell or Sh, theC-Shell orCsh developed at UCB byWilliamJoy,andanenhancedCshnamedTcshwhichhasreplacedCshfor themostpart.TheDashshellisabare-bonesandPOSIX-compliantimplementationofShusuallyusedonlyatsystemboottime.ThestandardShellforLinuxistheBash(Bourne-AgainSh),whichisapowerfulandmuchimprovedversionofSh.ThedefaultShellonmostLinuxdistributionsisBash.AttheShellprompt,enterthecommandecho$0+todisplaythenameoftheShellyouareusing.Hereechodisplaysthevalueof
theShellvariable$0.Don’tworry,3explainshowthisworks.YoucanchangethedefaultShellwiththechsh(changeShell)command.Forsecurityreasons,usuallyonlyapprovedShellscanbeused.InthistextwewillassumetheBashShell,althoughbasicfeaturesofallShells
areverysimilar.
EnteringCommandsIn Linux, you can give commands to the Shell to start application programs,managefilesandfolders,controlmultiplejobs(tasksthatarerunning),redirectI/O of programs from/to files, connect one program to another, and performmany other tasks. Virtually anything you want done in Linux can beaccomplishedbyissuingacommandtotheShell.Manydifferentcommandsareavailable,butsomegeneralrulesapplytoallof
them.Onesetofrulesrelatestocommandsyntax—thewaytheShellexpectstosee your commands.A command consists of one ormorewords separated byblanks.Ablankconsistsofoneormorespacesand/ortabs.Thefirstwordisthecommandname(inthisbookthenameofacommandwillappearinboldface);the remaining words of a command line are arguments to the command. A
command line is terminated by pressing the return (or enter) key. This keygenerates a newline character, the actual character that terminates a commandline.Multiplecommandscanbetypedonthesamelineiftheyareseparatedbyasemicolon(;).Forexample,thecommandlsfolderliststhenamesoffilesinafolder(directory)specifiedbytheargumentfolder.
Ifadirectoryisnotgiven,lsliststhecurrentworkingdirectory(Section1.5).Sometimesoneormoreoptionsisgivenbetweenthecommandnameandthe
arguments.Forexample,ls-Ffolderaddsthe-F(filetype)optiontolstellinglstodisplaythenameofeachfile,or
eachfilename,withanextracharacterattheendtoindicateitsfiletype:/forafolder,*foranexecutable,andsoon.AttheShelllevel,thegeneralformforacommandlookslikecommand-name[options]...[arg]...The brackets are used to indicate elective parts of a command that can be
given or omitted. The ellipses (… ) are used to indicate possible repetition.These conventions are followed throughout the text. The brackets or ellipsesthemselvesarenottobeenteredwhenyougivethecommand.Commandoptionsareusuallygivenasasingleletterafterasinglehyphen(-).
Forexample,thelonglistingoptionforthelscommandis-l.Suchsingle-letteroptions can sometimes be hard to remember and recognize. Many Linuxcommands also offer full-word options givenwith twohyphens. For example,the–helpoptiongivenaftermostcommandswilldisplayaconcisedescriptionofhowtousethatparticularcommand.Tryls–helptoseeasampledisplay.After receiving a command line, theShell processes the command line as a
character string, transforming it in various ways. Then, the transformedcommand line isexecuted.Afterexecution is finished, theShellwilldisplayaprompttoletyouknowthatitisreadytoreceivethenextcommand.Figure1.6illustratestheShellcommandinterpretationloop.Typeahead isallowed,whichmeansyoucantypeyournextcommandwithoutwaitingfortheprompt,andthatcommandwillbetherewhentheShellisreadytoreceiveit.
TryingaFewCommandsWhenyouseetheShellprompt,youareattheShelllevel.NowtypeechoHelloLinux
Figure1.6CommandInterpretationLoop
You’llseethattheechocommanddisplayswhatyoutype.Next,enterecho-n"HelloLinux";echouserThis command line contains two commands separated by the ; command
separator. (Ifyoumakeamistake typing thesecommands,glanceahead to thenext subheading on correcting typingmistakes.)The option -n causes echo toomitanewlinecharacterattheendofitsoutput,sotheworduserappearsonthesame line asHelloLinux.Note also the use of quotationmarks for the stringHelloLinuxwhichhasatrailingspace.Oneuseofecho istoexaminethevalueofaShellvariable.Forexample, if
youtypeecho$HOMEyou’llseethevalueoftheShellvariableHOMEwhichisthelocationofyour
home directory in the file system. Note that the value of a Shell variable isobtainedbyprefixing the variable namewith a dollar sign ($).MoreonShellvariablescanbefoundin3.A computer on a network is known as ahost and is usually identified by a
hostname.TofindoutyourLinuxsystem’shostname,givethecommandhostnameToidentifytheoperatingsystemversionrunningonyourcomputer,enterthe
commanduname–allAnothercommandiswho.Typewhoto list currentusers signed inon the system.Thisgivesyouan ideaofhow
manypeoplearesharingthecomputingfacility.Thelscommandwillnotlisthiddenfiles,anyfilewhosenamebeginswiththe
period(.)character,unlessthe-aoptionisgiven.ls-alists thenamesof allyour files, including thehiddenones.Hidden files are
usuallystandardoperatingsystemorapplicationfilesforconfigurationorother
prescribedpurposesandoughtnotbemixedwithotherfilescreatedbytheuser.For the Bash Shell, one standard file is .bash_profile in a user’s home
directory.YoucanplaceinthisfileyourpersonalinitializationtobeusedwhenbashstartsasaloginShell.Ifyouarecuriousaboutwhat’sinthefilebash_profile.,typethecommandmore.bashprofiletodisplay its contents.Press space tocontinue to thenextpageorq toquit
fromthemoredisplay.Don’tbediscouragedbywhatyoudon’tunderstand inthis file. When you have progressed further in this book, the contents willbecomeclear.TheLinux systemkeeps trackof the time anddate precisely, as youwould
expectanycomputertodo.Thecommanddatedisplays the current date and time as given by the following typical output
showingEasternDaylightTimeThuDec416:37:07EST2018TheLinuxsystemhasadictionaryofwordsforspellcheckingpurposes.The
commandspellfilewilldisplaysuspectedmisspellingsforyou.Oryoucanuseaspell-cfiletointeractivelyspellcheckthegivenfile.Tolookforwords,youcanuselookprefixonmostLinuxsystems,andallwordsinthedictionarywiththegivenprefix
aredisplayed.Anotherusefulcommandispasswd.Typepasswdtochangeyourpassword.ThiscommandwillpromptasfollowsChangingpasswordforyouruseridOldpassword:Newpassword:Retypenewpassword:pausingaftereachprompttowaitforinput.ManyLinuxinstallationsgiveout
newuseridswithastandardpassword,andthenewuser isexpectedtouse thepasswdcommandtochangetoapersonalpasswordassoonaspossible.Thecommandmanconsultsthemanualpagesformostcommands.Thus,mancommandwilldisplaythedocumentationforthegivencommand.Trymanpasswd
justtoseewhatyouget.LearnaboutmanwithmanmanDetailsonthemancommandcanbefoundinSection1.15.Themancommanddocumentsregularcommands(applicationprograms),but
normally not commands built in to Shells or other application programs. ForBashyoucanusehelpbuiltin_commandto see a summary of any particular Bash built-in command. Many Linux
systemsaddaBash_BuiltinsmanpagesothemancommandwillworkforBashbuilt-incommandsaswell.
CorrectingTypingMistakesAs you entered the preceding commands, you may have made at least onekeystroke error, or you may wish to reissue a command you have enteredpreviously. Linux Shells provide easy ways to correct typos and to reusepreviouscommands.Basically,youcanusethearrowkeystomovethecharactercursorleftandrightonacommandlineanduptoapreviouscommandordowntothenextcommand.The delete (backspace) key deletes the character under (before) the cursor.
Theenter(ret)keyissuesthecommandnomatterwherethecursorisontheline.The Bash Shell has great support for editing the command line. It actually
allows you to pick a text editor to help do the job.We will return to this inChapter3,Section3.3.
AbortingaCommandApartfromcorrectingtypingmistakes,youcanalsoexerciseothercontrolsoveryourinteractionwithLinux.Forinstance,youmayabortacommandbeforeitisfinished, or youmaywish to halt, resume, and discard output to the terminalwindow.Sometimes,youmayissueacommandandthenrealizethatyouhavemadea
mistake.Perhapsyougiveacommandandnothinghappensoritdisplayslotsofunwantedinformation.Theseareoccasionswhenyouwanttoabortexecutionofthecommand.To abort, simply type the interrupt character, which is usually ctrl+c. This
interrupts (terminates) execution and returns you to the Shell level. Try thefollowing
1. Typepartofacommand.2. Beforeyouterminatethecommand,pressctrl+c.
1.4
1.5
Itcancelsthecommandandgivesyouanewprompt.
EXERCISEA1. Howdoyoustartaterminalwindow?2. Whatcommandandoptionshouldbeusedtolistallthefilesinyourhome
directory?3. Setupctrl+alt+Tasthekeyboardshortcutforrunningaterminalwindow.4. What command is used to change your password? Can you change it to
something like 123 Why Make up a longer password and change yourpasswordtoit.Whydidyouhavetotypeyourpasswordtwicethistime?
5. TryinputeditingwiththearrowkeysunderBash.Afterdoingacommandls -l,pressup-arrowonceand left-arrow twice.Where is thecursornow?Now, pressRIGHT-ARROW once and the cursor should be over the letter lwhich is the last character on the command line. Can you press RIGHT-ARROWagaintomovethecursorbeyondlIfnot,canyoufindaway?(Hint:Limityourselftousingonlythearrowkeys.)
6. What is theLinuxdistributionyouare running?What is thehostnameofyourLinuxcomputer?Howdoyouobtainthisinformation?
Figure1.7ASampleFileTree
USINGFILESANDDIRECTORIESLikeothermodernoperatingsystems,Linuxstoresfilesforusers,applications,andtheoperatingsystemitselfonharddisksforreadyaccess.Thestructureusedto store and manage such files is called a file system. Files under Linux areorganizedintoatreestructurewitharootnamedbythesinglecharacter/.Aregularfilestoresaprogramordata.Adirectoryorfoldercontainsfilesand
possibly other directories. Internal nodes on the Linux file tree representdirectories; leafnodesrepresent regular files.Thishierarchical filestructure is
widelyusedbydifferentoperatingsystems.AsampleLinuxfiletreeisshowninFigure1.7.Avisualfilebrowser(Figure1.8)utilityallowsyoutonavigatethefilesystem
and perform operations on files and folders. Two popular file browsers areKonquerorandNautilus.Forexample,thecommandnautilusfolderlaunchesNautilusandpositionsitatthegivenfolder.Whilethefilebrowsermakesmovingaboutthefilesystemmorevisual,many
LinuxusersstillfinddealingwithfilesandfoldersviatheShellcommandlinemoreefficient.
CurrentWorkingDirectoryandFilenamesWhen you get a userid and account on your Linux system, you are given apersonalfiledirectoryknownasyourhomedirectory.Yourhomedirectorywillhaveyouruseridasitsname,anditwillusuallybeachildofadirectorycalledhome.Yourfilesandfoldersarekeptinyourhomedirectory.
Figure1.8LinuxFileBrowser
Toaccess a fileor directory in the file system from the command line, youmustcall itupbyitsname,andthereareseveralmethodstodothis.Themostgeneral,andalso themostcumbersome,waytospecifya filename is to listallthenodesinthepathfromtheroottothenodeofthefileordirectoryyouwant.This path, which is specified as a character string, is known as the bsolutepathname,or fullpathname, of the file.After the initial /, all components inapathname are separated by the character /. For example, the file note.txt inFigure1.7hastheabsolutepathname/home/pwang/note.txt
The full pathname is the complete name of a file. As you can imagine,however, this name often can be lengthy. Fortunately, a filename also can bespecified relative to thecurrentworkingdirectory (alsoknownas theworkingdirectory or current directory). Thus, for the file /home/pwang/note.txt, if thecurrent working directory is /home, then the name pwang/note.txt suffices. Arelative pathname gives the path on the file tree leading from the workingdirectorytothedesiredfile.Thethirdandsimplestwaytoaccessafilecanbeusedwhentheworkingdirectoryisthesameasthedirectoryinwhichthefileisstored. In this case, you simplyuse the filename.Thus, aLinux file has threenames
Afullpathname(forexample,/home/pwang/note.txt)Arelativepathname(forexample,pwang/note.txt)A(simple)name(forexample,note.txt)
The ability to use relative pathnames and simple filenames depends on theabilitytochangeyourcurrentworkingdirectory.If,forexample,yourworkingdirectory is /tmpandyouwish toaccess the filenote.txt,youmayspecify theabsolutepathname/home/pwang/note.txtoryoucouldchangeyourworkingdirectorytopwangandsimplyrefertothe
filebyname,note.txt.Whenyoulogin,yourworkingdirectoryisautomaticallysettoyourhomedirectory.Thecommandpwd(printworkingdirectory)displays the absolute pathname of your current working directory. The
command(Bashcommand)cddirectory(changeworkingdirectory)changesyourworkingdirectorytothespecifieddirectory(givenbyasimple
name,anabsolutepathname,orarelativepathname).Twoirregularfilesarekeptineverydirectory,andtheyserveaspointers
File . isapointertothedirectory(directoryselfpointer)inwhichthisfileresides.
File .. isapointertotheparentdirectory(parentdirectory)ofthedirectoryinwhichthisfileresides.Thesepointersprovideastandardabbreviationforthecurrentdirectoryandits
parentdirectory,nomatterwhereyouareinthefiletree.Youalsocanusethesepointersasashorthandwhenyouwanttorefertoadirectorywithouthavingtouse,orevenknow,itsname.Forexample,thecommandcd.
hasnoeffect,andthecommandcd..changestotheparentdirectoryofthecurrentdirectory.Forexample,ifyour
workingdirectory is jdoe, andyouwant to access the file sort.c in thepwangdirectory,youmayuse../pwang/sort.c.Whydoesthiswork?Yourhomedirectoryalreadycomeswithaname,youruserid.However,you
nameyourfilesandsubdirectorieswhenyoucreatethem.Linuxislenientwhenitcomestorestrictionsonfilenames.InLinuxyoumaynameyourfilewithanystring of characters except the character /. But, it is advisable to avoid whitespacecharactersandanyleadinghyphen(-).
HandlingFilesandDirectoriesGenerally,therearetwokindsofregularfiles:textandbinary.ALinuxtextfilestorescharacters inASCIIorUNICODEandmarks theendofa linewith thenewline character. 1 A binary file stores a sequence of bytes. Files may becopied, renamed, moved, and removed; similar operations are provided fordirectories.ThecommandcpwillcopyafileandhastheformcpsourcedestinationThe file source is copied to a file named destination. If the destination file
does not exist, it will be created; if it already exists, its contents will beoverwritten.Themv(move)commandmvoldnamenewnameisusedtochangethefileoldnametonewname.Nocopyingofthefilecontent
is involved. The new namemay be in a different directory—hence the name“move.”Ifnewnamealreadyexists,itsoriginalcontentislost.Once a file or subdirectory has outlived its usefulness, you will want to
removeitfromyourfiles.Linuxprovidesthermcommandforfilesandrmdirfordirectoriesrmfilenamelfilename2rmdirdirectoryname1directoryname2...Theargumentofrmisalistofoneormorefilenamestoberemoved.rmdir
takesasitsargumentalistofoneormoredirectorynames;butnote,rmdironlywilldeleteanemptydirectory.Generally, toremoveadirectory,youmustfirstcleanitoutusingrm.To create a new directory, use the mkdir command, which takes as its
argumentthenameofthedirectorytobecreatedmkdirnameWhenspecifyingafileordirectorynameasanargumentforacommand,you
1.6
mayuseanyoftheformsoutlined.Thatis,youmayuseeitherthefullpathname,therelativepathname,orthesimplenameofafile,whicheveryouprefer.
StandardPersonalDirectoriesItiseasytochangetoahomedirectory,justdocd (goestoyourhomedirectory)cd (goestothehomedirectoryofuserid)In Linux, there are a number of standard folders under each user’s home
directory,usuallyincluding
Desktop—Files in this folder appear as icons on your graphical desktopdisplay, including regular files and application launchers (with filenamesuffix.desktop)Documents—TextualdocumentssuchasPDFfilesandthosecreatedusingtoolssuchasApacheOpenOfficeandLibreOffice.Download—FilesdownloadedfromthenetworkMusic—SoundandmusicfilesPictures—Picturesfromdigitalcameraspublic_html—FilesunderthisfolderaremadeavailabletotheWebviaanHTTPserveronyourLinuxsystemVideos—Filesfromvideocamerasandrecorders
In addition to these, you may consider setting up a bin/ for your ownexecutables, a tmp/ for temporary files, a templates/ for reusable files, ahomework/foryourclasses,andsoon.
PROTECTINGFILES:ACCESSCONTROLEvery file has an owner and a group designation. Linux uses a 9-bit code tocontrol access to each file. These bits, called protection bits, specify accesspermission toa file for threeclassesofusers.Ausermaybeasuperuser, theownerofafile,amemberinthefile’sgroup,ornoneoftheabove.Thereisnorestrictiononsuperuseraccesstofiles.u (Theownerorcreatorofthefile)g (Membersinthefile’sgroup)o (Others)Thefirst threeprotectionbitspertain touaccess, thenext threepertain tog
access, and the final three pertain to o access. The g type of user will be
discussedfurtherin6.Each of the three bits specifying access for a user class has a different
meaning.Possibleaccesspermissionsforafilearer (Readpermission,firstbitset)w (Writepermission,secondbitset)x (Executepermission,thirdbitset)
TheSuperUserRoot refers toa classof superusers towhomno file access restrictionsapply.The root status is gained by logging in under the userid root (or some otherdesignatedrootuserid)orthroughthesucommand.Asuperuserhasreadandwrite permissionon all files in the system regardlessof theprotectionbits. Inaddition, the superuserhasexecutepermissiononall files forwhichanybodyhas execute permission.Typically, only system administrators and a fewotherselected users (“gurus” as they’re sometimes called) have access to the superuserpassword,which,forobviousreasons,isconsideredtopsecret.
ExaminingthePermissionSettingsThenineprotectionbitscanberepresentedbya3-digitoctalnumber,whichisreferredtoastheprotectionmodeofafile.Onlytheownerofafileorasuperusercansetorchangeafile’sprotectionmode;however,anyonecanseeit.Thels-llistingofafiledisplaysthefiletypeandaccesspermissions.Forexample,-rw-rw-rw-1smith127Jan201:24primer-rw-r–r–1smith58Jan243:04updateisoutputfromls-lforthetwofilesprimerandupdate.Theownerofprimeris
smith,followedbythedate(January20)andtime(1:24A.M.)ofthelastchangetothefile.Thenumber127isthenumberofcharacterscontainedinthefile.Thefiletype,accesspermissions,andnumberoflinksprecedethefileowner’suserid(Figure 1.9). The protection setting of the file primer gives read and writepermission tou,g,ando.Thefileupdateallowsreadandwrite tou,butonlyreadtogando.Neitherfilegivesexecutionpermissions.Therearetenpositionsin theprecedingmodedisplay(of ls).The firstpositionspecifies the file type;the next three positions specify the r, w, and x permissions of u; and so on(Figure 1.9). Try viewing the access permissions for some real files on yoursystem.Issuethecommandls-l/bintoseelistingsforfilesinthedirectory/bin.
1.7
Figure1.9FileAttributes
SettingPermissionsA user can specify different kinds of access not just to files, but also todirectories.Auserneedsthexpermissiontoenteradirectory,therpermissiontolistfilenamesinthedirectory,andthewpermissiontocreate/deletefilesinthedirectory.Usually,afileiscreatedwiththedefaultprotection-rw——-soonlythefileownercanread/writethefile.Tochangetheprotectionmode
onafile,usethecommandchmodmodefilenamewheremodecanbeanoctal(base8)number(forexample,644forrw-r–r–)to
set all 9 bits specifically or can specify modifications to the file’s existingpermissions,inwhichcasemodeisgivenintheformwhooppermissionop2permission2...Who represents the user class(es) affected by the change; it may be a
combinationofthelettersu,g,ando,oritmaybetheletteraforallthree.Op(operation)representsthechangetobemade;itcanbe+toaddpermission,-totake away permission, and = to reset permission. Permission represents thetype(s)ofpermissionbeingassignedorremoved;itcanbeanycombinationofthelettersr,w,andx.Forexample,chmodo-wfilenamechmoda+xfilenamechmodu-w+xfilenamechmoda=rwfilenameThe first example denies write permission to others. The second example
makesthefileexecutablebyall.Thethirdexampletakesawaywriteandgrantsexecute permission for the owner. The fourth example gives read and writepermission(butnoexecutepermission)forallclassesofuser(regardlessofwhatpermissionshadbeenassignedbefore).AdetaileddiscussionontheLinuxfilesystemcanbefoundin6.
EXERCISEB
1.8
1. Gotoyourhomedirectoryandlistallfiles(hiddenonesincluded)togetherwiththepermissionsettings.
2. Usingthelscommand,listyourfilesintimeorder(mostrecentfirst).3. List the permission settings of your home directory. Use the chmod
commandtomakesuretoforbidreadandwritefromgando.4. Createa folderpublic_htmldirectlyunderyourhomedirectoryandmake
sureyouopenreadandexecutepermissionsonthisfolder.5. Connect your digital camera to your Linux box and download pictures.
Where are the pictures placed? Can you find them under your Picturesfolder?
Figure1.10Gedit
TEXTEDITINGCreatingandeditingtextfilesisbasictomanytasksonthecomputer.TherearemanytexteditorsforLinux,includinggedit,nano,vim/gvim/vi,andemacs.Theeditorgedit (Figure1.10) comeswith theGNOMEdesktop. It requires
almost no instructions to use. Start it from the Start menu Text Editor or thecommandgeditfile&Aneditorwindowwilldisplay.Thenyoucantypeinput;movethecursorwith
thearrowkeysormouse;selecttextwiththemouse;removetextwiththedeleteorbackspacekey;andfind,cut,copy,andpastetextwiththebuttonsprovidedorwiththeeditmenuoptions.Itisveryintuitive.Thegedit isaGUIapplication.Ifyouwanta terminal-window–basededitor
then consider nano, which is very easy to learn but is less powerful orconvenient thanvimoremacs.Guides tovim andemacscan be found in theappendicesonthecompanionwebsite(mml.sofpower.com).Editingpoweraside,thereissomethingtobesaidaboutaneditorthatiseasy
andintuitiveforsimpletasks,especiallyifyouareabeginner.Inanycase,pickatexteditorandlearnitwell.ItcanmakelifeonLinuxsomucheasier.
Figure1.11Nano
Toinvoketheeditornanoforeditingfile,typefromtheShelllevelnanofile(startsnano)nano-wfile(startsnanowithoutlinewrapping)If the file exists,nano displays it for editing.Otherwise, you are creating a
newfilebythatname.Asyouentertext,nanowillstartanewlineautomaticallywhen the text line gets close to the right edgeof your editorwindow.The -woptionasksfornosuchautomatic linewrapping. It isalsoadvisable toalwaysuse the -zoptionwhichallowsyou tosuspendnanoandgetback to theShelllevel.Once insidenano, youareworking in a text-editingenvironment controlled
bynano, andyou can create text,make changes,move text about, and so on.Common operations are indicated by the nano editor window (Figure 1.11).Hereisalisttogetyoustarted.
Tosavethefile,typectrl+o.Toquitandterminatenano,typectrl+x.Youcanthenelectwhethertosavethebufferorcanceltochangeyourmindaboutquitting.Tomovethecursor,usethearrowkeys.Tocutandpastewholelines,ctrl+kcutsonelineatatimeandctrl+upastesthelinescut.To cut and paste selected text, type ctrl+6, move the cursor to highlightselectedtext,andthenusectrl+kandctrl+u.Tolookfortextintheeditingbuffer,typectrl+w(where),thetexttofind,andenterorreturn.Togethelponoperations,typectrl+g.
1.9
1.10
GETTINGHARD/SAVEDCOPIESTogetaprintedcopyofafileuselpr[options]filenameThiscommandsendsfilenametoaprinter.Yourprintingrequestjoinsaqueue
ofsuchrequeststhatareprocessedinorder.Notethatonlysupportedfiles(plaintext,postscript,orpdf)canbeprintedthisway.Donotsendunsupportedfiles,suchasacompiledprogram(.ofile),oracompressedfiletoaprinterthisway.The print option on the filemenu of application programs, such as yourWebbrowser,PDF(PortableDocumentFormat)reader,ordocumenteditor,canalsobeused.Often,youcanavoidwastingpaperbyusingtheprinttofileoption.Youcan
easilysavetheresultingfile(mostlyinPDF)andsharewithothersbyemailorSFTP(SecureFileTransferProtocol,5,Section5.20).
COMMUNICATINGWITHOTHERSAssoonasyoulogin,youcanpotentiallyinteractwithothers,whethertheyareuserson thesameLinuxcomputeroronotherhosts (computers)connectedbynetworking. Commands such aswho (who is logged in) and finger help toidentifymembersofyourusercommunity;emailapplicationsallowthesendingand receiving of messages and files; and instant messaging (IM) programsenableimmediateinteractionamongon-lineusersanywhereontheInternet.
Who’sWhoontheSystem:fingerIf you are a new user, you may not know many people on the system, andalthoughtheinformationprovidedbywhoandwisuseful,youdon’tknowwhothese users are. You only know their userids, which may not resemble theiractualnamesevenfaintly.Thecommandfingerwillgiveyousuchdataasfullname, office, address, and phone number for each user; this is sometimesreferredtoasthefingerdatabase,becausefingerisusedtolookupinformationfromthisdatabase.Thegeneralformisfingername...This commandwill display all entries in the finger database that contain a
useridandfirst,middle,orlastnamematchinganyofthegivenarguments.Forexample, either finger smith or finger clydewill result in the entry shown inFigure1.12.
Figure1.12ASamplefingerOutput
This multiline output includes a project field, which is the first line in the.projectfileintheuser’shomedirectory.Theplanfielddisplaystheuser’s.planfile. These two files supply additional information about a user for the fingerdatabase.Thenoplanlineintheexampleindicatesthatcsmithhasno.planfile.Onsomesystems,fingergivesonlyaveryshortsummaryunlessthe-loptionisgiven.Usedwithanargument,fingerwillaccessinformationonanyuserknownto
thesystem,whether thatuser is loggedonornot. If finger isusedwithoutanargument,anabbreviatedfingerentryisdisplayedforeachusercurrentlyloggedin.Thefcommandissometimesavailableasashorthandforfinger.
Figure1.13ThunderbirdEmailProgram
EmailElectronicmailgivesyoutheability tosendandreceivemessages instantly.Amessage sent via email is delivered immediately and held in a user-specificmailboxforeachrecipient.YoucansendemailtousersonthesamecomputeroronothercomputersontheInternet.Many utilities are available on Linux for email, including the popular
thunderbird (Figure 1.13), Evolution, and Kmail. These full-GUI email
1.11
programs are nice when you are at a Linux console. Command-line emailprogramssuchaselmandmuttareusefulfromaterminalwindow.Let’sexplainhowtousemutt.muttuserid@host-address(Internetmail)muttuserid(localmail)Thenjustfollowinstructionsandenterthemessagesubjectandtype/edityour
message.Mutt lets you edit yourmessage using your favorite text editor. Formuttandmanyotherapplicationsthatneedtextediting,setyourfavoriteeditorbygivingavaluetotheenvironmentvariableEDITOR(3,Section3.10).EDITOR=vim+orEDITOR=emacsexportEDITORWhenyoufinisheditingyourmessage,itwillbesentoutautomatically.mutt–help|moredisplaysmore informationonmutt usage.Here, the output is piped via the
+|+notation(Chapter3,Section3.5) to themorepaginatorwhichdisplays theinformationonepageatatime.To receive email (to checkyourmailbox), typemuttwith no argument and
follow instructions. Try to send yourself some email to get familiar with theusage.Emailisfast,butnotinstantorinteractive.OnLinux,youcandoIM.Skype
andGoogleHangoutareverypopular.BothworkwellonallkindsofsystemsincludingLinux.ForIMonLinux,youcanalsochoosepidginorempathy.
BROWSINGTHEWEBOneofthemostimportanttoolsonanycomputersystemistheWebbrowser.OnLinux you have a choice of differentWeb browsers. Popular choices includeGoogle Chrome and Firefox. The command firefox comes with most Linuxdistributions.InFigure1.14weseethehomepageofourtextbookwebsite.
1.12
1.13
Figure1.14AccessTextbookSite
You can enter aURL (UniformResource Locator) in the browser Locationwindow to visit a specific Web address. A local file URL, taking the formfile://full_pathnamecanbeusedtovisityourlocalfilesystem.Normally,Webbrowsersarefull-GUIprogramsusedinteractively,butLinux
also offers a command-lineWeb browser called lynx, a text-only browser thatdoesnotdisplayimages.However,lynxcanbeusedinsideaterminalwindowtointeractivelybrowsetheWebusingthearrowkeysortodownloadfilesfromtheWeb.
EXERCISEC1. Trythemuttemailprogram.Useit tosendanemailandattachafile.Do
thesameusingThunderbird.2. Createatextfileusingnano.3. Try the vim or emacs editor. Read the related appendix on the book’s
website.4. If possible, set up Thunderbird as your email program and Firefox or
ChromeasyourWebbrowser.5. DownloadafileusinglynxfromtheWeb.
CREATINGANDRUNNINGYOUROWNPROGRAM
Skip this section if you have no immediate interest inwriting a program in ageneral-purposeprogramminglanguagesuchasC.Youcanalwaysreturntothissection later.TheLinuxsystemoffersmany languages:C,C++,Java,Fortran
77/95, Python, Ruby, and Perl, just to name a few. You can also write Shellscripts(5)toautomatefrequentlyusedoperations.FileNameSuffixesLinuxfollowsasetofconventionsfornamingdifferentkindsoffiles.Table
1.1illustratessomecommonlyusedfilenamesuffixes.Asourcecodefilecannotbeexecuteddirectly.Theprogramusuallymustbecompiledintomachinecodebeforeexecutioncan takeplace.Analternative tocompilation is to interpret ahigh-levellanguageprogramdirectlyusinganinterpreter.
Table1.1PersonIn-ChargeAttheOrganizationRespondents
We shall follow an example of creating and running a simple C-languageprogram. Use your favorite text editor and create a C source file try.c (Ex:ex01/try.c)withthecode
#include>stdio.h<intmain(){printf("runningmyC
program\n");return0;}
This is a simple source program inC that displays the line “runningmyCprogram.”ThenotationnstandsfortheNEWLINEcharacter.
CompilingBefore try.c can be run, it must be compiled. Compiling is the process oftranslatingaprogramwritteninahigh-levellanguagesuchasCorPascalintoalow-levellanguageforexecutiononaparticularcomputer.Onmanysystemsthecompilerwilloutputafileofobjectcode,whichmustbeinked(combinedwithroutines suppliedby the system library)bya separateprogramcalleda linker.Once linkage is complete, the file is considered executable and is ready to beloadedintomemoryandexecuted.Linux-based compilers will handle both the compiling and the linking of a
program unless you specifically tell them not to, and their output will be anexecutablefile.Availablecompilersincludeproduceobjectfiles(.o).gcc GNUCcompilerg++ GNUC++compiler
javac Javacompilergfortran GNUFortran77/95compilerLet’strycompilingthesampleprograminthefiletry.cgcctry.cThiswillproduceanexecutablefile+a.out+whichcanbeinvokedsimplyby
typingitasacommand(Chapter3Section3.5).a.outNotethatinLinuxthecommandtorunaprogramissimplythepathnameof
theexecutablefile.Thus,./a.out(runstheexecutable)At some point, you probably will want to name your executable file
something other than a.out, especially since a.outwill be overwritten the nexttimeyouinvokeacompilerinthisworkingdirectory.Wealreadyknowthatthemvcommandcanbeusedtorenameafile,butthe-ooptiontogccorg++canbeusedtoprovideanametouseinsteadofthedefaulta.out.Forexample,gcc-omytrytry.cproducestheexecutable+mytry+.Nomatterwhichlanguageprogramyourun,youprobablywillwantarecord
ofyourresults(tosubmitasahomework,forexample).Onewaytodothisistouseoutputredirection.Forexample,./a.out>results+The symbol > tells the Shell to redirect output of a.out from the terminal
screenintoanewfilenamedresults.Thus,youwillseenooutputintheterminalwindowaftertheprecedingcommand.Instead,you’llfindtheoutputinthefileresult.AfullaccountofShellI/Oredirectioncanbefoundin3,Section3.5.Anotherwaytodothisistousethescriptcommandscriptrecord_fileto recordyour terminal session intoa record_file.Whilescript is active, all
I/Otoandfromyourterminalwindowiswrittentothefileyouspecified(ortoafilenamedtypescriptifyouenteredscriptwithoutanargument).Therecordingcontinuesuntilyoutypectrl+datthebeginningofacommandline.Thefilethencanbeviewed laterwitha texteditororemailed to someone.Forexample, torunaCprogramwithscript,thefollowingsequenceofcommandsmaybeusedscriptdisplay_record?ccmyprogram.c?a.outctrl+dThescriptcommandrequests thatall subsequent I/Oberecorded in thefile
1.14
1.15
displayrecord.Thectrl+donthelastlinestopstherecordingandgetsyououtofscriptandbacktotheShelllevel.An advantage of using script over simply redirecting output is that the file
producedbyscriptwillcontainbothinputtoandoutputfromtheprogram.Thefilecreatedbyredirectingoutputwillcontainonlyoutput.
EXERCISED1. Typeinasimpleprogram(inyourfavoriteprogramminglanguage)totype
outthemessage:Linuxisniceonceyouknowit.Compileitandrunit.Usescripttogetadisplayrecordoftheprogram’sexecution.
2. Usemoreornanotoviewthefileproducedbyscriptandthensendthefiletosomeonebyemail.
CONSULTINGLINUXDOCUMENTATIONThe command yelp (regular command) provides a GUI for browsing Linuxdocumentationandwhengivenwithnoarguments launches theGNOMEHelputilitywhichisagoodguideforbeginners(Figure1.15).
Figure1.15GNOMEHelp
The Linux Documentation Project website http://tldp.org providescomprehensivedocumentationforalmostallaspectsofLinux.You’llfindFAQs,topic-specific step-by-step instructions called HOWTOs, guides, and manualpagesforcommands.Asearchfeaturemakesiteasytofindwhatyouneed.You can also find documentation provided by your own Linux. The Help
1.16
1.17
1.18
menuonthetoolbarofGUIapplications,suchastheFileBrowser,theTerminalEmulationWindow,andPidgin,providestool-specificdocumentation.Command-line programs often provide brief and concise usage information
withthe–helpcommandoption.Forexample,tryls–helpTheman command displays manual pages set in standard UNIX-defined
formats. See Section 3, Section 3.11 for more information on Linux manualpages.
EXERCISEE1. Howdoyouaskacommandtohelpyouuseit?2. Accessthemanpageforlsandreaditfrombeginningtoend.3. Explain how to display the introduction section to the user commands
chapteroftheLinuxmanpages.
ROUNDINGUPUSEFULCOMMANDSIn this chapter, we have run into only a small number of themany availableLinux commands. The richness and variety of Linux commands are majorstrengths of the system. It is doubtful, however, that many users know all ofthem; you learn the commands that accomplish what you need. This sectioncollectsthecommandsthatshouldbeinanewuser’sbasicrepertoire.InLinux,bothuppercaseandlowercasecharactersareused,andtheyarenot
interchangeable.Allsystem-definedLinuxcommandsareenteredinlowercase.Also, there are two kinds of commands: (1) built-in Shell commands that aresubroutines in the Shell and (2) regular commands that are initiated as jobscontrolledbytheShell.Theimportanceofthisdistinctionwillbecomeclear.Inthefollowinglistingofcommands,user-suppliedargumentsareshowninitalics.Optional arguments are enclosed in square brackets [ ]. Possibly repeatedarguments are indicated by ellipses (...). These conventions will be followedthroughout the book. Only the most common usages of these commands aregiven.The informationhere is intended togetyou startedand isbynomeanscomplete.Detailsareprovidedinlaterchapters,andyoushouldconsulttheon-linemanualforfulldescriptionsofeachofthesecommands.
SUMMARY
1
Linux provides both full-GUI applications and command-line programs. TheGUIisvisualandmoreintuitivetouse,butmanybasicLinuxutilitiesaremoreconvenientonthecommandline.AShell(typicallyBash)runninginaterminalwindowprovidesaCLI to enter andexecute commands.Learning touseboththeGUIandtheCLIeffectivelywillmakelifemucheasieronLinux.TheCLIisespeciallyimportantforremoteaccessofLinuxusingSSH.Thedesktopmainmenuleadstomanyusefuloperations.2presentstheLinux
desktop.AShellexecutescommandsyouinputfromthekeyboardanddisplaysresults
inyourterminalwindow.Typingerrorscanbecorrectedthroughinputediting.Both the systemand theusers storedata in filesmanagedby theLinux file
system, which has a tree structure. Each file can be referred to by a full(absolute)pathname,arelativepathname,orasimplefilename.Eachuserhasahome directory in which personal files and directories can be kept. Files anddirectoriescanbecreated,moved,copied,listed,anddestroyed.Read,write,andexecute permissions are used to control file access by u (owner), g (groupmember),ando(others).Theownercansetandchangetheaccesspermissionsofafile.Youcancommunicatedirectlywithotherusersbyusingtalktochatdirectly,
byemail,andbyinstantmessaging(Skype,GoogleHangout,Pidgin).Linuxoffersseveral texteditors.Thefull-GUIgedit isagoodchoice.Fora
terminal window, the simple and easy nano is good for beginners and lightediting tasks. Serious editing is more efficient with an editor such as vim.Editing,compiling,andrunningofasimpleCprogramhavebeenpresented.Linux offers many facilities and a complete set of manuals. The man
command can be used to consult the manual pages, and the LinuxDocumentation Project website provides a variety of comprehensive Linuxdocumentations.RefertoSection1.17foralistofusefulcommandsforLinuxbeginners.
OnWindowsorDOSsystems,endoflineisindicatedbyreturnfollowedbynewline.
2.1
Chapter2
TheDesktopEnvironment
Itisimportantforanoperatingsystemtoprovideaconvenientinterfaceforuserstoperformandmanagetasks.Inthebeginning,Linux/UNIXsystemswereusedexclusively from the keyboard via the command-line interface (CLI) (3). Byaddinggraphicaluserinterfaces(GUIs),LinuxsystemsaremademoreintuitivetouseaswellaseasiertolearnforpeoplefamiliarwithMicrosoftWindows®orMacOS®systems.Agraphicaluserinterface(GUI)employsapixel-basedgraphicaldisplayand
addsapointingdevice,suchasamouse, touchpad,or touchscreen, to interactwiththeuser.ThefirsteffectiveGUIonanaffordablepersonalcomputer1wasintroducedbytheAppleLisaintheearly1980s.ADesktopEnvironment (or simplyDesktop) supports theGUIbyproviding
workspaces,windows,panels, icons, andmenus, aswell as clicking, scrolling,copy-and-paste, and drag-and-drop operations. Today, it is hard to imaginecomputerusersdoingwithoutadesktopGUI.Nevertheless,whenyoubecomemore of a Linux expert, you may find the CLI more convenient in manysituations.TherightapproachistocombineGUIandCLItogetthebestofbothworlds.
DESKTOPOVERVIEWAfter login at aworkstation, the first thingyou see is thedesktop fromwhichyou can launch applications, manage files, control your Linux system, andperform many other tasks. A desktop makes operating your computer moreintuitive through a desktop metaphor by simulating physical objects.Overlappingwindowscanbemovedandshuffledlikepiecesofpaper.Buttons(icons)canbepushed(clicked)toinitiateactions.Unlike MS Windows® or the Mac OS®, Linux offers a good number of
2.2
alternativedesktopswithahighdegreeofusercustomization.PopulardesktopsincludeKDEPlasma,GNOME3,Cinnamon,MATE,Unity,Xfce,andothers.Adesktop is often built on top of awindowing system, such as theX WindowsSystemorthenewerWayland.InadditiontoofferingacompletedesktopGUI,aLinuxdistributionoftenalsosuppliesasetofessentialapplicationsincludingaclock/calendar, sound volume control, email client, Web/file browser, instantmessenger, image displayer, media player, address book, PDF reader, photomanager,preference/configurationeditor,andmore.Agoodunderstandingofthedesktopandhowtouseiteffectivelycanmake
life onLinuxmuch easier. It is perhaps safe to assume that you already havegood working experience with MSWindows® or Mac OS®. Our discussionhereoftenfeaturestheGNOMEDesktop,oneofthemostpopulardesktopsforLinuxdistributions.OtherLinuxDesktopsfunctioninsimilarways.
Figure2.1LinuxMintMenuandPanel
DESKTOPCOMPONENTSAdesktopusuallydisplaysthefollowingcomponents:
RootWindow—After login, the entire graphical display screen is coveredby the rootwindow of yourDesktop. It is the spacewhere all otherGUIobjects (desktop components and application windows) are placed andmanipulated.DesktopPanel—ADesktopnormallydisplaysaPanelintheformofataskbaralongthetoporthebottomedgeoftherootwindow.ThePaneldisplaysimportant controls and the start icon which leads to the start menu foralmost all tasks, including checking system status and customization. APanelmayalsodisplayapplicationsthatarestillrunningsoyoucaneasilygo back to them. A Panel can display launchers to invoke specificapplications aswell aswidgets (small applications) such as a clockor anaudiovolumecontrol.SeeFigureforaMintPanelandstartmenu.YoumayalsobeabletoaddobjectstothePanelbyrightclickinganyemptyspaceinit.StartMenu—TheStartmenuisexposedbyclickingthestarticon(oftenintheformofalogoforGNOME,RedHat,Fedora,orUbuntudependingonyourLinuxversion)placedattheendofthePanel.ThekeyboardshortcutfortheStartmenuisusuallyALT+F1.FromtheStartmenu,youcanperformalmost all operations and can access files, the network, installedapplications, commands, andpreferenceoptions.Theremaybe additionalmenusonthePanel.NotificationArea—UsuallypartofthePanel,thenotificationareadisplaysstatusiconsforcertainapplications;forexample,aspeakervolumeicon,anetwork status icon, a system update available icon, an incoming emailicon, and so on. Mouse-over or click a notification icon to see moreinformation and available actions. The notification area is usually at therightendofthePanel.Application Program Windows—Each GUI application works within itsown window. Such windows are displayed in the root window as childwindows.Multipleapplicationwindowscanoverlap.Youcanswitchfromone app to another by changing the input focus (Section 2.4) from onewindow to another, as well as move, resize, maximize, minimize,unmaximize,orcloseeachwindowasyoulike.WindowsList—Alist of buttonsdisplayedon thePanel (Figure2.1)eachrepresenting an application wiondow in the root window. Clicking on awindow button minimizes or restores the window. On newer GNOMEDesktopspressingtheSUPERkey2revealstheactivityoverviewscreenwithlargericonsforallactivewindowsforeasyswitching(Section2.3).Moving
2.3
themousequickly to theupper-left cornerof theDesktopcanalso revealtheactivityoverviewscreen.Workspace Switcher—A workspace switcher enables you to work withmultiple workspaces and to switch from one workspace to another. Aworkspaceisessentiallyaduplicaterootwindowtoprovidemorespaceforplacingadditionalapplicationwindows.Withseveralworkspaces,youcanspreadoutyourapplicationwindowsforeasieruse.AworkspaceswitchercanbefoundonthePanelorbygoingtotheactivityoverviewscreen.Yourrootwindowcandisplayoneworkspaceatatime.Theworkspacevisibleisyour current workspace. Click on the switcher to change the currentworkspace.Desktop Objects—In the vast space left open by the Panel on the rootwindow, you often can place objects (icons) such as files, folders, andapplicationlaunchersforeasyaccess.Desktopobjectsusuallyinclude(1)aComputerorPlacesicontoaccessfilesandremovablemedia,(2)aHomeiconforyourhomefolder,and(3)aTrashiconforrecoveringdeletedfiles.Thesedesktopiconsrepresentfiles in theuser’s$Desktopfolder. Inotherwords,placingafileinyourDesktopfoldercancreateacorrespondingiconontheDesktop.
THEGNOME3DESKTOPRadicallydifferentfrompreviousdesigns,theGNOME3(version3.22released09/2016)DesktoptookaholisticapproachinGUIdesign.Asaresult,GNOME3works somewhat differently frommany traditional desktop environments. ItintroducedthegraphicalGNOMEShellastheoneplaceforuserstocontroltheGNOME 3 desktop, providing easy and efficient operations such as appfinding/launching,window/workspaceswitching,andfavoriteappaccessing.
Figure2.2GNOME3PanelwithShellExtensions
Underthetoppanel(Figure2.2),theGNOMEShellusestwofull-sizedisplayareas:
App Windows Display—This is the root window which now showsexclusivelyoverlapingwindowsofcurrentlyactiveapps.Beforeanyappislaunched,bydefault,you’llseeonlyemptyspacebelowthepanel.
2.4
Activities Overview—Showing a left-side Dash for launching favoriteapps, an app search box, iconified active app windows, and a right-sideworkspaceswitcher(Figure2.3).
Figure2.3GNOME3ActivityOverview
Movingthemousequicklytotheupper-leftcornerofthescreen(orpressingtheSUPERorWINDOWSkey)switchesbetweentheabovetwodisplays.TheGNOMEShellcanbeconfiguredandcustomizedinmanyways.Youcan
use the gnome-tweak-tool to configure settings and add Shell Extensions(Figure2.2). For example, you can enableDesktop icons to show files in theyourDesktopfolder.InGNOME3 theversatileShellExtensionsmechanism replaces applets for
addingwidgets andmenuson thePanel.For example, youhave extensions toprovideanApplicationsmenu,aPlacesmenu,aWeatherindicatorandsoontothetoppanel.A preferences widget is shown on the Panel for managing extensions. The
commandgnome-shell-extension-prefsservesthesamepurpose.Distributions that come with integrated GNOME 3 include openSUSE,
Fedora,UbuntuGnome,Debian,andLinuxMintCinnamon.
UNDERSTANDINGGUIA graphical user interface allows user control via mouse movements, buttonpressing, touchpad (touch screen) gestures, as well as key strokes from thekeyboard.Theseareknownasuserinputevents.TheGUIworksbymonitoringsuchevents,relatingthemtothegraphicalelementsdisplayedonthescreen,andprocessingeacheventquicklytoachievethedesiredeffects.
2.5
When launched, a GUI application, such as aWeb or file browser, usuallymakes a nice graphical display in a new window then does nothing exceptwaitingforuserinput.Whensuchaninputeventoccurs,theGUIappreactstotheevent,handles itquickly,andgoesback todoingnothing—beingreadyforthenextevent.Whenaninputeventtakesplace,theGUIenvironmentcreatesaneventobject
(a data structure) that contains the event attributes (for example, event type,mouse position, button id, or keycode) and delivers the event object to theprogramthatshouldreceiveandhandletheevent.Whenworkingwithmultiplewindows, one of them always has input focus
andisknownasthecurrentwindow.Allinputeventsaretobedeliveredtothecurrent window and the app running within it. The desktop allows a user toeasilychangethecurrentwindowbyclickingorselectingit.Acertainpartwithin thecurrentwindowmayactuallygain input focus.For
example,clickingononeofthreedisplayedbuttonssendstheeventtothebuttonbeing clicked and not the other buttons. In filling a form, key strokes aredeliveredtotheinputfield(suchasname,email,oraddress)thathasinputfocus.Usually clicking themousemoves input focus to the locationof the click andpressingtheTABkeymovesinputfocustothenextelementinsideawindow.Whenithappensthatthereisnoresponsetoyourinput,itisusuallybecause
yourinputissenttoaplacethatdoesnotunderstandit.Inotherwords,theinputfocusisatthewrongplace.Correctyourinputfocustofixtheproblem.
WORKINGWITHTHEDESKTOPOneof thebestways toget familiarwithadesktop is to learnhowtoachievespecifictaskswithit.HerearesometasksimportantforanewLinuxuser.
SessionControlandSystemSettingsYoucancontrolyourloginsessionbyclickingonthepowerswitchicononthePanel (usually toward the right side) allowing you to manage networkconnectedness,currentuser,soundvolume, lockscreen,session logout, restart,andsystemshutdown.Figure2.4showsLinuxMintsessioncontrol.
Figure2.4SystemControl
You can also controlmany other system settings such asmouse, keyboard,display, and devices (printers, scanners, webcam ...). Click on the time/datedisplaytosetit.Figure2.5showsasystemsettingsdisplay.
Figure2.5SystemSettings
LaunchingApplicationPrograms
Perhaps thesinglemost importantpurposeofadesktop is tohelpyourunandmanage application programs. Linux offers a large number of applications.You’ll findmany of them from the Startmenu organized into several generalgroupssuchasaccessories,office,graphics,Internet,programming,multimedia,games and so on. In addition, there aremany commands that can be invokedfromthecommandlineinaterminalwindow.Infact,youhavemultiplewaystostartapplications:
Using an app search to find and launch available apps. If an app is notinstalledonyoursystem,downloadandinstallitfromyourLinuxappstore.Singleclickingalaunchericon(Youmayaddalauncherforanyapplicationyoulike.)Single or double clicking an executable object/file, depending on yourpreferencesetting,displayedonthedesktoporafilebrowser.SelectinganapplicationfromtheStartmenuorasubmenuthereof.Issuing a command in a terminal window or from the command pop-updialogue(tryALT-F2).
Ifyouissueacommandforanappnotyetinstalled,yourLinuxmayevenoffertoautomaticallydownloadandinstallitforyou.Toinitiateagraphicalapplication,saygedit,fromthecommandlinewithout
the Shell waiting for it to finish or the Shell job control mechanism gettinginvolved,use(geditfilename&)This way, a subshell puts the graphical application in the background,
disassociates it from the terminalwindow, andgives youbackyour commandprompt.
ManagingFilesManagingfilesisimportantforanyuser.ClickonPlaces->Hometobrowsetheuser’s files/folders. Click on Places->Computer to browse files/folders for thewholesystem,includinginserteddrives(Figure).Eachuser’shomedirectoryoftencontainsthesestandardfolders:Documents,
Downloads,Music,Pictures,Videos,andTrash. IfyourLinuxserves theWeb,eachusermayalsohaveafolderpublic_htmlwhereper-userWebpagesreside.OtherfilesandfolderscanbesetupasyouworkonyourLinuxcomputer.FilesandfolderscanbemovedtoTrashandtheneventuallydiscardedwhen
youemptytrash(fromFilemenuofTrash).ItisalsopossibletoretrieveitemsfromTrash.TheTrashfolderiskeptwithinauser’shomedirectory,sometimes
underahiddenfolder(.local/share/Trashforexample).Right click a desktop object to rename it, move it to Trash, set its access
permissions,orotherwisemanageit.
Figure2.6APlacesMenu
MultipleWorkspacesThe desktop workspace can get crowded with multiple application windowsquicklywhenyouworkonyourLinux.Fortunately,youcansetupmore thanoneworkspaceandswitch fromone toanother toworkondifferent tasks.Forexample,youmayhaveoneworkspaceforWebbrowsing,anotherforemailandinstantmessaging,yetanotherfortexteditingandwordprocessing,andsoon.ForGNOME3basedsystems,aworkspaceswitchercanbefoundontheright
side of the activity overview. Use it to add, switch, and delete workspaces.Keyboardshortcuts,usuallyALT-UPandALT-DOWN,canbeusedtoquicklyswitchworkspaces.Asyougotoadifferentworkspace,theactivitydisplaychangestoshowthe
windowsinthenewcurrentworkspace.Inotherdesktopsystems,youmayaddaWorkspaceSwitchertoyourPanel.
Thiswill give you fourworkspaces by default.Right click on the switcher tocustomizeittoyourliking.
DesktopAppearanceYoucancustomizethelookandfeelofyourdesktopdisplay.Rightclickonanempty spot on your desktop working area and choose change desktop
2.6
backgroundtofindabackgroundyoulike.FortheGNOMEShell,youcanusetheGNOMETweaktool(gnome-tweak-
tool)tocustomizemanyaspectsoftheGNOMEDesktopappearanceincludingpickingfrommanywell-designedthemes.
WINDOWSTheXWindowSystemIn Linux/Unix, graphical applications use the X Window System (originallydeveloped at the Massachusetts Institute of Technology) to create GUIs.Windowing software suchasXenablespixel-basedgraphicaldisplays and theuse ofwindows,menus, and themouse to interactwith application programs.TheXWindowSystemworksasaGUIserver(theXserver)thatenablesclientprograms(Xclients)tointeractwithusersthroughGUIs.XclientscanconnecttothelocalXserverrunningonthesamehostoraremoteXserverrunningonahostacrossthenetwork.Furthermore,theXservercanhandlemultiplestations,eachpotentiallywithmultipledisplays.(Twoorthree20-inchLCDdisplaysforyourLinuxdesktopcomputer,anyone?)ForLinux,theXserverisbasicandisstartedwithinthebootsequence.Ifthe
X server is not running, no GUI programs will work. Figure shows the XWindowSystemarchitecture.
Figure2.7XWindowSystemArchitecture
When anX client starts, it needs to connect to anX server running on thelocalhostoraremotecomputer.TheXserverisalwaysspecifiedbythedisplayoption.Forexample,xclock-displayhostname:s.m
saysthexclockdisplaywillberenderedonhostname,stationnumbers,andmonitornumberm.Astationonacomputerconsistsofakeyboard,apointingdevice (mouse), andoneormoregraphicaldisplaymonitors.Acomputermayhaveoneormorestations,eachwithoneormoremonitors.IftheXserverislocal(onthesamehostastheclient),thehostnamepartcan
beomitted.Forasingle-displaycomputer,themonitor-stationdesignationwouldbe:0.0andcanusuallybeomittedalso.TheShell environmentvariableDISPLAYspecifies thedefaultX server for
anyclientprogramstartedwithoutanexplicit-displayoption.Trythecommandecho$DISPLAYtoseethevalue.Mostlikely,itwillbethestring:0.0.X11 has been in use for many years and experience told us that it can be
improvedinmanyrespects.Wayland,oneofthenewwindowingsystemsbeingdeveloped, aims to be compact, efficient, and better connected to hardware.Fedora is a Linux distribution that is moving to support Wayland as analternative toX11.However,at thispoint,X11 is still thestandardwindowingsystemformostLinuxdistributions.
WindowManagerYou control windows displayed on your desktop through awindowmanager.Thewindowmanageristhepieceofsoftwarethatcontrolsthedisplaywindowsin theXWindowSystem environment.The opening, closing, size, placement,borders,anddecorationsofanywindowaremanagedbythewindowmanager.TheXWindowSystemcallsforthewindowmanagertobeaclientratherthanabuilt-inprogram.InthiswayXcanworkwithmanydifferentkindsofwindowmanagers.Oneoftheoriginalwindowmanagersistwm.Your desktop environment works with the window manager to display
applicationwindowsonyourscreen(Figure).MutteristhedefaultwindowmanagerforGNOME3replacingMetacity.The
GNOME Shell (Section 1.3), implemented as a plugin forMutter, is speciallydesigned to work on large-screen workstations. Examples of the many otherwindow managers include i3, XMonad, AwesomeWM , Enlightenment andFluxbox.Youcaneasilyminimize,maximize/unmaximize,orcloseawindowusingthe
usualbuttonsonthetitlebar.Ifyoucloseallthewindowsofanapplication,thenthe application will be closed. Right clicking a window’s title bar usuallydisplaystheWindowMenufromwhichyoucancontrolthewindow.Awindowcanbemovedbyholdingontothetitlebaranddraggingitandcanberesizedby
2.7
holdingontoanddraggingasideoracorneroftheframe.Awindowisintheworkspacewhereitiscreatedbutcanbemovedtoanother
workspaceanytimeyouwish.Windowsinaworkspacemayoverlap.Clickingonawindoworitsactivitiesimageshiftsinputfocustothatwindowandbringsittothetop.Inadditiontoregularwindows,anapplicationwillsometimesdisplayadialog
window. Such popup windows are used to display alerts, to solicit userconfirmation,or toobtainuser input.Forexample,anapplicationmayaskifauser really wants to quit before closing. A dialog window can be modal ortransient.Amodaldialogwillnotallowyoutointeractwiththemainapplicationwindowuntilthedialogisclosed.Thus,youmustdealwithamodaldialogandcloseitbeforeyoucanresumeworkingwiththeapplication.
WindowInformationUnder XWindows, all windows form a containment hierarchy with the rootwindow sitting at the top. Each window has a unique window ID. The rootwindow’sIDisroot.ThecommandxwininfodisplaysthewindowIDandmanyotheritemsofinformationaboutanywindow.Runxwininfofirst,thenclickonanytargetwindowtogettheinformationdisplay.HereisasampleontheFirefoxwindow(Ex:ex02/xwininfo).
xwininfo:Windowid:0x1800010Absoluteupper-leftX:470Absolute
upper-leftY:64Relativeupper-leftX:10Relativeupper-leftY:
45Width:1448Height:984Depth:24Visual:0x304VisualClass:
TrueColorBorderwidth:0Class:InputOutput...Corners:+470+64
-2+64-2-32+470-32-geometry1448x984--8+19
NotethatthewindowIDisahexnumber0x1800010.Nowlet’stakealookatsomeusefulGUIapplicationsonLinux.
THEFILEBROWSERAnimportantaspectofanyoperatingsystemistheabilitytostoreandmanagefiles.TheLinuxfilesystemhasahierarchicalstructure.Eitheraregularfile,adirectory(folder),orahardwaredevice(special file) isconsidereda fileintheLinux file system. A directory (folder) is a file that records the names andattributesoffilesitcontains.Anyofthecontainedfilescan,inturn,befoldersaswell.TheLinuxfilesystemisallencompassingbecausealmosteverythinginLinux
hasarepresentationinthefilesystem.Chapter6discussestheLinuxfilesystem
indetail.Afilebrowser,orfilemanager,providesaGUItomanagefiles.Ithelpsyou
navigatetheLinuxfilesystemandmanageyourfilesvisually.Nautilus(Figure)isthefilebrowserthatcomeswithGNOMEShell.
Figure2.8NautilusFileManager
UseacommandsuchasnautilusfolderLocationto browser any desired folder. Special URIs (resource identifiers) such as
thesealsowork:nautiluscomputer:///nautilustrash:///nautilusnetwork:///nautilussmb:///OtherfilemanagersincludeKonqueror,Dolphin,GNUMidnightCommander,
andPCManFM. A file browser enables you to interactively navigate the filesystem,manage files and folders, access special places onyour computer, useopticaldrives,andreachavailablenetworkingplaces.
NavigatingtheFileTreeYoubrowsethefilesystembyfollowingthefolder-subfolderpathuntilyoufindyourtargetfile(s).Thus,atanygiventimeyouarelocatedatacurrentdirectory.Thecontentsofthecurrentdirectorycanbedisplayedinalistvieworaniconview,andyoucanswitchbetweenthemeasily.Theiconviewisrecommendedforeasiervisualinteractions.Double click a folder to open it and see the files in it, and click on the up
button tomoveup to theparent folderof thecurrent folder.TheLocationboxshows thepathname leading from the root directory / to the current directory.Normally,anyfilewhosenamebeginswithaperiod(.)ishidden.SelectView->ShowHiddenFilestorevealthem.
OpeningaFileorFolderDouble click a folder to open it and display its contents. Double click anordinaryfiletoopenitwiththedefaultapplication,forexample,PDFfileswithevince , .txt fileswithgedit, or .html files with your preferredWeb browser.Rightclickanordinaryfiletoopenitwithanyapplicationyouchooseandthatapplicationwillberememberedasapossibilitytorunthatparticulartypeoffile.Byrightclicking,youcanalsoelecttoremoveafile/foldertoTrash,toopenitwith a desired application, or to change its properties, including accesspermissions(seeSection1.7.5).
FindingFilesClickontheSearchbuttontogetaSearchbox.Typeastringofcharactersinthename or contents of the file(s) youwish to find and press ENTER. The searchresultswillbedisplayed.Iftoomanyfilesarefound,youcannarrowyoursearchbyfiletypeortimeofcreation(Figure).
Figure2.9NautilusFileSearch
ManagingFilesandFoldersFromthefiledisplayarea,youselectoneormorefilestomanage.Click(orCTRLclick)afileorfoldertoselectit.UseCTRLclicktoselectadditionalitems.Theselecteditemsarehighlighted.Iniconviewyoumaydragarectanglearoundagroupoficonstoselectthem.Makinganewselectioncancelsthepreviousselection.IfyouCTRLclickona
highlighteditemorclickonanemptyspotinthefiledisplayarea,theselectionisalsocanceled.Aftermakingaselection,youcanperformoperationsontheselectedfiles.
Draganddrop the selected items into a folder (or thedesktopwhich is afolderanyway)tomovethemthere.Grab the selection, then hold down ALT, and drag to a new folder andreleasethemouse.Thenselecttheoperationyouwish,includingmovehere,copyhere, or linkhere.A link is a shortcut or a pointer to an actual file(Section).Rightclickonyourselectiontoseetheavailableoperations.Theseincludemovingtotrash,openwith,copying,send to,andchangingfileproperties(name, permissions, list of applications to open files of this type, and soon).
*********************
AccessControlforFilesandFoldersOnLinuxafileisownedbytheuserwhocreatedit.Thefileownercansetthepermissionsforthefiletocontrolwhichuserswillhavewhataccessestoit.Users are alsoorganized intogroups.Auser canbelong tomultiplegroups.
Thefile/etc/groupslistsgroupsandgroupmembers.Thefileownercanalsosetthegroupattributetoanygrouptowhichtheownerbelongs.As a file owner, you can set the read (r), write (w), and execute (x)
permissionsforthreedisjointsetofusers: thefileowner(u),otherusersinthefile group (g), and all others (o). Each permission is independent of otherpermissions.Forexample,afilemayhavenopermissionsforu,butrandwforo.Itmaynotmakemuchpracticalsense,butitdoesdrivehomethepointoftheu,g,andopermissionsbeingindependent.Themeaningofeachpermissionisasfollows:
r—Permissiontoopenafileortoinspectcontentsofafolderw—Permissiontomodifyafileortocreateordeletefilesinafolderx—Permissiontorunafileasaprogramortoenterafoldertoreachfilescontainedinit
Youset/changepermissionsandmanageapplicationsassociatedwiththefilebyrightclickingthefileandselectingthepropertiesdialog(Figure2.10).
Figure2.10FileProperties
Therootusersaresystemmanagersandhaveallpermissionstoallfilesinthefilesystem,regardlessofanyfilepermissionsettings.
WritingCDsorDVDsTocreateadataCD,clicktheCD/DVDCreatoroptiononthegomenuorsimplyinsertablankdiscintotheopticaldrive.Thisleadstoaspecialfolder(burn:///).Dragfilesandfoldersyouwishtowrite to thedisc into theburnfolder.Whenready,clickontheWritetoDiscbutton.TocopyaCD/DVD,simplyinsertthediscintheopticaldrive,rightclickon
theresultingdesktopicon(orfinditfromtheComputericon),andchooseCopyDisc.Or you can launch aCD/DVD tool such asBrasero to perform the desired
operations.
ChangingYourDefaultFileBrowserThedefaultfilebrowserisusedautomaticallywhenyouclick/double-clickafile,a folder, Places, orComputer. If you like to use a file browser other than thecurrent default, you can change your default file browser by following thesesteps.
1. Downloadandinstallyourfavoritefilebrowser.Themostconvenientwayistouseapackagemanager(Section8.2).Ifyoudon’thaverootprivilegeasksystemadmintodoitforyou.
2. Find the installed file browser in /usr/share/applications. For example, ls
2.8
/usr/share/applications
mayshowtwoavailablefilebrowsersorg.gnome.Nautilus.desktoporg.kde.dolphin.desktopTochangeyourdefaultfilebrowsertodolphin,issuethiscommandxdg-mime default org.kde.dolphin.desktop inode/directory application/x-
gnome-saved-searchwhich sets Dolphin as the default application to open the file type
inode/directoryofanyfolder.
TERMINALWINDOWBecause Linux is based on UNIX and offers many programs that run undercharacter-based terminals, character terminal emulation programs are providedto run such applications. Popular terminal emulators includeGnomeTerminal,konsole,Terminator,Sakura,GnomeGuake,andthemorerecentTilix.Onsomesystems,thekeyboardshortcutCTRL+ALT+Tcanlaunchthedefaultterminal.Linux commands requiring a terminal window include bash (the default
Linux Shell), vim (text editor), ssh (secure remote login), sftp (secure filetransfer), and many other command-line applications such as ls (directorylisting),rm,mv,more,man(displayingmanualpages),andsoon.TheclassicxtermterminalemulatorisavailableonmostLinuxdistributions,
butusersnormallyprefertouseanalternativesuchasgnome-terminal.We will discuss the GNOME Terminal here. Other terminals work in very
similarways.
Figure2.11StartingaTerminal
StartingaGNOMETerminalAGNOMETerminal emulates a character-based computer terminal and allowsyou to run a Shell or command-line applications. Here is how it works. AGNOME Terminal emulates the xterm from the X Consortium which can, inturn, emulate the various DEC (Digital Equipment Corp.) terminals such asVT220andVT320.From the Startmenu, theApplicationsmenu or theActivity Dash you can
easilylaunchaGNOMEterminalwhichwillrunBashbydefault(Figure2.11).Withoutcustomization,theterminalattributesarespecifiedbyadefaultprofile.You can customize the size, font, colors, and other attributes for the terminalwindowfromtheEdit->CurrentProfilemenuoption.By creatingdifferent terminalwindowprofiles andgiving themnames, you
canusethemfordifferentinstancesofGNOMETerminalsyourun.Let’sassumethatyouhavesavedyourfavoritewindowprofileunderthenamemain.Becauseeasyaccesstoaterminalcanbesuchaconvenience,werecommend
that you add a terminal launcher to start your customized terminal window.Followthesestepstocreateanewapplicationlauncher:
1. Fromthecommandlinecd$HOME/.local/share/applications+2. Tocreateanewlaunchervimy-new-launcher.desktop3. Putinthefiletheselines
[DesktopEntry]Type=ApplicationEncoding=UTF-8Name=My-Gnome-
TerminalComment=definedmanuallybyPaulWangExec=gnome-terminal--
window-with-profile=monkey--title=Main-WindowIcon=utilities-
terminalTerminal=falseStartupNotify=true
The value given to the Exec attribute is the command invoked when thelaunchericonisclicked.Foradditionalexamplesoflaunchers,checkoutthe.desktopfilesinthefolder
/usr/share/applications.
TerminalWindowandtheShellWhenyoustartaterminalwindow,yourdesignatedShell(bashbydefault)willbetheapplicationrunninginthewindow.TheShellcanruninthewindowasaregularShellora loginShell.TheGNOMETerminalallowsyou tomake thischoiceaspartofthewindowprofilesetting.ThedifferenceisthataregularShell
readsonly theShell initialization file,whereas a loginShellwill also read theShelllogininitializationfile(Section3.13).Insomesituations,youmaywanttostartaterminalwindowtorunsomething
otherthanaShell.Forexample,gnome-terminal-e"[email protected]"givesansshcommandtorunintheterminalwindowinsteadoftheShell.The
result is a GNOME Terminal connected to the remote host pwangtiger.cs.kent.edufortheuserpwangtologin.The terminalwindow closeswhen the application,whether a Shell or some
otherprogram,terminates.
Select,Copy,andPasteIt is generally a good idea to usewhat is alreadydisplayedon the screen andavoidtypingtheinformationagaintosavetimeandpreserveaccuracy.WiththeGNOMETerminal, andother text-displayingwindows such as aWebbrowserwindoworatexteditorwindow,youcanselect,copy,andpastetextwithease.
Select—Presstheleftmousebutton,clickacharacter,doubleclickaword,or triple click a line and then drag the mouse over a section of text tohighlightandselectthedesiredtext.Copy—Simply selecting the text copies it into a clipboard.You can alsoright click the highlighted text (or use theEdit->Copymenu option) toexplicitly copy the text into a copy buffer. Any previous content in theclipboardorcopybufferislost.Paste—Go to the target application, position the input cursor where youwish,andthenclickthemiddlemousebuttontopastefromtheclipboard.Oruse theEdit->Paste option of the target applicationwindow to pastefromthecopybuffer.
AGNOMETerminal remembersdisplayed text lines (500bydefault).Use thescrollbartogobackandforthonthetext.
WebandEmailLinksThe GNOME Terminal recognizesWeb and email addresses. For example, itrecognizeshttp://[email protected] yourmouse cursor over such an address and itwill be automatically
underlined, signaling that the GNOME Terminal has recognized the address.Rightclickandselect theOpenLink (SendEmailTo)option to launchaWebbrowser(anemailapplication)directly.Thisfeatureisveryconvenient.
2.9
2.10
2.11
The application launched in response to such usage is defined by yourPreferredApplicationssettingundertheStartmenu.
ACCESSINGHELPANDDOCUMENTATIONThe command gnome-help, or equivalently yelp, gives you easy access toinformation for the GNOME Desktop. A Web search will lead you todocumentationforyourparticularLinuxdistribution.Ingeneral, theHelpbuttononthemenubarofanyapplicationprogramwill
leadtodocumentationanduserguidesforthatparticularapplication.
SUMMARYLinuxGUI support is provided via theDesktopwhich has been evolving andimprovingsteadily.ModernLinuxdistributionsreceivewideracceptancedue,innosmallpart,togoodGUIandusefulGUIapps.GNOME and KDE are the two most widely used GUI environments for
Linux.Theybothrelyonanunderlyingwindowingsystemforgraphicaldisplayandwindowingsupport.KnowledgeandskillfuluseoftheGUIcanmakelifeonLinuxeasierandyoumoreproductive.GNOME3isthemostup-to-datedesktopthatispowerfulandpopular.Major desktop components are the Panel, Start button, apps menu, system
preferencesmenu,rootwindow,activeappwindows,theDash,appsearchbox,andtheworkspaceswitcher.The gnome-terminal can be customized and your settings can be saved in
profilesforreuse.AcomfortableterminalwindowcanmakelifeonLinuxmucheasier.TheGNOMENautilusfilebrowserprovidesavisualenvironmenttonavigate
the file tree and tomanage files and folders as well as their attributes. Otheruseful GUI applications include image processing, documentpreparation/viewing, audio-video playing, and creating CDs andDVDs.MoreGUIappsarediscussedinChapter4.
EXERCISES1. HowdoyoumoveyourPaneltothetoporbottomoftherootwindow?Is
thispossiblewithGNOME3?2. HowdoyoumakeyourPanelspantheentirewidthoftherootwindowor
12
becenteredatthetop/bottom?3. Findthelogouticonon/inthePanel.Isthereawaytoswitchuser?4. Findouthowtolockscreenandhowtoputyoursystemtosleep.5. Doyouhaveaworkspaceswitcherforyourdesktop?Ifnot,describehow
tocreateone.6. Isitpossibletoadd/removeworkspaces?How?7. Describehowtofindandlaunchappsonyourdesktop.Doyouhaveaccess
toan“appsearch"function?Howdoyoufindappsthatcanbeinstalledonyourdistribution?
8. Howdoesoneplaceananalogueclockonthedesktop?9. Describehowtoplaceapowermanagementlauncheronthedesktop.10. Whatiseog?Placealauncherforitonthedesktop.11. Whatisevince?Placealauncherforitonthedesktop.12. Settingupa terminalwindowcorrectlycanmakeabigdifference inhow
comfortableyouwillbeusingtheLinuxcommand-lineinterface.Considerthe command gnome-terminal –geometry=80x30+350+150? –window-with-profile=mainandexplainitsmeaning.Createanamedprofileofcolor,font,andotherpreferencesforyourself.Makeyourselfapanellaunchertostartyourcustomizedgnome-terminal.
13. On Linux, which GUI application can be used for MSWord and Exceldocuments?Installitonyourcomputerandplaceanicononthedesktop.
14. BurnadataCDanddescribeyourprocedure.Costabout$10Kin1983.AlsoknownastheWINDOWSkey.
Chapter3
InteractingwiththeBASHShell
Aswehavestated,inadditiontoGUIs,Linuxalsooffersefficientandpowerfulcommand-line interfaces (CLIs). In thebeginning,Unix/Linux systemshadnoGUI and only the CLI. Increasingly, many commands also have GUIcounterpartssotheyareeasiertouse.TheCLI,providedbyaprogramcalledaShell,remainsanimportantinterface
forLinuxusers.AShellnormallyrunsinsideaterminalwindowsuchasgnome-terminal or konsole (Chapter 1, Section 1.2). It takes input from the user(keyboard) and serves as a command interpreter to start applications and toperformallotheravailableoperations.ManyLinuxapplicationprogramscamefromUNIXandwerewrittenbefore
thegraphicaldisplaybecamestandard.OtherschosenottouseanyGUI.Thesecommand-line applications tend to be more efficient (less taxing on thecomputer,easiertocombinewithotherapplications,andsimpletoaccessacrossanetwork),butcanbeharder fornoviceusers.GUIapplicationsaregenerallymoreintuitivetolearnanduseinteractively,buttheycanbehardertocontrolorrunwithinotherprograms.WhenaccessingaLinuxsystemfromanotherhost,suchasaPC(Windowsor
Mac) orLinuxbox, through a remote login program such asSSH (Chapter 1,Section1.2)orTelnet,thefull-GUIofadesktop(Chapter2)ishardtoachieve,andtheShellisusuallytheonlyfeasibleuserinterfacechoice.WealreadyknowthatLinuxoffersanumberofdifferentShellsincludingSh
(theoriginalBourneShell),Ksh (theKornShell),Csh (theBerkeleyCShell),Tcsh(TCShell,animprovedCShell),andBash(theBourne-AgainSh).AusercanpickwhichShelltouse.AlthoughtheseShellsarecomparable,BashisthestandardandpreferredShellonLinuxsystems.WewillpresentinteractiveuseofBashinthischapter.ProgramminginBash
ispresentedinChapter5.
3.1
3.2
BASHDevelopedin1987fortheGNUProject(FreeSoftwareFoundation),BashisafreelyavailableShellbasedupontheoriginalSh(BourneShell,1978).TheBashShellincorporatesfeaturesfromSh,Ksh,Csh,andTcsh;addsnewfeaturessuchasShell-definedfunctions;andconformstotheIEEEPOSIX(pronouncedpahz-icksforPortableOperatingSystemInterface)specification.Today,BashisthemostpopularShellonLinuxsystems.Improvedversionsof
Bashhavebeenreleasedregularly.Normally,yourdefaultShell is /bin/bash.Ifnot,youcanalwayssetyourdefaultShellto/bin/bash(recommended)withthecommandchsh-s/bin/bashInaBashShell,thecommandecho$BASH_VERSIONdisplaysitsversioninformation.ItisagoodideatohaveyourBashreadyfor
experimentationwhenreadingthischapter.
INTERACTINGWITHBASHInside a terminal emulatorwindow, Bash serves as your command interpreterandcontinuallyexecutesthecommandinterpretationcycle:
1. Displaysaprompt2. Enablestheusertotype,edit,andenterthenextcommandline3. Breaks thecommand line into tokens (wordsandoperators)andperforms
well-definedShellexpansions,transformingthecommandline4. Carries out (by calling Shell-level functions) or initiates (by starting
externalprograms)therequestedoperations5. Waitsforinitiatedoperationstofinish6. Goesbacktostep1
The default prompt for Bash is $, but it can be customized to become moreuseful(Section3.9).Acommand lineconsistsofoneormorewords separatedbywhitespace or
blanks (spaces and/or tabs). Pressing the ENTER (RETURN) key completes inputtypingandsendstheShelltothenextstep.TheENTERkey(generatingaNEWLINEcharacter)completesacommandlineunlessprecededbyabackslashcharacter(),inwhichcasetheENTERisescaped(Section3.14)andbecomesablank.Thefirstwordinacommandisthecommandnameandindicatestheprogramtobeexecuted;theotherwordsareargumentstothecommand.Therearetwotypesof
commands: Shell built-in commands and regular commands. A built-incommandinvokeseitheraroutinethatispartoftheShell(cd,forexample)orafunctionoraliasdefinedbytheuser.Toexecuteabuilt-incommand, theShellsimplycallsup theappropriate subroutinewithin itself.A regular command isany other executable program in Linux that is not built into the Shell. Theseinclude system commands such as ls, rm, and cp, as well as your ownexecutableprogramssuchasa.out.Eachexecutingprogramisknownasaprocesscontrolledandmanagedbythe
operating system. Your interactive Shell is a process. The Shell spawns(initiates) a separate child process, known as a subshell, to execute a regularcommand. The distinction between built-in and regular commands is animportantone,asyouwilldiscover.A simple command is just the command name followed by its arguments.
Severalcommandscanbegivenonasinglecommandlineiftheyareseparatedbysemicolons(;).TheShellwillexecutethecommandssequentially,fromlefttoright.Twocommandsseparatedbyaverticalbar(|)formapipe(Section3.5).Theoroperator(||)andtheandoperator(&&)specifyconditionalexecutionofcommands:cmd1||cmd2(executescmd2onlyifcmd1fails)cmd1&&cmd2(executescmd2onlyifcmd1succeeds)Theseareexamplesofcompoundcommandswhereseveralsimplecommands
aregroupedtogethertoformasinglecommand.InLinux,acommandreturnsanexitstatusofzerowhenitsucceedsandnon-
zerootherwise.If you enclose one or more commands inside a pair of parentheses (), the
commandswillbeexecutedasagroupbyasubshell.After issuing a command, it is not necessary to wait for a prompt before
typinginadditionalinput.Thisfeatureisknownas typeahead.Whatyou typeaheadwillbetherefortheShellwhenitisreadytoreceivethenextcommand.YoualsocaninstructtheShellnottowaitforacommandtofinishbytyping
an AMPERSAND (&) at the end of the command. In this case, the Shellimmediately returns to process your next command, while the previouscommand continues to run detached from the Shell. Such detached processescontinuetoexecuteandaresaidtoberunninginthebackground.Forexample,firefox&will start the browser and return you to the Shell levelwithoutwaiting for
firefox to finish, which is not likely to be any time soon. Basically, theAMPERSAND instructs the Shell to skip step 5 in the command interpretationcycle.Abackgroundprocessalsogivesupreadaccesstothekeyboard,allowing
3.3
youtocontinueinteractingwiththeShell.Abackgroundprocesscanbereattachedto theShell—that is,brought to the
foreground—bythecommandfgjobidPleaserefertoSection3.6forjobIDsandjobcontrol.A foreground program receives input from the keyboard. If we bring the
firefoxjobtotheforeground,wecantypeaCTRL+Ctoabortit,forexample.Therecanbeonlyonerunningforegroundprogramatanygiventime.
COMMAND-LINEEDITINGANDCOMMANDCOMPLETION
Let’slookattypinginputtoBash.WehaveseeninChapter1(Section1.3)howthearrowkeystogetherwithDELETEandBACKSPACEcanbeusedtocorrectinputerrorsandtoreusepreviouscommands.Theseandothercommand-lineeditingfeaturesareprovidedbythereadlinelibrary.You, in fact, have a choice of usingvi oremacs (see the appendices at the
companionwebsite)foreditingthecommandlinewithset-oviset-oemacsIncaseofvimode,youwouldtypeESCtogetintothevicommandmodeand
then use vi commands to do any editing. When you are done editing thecommandline,pressRETURN(orENTER)toissuethecommand.While entering a command line, Bash helps you complete your typing in
varioususefulways.Basically, you engage the completion feature bypressingtheTABkey.Ifthereisauniquecompletion,itwillbedone.Iftherearemultiplewaystocompleteyourtyping,asecondTABwillrevealthechoices.Forexample,ifyouenterunfollowedbytwoTABs,alistofchoices
unaliasuniqunlinkunstrunameuniqleafunopkg...
willbedisplayed.Thetechniquenotonlysavestyping,butalsoshowsyoualltheBashbuilt-inandregularcommandswithagivenprefix,whichcanbeveryhandyifyouforgottheexactcommandnametouse.Some users prefer getting the choices listed directly with the first TAB by
puttingsetshow-all-if-ambiguousoninthereadlineinitfile/.inputrcStandardcompletionsperformedare
3.4
Commandnamecompletion—CompletingShellbuilt-incommands,aliases,functions,aswellasregularcommands;performedonthefirsttokenofthecommandlineFilename completion—Completing names for files; performed onargumentstoacommandUser name completion—Completing userids for all users on your Linuxsystem,performedonanywordstartingwithaHostnamecompletion—Completingdomainnames;performedonanywordstartingwith@Variablenamecompletion—CompletingnamesforexistingShellvariables;performedonanywordstaringwith$
Further, thebash-completionpackage (includedwithmostLinuxdistributions)enablesyoutoTAB-completecommonargumentstooften-usedcommands.Forexample,theargumenttothesshcommandsshpwang@[email protected]@[email protected]@mathedit.orgpwang@monkey.zodiac.cs.kent.eduOntopofthesecompletions,youcandefineyourownwiththeBashbuilt-in
completecommandwhichimplementsaprogrammablecompletionAPI.Seethecompletedocumentationfordetails.ThereadlineescapecharacterCTRL+Visusedtoquotethenextcharacterand
preventitfrombeinginterpretedbyreadline.Thus,togetaTABintoyourinputinsteadofinvokingthecompletionfunction,youwouldtypeCTRL+VfollowedbyTAB.Forexample,youcandefinetheCTRL+Laliaswiththefollowing:aliasCTRL+VCTRL+L=clear
BASHCOMMANDEXECUTIONThe first word of a command line is the command name. It can invoke aprocedurewithintheShell(inorder):analias(Section3.7),afunction(Section3.15),orabuilt-incommand.Ifnot,thenthecommandnameinvokesaregularcommandimplementedbyaprogramindependentoftheShell.In a regular command, the command name indicates an executable file and
canbe inoneof two forms. It canbe theabsoluteor relativepathnameof theexecutable file, or if the executable file is on the command search path, thesimple filename itselfwill suffice.Theprocedurebywhich theShell finds the
executablefileisasfollows:
1. Ifthecommandnameisanabsoluteorrelativepathname,thenthenameoftheexecutablefilehasbeengivenexplicitlyandnosearchisnecessary.
2. Ifthecommandnameisasimplefilename(containingno/character),theexecutable file is found by searching through an ordered sequence ofdirectoriesspecifiedbythecommandsearchpath.Thefirstfilefoundalongthissearchpathisused.
Iftheexecutablefilecannotbefound,orifitisfoundbuttheexecutepermissiononthefile isnotset, thenanappropriateerrormessage isdisplayed.Theerrormessagemostlikelywillbefilenotfoundorpermissiondenied.The Shell environment variable PATH (Section 3.10) defines the command
search path, a list of directories containing executable commands. The Shelllooks sequentially through thesedirectories for any commandyougiveon thecommand line. The PATH usually includes the system folders /bin, /sbin,/usr/bin,and/usr/sbin,wheremostsystem-suppliedexecutableprogramscanbefound. The search path can bemodified to include additional directories. Forexample,exportPATH=$PATH:/usr/local/bin:$HOME/binaddstwodirectoriesat theendofPATH:a/local/binwhereyouinstallextra
applicationstoyourLinuxandabininyourhomedirectory.1Now,youcanusea simple filename to run a program whose executable file resides in the$HOME/bindirectory.Bashusesahashtabletospeedupcommandsearchandonlyneedstosearch
through$PATH(andupdatethetable)whenacommandisnotfoundinthetable.Thebuilt-inhashcommandallowsyoutodisplayandmanipulatethistable(seehelphash).Thespecialperiodsymbol(.)isoftenplacedattheendofthesearchpathto
enable you to invoke any command in the current directory with a simplefilename.exportPATH=$PATH:.The built-in export command tells the Shell to transmit this value to the
execution environment (Section 3.10) that will be inherited by subsequentregularcommands.Becauseofaliasing(Section3.7),user-defined functions (Section3.15), and
commandsearch,thecommandactuallyexecutedmaynotbeexactlywhatyouintended.Tobesure,youcancheckbyissuingwhichcommand_nameto display the alias/function or the full pathname of the executable file
3.5
invokedbythecommand_name.Forexample,whichgnome-terminaldisplays/usr/bin/gnome-terminalOnceanexecutable filehasbeen found, theShell spawnsachildprocess to
runtheprogramtakingthesethreesteps:
1. Anew(child)processiscreatedthatisacopyoftheShell.2. Thechildprocess isoverlaidwith theexecutable file.Then thecommand
nametogetherwithanyargumentsarepassedtoit.3. The interactive Shell waits for the child process to terminate before
returningtoreceivethenextcommand,unlessthecommandhasbeengivenwith a trailing ampersand (&). If the command ends with &, the Shellreturnswithoutwaiting,andthecommandisruninthebackground.
BASHINPUT/OUTPUTREDIRECTIONUntilnow,ouruseofLinuxhasbeenlimitedtoissuingcommandsandobservingtheir output. However, you certainly will want results in amore useful form,either as hard copy or stored in a file. Furthermore,many instanceswill arisewhenyouwantinputtocomefromsomewhereotherthanthekeyboard,suchasafile,orperhapsevenfromanothercommandorprogramrunningconcurrently.Linuxprovidesanelegantsolution:input/outputredirection.When processing a command line, the Shell arranges any I/O redirections
beforeexecutingcommandscontainedinthecommandline.
StandardInputandOutputAs an operating system, Linux provides input and output (I/O) services forprocesses.Foreachprocess,asetoffiledescriptorsnumbered0,1,2,andsoonisusedforI/Otransactionsbetweentheprocessandtheoperatingsystem.Whena process opens a file or a device for I/O, a file descriptor is assigned to theprocess to identify the I/O channel between the process and the open file ordevice. When a new process is created, its first three file descriptors areautomaticallyassigneddefaultI/Ochannels.
File descriptor 0, the standard input or simply stdin, is connected to thekeyboardforinput.Filedescriptor1,thestandardoutputorsimplystdout,isconnectedtotheterminalwindowforoutput.
File descriptor 2, the standarderror or simply stderr, is connected to theterminalwindowforerroroutput.
Most CLI commands receive input from standard input, produce output tostandardoutput,andsenderrormessages tostandarderror.TheShell-providedI/OredirectioncanbeusedtoreroutethestandardI/Ochannels.
I/ORedirectionThespecialcharacters>,<,and|areusedbytheShelltoredirectthestandardI/OchannelsofanycommandinvokedthroughtheShell(Figure3.1).
Figure3.1I/ORedirection
Let’slookatasimpleexample.Thecommandlinels>filelistcreatesinyourcurrentdirectoryafilenamedfilelistcontainingtheoutputof
the ls command. The symbol > instructs the Shell to redirect the stdout of lsaway from the terminal screen to the file filelist. If a file by the same namealready exists, itwill bewiped out and replaced by a new filewith the samename,unlessyousetthenoclobberoptionwiththeBashbuilt-insetcommand
set-onoclobber(turnsonthenoclobberoption)set+onoclobber
(turnsoffthenoclobberoption)set-o(displaysalloptions)
Whenthenoclobberoptionison,redirectingoutputwith>toanexistingfilewill result in an error. This feature protects against accidental loss of a filethroughoutputredirection.Ifyoudomeantowipeoutthefile,addaverticalbar(|)afterthe>.Forexample,ls>|filelistMany users set the noclobber variable in their Bash initialization file
.bash_profile(seeSection3.13).Oneexceptionisthat/dev/nullisaspecialdatasink. Output redirected to it disappearswithout a trace. It is usefulwhen you
wishtodiscardoutputfromacommand.Thesymbol>>operatesmuchthesameas>,butitappendstotheendofa
fileinsteadofoverwritingit.Forinstance,catfile1>>file2appendsfile1totheendoffile2.Iffile2doesnotexist,itwillbecreated.So far, we have only talked about redirecting the standard output. But
redirectingthestandarderrorfollowsthesamerules,exceptyouneedtouse2>and2>> instead to explicitly indicate the file descriptor being redirected.Toredirectbothstandardoutputandstandarderror,use
someCommand>file2>&1(stderrjoinsstdoutintofile)someCommand
>file12>file2(sendstodifferentfiles)
Let’slookatanotherexample.cat>fileAfter giving this command, what you type on the keyboard (or copy and
paste) is put into file. Keyboard input is terminated by CTRL+D given at thebeginningofaline.Next,let’sconsiderredirectionofstdin.Usingtheoperator<,acommandthat
takesinteractiveinputfromthekeyboardcanbeinstructedtotakeinputfromafileinstead.Forexample,vitextfile<cmd-filewhere cmd-file contains commands to the vi text editor. Let’s say cmd-file
containsddZZthenthefirstlineoftextfilewillbedeleted.ManyLinuxcommandstakeinput
fromafileifthefileisgivenasanargument(sortfile,forexample);theusagesort<fileiscorrectbutunnecessary.
PipesInadditiontobeingabletoredirectI/Otoandfromfiles,youalsocanredirecttheoutputofoneprogramasinputtoanotherprogram.Theverticalbarsymbol(|)isusedtoestablishapipe,whichconnectstheoutputofthefirstcommandtotheinputofthesecond(Figure3.2).
3.6
Figure3.2APipe
Thus,ls-lt|morepipesthestandardoutputofls-lttothestandardinputofmore.Theresulting
commandiscalledapipeline.Sometimes,fornewusers,itishardtounderstandthedifferencebetween|and>.Justrememberthatthereceivingendofapipe|isalwaysanotherprogramandthereceivingendofa>or>>isalwaysa file.You can pipe the standard error together with the standard output using |&instead of |.More elaborate examples of pipelines are described inChapter 4,Section4.7.BashI/ORedirection
Table3.1summarizesBashI/Oredirection.Optionalpartsinthenotationareenclosedinsquarebrackets.
BASHJOBCONTROLOnthedesktop,weknowwecanrunmultipleapplications,each inadifferentwindow,andwecanswitchinputfocusfromonewindowtoanother.Within a single terminal window, the Shell also allows you to initiate and
controlmultiplecommands(calledjobs).Atanytimethereisonejobthatisinthe foreground and connected to the keyboard input for the terminal window.Other jobs are in the background. We already mentioned that if you add atrailing & to a Shell-level command, the Shell will run the job in thebackground.Hereisanotherexample.xclock(runsxclockinthebackground)Thenyoumaystartanotherjob,say,fortextediting,bythecommandnano-znotes.txtThis job is in the foreground, enabling you to control nano and perform
editing functions using the keyboard. At any time, you can type CTRL+Z to
suspendtheforegroundjobandgetbacktotheShelllevel.Ifyoudothat,thenyou’llsee2[2]+Stoppednano-znotes.txtand a new Shell prompt will appear in your terminal window to provide
confirmation that the current job has been suspended and will be in thebackground waiting to be resumed. Now you can issue any Shell-levelcommand, including one to start another job (which may itself be suspendedwithCTRL+Zinthesameway).Let’ssaythatyouthenstartathirdjob,gimppicture.jpgtodoimageprocessingonapictureandthensuspenditalso.Inthisway,itis
possibletostartthensuspendorputinthebackgroundquiteafewjobs,anditiseasytoseehowthiscanbecomeunmanageablequickly.Fortunately,ifyouissuetheShellbuilt-incommandjobsyou’llseeallyourjobsdisplayed[1]13519Runningxclock&[2]-12656Stoppednano-znotes.txt[3]+13520Stoppedgimppicture.jpgInthiscase,therearetwosuspendedjobswithjobnumbers2and3,andone
jobrunninginthebackgroundwithjobnumber1.TheShellalsoallowsyoutoresumeasuspendedjob,pullabackgroundjobintotheforeground,orkillajobentirely.To identifya job,a jobid isused,whichcanbegiven inanumberofways:
%job-number,%name-prefix,%+,and%-.Forexample,thejobids%3,%+,and%g all refer to same job in the preceding example.The job%+ is always themost recently suspended (the current job), and %- is always the next mostrecentlysuspended(thepreviousjob).The%-isusefulwhenyouaregoingbackand forth between two jobs.When using the name-prefix form, you need justenough prefix of the command name to disambiguate it from other jobs. Forexample,%vim,%vi,or%vallrefertojob2.Ajobcanberesumed(broughttotheforeground)bytheShell-levelcommandfgjobidYoucanabbreviatethecommandtojustjobid.Forexample,%1willbringjob
1totheforeground,%+(orsimplyfgbyitself)resumesthecurrentjob,and%-resumes theprevious job. Ifno jobid isspecified, themost recentlysuspendedjobwillbeactivatedandruninthebackground.If a background job produces output to stdout, it will be displayed in the
terminalwindowandinterferewithoutputfromanyforegroundjob.Further,if
thebackgroundjobrequiresinputfromtheterminal,itwillstopitselfandwaittobebroughttotheforegroundtoreceivetheinputitneeds.Thus,forjobstorunefficiently in the background, redirecting standard I/O to files usually isessential.Whenabackgroundjobterminates,theShelldisplaysamessagetonotifythe
user:[jobnumber]DonecommandasgivenThemessage isdisplayedafternormalcompletionof abackgroundprocess.
The following message is displayed when a background process terminatesabnormally:[jobnumber]Exit1commandasgivenToswitchasuspendedjobtoruninthebackground,usethecommandbgjobidSuspendingajobusingCTRL+Zisnotthesameasexitingorterminatingit.Itis
goodpracticetoexitalljobsproperlyandcloseallwindowsbeforeyoulogout.Each job provides its ownway for exiting (quitting); for example,CTRL+X fornano,:q!orZZforvim,qformutt,andexitfortheShell.Sometimes youmay need to force a program running in the foreground to
terminate.Thiscanbedoneby typing the interruptcharacter, usuallyCTRL+C,whichabortstheexecutingjobandreturnsyoutotheShelllevel.Iftheinterruptcharacterdoesnotstopyourprogramforsomereason,yourlastresortisthekillcommand.UseCTRL+ZtosuspendthejobandgettotheShelllevel,thentypekill-9jobidtoterminatethejob.Theoptionalargument-9instructskilltosendaspecific
signaltotheprocess,whichforcesittoterminate.SignalsaredescribedfurtherinChapter11,Section11.6.Inadditiontojobids,killcanalsotakeprocessnumbers.Thecommandjobs-lgivestheprocessnumbersforalljobs.Thekill commanddiscussedhere is built intoBash.There is also a regular
command,/bin/kill, thatcanbeused.Amongotherdifferences,/bin/killworksonlyonprocessnumbers.Table3.2listsusefuljobcontrolcommands.JobControlCommands
3.7
To sumup, a jobmay be in one of three states: running in the foreground,running in thebackground,orstopped(suspended).Nomore thanone jobcanrun in the foreground at any time, butmany jobs can run concurrently in thebackground. Many also may be stopped. To see the states of the jobs undercontrol of your Shell, use the command jobs. Use fg alongwith the jobid tobring a particular job from suspension or from the background into theforeground.Usethesuspendcharacter(usuallyCTRL+Z)tosuspendaforegroundjob.Usetheinterruptcharacter(usuallyCTRL+C)tokillaforegroundjob.Ifajobisstoppedorrunninginthebackground,itcanbekilledbyissuingthecommandkill[-9]jobid.Ifyougivetheexit(logout)commandwhiletherestillareunfinishedjobs,the
Shellwillremindyouofthefact.Itisbesttoterminateallunfinishedjobsbeforeexiting the Shell. However, if you insist by issuing an immediate second exitcommand, the Shell will abort all your unfinished jobs, and your terminalwindowwillclose.
BASHSHELLEXPANSIONSEachcommandlineundergoesanumberoftransformationsbeforeitisexecutedbytheShell.ThesetransformationsarecalledShellexpansionsandaredesignedtoprovidepowerandconveniencetotheuser.Forexample,youcanusels-l*htmlto see a listing of all files with a name that ends with html. This works
becauseofFilenameExpansion.Let’sseehowtheseexpansionswork.
Figure3.3BashExpansions
Bashtransformseachcommandbyapplyingthefollowingexpansions(Figure
3.3)insequence:
1. Historyexpansion—Allowsreuseofpartsofpreviouscommands2. Aliasexpansion—Replacescommandaliasesbytheirdefinitions3. Braceexpansion—Treatsexpressionswithincurlybraces4. Tildeexpansion—Expandsaprefixedwordtoacertaindirectoryname5. Variableexpansion—Replacesvariablesbytheirvalues6. String expansion—Interprets standard escape characters, such as n
(NEWLINE), r (RETURN), and t (TAB), in strings of the form $ ’xyz’; forexample,$’NametAgern’
7. Commandexpansion—Insertstheoutputofacommandintothecommandline
8. Arithmetic expansion—Includes results of arithmetic expressions in acommand(thisfeatureismostlyusedinShellscriptsandwillbecoveredinChapter5,Section5.11)
9. Processexpansion—Specifiesoutputproducedbyacommandtobecomeafilenameargumentforanothercommand
10. Filename expansion—Adds filenames to the command line by patternmatching
After all transformations, the resulting command line gets executed. You areencouraged to experiment with the expansions as you read their descriptions.The built-in command echo which displays the after-expansion state of itsargumentscanbeveryuseful.Byputtingtheecho infrontofacommandline,theeffectsofallbutaliasexpansioncanbeexamined.
HistoryExpansionTheBashhistorymechanism recordspreviouscommands foreasy reuse.Eachcommandlineissuedbyyou,whethersuccessfulornotandwhetherconsistingofoneormorecommands, iskeptasan item in thehistory list,whichcanbedisplayed using the built-in commandhistory. Each item in the history list isknownasahistoryevent,andeacheventisidentifiedbyasequencenumber.Thetotalnumberofeventskeptonthehistorylisthasalimit(defaultsto500)whichissetbyCommonHistoryExpansions
HISTSIZE=numberNormally, keeping 50 events or so is quite enough. Entering your own
HISTSIZEsettinginthe.bash_profilefile(Section3.13)makesgoodsense.WealreadyknowfromChapter1thatyoucanusetheupanddownarrowkeystogoback and forth on the history list and reuse previous commands.Furthermore,history expansion enables you to substitute history events into the currentcommandlinewithjustafewkeystrokes.Italsomakesmodifyingandreissuingpastcommands,orpartsofthem,easytodo.Historyexpansioniskeyed(activated)bytheexclamationpointcharacter(!),
anditworksbyrecallingitemsfromthehistorylist.Itemsthatcanberecalledfromthelistandsubstitutedintothecurrentcommandincludeanyhistoryevent,anywordorwordsofanyevent,andpartsofcertainwords.Theseitemsalsocanbemodifiedbefore their inclusion into the current command.Table3.3 showssome common history expansions. Table 3.4 contains some applications ofhistoryexpansionincommands.HistoryExamples
Eachexampleisdescribedhere,andthenumberscorrespondtothenumbersinTable3.4.
1. Reusethenamefile3.2. Nameturnsouttobeadirectory.3. Mistypedthecommandnamesort.4. Thedesiredfileisnotinthecurrentdirectorybutinthedirectorydir.
5. Thedirisnotinthecurrentdirectorybutinthehomedirectory.6. Notethatblanksareallowedinthestringreplacement.
Havingseenanumberofexamples,youarereadytoproceedtothegeneralformofahistoryexpansion:event[:worddesignator][:modifier…]Theeventisgiveninoneofthefollowingways:
Eventnumber !12givesevent12onthehistorylist.
Relativeposition
!-2givesthesecondmostrecentevent.Aspecialcaseis!!,whichreferstothelastevent.
Commandprefix !nanogivesthemostrecenteventprefixnano.
Matchingstring
!?string?givesthemostrecenteventcontainingstringanywherewithintheevent.
str1str2 Repeatsthelastcommand,butwithstr1replacedbystr2.Followingtheeventaretheoptionalworddesignators.Thepurposeofaword
designator is to choose certain words from the history event. If no worddesignators are used, the entire event will be selected. The following worddesignatorscanbeused:Anoptionalsequenceofmodifiersalsocanbeused.Onefrequentusageisevent:s/xx/yy/tosubstitute thestringxxbyyy inevent. If aword isa longpathname, it is
sometimes convenient to use a modifier to extract a portion of it, but mostmodifiers are seldomly used interactively. Writing programs in the Shelllanguage(Shellprocedures)isdiscussedinChapter5,andatthatpointyouwillbe able to seewhymodifiers are needed.A number ofmodifiers are listed inTable3.5;refertotheBashmanualforacompletelist.Onceacommandlinehasgone through history expansion, it too becomes part of the history list as themostrecentevent.HistoryModifiers
TheBashbuilt-incommand fc (fix command)puts a rangeof history items
intoyour favorite text editor, allowsyou tomodifyanyparts atwill, and thenexecutestheresultingcommandsautomaticallywhenyouexittheeditor.fcfirst_eventlast_eventFinally, when you are finished interacting with it and exit, Bash saves the
command history to the history file specified by the environment variable$HISTFILE,whichdefaultsto.bash_historyinyourhomefolder.NexttimeyoustartBash,thesavedhistorywillberestoredfromthehistoryfile.ThehistoryfilefeaturecanbedisabledbyexportHISTFILE=
AliasExpansionThe alias feature allows you to define shorthands for often-used commands,making themeasier toenter.Tocreateanalias (anysingleword)andgive itavalue(acharacterstring),usetheBashbuilt-incommandalias.Thenotationaliasname=value…defines the given name as an alias for the specified string value. Multiple
name-valuedefinitionsareallowed.ThevaluepartoftenrequiresquotesaroundittopreventunwantedShellexpansions(seeSection3.14forwhenandhowtousequotes).Herearesomesimplebutusefulaliasdefinitions.aliasdir="ls-l"back=’cd$OLDPWD’aliasmonkey="ssh-lpwangmonkey.cs.kent.edu"aliasappend2end="cat>>"Withthesealiasesdefined,thecommanddirworksbecauseitexpandstols-l.
Thealiasbackworks itsmagicbecause theBashvariable$OLDPWDalwaysholdsontothepreviousworkingdirectory.Aliasexpansionmeansthatifthefirstwordofasimplecommandisanalias,
Bash will replace that first word with the alias value. The first word of thereplacementtextisagaintestedforaliases,butawordthatisidenticaltoanaliasbeing expanded is not expanded a second time. This means the following iscorrectanddoesnotresultinaninfiniteloop.aliasls=’ls-F’Thus, the ls command always is given with the -F option, which causes,
among other things, directory names to bemarkedwith a trailing /, symboliclinks to bemarkedwith a trailing@, and executable files (fileswith executepermission;seeChapter1,Section1.6)tobemarkedwithatrailing*.Todisplayexistingaliases,use
alias(displaysallaliases)aliasname(displaysthealias)
Toremovealiasdefinitions,useunaliasname...
BraceandTildeExpansionsBraceexpansionprovidesashorthandforsimilarwordson thecommand line.Withbraceexpansion,thecommandlinenanomemoSep,Oct,Nov2018.txtbecomesnanomemoSep2018.txtmemoOct2018.txtmemoNov2018.txtandlprchap2..5.pdfbecomeslprchap2.pdfchap3.pdfchap4.pdfchap5.pdfThesequencenotation(..)worksfornumbersandsingleletters,forexample,
a..z.ThecharacterTILDE ( )expands to theuser’sownhomedirectory, useridto
thehomefolderofsomeotheruser,+tothecurrentfolder,and-tothepreviousfolder.Thus,thealiasbackearliercanalsobedefinedasaliasback="cd-"
VariableExpansionTheShell allows the use of variables, also known as parameters.A variable’svalue is a character string. Some variables are reserved for Shell use. Forexample, USER, HOME, PATH, and HISTSIZE are Shell variables havingprescribedmeaninginBash(seeSection3.9).Inaddition,youcanalsosetanduseyourownuser-definedvariables.Generallyspeaking,avariableidentifiercanbeanywordwhosefirstcharacter
isaletterandtherestconsistsofletters,digits,andunderscorecharacters.Usevar=value(setsvariablevalue)to assign a value to a variable. The value can be a singleword ormultiple
wordsinquotes,andnowhitespaceisallowedimmediatelybeforeoraftertheequalsign(=).Afterbeingset,avariablecanbeusedinsubsequentcommands.Forexample,ldir=/usr/localgivesthevariableldirastringvalue/usr/local.Withthisvariableset,youcan
inputcd$ldirwhich is a command with a variable in it. After variable expansion, this
commandbecomes
cd/usr/localAsyoucansee,variableexpansioniskeyedbythecharacter$.Thatis,aword
thatbeginswitha$ isavariable.If$ isfollowedbyablankorprecededbyabackslash(),thenitstandsforitself.Theechocommandcanbeusedtodisplaythevalueofavariable.Forexample,echo$ldirdisplays/usr/local.Useunsetvartoremoveanyvariablevar.The extent of a variable name can be delineated by braces ( and ). For
example,x=abcecho$xdedisplaysthestringabcde,whereasecho$xdedisplaysanemptylinebecausethevariable$xdehasnovalue.Variables often have string values. However, they may also have integer
values.Inside$((...)),youmayperformintegerarithmeticoperations(including+-*/+%**++–)onvariablesusingC-languagesyntax.Forexample,
count=7echo$((3*count))(displays21)echo$((count%5))
(displays2)echo$((count++))(displays7,setscountto8)
You can display variables (Shell built in and user defined) and functiondefinitions(Section3.15)with
set(displaysallvariablesandfunctions)declare(displaysall
variablesandfunctions)declare-f(displaysallfunctions)
CommandExpansionCommandexpansionmakesitpossibletousethestandardoutputofacommandas a stringofwords in another command.Either $(command) or ‘command’(notetheBACKQUOTE)canbeusedforcommandexpansion.Forexample,
dir1=$(pwd)(ordir1=`pwd`)
assignstheoutputofthepwd(printworkingdirectory)commandtotheuservariabledir1.Anotherexample,files=$(ls)assignstofileswordsproducedbythelscommand,namely,thefilenamesin
the current directory. The substitute string of a command expansion also canformpartofasingleword,asinfile1=$(pwd)/test.c
The substitute string isnormallybroken into separatewords atblanks, tabs,andNEWLINEs,withnullwordsbeingdiscarded.
ProcessExpansionBash extends the ideas of I/O redirection one step further by allowing thenotation
<(commandargs...)
tobeusedwhereafilenameargumentisexpectedforacommand.Thus,thenotation < (...) produces a temporary file, with the output produced by thecommandinside,whichcanbegiventoanothercommand.Forexample,
nano<(ls-l-F)
opensnanotoview/edittheresultsproducedbythegivenlscommand.Thisabilitycanbehandysometimes.Itispossibletosupplymultiplefilesinthisway.Forexample,
diff-u<(ls-F/usr/bin)<(ls-F/usr/bin.old)
displaysthedifferencesbetweenthetwodirectorylistings.
FilenameExpansionBecause command arguments often refer to files, the Shell provides filenameexpansion tomake it easier to specify files.When a filename pattern orglobpattern is used in a command line, the pattern is expanded to become all thefilenamesmatching the pattern.A patternmaymatch simple filenames, in thecurrentworkingdirectory,aswellasfullorrelativepathnames.Ifapatterndoesnotmatchanyfile,thenitstandsforitselfandisnotexpanded.Globpatternsarespecifiedusingthespecialcharacters*,?,and[].The*matchesanysequenceofzeroormorecharacters.Forexample,
ls-l*.c
producesa listingofall fileswithanameending in .c.The*.c isapattern,and it expands tomatch all filenames in the currentworking directory endingwith.c.Thecommand
ls-l../*.c
doesthesameforfilesintheparentfolder.Thecommand
ls~/Pictures/2018*/*.jpg
conveniently displays a listing of all pictures, ending in .jpg, under folderswithanameprefix2018,inthe/Picturesdirectory.Filenamepatternsarematchedagainstexistingfilenames.Rulesforfilename
patternsareasfollows:
*Matchesanycharacterstringoflengthzeroormore(the
“wildcard”).?Matchesanysinglecharacter.[…]Matchesanyoneof
thecharacterscontainedbetween[and](arangepattern).For
instance,a[rxz]bmatchesarb,axb,orazb.Thepatternchapter[0-9]
matcheschapter0,chapter1,andsoon.[^…]Matchesanycharacter
notin[and].The!charactercanbeusedinsteadof^.[[:class:]]
Matchesanyinaclassofcharacters.Theclasscanbealnum
(alphanumeric),alpha,digit,lower,orupper.
Forexample,inthecommand
ls[[:digit:]]*
thepatternmatchesallfileswhosenamestartswithadigit.Filename expansion is also known asglobbing. Filename expansion can be
deactivatedwiththeBashbuilt-incommand
set-f(or-onoglob,filenameexpansionoff)set+f(or+onoglob,
filenameexpansionon)
FilenameexpansionshouldnormallybeonwhenusingtheShellinteractively.The character . at the beginning of a filename must be matched explicitly
unlessthedotgloboptionisset.
shopt-sdotglob(enablesmatchingleadingdot)shopt-udotglob
(disablesmatchingleadingdot)shopt(listsBashoptions)
Hence,thecommandls*normallydoesnotlistanyfileswhosenamebeginswithadot.Additionally,thecharacter/inafilenamemustbematchedexplicitly.Afilenamepatterncancontainmore thanonepatterncharacter.Whenmore
thanone filename ismatched, the pattern is expanded into a sorted list of thematched filenames. Matching is case sensitive unless you do shopt -snocaseglob. If a pattern matches no filenames (match failure), then it is notexpanded(staysunchangedinthecommandline)unless
3.8
3.9
shopt-sfailglob(matchfailurecausesanerror)shopt-snullglob
(matchfailureexpandstoemptystring)
BASHBUILT-INCOMMANDSWehaveseenanumberofBashbuilt-incommands.Afewmoreareintroducedinthissection.ToseealistofallBashbuilt-incommands,youcanusethebuilt-inhelp.
help(listsallbuilt-incommands)helpcommandName(describesthe
givencommand)helphelp(tellsyouhowtousehelp)
Bashmaintainsadirectorystackthat,bydefault,containsthecurrentworkingdirectory.The built-inpushddir changes to the given directory and pushes itontothestack.Thebuilt-inpopdchangestothetopdirectoryonthestackafterpoppingitoffthestack.Thus,thesequencepushddirpopdbringsyoubacktowhereyouwerewithoutchangingthedirectorystack.The
built-indirsliststhefoldersonthestack.While interactive inputusuallycomesfromthekeyboard, it isconvenient to
save and edit commands in a file and then ask the Shell to execute thosecommandsfromthatfile.TheBashbuilt-incommandsource(orsimplyadot.)canreadafileofBashcommandsandprocessthemonebyone.AfileofShellcommandsisknownasaShellscript.Thus,eitherofsourcescript.scriptcauses your interactive Bash to read commands from the given script as
though they were entered from the keyboard individually. Since source is abuilt-incommand,thescriptisnotreadbyasubshell(Section3.2).
SHELLVARIABLESBashusesanumberofspecialvariables,withalluppercasenames,forspecificpurposes.SettingspecialvariablescontrolsthewaycertainBashoperationsarecarriedout.Forexample,settingtheCDPATHtoalistofoften-useddirectoriesenables you to use simple folder names with the cd command (cdsimpleFolderName).BashwillthensearchforthetargetfolderunderdirectoriesontheCDPATH.Besuretoincludethe.ontheCDPATH.SomevariablesthataffectinteractiveuseoftheShellarelistedhere.Otherspecialvariablesaffecting
3.10
theprocessingofShellscriptsarediscussedinChapter5.
ENVIRONMENTOFAPROGRAMThe exact manner in which a program works depends on the executionenvironment withinwhich it is supposed to do the job. For example, the texteditornanoorvimneedstoknowthecapabilitiesoftheterminalemulatoritisdealingwith,andsodoesthecommandmore.Thecurrentworkingdirectoryissomethingalmostallprogramswillwanttoknowwhentheyrun.Forfileaccesspermissionpurposes,anyprogramthataccessesfilesneedstoknowtheuseridoftheuserwhoinvokedit.Theexecutionenvironmentofeveryprocessconsistsoftwo parts: user defined and system defined. The userid, current workingdirectory,openfiles,etc.aredeterminedbythesystemandpassedonfromyourShelltoanyinvokedapplication;whereasquantitiessuchasthehomedirectory,thecommandsearchpath,andthedefaulteditoraredefinedbytheuser.Theseare known as environment variables. Many applications use certain specificenvironment variables of their own; for example, DISPLAY for any GUIapplication, CLASSPATH, and JAVA_HOME for the Java compiler,MOZILLA_HOMEforFirefox,andEDITORformutt.
CommandExecutionEnvironmentA principal task of a Shell is to launch applications by interpreting usercommands. When Bash launches an application, it creates a child process(anotherrunningprogram)andtransmitstoitanexecutionenvironment(Figure3.4)thatincludesthefollowingattributes:
StandardI/OandotheropenfilesCurrentworkingdirectoryFilecreationmask(Section3.12)Environment variables already in the Shell’s own execution environmentandadditionalonesdefinedbytheuser
A child process (an application, for example) is said to inherit its initialenvironmentfromitsparentprocess(theShell,forexample).Anychangesintheenvironmentofthechildprocessdoesnotaffectthatoftheparent.
3.11
Figure3.4ExecutionEnvironmentofaProcess
LetXYZbeanyvariable.YoucanmakeitpartoftheShell’senvironmentbyexportXYZthereforemakingitavailabletoanychildprocesstheShellinitiateslater.Ifa
variable is unset, then it, of course, is also removed from the Shell’senvironment.Insteadofexportingandthenunsettingavariable,youcanaddvariablestothe
environment on a per-command basis.When you issue any regular command,you can set variables in front of the command name to add them to theenvironment passed to the command without affecting the environment of theShellitself.Forexample,ifwestartasubshellwithYEAR=2018bashThesubshellwillhaveanenvironmentvariableYEARsettothevalue2018
whileyourShellremainsunchanged.TheenvironmentvariableTERMrecordstheterminaltype.ForLinuxusers,
TERMismostlikelysettoxterm(XTerminal)byaterminal-windowprogramsuchasgnome-terminal(Chapter3,Section2.8).Thecommandsearchpathisanother environmental parameterwhose value is contained in the environmentvariablePATH.Also,XWindowsclientprogramsusethesettingofthevariableDISPLAY (Chapter 2, Section 2.6). The Bash built-in commandprintenv (orenv)displaysallcurrentlysetenvironmentvariablesandtheirvalues.Hereareafewmorecommonenvironmentvariables.TERM TypeofterminalEDITOR DefaulttexteditorDISPLAY XserverandphysicaldisplaydevicedesignationMANPATH SearchpathforthecommandmanRemember,inBashanyvariablecanbecomeanenvironmentvariablebythe
export command.However, it isgoodpractice touse alluppercasenames forenvironmentvariables.
EXAMPLESOFBASHUSAGE
By studying examples, you can gain a deeper understanding of how the Shellworksandhowthevariousexpansionscanbeused.Almostallexamplesgivenhereareofpracticalvalue,andyoumayconsideradoptinganyorallofthemforyourownuse.
CustomizedPromptTheShelldisplaysapromptwhenitisreadyforyournextcommand.ForGNULinux, thedefaultBashprompt isPS1=’ s- v$ ’,meaning -Shell_base_name-version$.Forexample,-bash-3.2$Thetrailing$isautomaticallyreplacedby#iftheuserisroot.Manyuserschoose tocustomize theprompt todisplaymore information.A
goodexampleisPS1=’u@h:W$’which specifies userid@hostname:current_folder history_number$ and
produces,forexample,thepromptpwang@acerwang:ch03361$You may also set the special variable PROMPT_COMMAND to any
command to be executed before displaying each prompt. See the Bashdocumentationformoreinformationonsettingtheprompt.
RemovingFilesSafelyDeleting files accidentally is not unusual. This is especially true with thepowerful and terse notation of the Shell. It is entirely possible tomistype thecommand
rm*.o(deletesallfileswiththe.osuffix)asrm*.o(deletesall
filesandthefile.o)
byaccidentallytypinganextraSPACEinfrontofthe.o.Itisrecommendedthatyoudefineanaliasaliasrm="rm-i"The -i option requires interactive confirmation before deleting any file.
Considerplacingthisaliasinyour.bash_profile(Section3.13).Someuserspreferanevensaferalternative,movingunwantedfilestoatrash
folderratherthanactuallydeletingthem.YoushouldalreadyhaveaTrashfolderinyourhomedirectoryoryoucancreateonewithmkdir/TrashNow, define a function rm (Section 3.15) that usesmv tomove any given
filesto/Trash(seeExercise20).
Copy,Paste,andI/ORedirectionYoucancombinecopy-and-paste(usingthemouse,seeChapter2,Section2.8)with I/O redirection to make certain operations easier. For example, you canmarkandcopydisplaytextcontaininginformationyouwishtosaveandenteritdirectlyintoafile.Justtypecat>notes.txtandpastethemarkedlinefollowedbyCTRL+Donanewline(tosignalendof
inputtocat).Tomailsomescreenoutputtoanotheruser,simplydocat|mailuserid-ssubjectandthenpastethematerial.
DisplayingManualPagesEach manual page provides a concise description of a Linux command. Themain body of the manual pages is divided into chapters. Although the exactorganization of the chapters may vary with the particular Linux system, thefollowingisatypicalorganization
1. User-levelcommands2. LinuxsystemcallsintheClanguage3. SystemlibraryfunctionsforC,Fortran,networking,andotherpurposes4. Specialfiles,relateddevicedriverfunctions,andnetworkingsupport5. Formatsforexecutableandsystemdatabasefiles6. Miscellaneoususefulinformation7. Linuxmaintenance,operation,andmanagement
Youcanusethecommandmanmanto see the organization of yourmanual pages.To display an introduction to
chaptern,typemannintroTodisplaythemanualforcommand_name,typeman[n]command_namewherethechapternumbernisoptional.A typical manual page contains the following information: NAME (and
principal purpose), usage SYNOPSIS, DESCRIPTION, OPTIONS, relatedFILES,andSEEALSO(relatedcommands).Ifthemanualpageistoolargetofitononescreen,theprogramwilldisplay
onepageatatimeuntiltheentireentryhasbeenshown.YoucantypeqtoquitmanandreturntotheShellprompt.Thisisespeciallyusefulifthemanpageislargeandyoudon’twanttoseeitall.TheSYNOPSISpartofthemanualpagegives a concise description of the command syntax. In the synopsis, certaincharactersaretobetypedliterallywhenusingthecommand;othercharactersorstringsaretobereplacedbyappropriatewordsorfilenamessuppliedbytheuser.Portionsenclosed inbracketsareoptional,and thebrackets themselvesarenotpartofthecommand.Ellipses(…)areusedinthesynopsistoindicatepossiblerepetitions.MostLinuxcommandsreceiveoptions thatmodify thebehaviorofthe command. As mentioned earlier, an option is usually given as a singlecharacterprecededbyadash(-),butmoreverboseoptionsarealsopossible.TheFILESsectionof themanualpagegives the locationsof files related to
the particular command.TheSEEALSO section gives related commands thatmay be of interest. The BUGS section lists some known problems with thecommand.Thecommandmanalsocanperformakeywordsearchthroughthenameand
purpose part of themanual pages, displaying each line containing any of thegivenkeywords.Thisisdonebyusingthe-koptionman-kkeyword...Thisfeatureisusefulwhenyouwanttodosomething,butcan’trememberthe
appropriatecommand.Forexample,tofigureouthowtocopyafile,youcouldtryman-kcopy.Thekeywordcanbeanycharactersequence.Soyoucanfindacommandifyourememberonlyapartofitsnameordescription.There are also Web page versions of the Linux man pages (for example,
linuxmanpages.com)thatcanbemucheasiertouseasareference.AlsoyoumayusetheYelpdocumentbrowseryelp’man:name_of_command’to conveniently view anymanual page. For example, Figure 3.5 shows the
displayofyelpman:chmod.
3.12
Figure3.5BrowsingManpage
SettingUpYourPersonalWebFolderOften,theLinuxsystematschoolortheofficewillalsoservetheWeb.Ifso,theLinuxsystemoftenalsosupportsper-userWebpages.Thismeansyoucansetupapublic_htmlfolderinyourhomedirectoryinthefollowngway:
cd(goestohomedirectory)chmoda+x.(allowsWebserver
access)mkdirpublic_html(createsnewfolder)chmoda+xpublic_html
(allowsWebserveraccess)
Now you may create Web pages (filename.html) in your public_html andmakeeachoneWebreadable:chmoda+rpublic_html/filename.htmlYoucanthenaccessthemovertheWebwiththeWebaddresshttp://hostname/your_userid/filename.html
DEFAULTFILEPERMISSIONSFileprotectionwasdescribedinChapter1,Section1.6.Whenyoucreateanewfile, Linux gives the file a default protectionmode.Often, this default settingdenieswritepermissiontogandoandgrantsallotherpermissions.Thedefaultfileprotection setting iskept in a systemquantityknownasumask.TheShellbuilt-in command umask displays the umask value as an octal number. Theumask bit pattern specifies which access permissions to deny (Chapter 11,Section11.4).The positions of the 1 bits indicate the denied permissions. Forexample, theumaskvalue0022(octal022)hasabitpattern000010010,anditspecifies denial ofwrite permissions for g ando.TheShell built-in command
3.13
umaskalsosetstheumaskvalue.Forexample,umask0077setstheumasktodenyallpermissionsforgando.Ifyoufindyourselfusing
chmodgo-rwxalot(Chapter2,Section2.7),youmightwanttoconsiderputtingumask0077intoyour.bash_profileand.bashrcfiles(Section3.13).
SHELLSTARTUPANDINITIALIZATIONAsmentioned,theShellitselfisauserprogram.Thetermuserprogramreferstoprogramsnotbuilt into theLinuxoperatingsystemkernel.Examplesofkernelroutines are file system routines, memory management programs, processmanagementprograms,andnetworkingsupport.Thecommandsls,nano,mail,andcat,aswellasShellsbash,csh, andsoon,areuserprograms. In fact, allLinuxcommandsareuserprograms.TheloginShellisselectableonaper-userbasisandisspecifiedintheuser’s
passwordfileentryinthepasswordfile/etc/passwd.Thisfilecontainsaone-lineentryforeachauthorizeduseronthesystem.Eachpasswdentryconsistsofthefollowingfields:
Loginname(containsnouppercaseletters)EncryptedpasswordorxNumericaluseridNumericalgroupidUser’srealname,office,extension,andhomephoneUser’shomedirectoryProgramtouseastheShell
The fields are separated by colons (:). For example, a passwd entrymay looklikethefollowing:pwang:x:500:500::/home/pwang:/bin/bashThexpasswordindicatesthatashadowpasswordfileisusedtobetterprotect
andmanageuserpasswords.The/bin/bashat theendspecifies theuser’s loginShell.Immediately after a login window starts, the user’s login Shell is invoked
(Chapter2,Section2.8).The loginShell specified in thepasswdentry canbechangedusingthecommandchsh(changeShell).Forexample,chsh-s/bin/bashwillchangeyourloginShellto/bin/bash.AttheShelllevel,thecommandecho$0
displaysthenameofyourcurrentShell.When a Shell starts, it first executes commands in Shell initialization files,
allowingaLinuxinstallationandindividualuserstocustomizetheShelltosuittheirpurposes.ExactlywhichinitializationfileBashloadsdependsonhowitisinvoked.
LoginBash—Ifbashisinvokedviaaloginwindoworgiventheoption-lor –login, then it is a login Shell. As a login Shell, Bash first loads thesystem-wide initialization file /etc/profile which defines environmentvariablessuchasPATH,USER,HOSTNAME,andTERM.Thenitloadsaper-user initialization file which is the first of .bash_profile, .bash_login,and .profilefoundin theuser’shomedirectory.Theper-userclean-upfile.bash_logoutisexecutedwhenaloginBashexits.Non-logininteractiveBash—WhenBashisrunfromthecommandline,itisaninteractiveShell(withstandardI/Oconnectedtotheterminalwindow)butnota loginShell.SuchaBash loads thesystem-wide /etc/bash.bashrcfirstandthenloadstheper-user/.bashrc.Non-interactiveBash—Bashstartedtorunacommand(bash-ccmd)orascript (Chapter 5) is non-interactive. Such a Bash does not load any initfiles by default. It will load a file specified by the environment variableBASH_ENV.
There are some differences among Linux distributions on Shell initializationfiles. For example, CentOS/Fedora/Red Hat also provides the system-wide/etc/bashrcfileforuserstoloadifdesiredwithaconditionalexpression:
if[-f/etc/bashrc];then./etc/bashrcfi
Notethatthe.commandisthesameassource.WritingBashprogramsisthetopicofChapter5.Amongotherthings,the/etc/bashrcusuallysetstheumasktoadefaultvalue
(Section3.12). It is agood idea to include /etc/bashrc ifyour systemprovidesone.Hereisasample.bashrcfile.
#Sourcesystemdefinitionsif[-f/etc/bashrc];then.
/etc/bashrcfiset-onoclobberumask0007
The.bashrcisusuallyincludedinthe.bash_profile,whichaddsothersettingsimportant for interactive use of the Shell. Figure 3.6 shows a sample.bash_profile(Ex:ex03/bash_profile).
3.14
Figure3.6ASample.bash_profile
A non-interactive Bash is a subshell, and the execution of any Bash script(Chapter 5) involves a subshell Bash. Therefore, the setting for aliases,functions, and PATH used for Shell procedures ought to be placed in .bashrcinsteadofin.bash_profile.
SHELLSPECIALCHARACTERSANDQUOTING
TheShell usesmany special characters in establishing the command languagesyntaxandaskeysforthevariousexpansionsprovided.Someoften-seenspecialcharactersarelistedinTable3.6.BashSpecialCharacters
SpecialcharactershelpachievemanyShellfunctionalities.However,becausetheShell interpretsaspecialcharacterdifferentlyfromaregularcharacter, it isimpossible for a special character to stand for itself unless additional
arrangementsaremade.Forexample,ifthereisafilenamedf&g.c,howcanyourefertoitinaShellcommand?Thesolutiontothisproblemistheuseofmorespecialcharacters,knownasquotecharacters.IfyouaregettingtheimpressionthattherearemanyspecialcharactersinLinux,youareabsolutelyright.Infact,any character on the keyboard that is not alphabetic or numeric is probablyspecial in someway.Notable exceptions are the period (.) and the underscore(_).
QuotinginBashBash provides the backslash ( ) escape character, single quotes (‘...’), doublequotes(“...”),andANSI-Cquotes($‘...’).Thecharacterquotesorescapesthenextcharacter.Forexample,nanof&g.candgrepUS$report.*Thecharacters&and$losetheirspecialmeaningwhenprecededbyİnstead,
theystandfortheliteralcharactersthemselves.Ifaspaceortabisprecededbyathenitbecomespartofaword(thatis,itlosesitsspecialmeaningtodelineatewords). If theNEWLINE character isprecededbya it isequivalent toablank.Thus,usingaattheendofalinecontinuestheShellcommandtothenextline.Togetthecharacterwithoutescapingthenextcharacter,use.Whereas theescapes thenextcharacter,apairofsinglequotationmarks(’)
quotestheentirestringofcharactersenclosed.
echo´a+b>=c*d´
When enclosed by single quotation marks, all characters are escaped. Thequotedstringformsallorpartofaword.Intheprecedingexample,thequotedstringformsonewordwiththespacesincluded.Thecommand
cat/user/pwang/´my>=.c´
is used to type out a C program in the file /user/pwang/my > =.c. In thisexample, thequotedstringformspartofaword.To includeasinglequotationmarkinastring,theisused,asinechoIt’sagooddayThefollowingrulessummarizequotationwithsinglequotationmarks:
1. All quoted characters, including are taken literally. Thus, escaping thesinglequotewithbackslashwithinasingle-quotedstringdoesnotwork.
2. Thequotedstringformspartorallofoneword.
3.15
Sometimes it is desirable to allow certain expansions within a quoted string.Quotingwithdoublequotationmarks (") serves thispurpose.Apairofdoublequotation marks functions the same as a pair of single quotation marks withthreedifferences:
First, variable and history expansions are performed within doublequotationmarks;thatis,variableexpansionkeyedbythe$signandhistoryexpansions keyed by the ! signworkwithin double quotationmarks. Forexample, echo "your host name is $HOST" echo "Last command is !-1"workasexpected.Second, command expansions are allowed inside double quotationmarksand are treated slightly differently from normal command expansions.Normally, the output of a command expansion, via $(...) or ‘...‘ (Section3.7),isbrokenintoseparatewordsatblanks,tabs,andNEWLINEs,withnullwords being discarded; this text then replaces the original backquotedstring. However, when command expansion is within double quotationmarks,onlyNEWLINEsforcenewwords;blanksandtabsarepreserved.Thesingle,finalNEWLINEincommandexpansiondoesnotforceanewwordinany situation. For example, date=‘date‘ and datestring="‘date‘" aredifferentinthat$dateincludesmultiplewords,but$datestringisoneword.Third, escaping " with backslash within a double-quoted string works.Actually,withinadouble-quotedstring,thebackslash()escapesonly$,‘,", orNEWLINE.Within a double-quoted string, the combination escapeshistory expansion, but the backslash is not removed from the resultingstring.
Now,westillneedaneasywaytoincludehard-to-keyboardcharactersinstrings.ThisiswheretheANSI-Cquotesareuseful.Astringintheform$’str’allowsyou to use ANSI-C escape characters in str. For example, you can useBACKSPACE),f(FORMFEED),n(NEWLINE),andsoon.Forexample,alias$’f’=cleardefines a convenient alias, allowing you to clear your terminal screen by
typingCTRL+Lasacommand.
SIMPLEFUNCTIONSYoucantakeahard-to-entercommandorasequenceofcommandsforacertaintask and build a function tomake the task easy.To define a function, use thesyntax
functionfnName(){command1;command2;...commandn;}
A command in a function can be a Shell built-in command, a regularcommand, or a call to another function.Aliases don’twork inside a function.Eachcommandinthefunctiondefinitionmustbeterminatedbyasemicolon.Somewhitespacebetweentheandcommand1isnecessary.Oncedefined,youcanuse the functionnameas a commandnameandalso
passthefunctionarguments.Forexample,
functionoffice(){/usr/bin/libreoffice$1;}
defines the function office.You can then invoke libreoffice on a documentwiththecommandofficenote.docThespecialvariable$1inthefunctiondefinitionreferstothefirstargumentin
the function call. In general, the positional parameters $1, $2, ... are used toaccessargumentspassedinafunctioncall.Infact,thekeywordfunctionisnotnecessaryif()aregiven.Forexample,
dir(){ls-lF--color=auto--color=always"$@"|less-r;}
givesyouaDOS-likedircommand.3Thespecialvariable$@referstoalltheargumentsinthefunctioncall(Chapter3,Section5.3).AfunctionisnormallynotinheritedbychildShellsunlessitisexportedwith
export-ffunctionName.Youcanremoveafunctionwithunset-ffunctionNameanddisplayallfunctionswithdeclare-fThereisnobuilt-incommandtodisplayaspecificfunction,butthefollowing
functionwilldothejob
functionwhich(){(alias;declare-f)|\/usr/bin/which--tty-only
-i\--read-functions$@;}
Thepairofparenthesesaround(alias;declare-f)groupscommandsjustlike,except it calls for a subshell to execute the commands. The stdout of thatsubshellisfedtothe/usr/bin/whichcommand.Withthisfunctiondefined,thecommandwhich fnamewillnowdisplayany
aliasor functiondefinitionfor thegiven fname. If there isnosuch functionoralias,itwillalsolookforfnameon$PATH.Thespecialvariable$@evaluatestoall the arguments passed to the function. Also note we used /usr/bin/which
3.16
3.17
instead of just which because it is not our intention to call the functionrecursively.4Hereisthedisplayproducedbywhichwhich.
which(){(alias;declare-f)|/usr/bin/which--tty-only-i\--
read-functions--show-tilde--show-dot$@;}
MorewillbesaidaboutfunctionsinChapter5,Section5.18.
FORMOREINFORMATIONYoucanusetheBashcommandhelp|moretogetalistingofbuilt-incommandsandhowtogetmoredetailsonthem.TheBashmanpagemanbashisagoodreferenceontheBourne-AgainShShell.TheBashManualfromGNUcanbefoundatwww.gnu.org/software/bash/manual.
SUMMARYRunning in a terminal window, the Bash Shell provides a CLI to your Linuxsystem.You interact with the Shell via the input-processing-execution-promptcycle.Thecommandlinegoesthroughawell-definedsetofexpansionsbeforegettingexecuted.AShellbuilt-incommandiscarriedoutbytheShellitself.Anon-built-inorregularcommandinvolveslocatinganexecutableprograminthefile system, running it in a child process, and passing to it any command-lineargumentsandanyenvironmentvalues, includingexportedShellvariablesandfunctions.Acommandnamecanbeeitherasimplenameorapathname.Intheformer
case,thecommandmayinvokeaShellaliasorfunctionifoneexists.Otherwise,thecommandisfoundbysearchingthroughthecommandsearchpath—alistofdirectoriesgivenbytheenvironmentvariablePATH.I/Oredirectionenablesyoutodirectthestdin,stdout,andstderrofcommands
to/fromfilesandothercommands(formingpipes).Jobcontrolmakesitpossibletostartmultipletasks,suspendthem,puttheminthebackground,orbringanytotheforegroundtoreassertterminalcontrol.Entering of input is helped by input editing, TAB-completion, history
3.18
substitution,andfilenameexpansion.Bash loads initialization files at start-up time. It is important to keep your
favoritesettingsintheappropriateinitfiles.bashrcand.bash_profile.TheShellusesmanyspecialcharacters,suchas*,=,(),[],blanks, ;,andso
on. Quoting with single and double quotes and character escaping with arenecessarytocountertheeffectsofsuchcharacters.Thisisespeciallyimportanttorememberwhenissuingcommandsthatrequiretheuseofsuchcharacters.Bash also supports function definition. A function becomes a new built-in
command. A function can take arguments and access them as positionalparameters.IfyoulikeShellaliases,you’lllovefunctions.MoreaboutfunctionscanbefoundinChapter5.
EXERCISES1. ThecommandcdisbuiltintotheShell.Whycan’titbeimplementedasa
regularcommand?2. Findanddescribeawaytodoakey-wordsearchoftheLinuxmanpages.3. WherecanyoufinddocumentationforacommandbuiltintoBash?4. Consider thespecialdirectorysymbol .and its inclusionon thecommand
search path ($PATH).What difference does it make if you do or do notinclude .? If youdo include .,where should it be placed relative to otherdirectorynamesonthesearchpath?Why?
5. You have written a program that takes input from stdin and writes it tostdout.Howcouldyourunthisprogramifyouwantedinputtocomefromafilenamedinandoutputtobestoredattheendofafilenamedoutandanyerrortostderrberecordedinafilenamederrlog?
6. Whatifyouwishtohavestdoutandstderrsenttothesamefile?7. Johnwantedtoappendthefilefbtotheendofthefilefa,sohetypedcatfa
fb>|faWhatreallyhappenedhere?Howwouldyoudoit?8. Johnthenwantedtosendaline-numberedlistingoffilefatotheprinter.He
typedcat-nfa>lprbutnoprintoutappeared.Why?Whathappenedhere?9. Johnmadeatyposrotfile1file2SpecifytwowaysusingtheShellhistory
mechanismtocorrectsrottosortandreissuethecommand.10. HowdoesonesettheeditorusedinBashcommand-lineediting?Showthe
code.11. NameatleasttwocommandsthatarebuiltintoBashbutalsoareregular
Linuxcommands.12. Give a command to edit, using nano, every file in the current directory
1234
whosefilenameendsin .txt thatcontainsthestringLinux.(Hint:considerthe-loptionofgrep.)
13. What isa foreground job,background job,andsuspended job?Howdoesonedisplayalistofalljobs,orswitchfromonejobtoanother?
14. HowdoyouexitfromyourinteractiveShell?Specifyatleastthreeways.15. WhathappensifyouexitfromyourShellandthereareunfinishedjobs?16. Explain the difference between these two commands: ls chap[0-9] ls
chap0..917. What is command expansion in Bash? Give the two notations used for
commandexpansion.18. WhatisstringexpansioninBash?Explainandgivetwoexamples.19. Try
country="usa";echo${country^^}
FindoutmoreaboutBashcasemodificationinvariableexpansion.Explainandgivetwoexamples.
20. Consider the twoBash initializationfiles: .bashrcand .bash_profile.Whatinitializationcommandsshouldbekeptinwhich?Why?
21. What is the syntax for function definition in Bash? After defining afunction,canyouundefineit?How?
22. InBash,whatarepositionalparametersofafunction?Howdoyouexportafunctionintotheenvironment?Whatgooddoesitdo?
23. WriteaBashfunctionrm tomoveitsargumentfilestothe/Trashfolder.(Hint:Usemv-i.)
24. Explainthecodeman()yelp"man:$@";andgiveusageexamples.25. Find the Linux version running on your computer. (Hint: The uname
command.)ThevalueoftheShellvariable$HOMEisthefilenameofyourhomefolder.NotethatnanoignoresCTRL+Zunlessgiventhe-zoption.NotethatLinuxalreadyhasaregularcommanddirforlistingdirectories.IfyourBASHShellcomeswithanaliasforwhich,unaliasitsoyoucanreachthefunction.
Chapter4
PuttingCommandsandApplicationsto
Use
Linuxoffersarichsetofcommandsandapplications,todoalmostanythingyoudesire.EffectiveuseofLinux involvesknowingwhatappsandcommandsareavailable.Through software searchandpackagemanagement, apps, aswell assystemsoftware,canbeeasilyfound,installed,updated,andremoved.Furthermore, existing commands can be combined easily to form new ones
eitheronthecommandlineorinShellscripts(Chapter5).WewilldiscussasetofoftenusefulGUIappsandCLIcommands.We’llalso
show you how to combine commands into new commands, and selecting therightcommandstoapply.Throughoutthisbook,wewillintroducemanyusefulLinuxcommandsanddemonstratehowtheycanbeputtogooduseindividuallyandincombination.Many commands are filters. A filter usually performs a simple and well-
defined transformation of its input and follows certain conventions tomake iteasy to connect to other programs. Filters can be strung together using pipes(Chapter3,Section3.5)tobecomepipelinesthatcanperformcomplexfunctionsondata.Manyusefulfiltersarepresentedinthischapter.Examplesshowhowtobuildpipelinesinpractice.For instance, the command sort is a filter that orders its input lines. The
commandtrtranslatesspecificcharactersintheinputintoothercharacters.Youcan combine these two filters with others to create and maintain a simpledatabaseofaddresses.Utilizingandprocessinghumanreadabletextualdatahavebeenanemphasis
ofLinux.Withintextualdata,weoftenneedtoidentifytheexactplaceswheretransformationsormanipulationsmust takeplace.Regularexpressions provide
4.1
standard ways to specify patterns in textual data. It is important to becomefamiliarwithregularexpressionsbecausetheyoccurfrequentlyandarebasictoprogramming. We explain the regular expression notations and how they areusedinapplicationssuchasgrepandsed/vi.
USEFULGUIAPPSLinuxoffersalargenumberofapps.Usersjustneedtoknowwhat’savailable,makesureanappisinstalled,andthenlearntouseit.GUI tools for finding, installing and managing apps, such as GNOME
Software (gnome-software) and Ubuntu Software Center (software-center)whereappsaregroupedintosearchablecategories,makeappmanagementeasy(Figure 4.1). In addition, there are also the command-line oriented DNF andAPTpackagemanagers(Chapter8,Section8.2).
Figure4.1GNOMESoftware
Wewilltalkaboutafewappsthatcanoftenbehandy.
WordProcessingForschoolorwork,wordprocessingisafrequenttask.OnLinuxwedon’thavethe expensive Microsoft Office but we do have Apache OpenOffice andLibreOfficethatarefree.Thelibreofficecommand(Figure4.2)comesinstalledonmanyLinuxdistributions.
Figure4.2LibreOffice
Use the free software forwordprocessing, spreadsheets, slidepresentations,drawings, editing PDF files, import and export documents from/to differentformatsincludingtext,PDFandMicrosoftformats.To view PDF on Linux you can also use Okular, Evince, or Foxit Reader.
Often evince is pre-installed and the default viewer for PDF. To select, split,mergeanddeletepagesofaPDFfile,considerPDFmod.Also,qpdfcanselectand combine pages from one or more PDF files as well as add passwordprotectiontoPDFfiles.
DocumentFormattingandTypesettingTo prepare larger documents such as technical papers and even books, theLaTeX system is often helpful and may even be required by scientificconferences and publishers. LaTeX is a high-quality typesetting system and amarkuplanguagethatallowsyoutouseplaintext tomarkupdocument layout,font,color,andmore.Forexample,thefilemydoc.texEx:04/mydoc)
\documentclass{article}\title{ASampleLaTeXDocument}\author{Paul
S.Wang}\date{2018-09-
01}\begin{document}\maketitle\section{Introduction}Moretexthere
...\section{Background}Moretexthere...\end{document}
canbeprocessedbythecommand1pdflatexmydoc.textoproducemydoc.pdfasdisplayedinFigure4.3.
Figure4.3ALaTeXProducedDocument
In addition to all the usually expected document formatting capabilities,LaTeXexcelsinfeaturessuchas
ExperttypesettingofmathematicalformulasAutomaticnumberingofchapters,sections,pages,tablesandfiguresAutomaticcrossreferencingGenerationoftableofcontentsandindices
InfactthisverytextbookhasbeensetinLaTeXusingatemplateprovidedbythepublisher.HereishowtoinstallLaTeX(chooseeithertexliveortexstudio):dnfinstalltexlive-scheme-fullapt-getinstalltexlive-scheme-fulldnfinstalltexstudioapt-getinstalltexstudio
DrawingandDiagrammingTools for drawing points, lines, arrows, rectangles, circles/ellipses, and othercurves to form diagrams often use vector graphics where coordinates, angles,anddistancesareusedtoformdrawings.Unlikerastergraphicswherepixelsareusedtoformshapes,vectorgraphicscanscaledrawingsupordown(zoominorout)withoutlosingclarity(becomingpixelated).TheLibreOfficeDrawdoesaverygood jobasadrawinganddiagramming
tool. The dia (Figure 4.4) is a Microsoft Visio like program very good formakingmanykindsofdiagrams.
Figure4.4DiagrammingwithDia
Xfig is another simple and efficient tool for diagram drawing. Inkscape(Figure4.5)isapowerfultool,notunlikeAdobeIllustratororCorelDraw,givingyoutheabilitytocreatebeautifuldrawingsbasedonvectorgraphics.
Figure4.5Inkscape
ThesK1isacapablevectorgraphicstooltopreparehighqualitydrawingsforprofessionalpublishing.Flowisanotherappforflowcharts,networkgraphs,andsoon.Asymptote isavectorgraphics languageandsystemallowingyou tospecify
drawings using plain textual instructions and producing graphics in files ofdifferent formats (Postscript,PDF,SVG,3DPRC)aswell as forviewing.Forexample,thetextfileline.asy(Ex:ex04/line.asy)
draw((0,0)–(50,50));says todrawa straight line from theorigin to (50,50).The file can thenbe
processedbythecommandasy:asy-Vline(producesline.epsanddisplaystheresult)asy-fpdfline(producesPDFfileline.pdf)
RasterGraphicsandImageProcessingOnLinux you can use eog (Eye of Gnu) for viewing photos and shotwell tomanagethem.Perhaps the best known and most widely used raster image creation and
processing application on Linux is GIMP (the GNU Image ManipulationProgram). Comparable to Adobe Photoshop, GIMP is feature rich and verycapable. Additionally, GIMP is designed to be augmented with plug-ins andextensions.Figure4.6 shows thegimp commandbeingused todesign the topbannerforthisbook’swebsite.GIMP can also directly create an image to process by taking a screenshot.
Alternatively you can use the PRINTSCREEN key (for the entire screen),ALT+PRINTSCREEN(forthecurrentwindow)andSHIFT+PRINTSCREEN(foracursor-selectedarea).The result image isplaced inyourPictures folder.Thegnome-screenshotcommandgivesyoumorecontrolovermakingscreenshots.Alsotheimportcommandcanbehandyforscreencaptures.
Figure4.6ApplyingGIMP
ForscanningyoumayusetheeasySimpleScan(simple-scan)orthefancierXSane(xsane).
FileUploadandDownloadForfileuploadanddownloadfromthecommand linewecanuse ftpandsftp(Section5.20).
Wealreadyknowthatnautilussupportsbothlocalandremoteaccessoffiles(Section2.7).FileZillaisaGUItoolforFTPandSFTPwhichcanbeeasiertouseforbeginnersandoccasionalusers.InstallFileZillawithdnffilezillaapt-getfilezilla
Figure4.7FTPToolFileZilla
andinvokeitwithfilezilla(Figure4.7).
PasswordManagerWeallhave toomanyaccountsanddevicesrequiring login toaccess.Keepingall those userids and passwords safe and easy to access becomes a problemneedingasolution.You can use a Web browser’s auto-login feature, where your browser
remembersyouruseridsandpasswordsfordifferentwebsites.But,youmustfirstset a browser master password to protect the saved login information fromotherswhomay gain access to your browser. Select your browser’s advancedsecurityoptiontosetitsmasterpassword.There are a number of password manager programs designed to store and
retrievepasswordssecurely. IfyouareusingGnome, theSeahorse tool,whichcansavenotonlypasswordsbutalsoencryptionkeys(Chapter7,Section7.10),isusuallyalready installed.Look forApplications>Utilities>Passwords andKeysorgivethecommand
seahorse
Figure4.8TheSeahorseTool
to launchSeahorse(Figure4.8).SelectLoginandclick the sign toaddnewentries.Click the lock icon to lock/unlockstored information.Makesureall islocked before you quit Seahorse. To unlock something in Seahorse, use yourLinuxloginpassword.Anotherablepasswordmanageriskeepassx2.Toinstalldodnfinstallkeepassxapt-getinstallkeepassx
CloudStorageStoringfilesandfoldersinthecloud(onserversovertheInternet)cangiveyouadditionalstoragespace,makesharingfileswithotherseasy,andprovideawaytobackupimportantfilesforsafety.ForLinux,oneparticularcloudstorageservicestandsout,namelyDropBox.A
freeaccountgivesyou2GBofstorage that isaccessibleon theWebandfromyourlocalfolder$HOME/Dropbox.InstallingDropBoxissimple:dnfinstalldropboxapt-getinstallnautilus-dropboxOnceinstalledyoucanlaunchDropBoxwiththecommanddropboxstart-iwhichwill leadyou to install theDropBoxdaemon (dropboxd) and to link
yourLinuxwithyourDropBoxaccount.Adaemonisaprogramrunninginthebackground ready to instantly provide a specific service. A Linux systemnormallyhasmanydaemonstosupportmanyusefulservices.
Now you can use your $HOME/Dropbox folder which is automaticallymirrored on your DropBox cloud storage. Read the Getting Started guideincludedinthesamefolderforusageinformation.Manyothercloudstorageoptionsareavailable forLinux includingAmazon
S3,GoogleCloudStorage,SpiderOak,andSeaFile.
3DModelingand3DPrintingFree apps for 3Dmodeling and 3D printing are also available on Linux. Thepowerful Blender supports animation, modeling, game development, 3DPrinting,andmore.Installitandyou’llhavetheblendercommand.While Blender is more for artistic projects, freeCAD helps you make
parametric 3Ddesigns in engineeringprojects.FreeCADcan input andoutputmanystandardfileformatsandcanbeprogrammedusingaPythoninterface.ItisagoodchoiceforbeginningandexperiencedCAD(ComputerAidedDesign)engineers.Figure4.9showsascreenshotfromtheFreeCADwebsite.
Figure4.9FreeCAD
MathematicalCalculationsThe expr command can perform simple calculations and comparisons withintegers.Seemanexprfordetails.
4.2
Figure4.10GnomeCalculator
For a desktop calculatorgnome-calculator is handy for interactive use andprovides several different input modes (Figure 4.10). The gcalccmd is acommand-lineversion.For awonderfully powerfulmathematics tool, considerMAXIMA, a freely
availablesymboliccomputation system derived fromMIT’sMacsyma (ProjectMAC’s SYmbolicMAnipulator) which is a general purpose computer systemdesigned to perform exact aswell as approximatemathematical computations(Figure4.11).MAXIMA offers an impressive collection of mathematics capabilities that
rivals well-trained engineers and mathematicians. Part of the author’s Ph.D.work contributed to the development of Macsyma including polynomialfactorization and GCD, complex numbers, limits, definite integration, linearalgebra,FortranandLaTeXcodegeneration.Seetheonlinedemosatthebook’swebsiteforacloserlook.
Figure4.11MAXIMA
ItissimpletoinstallMAXIMA.dnfinstallwxmaximaapt-getinstallwxmaximaThen use wxmaxima to invoke the GUI version ormaxima for the CLI
version.Now let’s turn our attention to command-line applications and how to put
themtouse.
COMMANDSANDFILTERS
Simply put, a filter is any command that produces output by transforming itsinput by following a set of well-defined conventions. The conventions makefilterseasytocombinewithotherprogramsinapipeline(Figure4.2).
Figure4.12AFilter
A filter is distinguished from other commands by the followingcharacteristics:
1. Afiltertakesinputfromthestandardinput(stdin).Thus,whenweinvokeafilter,itdoesnotneedafileargument.
2. Afiltersendsitsresultstothestandardoutput (stdout).Therefore, itdoesnotneedanoutputfileargument.
3. Afilterperformsawell-defined transformationon the inputandproducestheoutputwithnoheader,trailer,label,orotherformatting.
4. Afilterdoesnotattempttointerpretitsinputdatainanyway.Thus,itnevertreatsitsinputasinstructionsorcommands.
5. Withfewexceptions,a filterdoesnot interactwith theuser foradditionalparametersotherthanthosesuppliedonthecommandline.
6. Anyerrorordiagnosticoutputproducedbyafilter issent tothestandarderroroutput (stderr).Hence, errormessages are nevermixedwith resultsproduced.
Thesecharacteristicsmakeafiltereasytofitintoapipeline.Theoverallpurposeis tomake a programproduce output that can be fed into another program asinputandthatcanbeprocesseddirectly.Typically,suchinputcontains linesoftextwithnodecorativelabels,comments,orextraformatting.Aseparatelineisused for each data entry. For example, if the data entries arewords, then theinput should be one word per line. For more complicated data entries (forexample,thoseproducedbyls-l),thelinemayconsistofseveralfieldsseparatedbyspaces,tabs,orcolons(forexample,/etc/passwd).ManyLinuxcommandsarefiltersthatcanalsoworkonfiles.Theconvention
is If filenames are supplied as arguments, a command can use them forinput/output.Otherwise,ifnofilesaregiven,thecommandactsasafilter.The process expansion (Chapter 3, Section 3.7) feature of Bash makes it
possibletotreatoutputfromfiltersasinputfilestoothercommands.
Let’slookatsomefiltersandthenshowhowtobuildpipelineswiththem.
LeadingandTrailingLines:headandtailThecommandsheadandtailareavailablefordisplayingtheleadingandtrailinglinesofafile,respectively.Thecommandhead[-k][file...]outputsthefirstk(default10)linesofeachgivenfiletothestandardoutput.If
no file argument isgiven, the standard input isused.Thehead command is aquickway to examine the first few lines of a file, which are often all that isneeded.The command tail is the opposite, displaying the last part of a file on the
screen:tail[starting-point][file...]outputsthelastpart(fromstarting-pointtotheendor,bydefault,thelast10
lines)ofeachgiven file. Ifno file is specified, the standard input isused.Thestartingpointisspecifiedas+k(linekfromthebeginning)-k>(linekfromtheend)If the integer k is followed immediately by the characters b or c, tail will
countblocksorcharacters,respectively,insteadoflines.The-foptioninstructstail to continue, even after the end of the file has been displayed, repeatedlyprobingthefileincasemorelinesareappended.Thisoptionprovidesawayofmonitoringafileasitisbeingwrittenbyanotherprogram.Inpipelines,headandtailareusefulforselectingsomelinesfromtheinput
and excluding others. Themore (less) command can be used at the end of apipelinetomanagelongoutput.
CharacterTranslation:trThe command tr copies standard input to standard output, substituting ordeletingspecifiedcharacters.Forexample,
trA-Za-z<file1>file2
creates file2 as a copy of file1, with all uppercase letters translated to thecorrespondinglowercaseones.Anotherexampleis
tr'tab'%<file1>file2
where TAB must be escaped by CTRL+V when typing this command. ThismethodallowsyoutoseeeachTABinfile1asa%character in file2 (assuming
file1doesnotcontainany%characters).Generally,trstring1string2translates string1 characters to the corresponding % string2 characters,
assumingthetwostringsareofthesamelength.Ifstring2isshorter,itistreatedas if itwerepaddedwithenoughrepetitionsof its lastcharacter tomakeit thesamelengthasstring1.Arangeofcharacterscanbegiven,asinx-y.AcharacteralsocanbegivenbyitsASCIIcodeinoctal(forexample,040for%SPACE,011forTAB,and012forNEWLINE).Forexample,toreplaceastringofblankswithaNEWLINE,usetr-s’040011’’012’The-s(squeeze)optionshortensallstringsofconsecutiverepeatedcharacters
instring1 to justonecharacter.The -c (complement)option isused to specifystring1bynamingcharactersnotinit.Thus,tr-cs0-9A-Za-z’012’createsalistofallwords(oneperline)intheinput.Inthisexample,string1is
allcharactersexceptnumeralsandletters.Whentheoption-d(delete)isgiven,charactersinstring1aredeletedfromthe
output,andthereisnoneedforstring2.Forexample, toridtheinputofallCRcharacters,wecanuse
tr-d"\015"<file
TabExpansionTabs often need to be expanded into an equivalent number of spaces or viceversa. However, this transformation is not performed by tr because each TABmustbereplacedbyjustenoughspaces tomovetheoutputcolumnpositiontothenexttabstop.Tabexpansionanditsinversetransformationareslightlymorecomplicatedthansimplecharacter-for-characterreplacement.Thefiltersexpand(substitutesspacesfortabs)unexpand(substitutestabsforspaces)areusedforthesepurposes.Forexample,expand-t6<filereplaceseachTABinfilebyspaces,assumingthatTABstopsare6(default8)
spacesapart.
FoldingTextLinesIt issometimesnecessarytomakesure linesof textarewithinacertain lengthfor easy display, viewing, or printing. The fold filter breaks up long lines byinsertingaNEWLINEcharacterwherenecessary.
fold<fileThedefaultistolimitlinestoalengthof80characters.Usefuloptionsinclude-wn(setswidthtoncolumns)-s(breakslinesonlyatspaces)Forexample,fold-w72-sreport>new_reportcreatesnew_report as a version of report with all lines folded at spaces to
within72characters.
CalendarRemindersbyEmailOnLinuxyouhavefancyGUItoolssuchasEvolution,California,andLightningforThunderbird.However,usingasimplepipeline,thereisalsoanotherwaytogetcalendar remindersandhave themsent toyoubyemail thedaybefore thetargetevent.The calendar command is a filter that reads a calendar file (./calendar or
$HOME/.calendar/calendar)andwritesallevents for todayand tomorrow.ThecalendarfilelistsoneeventperlineintheformdateTABanytextdescriptionForexample,
4/15paytax1/21Reminder:ChineseNewYear1/28/201709/01
reminder:Sister'sbirthday(9/08)Saturdayweeklyfilebackup
Thepipelinecalendar|mail-sMyReminderemailAddresssendsthedesiredreminderbyemail.Seethecalendarmanpageforpossible
dateformats.
SortingTextLinesDataoften are sorted in somekindoforder for easyaccess andmanipulation.Youmay want to alphabetize a list of names and addresses, combine severalsuch lists into one, look an entry up in a list, or compare two lists already inorder.Thesortcommandtakesinputlinesandwritesthemtothestandardoutputin
sortedorder.Theunitsbeingsortedareentirelines.Eachlinemaycontainoneormore fields,which are separated by one ormore blanks (spaces or tabs). Forexample, a file called students (Ex: ex04/students)may contain the followinglines:
F.Smith213.75PhysicsJ.Wang232.00AccountingR.Baker203.20
ChemicalEngineeringS.Doe243.20BusinessP.Wang224.00Computer
Science
The first line contains five fields (separated by white space); the third linecontains six fields. The sort command allows you to use field positions tospecifysortkeysfororderingthelines.Asortkeyisdefinedbyastartingandanendingfieldpositionina line.Thesortkeys indifferent linesarecomparedtoorderthelines.Thus,ifyouspecifythesortkeyasthesecondfieldtosort thefilestudents,
thenthelineswillbeorderedbylastnameusing,bydefault,theASCIIcollatingsequence. In the absence of any specification, the sort key is the entire line.Multiplesortkeysaregiveninorderofimportance.Incomparinganytwolines,sortusesthenextsortkeyonlyifallprevioussortkeysarefoundtobeequal.Thecommandhasthegeneralformsort[options][–key=key...][file...]Alllinesinthegivenfilesaresortedtogether.Afilenamed“-”isthestandard
input. Ifnofile isgiven,sortuses thestandard input. Itwrites to thestandardoutputbydefault.Keysaregiveninorderofsignificance.Akeyisgivenbytwofieldpositions:begin[,end]which specify a sort key consisting of all characters between thebegin and
endpositions(fieldseparatorsexcluded).Whenomitted,endbecomestheendofline.Eachpositionhastheformf[.c]where f is a field number, and the optional c is a character number. For
example, the position 2.3 indicates the third character of the second field. Ifomitted,cis1.Thus,theposition3isthesameas3.1.Table4.1providessomeexamplesofsortkeyspecifications.SortKeys
Therefore,thecommandsort–key=2,3.0studentssortsthefilestudentsbylastname.Inthisandmanyothercases,theending
fieldcanbeomittedwithoutaffectingthesearch.SortkeysarecomparedusingASCIIordering,unlessoneofseveraloptionsis
4.3
used.Afewimportantoptionsarelistedhere:f Treatsalluppercaselettersaslowercaseletters
nSortsbyincreasingmagnitudeusingaleadingnumericalstringinthesortkeywherethenumericalstringmayhaveleadingblanksand/orasignfollowedbyzeroormoredigits,withanoptionaldecimalpoint
r ReversesthesenseofcomparisonsandsortsthelinesinreverseorderThese option characters can be given globally, affecting all sort keys, or
immediately after a key specification to affect only that sort key. Note someexamples:
ls-l|sort-n--key=5,6.0(sortbyincreasingbytecount)ls-l|
sort--key=5,6.0nr(sortbydecreasingbytecount)
Formultiplesortkeys,considersort–key=4,4.4nr–key=5studentswhich sorts by grade point average (4th field), highest first, and break ties
withthesecondkey,thedepartmentname(field5toendofline).Seemansortformoreinformation.
THEGREPCOMMANDThe grep command is a filter,fgrep, that provides the ability to search andidentify files containing specific text patternsor to find all lines ingiven filesthatcontainacertainpattern.Thecommandhasmanypossibleapplications.Youmay search for a name, a subject, or a phrase.Youmay search for somethingcontained ina filewhose filenameyouhave forgotten,oryoucanextract textlinesfromfilesthatpertaintoaparticularsubject.Thegrepfilterisoftenusefulinpipelines.Forexample,lookmen|grepgitisisacutewaytofindtheword“meningitis.”Thenamegrepcomesfromgeneralizedregularexpressionswhichareexactly
what grep uses to specify search patterns. The general form of the grepcommandisgrep[options][patterns][files]It searches for the given regular expression patterns (Section 4.4), using a
fairly efficientmatchingalgorithm, in thegiven files andoutputs to stdout thematchinglinesand/orfilenames.Makingitflexible,manyoptionscontrolhowexactlygrepworks.OptionsofthegrepCommand
Agrepcommandsearches thespecified filesor standard input for lines thatmatchthegivenpatterns.Alinematchesapatternifitcontainsthepattern.Eachmatched line is copied to the standardoutputunless specifiedotherwisebyanoption(Table4.2).Theoutputlinesareprefixedwithafilenameifmultiplefilesaregivenasarguments.Generallyspeaking,thegrepcommandisusedeithertoobtainlinescontainingaspecificpatternortoobtainthenamesoffileswithsuchlines.Forexample,let’ssayyouhaveafileofphonenumbersandaddresses.Each
lineinthefilecontainsthenameoftheperson,aphonenumber,andanaddress.Let’snamethisfilecontacts(Ex:ex04/contacts).Afewtypicalentriesfollow:
(330)555-1242BobSmithC.S.Dept.UnionCollege.StowOH
44224(415)555-7865JohnGoldsmithP.O.Box21951PaloAltoCA
94303(415)555-3217BertLin248HedgeRdMenloParkCA94025(617)
555-4326IraGoodman77Mass.Ave.CambridgeMA02139
Considerthecommandgrep-FstringcontactsorequivalentlyfgrepstringcontactsIfstring is aname, thenany linecontaining thegivenname isdisplayed. If
stringisanareacode,thenallentrieswiththesameareacodearedisplayed.Ifstringisazipcode,thenalllineswiththesamezipcodearedisplayed.Also,fgrep-vMAcontactsdisplaysalladdressesexceptthoseinMA.Here is an application dealing with multiple files. Let’s say you have a
directorynamedlettersthatyouusetofileawayelectronicmailforsafekeepingandlater reference.Supposeyouneed tofinda letter in thisdirectory,butyoudon’tremembertheletter’sfilename.Allyourecall isthattheletterdealswith
4.4
thesubject“salary”.Tofindtheletter,usecdlettersfgrep-i-lsalary*Thecommandsearchesall (non-hidden) filesunder thecurrentdirectory for
lines containing the string salary (ignoring case differences) and displays onlythenameofanyfilewithmatchinglines.TheShellvariable$?recordstheexitstatusofacommand(Chapter5,Section5.7).Thegrepcommandreturnsexitstatus0ifanymatchesarefound,1ifnone,and2iferror.
REGULAREXPRESSIONSInthegrepcommandandmanyothertextprocessingsituations,theneedtofindastringofcharactersmatchingaparticularpatternarises.Forexample,testingifafilenameendsin.pdf,checkingifaparticularuserinputrepresentsanumberwithanoptional leadingsign,ormakingsure thata lineof texthasnotrailingwhitespaces.Inordertodefinepatternstomatch,weneedanotationtospecifypatterns for programs. A regular expression is a pattern matching notationwidelyusedandunderstoodbyprogrammersandprograms.Thesimplest regularexpression isafixedstringsuchasUbuntuorCentOS.
Such a regular expression matches a fixed character string. However, regularexpressions are much more flexible and allow you to match strings withoutknowingtheirexactspelling.InLinux, the applicationsgrep,vi/vim,sed, egrep, andawk/gawk, among
others,use largely thesameregularexpressions.Table4.3gives thebasics forregularexpressionnotationsthatmostprogramsunderstand.Thegrepcommandaccepts many additional pattern notations (see Section 4.5 and the grep manpage).BasicRegularExpressions
Consider editing, with vim, a recipe that contains many steps labeledsequentiallybyStep1,Step2,andsoon.Inrevisingtherecipe,youneedtoadda few steps and renumber the labels.A search pattern can be specified by theregularexpressionStep[1-9]wherethenotation[1-9]matchesanysinglecharacter1-9.Inthevimeditor(seeappendices),youcansearchwiththecommand/Step[1-9]and make the appropriate modification to the number. After that, you can
repeat the search using the vim search repeat command n, change anothernumber,search,andsoonuntilallthechangeshavebeenmade.Let’s put the regular expression notations to use and look at some specific
patterns.In a regular expression, the * character indicates an occurrence of zero or
more times of the previous character/pattern. In Table 4.3, we see regularexpressionspecialcharacters:[,],*,,and$,eachhavingaprescribedmeaningasapatternspecifier.
QuotinginSearchPatternsThe use of special characters in any searching scheme inevitably leads to thequestionofhowtosearchforapatternthatcontainsaspecialcharacter.Let’ssaythat you are editing a report and you want to search for [9], which is abibliographicalreferenceusedinthereport.Becausetheregularexpression[9]matches the single character 9, you need to%quote the [ and ] so that theyrepresentthemselvesratherthanpatternspecifiers.Thesolution,ironically,istointroduce yet another special character, the backslash ( ), to serve as a quotecharacter that prevents the immediate next character from being treated as a
4.5
patternspecifierandforcingittostandforitself.Thus,thepattern
9
matches[9],andthepattern
[1-9]
matches the strings [1] through [9]. To match any such bibliographicalreference,usethepattern
[1-9][0-9]∗
.Herearesomemorepatternexamples:
... (matches...,namelythreedots)/* (matches/*) (matches)[0-9A-z] (matchesanyoftheindicatedcharacters)Quotingacharacterthatdoesnotneedquotingusuallycausesnoharm.
PATTERNSFORGREPMost of the basic regular expression patterns listed in Table 4.3 work inprogramsacceptingregularexpressionpatterns.Thegrepcommandalsoacceptsextended regular expressions available via the -E option or through the egrepcommand.ExtendedregularexpressionsaddnotationsdescribedinTable4.4tothebasicregularexpressions.ExtendedRegularExpressions
4.6
InTable4.4redenotesany regularexpression.Theprecedenceofoperatorsusedforextendedregularexpressionsis(),[],",+,?,concatenation,and|.Careshould be taken when entering patterns on the command line because manypatterncharactersarealsospecialShellcharacters.Itissafesttoalwaysenclosethe entire pattern in a pair of single quotation marks. Here are some moreexamples:
grep´\-s´(matches-s;the\prevents-sfrombecominga
commandoption)grep-i´^linux´(matcheslinuxatthefrontofa
line,ignoringcase)grep´ch[0-9]*´(matcheschfollowedbyany
numberofdigits)egrep\.html?\<(matchesawordendingin.htmor
.html)egrep´\>\w+\.docx?´(matchesanywordfollowedby.docor
.docx)
Thegrepcommandsareoftenusedinapipelinewithothercommandstofilteroutput.MoreexamplesofgrepwithinpipelinesarediscussedinSection4.7.Informationonregularexpressionspresentedhereformsabasisfor learning
moreelaborateregularexpressionsinlanguagessuchasPerl,Ruby,Javascript,andJava.
ASTREAMEDITOR:SEDThesedprogramisa filter thatuses line-editingcommands to transforminputlines, fromstdinora file, andproduces thedesiredoutput lines (Figure4.13).Sedisanon-interactive,line-orientededitor.Itappliesprescribededitingactionstolinesmatchinggivenbasicregularexpressionpatterns.
Figure4.13TheStreamEditorsed
In practice, sed is used for such chores as deleting particular lines, doublespacingaprogramlisting,andmodifyingalloccurrencesofsomepatterninoneormoretextfiles.Infact,sedandgrepcanperformmanyofthesamefunctions.However,sed
ismorepowerfulbecauseitsupplies texteditingcapabilities.Thesedprogrambuffersoneinputlineatatime,repeatingthefollowingstepsuntiltherearenomoreinputlines.Figure4.14showsthesedprocessingcycle.
1. If therearenomore input lines, terminate.Otherwise, read thenext inputlineintothebuffer,replacingitsoldcontent,andincrementthelinecount(initially0)by1.
2. Applyallgiveneditingactionstothebuffer.3. Writethebufferouttothestandardoutput.4. Gotostep1.
Figure4.14TheEditingCycleofsed
Eacheditingactionmaybeapplicabletoalllinesortojustafew.Therefore,itis possible for some lines to pass through sed unchanged; at the same time,otherscanbemodifiedordeletedentirely.Frequently,sedisusedinthesimpleformsedscript[file]…wherescript specifies one ormore editing actions separated by semicolons.
Forexample,sed’s/Website/website/’chapter1sed’s/Website/website/g’chapter1Thefirstcommandreadstheinputfilechapter1,substitutes(thesaction)any
firstoccurrenceofWebsiteineachlinewiththestringwebsite,2andoutputsalllines, changed or not, to the standard output. If any line contains multipleinstances ofWeb site, only the first instance in the line will be replaced. Toreplacealloccurrences,usethesecondcommandwheretheg(globalmodifier)doesthetrick.Ifnofile isspecified,sededits lines fromstdin.Thesinglequotationmarks
aroundscript prevent the Shell from interpreting any special characters in thescript.Thecommandsed’s/RedHat/Fedora/g;s/ubuntu/Ubuntu/g’chapter1appliestwostringreplacementactions,sequentially,toeachlineofchapter1.
Theoption-fscriptfile_fileindicatesafilecontainingthedesirededitingscript.Ifascriptfiledoublecontainsthetwolines
s/$/\/
thensed-fdoublefileaddsanemptylineaftereachline in file,producingadouble-spacedoutput.
Asingrep,thepattern$meanstheendofaline.Eacheditingactioncanalsobespecifiedtoactonarangeoflines.Hereisthe
generalform:[address1][,address2]action[args]wheretheaddressesspecifytherangeofinputlinestoapplythegivenaction.
Anaddresscanbealinenumberorapattern.
Noaddress—Thegivenactionappliestoeveryline.Oneaddress—Theactionappliestoeverylinematchingthataddress.Two addresses—Theaction is applied repeatedly to the next set of linesbeginningwitha line thatmatchesaddress1, up to and including the firstlinethatmatchesaddress2(butnotaddress1).
Forexample,sed’/$/d’fileappliestheactiond(deleteline)toeachlinematchingthesingleaddress/$/,
anaddressobtainedbysearchingforthenextemptyline.Theoutputwillbefilewithall empty linesdeleted.Anotherversionof thisexampledeletesallblanklinessed’/[⊘▹]*$/d’fileWe use the symbols⊘ and▹ to stand for a SPACE and a TAB, respectively.
Remember toescape theTABwithCTRL+V oryoucanuse instead.Theaddressmatchesalinecontainingzeroormorespacesandtabsandnothingelse.Let’s look at an example involving a two-address action. Say that in your
HTMLfilestablesaresandwichedbetweentwolines
<table...>
and
</table>
Supposeyouwish to removeall tables fromagivenHTMLdocument (Ex:ex04/remove_table).Youmayuse
sed´/<table.*>/,/<\/table>/d´try.html>notables.html
4.7
Thedeletelineactiondisappliedtoalltablelines.A useful sed option is -n, which skips step 3 of the sed cycle. Hence, the
commandsed-n’/pattern/p’withtheoutput-lineactionp,isequivalenttogrep’pattern’,andsed-n’12,20p’fileoutputsonlylines12–20ofthegivenfile.Hence, if youwish to extract all the tables from a givenHTML document
(Ex:ex04/extract_table),youmayuse
sed-n´/<table.*>/,/<\/table>/p´try.html>tables
tooutput linesbetween thebeginningand theendof each tableusing thepactionandthe-noption.Alternatively,youcanuse
sed´/<table.*>/,/<\/table>/!d´try.html>tables
The exclamation point (!) reverses the sense of the specified addresses; itapplies the specified action to every line except the lines matching the givenaddress.Also,theyactiony/string1/string2/when given two equal-length character strings, performs character
translations.Thus,sed’y/abc/ABC/’filefunctionsthesameastrabcABCfileSimplescriptsareeasy togiveon thesedcommand line.Morecomplicated
scriptsshouldbeplacedinfilesandappliedwiththe-foption.Storingscriptsinfilesmakesthemeasilyreusable.Thesedcommandoffersanumberofotherfeaturesandoptions.Pleaserefer
tothesedmanpagesforadditionalinformation.
BUILDINGPIPELINESWehave discussed a good number of filters and seen some pipelines already.Let’snowseeafewmoreexamples.Hereisapipelinetolookupthecorrectspellingsofwords:lookprefix|fgrepstringAll words in the dictionary /usr/dict/words with the specified prefix are
producedbylookandfedtofgrep,whichselectsonlythosewordsthatcontainthegivenstring.Forexample,lookdis|fgrepsiongivesthefollowingoutput:discussiondispersiondissensionAnother example is a pipeline that saves to a file those commands that you
havegiventoyourShell.TheBashcommandhistorydisplaysanumberedlistofyourmostrecentcommands.Toenterthelasteightcommandsintoafile,youcanusethefollowingpipeline:history|tail-8|sed’s/*[0-9]**//’>filewherethesedcommandremovestheleadingsequencenumbers.A third example collects a list of directorynames from the currentworking
directory:
ls-l|grep^d|sed´s/^d.*⊘//´
Herethesededitingcommanddeletesamaximal(longest)stringstartingwiththeletterdatthebeginningofalineandendingwithaspace(⊘)foreachline.Anotherwaytoaccomplishthesametaskis
ls-F|grep´/$´|sed´s/\/$//´
Afinalexamplehastodowithmaintaininganaddresslist.Let’sassumeyouhaveafileofaddresses,myaddr,inhuman-readableform.Itsentriesaremulti-line addresses, and a single empty line follows each entry. A typical addressentrywouldlooklikethefollowing(Ex:ex04/myaddr):Dr.JohnF.DoeGreatEasternCo.40NorthRd.Cambridge,MA02139This form is easy for a user to read, but hard to maintain using filters.
However, you can transform this address filewith the followingpipeline (Ex:ex04/toaddr):
sed´s/^$/@/´myaddr|tr´\012@´´:\012´\|sed´s/^://;s/:$//´|
sort-u-t:--key=1,2>|addr
Thefirstsedsubstitutesthecharacter@foreachemptyline.ThetrcommandtranslateseveryNEWLINEcharacterintoacolonandevery@intoaNEWLINE.At
4.8
this point, each address entry is on a separate linewith a colon separating thefieldswithineachaddress.Thesecondsedremovesanycolonatthebeginningortheendofaline.Thefinalsortcommandorderstheaddressentriesusingthefirstfieldandremovesanyduplicateentries.
AddressProcessingNowyour address file addr is sorted and contains one address per line in thefollowingform:Dr.JohnF.Doe:EasternCo.:40NorthRd.:Cambridge,MA02139Youcanextractanaddressbyusing(Ex:ex04/useaddr)grep’JohnF.Doe’addr|tr’:’’012’Youcandeleteanyaddressbyusingsed’/JohnF.Doe/d’addr>temp.filemvtemp.fileaddrYoucaninsertoneormoreaddressesbyusingsort-u-t:-key=1,2addr->temp.filewhich allows you to type in the entries from the standard input. You may
insertanotheraddressfile,addr2,byusingsort-mu-t:–key=1,2addraddr2>temp.filemvtemp.fileaddrIn the preceding example, the first field contains the title and name of a
person.Thesortedaddressfileisnotinalphabeticalorderwithrespecttonames,unless everyone has the same title. To avoid this problem, you may want tomodifytherecordformatto(Ex:ex04/newaddr)Doe:John:F:Dr.:EasternCo.:40NorthRd.:Cambridge,MA02139andsorttheaddressfileusingthefirst,second,andthirdfieldsaskeys.Then
thefollowingcanbeusedtodisplayanentry(Ex:ex04/usenewaddr):look’Doe’newaddr|gawk-F:’print$4,$2,$3".",$1;print$5;print$6;print$7’For largefiles, the lookcommand,whichusesabinarysearchmethodfora
line prefix, is much faster than the fgrep command. The gawk is a GNUimplementationofawk.
FORMOREINFORMATIONSeetheappsearchforyourLinuxandsearchontheWebforgoodappstouse.Seethemanpagesfor thecommandsandfilterscoveredinthischapter.For
filtersacceptingregularexpressions, theirmanpageswillspecifyexactlywhat
4.9
4.10
patternsarerecognized.AdetaileddescriptionofawkcanbefoundinAppendix:PatternProcessing
withawkatthebook’swebsite.
SUMMARYLinuxhas an abundanceof softwarepackagesmostly free.Agoodnumberofuseful GUI-based apps have been discussed. Find what’s available on yourLinuxdistributionanduse themto takefulladvantageofyoursystem.Systemadmins can easily install missing apps with the Linux package managementsystem.Filtersproduceoutputbyperformingasimple,well-definedtransformationon
their input and follow a set of well-defined conventions so they can becomestagesinpipelinesthatcombinethemtoperformmanyvariedtasks.FiltersandpipelinesareconcreteartifactsoftheUNIX/Linuxphilosophy.Linux filters range from simple character substitutions (tr and expand) to
findingstringpatternsintextlines(thegrepcommands),toorderingtextlines,and to complicated stream editing (sed). How these commands workindividually and in pipelines for realistic applications, such as creating,maintaining,andaccessinganaddressdatabase,havebeendiscussed.Regularexpressionsarewell-establishednotationsforcharacterstringpattern
matching.Theyareused,inverysimilarways,inmanydifferentprogramssuchas grep, egrep, sed/vim. In Chapter 5, you’ll see that the Bash Shell alsounderstandsregularexpressions.Itisimportanttobecomefamiliarwithregularexpressionconcepts.CommandsSummary
Table4.5summarizesthecommandsdescribedinthischapter.
EXERCISES1. FindoutifyourLinuxhasthegnome-softwareoranotherappcentertool.
CanyouuseitorthepackagemanagertoinstallMAXIMA?
2. Find the best media (image, audio, video) playing apps for your Linuxdistribution.
3. FindoutabouttheKeepassxtool.Whatisthemostrecentversion?Howisitinstalledandused?
4. Findthebestmediaformatconversion(image,audio,video)appsforyourLinuxdistribution.
5. Findouthowtousexfig.6. WhatwebcamtoolsareavailableonyourLinuxdistribution?Iscamorama
available?7. Findouthowtousethecommandsgimpanddisplay.8. Findouthow to installandusean Internet speed test tool foryourLinux
distribution.9. Consider how expand works. Write an algorithm for figuring out how
manyspacesshouldbegeneratedforaTAB.10. Writeapipeline,using lsandhead, to list the tenmost recent files in the
currentdirectory.11. Howcanyouusegrep to locate all lines in a file thatdonot contain the
pattern-option?12. WhatisaGlobpattern?Whatisaregularexpressionpattern?Whatisthe
difference?Giveapatternineachcasetomatchastringendingin.html.13. Specifyaregularexpressiontomatch(a)anywordendingin.html;(b)any
imagenameending in .jpg, .png,or .gif; (c)anyempty line (linewithnocharacters in it whatsoever); (d) any blank line (line that is empty orcontainsonlywhitespacecharacters);and(d)anynumber.
14. Explain the following regular expressions: (a) a+$, (b) http[s]*: and (c)[@]+@gmailċom.
15. Considerthefollowingsedcommand:sed-n’/begin/,/end/p’fileDiscussitseffectiffilecontainsmanylineswithbeginandorendinthem.
16. Considerbuildingpipelinestomanageanaddressfile.Supposeyouwishtohaveanaddress,anemail,andaphonenubmeroneachaddressline.Howwouldyoudesign the record format?Writeapipeline toextractadesiredemailorphonenumberfromtheaddressfile.
17. Following the previous exercise, write a pipeline to add/change a phonenumberoremailtoanexistingaddressentry.
18. Specifyansedcommandtoreplaceanysetofconsecutiveemptylinesinafilewith justoneempty line.Anempty line isonewithnothing in it,notevenblankcharacters.
19. Rot13isamethodtoencodeASCIItextfiles:eachletterinthealphabetAthroughzisreplacedbyanother13positionsaway(AbyNandnbyA,for
12
example).Writeatrcommandtoperformthisencoding/decoding.20. Theyfunctionofsed canperformmostof the same translationsas tr. Is
thereanythingtrcandothatsedcannot?Ifso,discuss.See/usr/binforversionsoftex/latexcommands.TheAP(AssociatedPress)stylebookrecentlymadethechange.
5.1
Chapter5
WritingBASHScripts
TheShellismorethanjustaninteractivecommandinterpreter.Italsodefinesasimpleprogramminglanguage.AprogramwritteninthislanguageisknownasaShellprocedureorShellscript,which,initssimplestform,isjustasequenceofcommands in a file. The file, when executed, performs the tasks as if eachcommandinthescripthadbeenenteredandexecutedindividually,butwithoutall the typing. Shell scripts can save you a lot of time if you find yourselfrepeating a sequence of commands over and over. The Shell language alsoprovides variables, control structures such as if-then-else, looping, functiondefinition,andmeansfor inputandoutput. Ifaparticular taskcanbeachievedbycombiningexistingcommands,thenconsiderwritingaShellscripttodothejob.AswithotherLinuxcommands, aShell script canbe invoked throughyour
interactive Shell and can receive arguments supplied on the command line.Sometimes,scriptswrittenbyindividualusersalsocanbeofgeneraluse.Suchscriptscanbeinstalledinasystemdirectoryaccessibletoallusers.This chapter covers Shell script writing and techniques for effective Shell-
levelprogramming.WewillfocusonBashscriptsbecauseBashiscurrentlythemostwidely used andmost advanced Shell. Csh, Tcsh,Dash and Sh 1 scriptsfollowmanysimilarrules.Thepresentationsinthischapterareorientedtowardscriptwriting.However,
most constructs discussed here can be used interactively aswell. Some topics(forexample,commandgrouping)areasrelevanttointeractiveuseastoscriptwriting.
INVOKINGSHELLSCRIPTSAsmentioned, a Shell script is a programwritten in the Shell language. The
5.2
program consists of variables, control-flow constructs, commands, andcomments.TheShellscriptiskeptinatextfilewhosefilenameissaidtobethenameofthescript.TherearetwowaystoinvokeaShellscript:byexplicitinterpretationandby
implicitinterpretation.Inexplicitinterpretation,thecommand
bashfile[arg…](forBashscript)tcshfile[arg…](forTcsh
script)shfile[arg…](forShscript)
invokesaspecificShelltointerpretthescriptcontainedinfile,passingtothescriptanyargumentsspecified.In implicit interpretation, the script file containing the script is first made
readable andexecutablewith the chmodcommand to turn on the appropriateprotectionbits (Chapter1,Section1.6).Then the script canbe invoked in thesameway as anyother command: bygiving the script nameon the commandlinefollowedbyanyarguments.In either explicit or implicit interpretation of a Shell script, two Shells are
involved:(1)theinteractiveShell(usuallytheloginShell)thatinteractswiththeuserandprocessestheuser’scommandsand(2)theinvokedShellthatactuallyinterprets thescript.The invokedShell isaprocessspawnedby the interactiveShell.Since thespawnedprocess isalsoaShell, it is referred toasasubshell.Theeffectofthiscanbeillustratedbythefollowingexperiment.Firstcreateafilenamedtrythatcontainsthesimplescriptcd/usr/libpwdTorunthisscript,typebashtryThescriptcalledtrydisplaysthestring/usr/lib,whichistheoutputofthepwd
contained in the script.However, once it is finished, if you typepwd in yourinteractive Shell, your old working directory will appear. Obviously, the cdcommandexecutedinthescripthasnotaffectedthecurrentworkingdirectoryofyourinteractiveShell.Thisisbecausecdisexecutedbyasubshell.ToexecutethecommandsintrywiththeinteractiveShell,useinsteadsourcetry
AFIRSTSHELLSCRIPTNowlet’sconsiderasimpleBashscript.Thepurposeofthisscriptistoconsultalist of email addresses that are kept in a file named myContactList (Ex:
ex05/myContactList) in a user’s home directory. Each line in the contact listgives the name of a person, email address, phone number, and perhaps someotherinformation.Thescript(Ex:ex05/contact_one.sh)is
#!/bin/bash##consultsmyContactListgrep-i"$1"~/myContactList
Wewillusethesuffix.shforBashscriptsasanamingconvention.Thefirstline is special. InLinux, the properway to begin anexecutable text file is #!,followedbythefullpathnameofanexecutablefiletogetherwithanyargumentsto it.Thisspecifies thecommand to invokean interpreter for the remainderofthe script.Make sure #! are the very first two characters in the file, with noemptyline,whitespace,oranyothercharacterbeforethem.The first line of contact.sh indicates a Bash script. Similarly, the line
#!/bin/cshbeginsaCshscript,andtheline#!/bin/shbeginsanShscript.Thesecondlineisacomment.InShellscripts, thepartofanylinefromthe
first#totheendoflineisignoredbytheShell.Thesymbol$1 iscalledapositionalparameter.Thevalueof thepositional
parameter$n is thenth command-line argument.Thus, if the first argument issmith,then$1hasthatvalue,andthescriptisequivalenttogrep-ismith/myContactListRecall that expands to your home directory. Now you should issue the
commandchmod+rxcontact.shtomakecontact.shreadableandexecutable.Nowthecommandcontact.shsmithrunsthecontact.shscript(inthecurrentdirectory).Theprecedingcommand
assumesthatthespecialperiodsymbol(.)isincludedinyourcommandsearchpath(Section3.4).Otherwise,youneedtouse./contact.shsmithIfthecontact.shscript isput inadirectorywhosenameisonthecommand
searchpath,thencontact.shsmithwillworknomatterwhatyourcurrentdirectoryis,withouthavingtospecify
thecontact.shcommandwithapathname.Usually,youwouldcreateadirectorybinorcmd inyourhomedirectory to
holdallscriptsandotherexecutablecommandswrittenorobtainedbyyou.Byincludingtheline
PATH=$PATH:$HOME/cmd:.
5.3
5.4
in your .bash_profile, you can invoke executable files in your own cmddirectoryjustlikeotherLinuxcommands.Ifyouinclude.onPATH,makesureitisat theveryend.Otherwise,youmayexecuteunexpected,orevenmalicious,codeinthecurrentfolder.
SHELLSCRIPTEXECUTIONA Shell script consists of a sequence of Shell built-in commands and regularLinuxcommandsseparatedbyNEWLINEorsemicolon(;)characters.Commentsare introduced by #, as previously mentioned. Commands in a script areexecuted insequence. If theexecutionofacommandresults inanerror,scriptexecution will be aborted if the offending command is a Shell built-in.Otherwise, for a regular command, the default action is to skip the offendingcommandandcontinuewiththenextcommandinthescript(Figure5.1).
Figure5.1BashShellScriptExecution
IndescribingtheShelllanguages,thetermcommandlistmeansasequenceofzeroormorecommandsseparatedbyNEWLINEorsemicolon(;)characters.Thetermwordlistreferstozeroormoreblankseparatedwords.
POSITIONALPARAMETERSIn Shell scripts, the variables $0, $1, $2, and so on are known as positionalparameters.Thevariable$0referstothefirsttokenofthecommandlinewhichinvoked the script. Thus, $0 may have the value contact.sh or ./contact.shdependingonthecommandgiven.Thevariables$1,$2,andsoonrefer to thecommand-linearguments.WhenaBashscriptisinvoked,thespecialvariable$0issettothecommand
name. The positional parameters $1, $2, etc. are set to the command-linearguments (use $n for n bigger than 9); $* (and $@) is set to the list ofargumentsgivenonthecommandline;and$#issettothenumberofpositional
5.5
parameters.TheBashscript(Ex:ex05/myecho.sh)
echo'$0='$0echo'$1='$1echo'$2='$2echo'$3='$3echo
'$#='$#echo'$*='$*echo'$@='$@
displaystheseparametervalues.Forexample,thecommandmyecho.shABCDproducestheoutput
$0=./myecho.sh$1=A$2=B$3=C$#=4$*=ABCD$@=ABCD
Tryityourself.
THEFORCOMMANDTheforcommandisusedtoexecuteasetofcommandsrepeatedly.Thegeneralformis
forvarinwordlistdocommandlistdone
Thelinebreaksareneededunlessyouusethe;commandseparatorasinforvarinwordlist;docommandlist;doneThe commandlist is executed once for each word inwordlist as, each time
through, the control variable var takes the next word for its value. As anexample, let’s rewrite the contact_one.sh script given in Section 5.2 as (Ex:ex05/contacts.sh):
#!/bin/bash##consultmycontactsforargsgivenforxin"$@"##
(0)dogrep-i"$x"~/myContactListdone
Bashhastwobuilt-invariables,$*and$@,referringtotheargumentsgivenonthecommandline.Eachisalistofwordsfromthecommand-linearguments.Consideracommandwiththreearguments:somecmdab"cd"The$*[email protected]
quotation "$*"makes it oneword,whereas thequotation "$@"makes it threewordsaband"cd".Itisimportanttounderstandthisdifference.Itturnsoutalsothatline0canbewrittensimplyasforx,whichmeansxwilltakeonsuccessivecommand-linearguments.Nowcontact.shcanbeusedononeormorenames,asincontact.sh"JohnSmith""PaulWang"Theforcommandcanbeusedtogothrougheachfileinadirectory.Trythe
5.6
followingscript:
#!/bin/bash##exampletogothroughallfilesinthecurrent
directoryforfilein*doecho$filedone
Execute this script, and you’ll see the filenames in the current directorydisplayed.Sincethefilenameexpansiondoesnotmatchanyfilenamethatbeginswithaperiod(.),thosefilenameswillnotbedisplayed.Togetallfiles,use
forfilein.**doecho$filedone
Bash supports another form of for loop that is similar to that of the Clanguage.
#!/bin/bashfor((i=0;i<9;i++))doecho$idone
The iteration control involves numerical expressions (Section 5.11). Suchloops areuseful for indexing througharrays (Section5.14) and, of course, fornumericalcomputations.
THEIFCOMMANDThe if construct provides for conditional execution of commands. The simpleformofifis
iftest-exprthencommandlist1elsecommandlist2fi
If the given test-expr is true, then commandlist 1 is executed; otherwise,commandlist2isexecuted.Theelsepartmaybeomitted.Forexample,thetestexpression[[-ffile]],knownasanextendedconditional
(Section 5.7), tests if file exists and is a regular file. We can improve thecontact.shasfollows(Ex:ex05/contact_check.sh).
#!/bin/bash##consultmycontactsforargsgivenif[[-f
~/myContactList]]##(A)thenforxdogrep-i$x
~/myContactListdoneelseecho"File~/myContactListnotfound."fi
Inatestexpression,theSPACEafter[[andtheSPACEbefore]]arepartof theconditionalnotation(lineA).Withintheifstatement,theelifconstructcanbeused.Thegeneralformis
iftest-expr1thencommandlist1eliftest-
expr2thencommandlist2elsecommandlist3fi
Iftest-expr1istrue,commandlist1isexecuted.Iftest-expr1isnottrue,andiftest-expr2istrue,thencommandlist2isexecuted.Therecanbeanynumberofelifconstructs.Ifalltestexpressionsarefalse,thentheelsepartisexecuted.Often,itisimportantforanyprogramtochecktheargumentsitreceives,and
aShell script isnoexception.Here is some typical argumentcheckcode (Ex:ex05/argCheck.sh).
#!/bin/bash##checkandsetcommand-lineargumentsif[[$#<2||
$#>1]]##(1)thenechousage:"$0[from-file]to-file"##
(2)exit1;##(3)elif[[$#==2]]##(4)thenfrom="$1"to="$2"else
##(5)to="$1"fi
Theexpression$#>2checks if thenumberof arguments isgreater than2.The || is logical or, whereas < is less than. This script expects one or twoarguments.Ifthenumberofargumentsisincorrect,itdisplaysanerrormessage(line2)and terminates thescriptwithanabnormalexitstatus1 (line3). Ifwehavetwoarguments(line4),wecansetthevariablesfromandto.Otherwise,wehaveonlyoneargumentand itbecomes thevalueof to.Argumentchecking iscriticalatthebeginningofeveryprogram.
Figure5.2ThecmdsearchScript
Now let’s look at a complete script using for and if constructs. The script(Figure 5.2) locates a command on the command search path ($PATH) anddisplays its full pathname (Ex: ex05/cmdsearch.sh). The first (and lone)argument is the target commandname (line a).On line b, each : in $PATH isreplaced by a SPACE with the tr command (Chapter 4, Section 4.2), and theresultingmultiwordstringisassignedtoavariablepathviacommandexpansion
5.7
(Chapter3,Section3.7).Foreach$diron$path(linec),weseeif$cmdisfound(line d). The conditional expression [[ -x file ]] is true if file exists and isexecutable (see Section 5.13 for more on file queries). If the program everreacheslinee,thenthetargetcommandisnotfound.Herearesomesampleusesofcmdsearch.cmdsearchgnome-terminalcmdsearchvimcmdsearchgcc
TESTEXPRESSIONSANDEXITSTATUSExitStatusInLinux,acommandindicateswhetherithassucceededbyprovidinganintegerexitstatustoitsinvokingenvironment.Azeroexitstatusmeansokay,andnon-zeromeanserror.TheShell, being a command interpreter, is a primary invoking environment
for commands. After executing a command, the exit status is available in thespecialShellvariable$?.InaShellscript,usethebuilt-incommandexitn toterminateexecutionand
returnnastheexitstatus.
TestExpressionsTestexpressionsareusedinifaswellasotherconstructs(while,until,etc.) toproducetrue/falsevaluesbytestinggivenconditions.ThetruthvalueofaBashtestexpressionisreallydeterminedbyitsexitstatus.
Atestexpressionistrueifitreturnsazeroexitstatus;otherwise,itisfalse.Nowlet’stakealookatthedifferentformsoftestexpressions.Atest-exprcanbealistofoneormoreoftheseexpressions:
Aregularorbuilt-incommand(Section5.7)Anextendedconditionalexpression[[]]A numerical expression (( )), with 0 being false and non-zero being true(Section5.11)(test-expr),using()forprecedencegrouping!test-expr“logicalnot”oftest-exprtest-expr1&&test-expr2“logicaland”ofthetwoexpressionstest-expr1||test-expr2“logicalor”ofthetwoexpressions
5.8
Hereisanexamplethatusesgrepasatestexpression(Ex:ex05/condEdit.sh).
#!/bin/bashforfilein*##foreachfileincurrentfolderdoif
grep-q"$1"$file##ifpattern$1isin$filethennano$file##
invokenanoon$filefidone
Anextendedconditionalisenclosedby[[SPACEontheleftandSPACE]]ontheright.2Table5.1liststestexpressionsforstrings.Withinthe[[conditional,Globpatterns(Chapter3,Section3.7)areallowedontheright-handsidesof==and!=.Furthermore,extendedregularexpressions(Chapter4,Section4.5)following=aresupported.BashStringConditions
Theextendedconditionalsalsosupportnumericaltest[[arg1roparg2]]tocomparetwointegersarg1andarg2witharelationaloperatorropwhich
canbe==,!=,<,>,-le,or-ge.3Often,programmersprefer tousenumericaltestsprovidedby(())(Section5.11)instead.Inside[[]]youcanalsousethelogicaloperators!(not),||(or),and&&(and)
on test expressions.Please refer toSection5.11 for numerical test expressionsandtoSection5.13forfile-relatedtestexpressions.
THESHIFTCOMMANDTheBashbuilt-incommandshiftleftshifts2to1,3to2,etc.Ingeneral,shiftnshiftsnto1,n+1to2,etc.The shift command is often useful after the first few positional parameters
havebeenprocessedandyouwanttousealoopsuchas(Ex:ex05/shift.sh)
forvardoecho$vardone
5.9
togoovertherestofthepositionalparameters.
THECASECOMMANDWhile the if-elif-else command enables logical branching in general, the casecommand provides branching based on simple pattern matching. The generalformofcaseiscase(str)in
case(str)inpattern1)commandlist1;;pattern2)commandlist2;;...esac
The given expression str is successivelymatched against the case patterns.EachcasepatterncanbeoneoralistofGlobpatterns(Chapter2,Section3.7)separatedby|andterminatedby).Onlythelistofcommandsforthefirstmatchwillbeexecuted.Nothingisexecutedifthereisnomatch.Forexample,thestringab.cmatchesthecasepattern*.corthepatterna*c.
Figure5.3TheappendScript
Asanexample,ascriptforappendingeitherthestandardinputorafiletotheendofanotherfileisshowninFigure5.3(Ex:ex05/append.sh).Thecommandappend.shfile1file2appendsfile1totheendoffile2.Thecommandappend.shfilefirstlinesecondlinethirdlineDappends the three lines to theendof file.Note thecatch-allpattern*as the
lastcaseclausetoprocessanyunmatchedcases.
5.10
5.11
THEWHILEANDUNTILCOMMANDSInadditiontotheforcommand,thewhileanduntilcommandscontroliterationwithanarbitrarycondition.Thegeneralformofthewhilecommandiswhiletest-exprdocommandlistdoneThetest-exprisevaluated.Ifitistrue,thencommandlistisexecuted,andtest-
expr is retested. The iteration continues until the test-expr tests false. For aninfiniteloop,usetheBashbuilt-incommand:(yes,thecharacterCOLON)asthetest-expr.The : command does nothing other than expand any arguments andgivea0exitstatus.In the following script (Ex: ex05/myfortune.sh), we continue to display a
fortunemessageuntiltheuserwishestostop.
#!/bin/bash##displaysfortuneuntiltheuserquitsgo="yes"while[[
"$go"=="yes"]]##(i)do/usr/bin/fortune##(ii)echo-n"****
Morefortune?[yes/no]:"##(iii)readgo##(iv)done
Thewhile condition is checked (line i). If true, the fortune command 4 isinvoked(lineii),andapromptisdisplayed(lineiii)toseeiftheuserwishestocontinue. The -n option tells echo not to output the usual line break after themessage. The user input is read into the variable go (line iv),whose value istestedagain.Ifwereplacethewhiletestexpressionwiththepatterncondition
[["$go"==y*]]
thentheusermayenteranythingstartingwithytocontinue.Theuntilloopisthesameasthewhileloop,excepttheiterationstopsassoon
astheuntilconditionismet.
NUMERICALEXPRESSIONSSinceShellvariablesarestring-valued,weneedtousethearithmeticexpansionnotation$((arith-expr))to perform integer arithmetic computations.TheShell built-in command let
canalsobeusedtoperformarithmeticoperations.letarith-expr1arith-expr2...
5.12
5.13
Herearesomeexamples(Ex:ex05/arith.sh).
#!/bin/basha=2echo$((a+3))##displays5letb=2*++aecho$b##
displays6echo$((a*b))##displays18letc=-8echo$((c<0?c
:-c))##displays8
Tocomparenumbersinnumericalconditionalsuse,forexample,if((a>b))(ifaisgreaterthanb)TheBashcommandhelpletdisplaysafulllistofoperatorsavailableforthenumericalexpressionsforlet
orinside(()).Hereisaloopthatdisplaysthecommand-lineargumentsinreverseorder(Ex:
ex05/echoback.sh).
#!/bin/bashoutput=""until(($#==0))dooutput="$1
$output"shiftdoneecho$output
THEBREAKANDCONTINUECOMMANDSThebreak command is used inside the iteration control structures for,while,anduntil.Whenbreakisexecuted,controlshiftstothefirstlineaftertheendofthenearestenclosingiteration.Thiscommandprovidesameansto“breakout”ofaniterationloopbeforeitsnormalcompletion.Thecontinuecommandisusedinthesamemanneras thebreakcommand,
except it transfers control to the beginning of the next iteration instead ofbreakingoutoftheloopentirely.Theexamplescriptclean.sh(seeSection5.20)involvessometypicalapplicationsofbreakandcontinue.Withinnestedloops,breakorcontinuecantakeanoptionalintegerargument
(1,2,3,etc.)tobreakorcontinueoutofthenthlevelofnestedloops.
FILEQUERIESTomakefileanddirectoryaccessandmanipulationeasy,Bashalsoprovidesasetofconditionstoquerystatusinformationforfiles.Filequeriesareintheform-xfile,wherexisasinglecharacter.CommonfilequeriesarelistedinTable5.2.Togetacompletelistingusehelptest.If the file does not exist or if it is inaccessible, all queries return false. For
example,thefollowingcodefragmentisvalid:
if[[-e$file&&-f$file&&-w$file]]thencat$1<<
5.14
5.15
$fileelseecho"accessproblemfor$file"fi
BashFileQueries
Inthefilesystem,anordinaryfileisonethatstoresapplicationdataandnotone that serves filesystem functions such as a directory (folder) or link(shortcut).SeeChapter6,Section6.2formoreinformationonLinuxfiletypes.
VARIABLESTherearedifferentkindsofvariables:
1. Positionalparameters($1,$2,...)andspecialvariables($0,$#,...).2. EnvironmentvariablessuchasDISPLAYandSHELL3. Ordinaryvariablesandarraysofyourownchoosing
Toassignvaluetoavariablevar=valueShell expansions and evaluations are performed on value, and the result is
assigned to thegivenvariable. Ifvalue is omitted, then the variable has valuenull.Variableattributescanbedeclared:
declare-ivar1var2...(holdingintegervalues)declare-rvar1
var2...(read-only)declare-aarr1arr2...(arrays)declare-x
var1var2...(exportedtotheenvironment)
To remove a variable use unset var. The special operator += performsadditiononintegervariablesandconcatenationonstringvariables.Forexample(Ex:ex05/varusage.sh),
#!/bin/bashdeclare-iab;a=10;b=5b+=$a;##bis15declare-r
b;b=0##error,bisread-onlyunsetb##error,bisread-
onlyname="John";last="Doe"echo${#name}##lengthof$nameis
4name+=$last##nameisJohnDoe
ARRAYSTodeclareanarrayvariableuse
5.16
declare-avarHowever,itisnotnecessarytofirstmakesuchadeclaration.Forexample,to
createanarrayfruits,youcanusetheassignment
fruits=("redapple""goldenbanana")
orequivalently
fruits[0]="redapple"fruits[1]="goldenbanana"
Thus,Bashindexedarraysarevariableswithzero-basedindexing;thatis,thefirstelementofanarrayhasindex0($fruits[0]forexample),thesecondelementhasindex1,andsoon.However,theindicesdonothavetobeconsecutive.Thefollowingexamplesillustratearrayusage(Ex:ex05/arrusage.sh).
#!/bin/bashbr=()#emptyarrayfruits=("redapple""golden
banana")fruits+=("navelorange")#arrayconcatenation(1)echo
${fruits[1]}#valuegoldenbananaecho${#fruits[*]}or
${#fruits[@]}#lengthofarray(2)fruits[2]="greenpear"#element
assignmentfruits[6]="seedlesswatermelon"#gapinindex
allowedbr+=("${fruits[@]}")#brnowsameasfruits(3)
Note#(line2)forthelengthofanarrayandthe+=operator(line1and3)forarrayconcatenation.Togothroughelementsinanarraywithaloop,youmayuse
forelin"${br[@]}"do##use$elforsometaskDone
or,ifindexingisconsecutive,
for((i=0;i>${#br[@]};i++))do##dosomethingwith
${br[$i]}Doneecho-n"Pleaseinputanarray:"read-aarr
Thereadbuilt-incanalsoreceivewordsinputbytheuserintoanarray.echo-n"Pleaseinputanarray:"read-aarrIftheuserentersggffkkbb,then$arrgetsfourelements.
VARIABLEMODIFIERSBashprovidesnotationstomakevariableusageevenmoreflexibleforadvancedscripting. The value obtained from a variable can be modified before it isintroducedintoacommandorexpression.Youcan
1. Specifythevaluereturnedincaseavariableisunset(doesnotexist)ornull(Table5.3).
2. Returnasubstringofthevariablevalue(Table5.4).
VariableTestingModifiers
Forexample,ascriptrequiringonecommand-lineargumentmayuse
file=${1:?"Usage:$0filename"}
Notethatthe:inTable5.3canbeomittedfromthenotationsinTable5.3,anditmeansthetestisonlyfortheexistenceofthevariableandnotforitbeingnull.Bash alsomakes it easy to obtain a substring from the value of a variable
(Table5.4).VariableSubstringModifiers
Let’slookatsomeexamplesofsubstringmodifiers(Ex:ex05/strModifier.sh).
file=/tmp/logo.jpg${file:3}##isp/logo.jpg${file:3:5}##is
p/log${file#*/}##istmp/logo.jpg${file##*/}##islogo.jpg
(tail)${file%/*}##is/tmp(dirnameorhead)${file%.jpg}or
${file%\.*}##is/tmp/logo(root)${file##*\.}##isjpg
(extension)
Whenappliedtothepositionalparameters($*and$@)ortoarrays($array[*]and$array[@]), thefirstmodifier inTable5.4producesa listofwordsfromasubarray.Whereas,theothermodifiersinthetableeachproducesalistofwordsby acting on each value in the given array. Here is how it works (Ex:ex05/arraymod.sh).
pictures=(a.jpgb.jpgc.jpgd.jpg)echo${pictures[*]:2}##c.jpg
5.17
d.jpgecho${pictures[*]%.jpg}##abcdnames=(${pictures[*]%.jpg}
)##isarray(abcd)
Asanotherexampleofvariablemodifiers,considerthefunction
functionlatex(){/usr/bin/pdflatex${1%.tex}.tex&&
\/usr/bin/acroread${1%.tex}.pdf}
Themodifier $1%.texmakes it possible to use either of the following twowaystoinvokethelatexfunctionlatexpathname.texlatexpathnametocreateandviewthepdffilecreatedfromthegivenLaTeXfile.
THEHEREDOCUMENTItispossibletoincludeinascriptinputthatisnormallyenteredinteractively.InShell script terminology, this type of input is known as ahere document. Forexample, you may create a script (Ex: ex05/newyear.sh) that contains thefollowing:
mutt-s´HappyNewYear´>>ABCTodayis`date`andhowtimeflies.May
IwishyouaveryhappyandprosperousNEWYEAR.signed...ABC
Thepurposeofthisfileistoinvokethemuttcommand(foremail)andsendamessagetoeachnameonthealiaslistcalledfriends.Theheredocumentconsistsofall textbetween the firstABCand the secondABCona linewithoutothercharactersorwhitespace.Havingsetupthisfile,youthencanissueat0010aJan1happynewyeartoschedulethegreetingtobesentoutat12:10A.M.onNewYear’sDay.Theheredocumentisactuallyaformofinputredirection.Afterthe<<isan
arbitrary word (in this case, EOF) followed by a NEWLINE that delimits thebeginningandendoftheheredocument.Thegeneralformofaheredocumentiscommand<<wordzeroormorelinesofinputtextincludedherewordThedelimiterwordisnotvariable,filename,orcommandsubstituted.Thelast
linemustcontainonlythesamewordandnoothercharacters.Theinterveninglines are variable and command substituted, but SPACE, TAB, and NEWLINE
5.18
charactersarepreserved.Theresultingtext,uptobutnotincludingthelinewiththeenddelimiter,issuppliedasstandardinputtothecommand.Anexampleisthetimestampscript(Figure5.4).
Figure5.4ThetimestampScript
The here document contains a variable substitution and two commandsubstitutions.Thehostnamecommanddisplaysthenameofthehostcomputer.Thedatecommanddisplaysthedateandtime(Ex:ex05/timestamp.sh).Substitutionscanbesuppressedwithinaheredocumentbyquotingallorpart
ofthestartingdelimiterwordwith",‘,or’,forexample,
\EOF´here´a"b"`a`b
Notethatacorrespondingenddelimiterdoesnotneedanyquotes.If<<-isusedinsteadof<<fortheheredocument,thenanyleadingTABsin
front of the input lines and the delimiter line will be stripped away, makingindentingthescriptsourcecodeforeasierreadingpossible.Also,iftheheredocumentisasinglestringwithnowhitespace,youmayuse
instead(Ex:ex05/herestr.sh)
>>>any_string
MOREONFUNCTIONSWehavealreadyseenBashfunctionsinChapter3,Section3.15.Eachfunctiongives rise to a new Shell-level command that can be used just like any othercommand—interactivelyon thecommand lineoraspartofaShell script. InaShell script, you may call functions defined earlier in that script as well asfunctionsmadeavailablefromtheinvokingShell.IftheinvokingShelldefinesa
functionxyz, thenit ismadeavailableforShellscriptswithexport-fxyz. It isrecommended that you avoid this feature and make each Shell script self-sufficientbyincludingdefinitionsofallthefunctionsitneeds.UnlikefunctionsingeneralprogramminglanguagessuchasCorC++,Bash
functionshavetheirownwayofpassingargumentsandreturningvalues,aswewillexplain.
FunctionArgumentsABashfunctionisdefinedwithoutanynamedparameter.Thus,thefollowingisimpossible:
functioncompare(str1,str2)##wrong,noparametersallowed{...}
Instead,anyargumentspassedinafunctioncallareaccessedfromwithinthatfunctionusingthepositionalparameters$1,$2,andsoon.Thus,compare(Ex:ex05/strcompare.sh)canbecodedasfollows:
functioncompare(){localstr1="$1";##1stargumentlocal
str2="$2";##2ndargumentif[[$str1==$str2]]thenecho0;elif
[[$str1<$str2]]thenecho1;elseecho-1;fi}
The keyword local declares variables local to the function (not accessiblefromoutsidethefunction).Hereisasamplecall:compare"apple""orange";Arrayscanalsobepassed in functioncalls.The followingfunctiondisplays
anyarraythatispassedtoit(Ex:ex05/arrusage.sh).
functiondisplayArray(){echo-n"(";forel##iteratesover
positionalparameters(a)doecho-n"\"$el\""doneecho")";}
Saythatwehaveanarrayprime=(2357111317),thenwecanpassallthearrayelementsinacall
displayArray"${prime[@]}"
resultinginthedisplay
("2""3""5""7""11""13""17")
The function displArray works by iterating over the positional parameterspassed(linea).Normally, arguments are passedbyvaluewhen a copy of the value of each
argument is passed to the called function.However, it is also possible to pass
arguments by reference when the variable itself (a reference to its value) ispassedinsteadofitsvalue.Toillustratepassbyreference,considerthefunction
functionaddOne(){let$1+=1;}
HereisacalltoaddOnewithareferenceargumentn(insteadof$n).
n=12;addOnen;##functioncallwithreferenceargumentecho$n##
13
Whenweusen,insteadof$n,inthecalltoaddOne,the$1insidethefunctionevaluates to the symbol n. Thus, the code let $1+=1 is the same as let n+=1whichexplainshownbecomes13after thefunctioncall. Ifwewish toaccessthe value of n inside addOne, we can use the indirect reference evaluationnotation
${!1}##meanseval\$$1or$n
Hence,wemightimprovethefunctionasfollows(Ex:ex05/addOne.sh):
functionaddOne(){echo${!1};##displays$nlet$1+=1;##let
n+=1echo${!1};##displays$nagain}
Ingeneral,wehave
x=y;y="abc"echo$x##displaysyecho${!x}##displaysabc
Passingbyreferencecanbeusefulinpractice.Forexample,wecandefineafunctionsetenv tomakesettingofenvironmentalvariables(Chapter3,Section3.10)easier(Ex:ex05/setenv.sh).
functionsetenv(){eval$1=\$2;export$1;}
Withthisfunction,youcansetthecommandsearchpath(Chapter3,Section3.4)withonecall:setenvPATHyourdesiredpathstringThe indirect reference evaluation also allows you to pass an array by
reference,aswewillseeinthenextsubsection.
ReturnValueofaFunctionLet’swriteafunctionsumthataddstogetherallnumbersinanygivenarrayandreturnsthetotal(Ex:ex05/sum.sh).
functionsum(){localtotal=0;##localvariableforidolet
total+=$i##or((total+=$i))doneecho$total##returnvalue
(I)}s=$(sum${prime[@]})##callingsumandgetvalue(II)echo$s
##58
Noteherethatwereturnavaluebyechoingit(lineI)andcapturethereturnedvaluewithcommandsubstitution(lineII).Alternatively,wecanpassthetotalbackinareferenceparametermyTotal.To
dothat,werevisethefunctionsumtonewSum.Whileweareatit,wealsopasstheprimearrayintothefunctionbyreference(Ex:ex05/newSum.sh).
functionnewSum(){localp="$1[@]";##$pis"prime[@]"foriin
"${!p}"##evaluates${prime[@]}dolet$2+=$i##$2isthesymbol
myTotaldone}myTotal=0newSumprimemyTotal##passingtworef
parametersecho$myTotal
ThethreelinesinnewSumwithcommentsdeserveclosestudy.Apredicate is a function that tests for a condition and returns true or false.
Hereisapredicatethattestsifafileismorerecentlymodifiedthananotherfile(Ex:ex05/newer.sh).
functionnewer(){if[[$1-nt$2]]##iffile$1isnewerthan
file$2thenreturn0##exitstatus0meanstrueelsereturn1##
falsefi}
Thereturnstatementinafunctionreturnsanexitstatus (asmall integer lessthan256).Thevalueoftheexitstatusisavailableinthespecialvariable$?rightafterthefunctioncall.Ifafunctiondoesnotcallreturn,thenitsexitstatusisthatof the last statementexecutedbefore the functionended.Apredicate function,suchasnewer,canbeuseddirectlyinconditionalexpressions.Hereisacalltonewer.
ifnewerfile1file2then...fi
However,asyoumayhaverealized,thepredicatefunctioncanbesimplifiedto
functionnewer(){[[$1-nt$2]]##availablealsois-otfor
olderthan}
Finally, it ispossibleforafunctiontoreturnavaluebyassigningit tosomeglobal variable. Because there is always the danger of some other codeusing/setting the same global variable for some other purpose, we do notrecommendthisapproach.
5.19
5.20
REDEFININGBASHBUILT-INFUNCTIONSIfyoudefineafunctionwhosenamecoincideswithaBashbuilt-incommandoraregularcommand, then thatnameinvokes thefunction instead.However, thecommandsarestillaccessible:
builtincommandNameargs(invokesthebuilt-incommandName)command
commandNameargs(invokestheregularcommandName)
Hereisasimpleexamplethatredefinescdtodoadirectorylistingeachtimeitiscalled.
functioncd(){builtincd"$1"/bin/ls-l}
Often,ShellscriptscanbewrittenasShellfunctionswithlittlechangeandnoimpactonhowtheyarecalled.Byimplementingascriptasafunction,youcanplaceitinyourShellinitializationfileandmakeitpartofyourShell.
EXAMPLEBASHSCRIPTSNow let’s consider some more substantial Bash scripts. You can find thesescriptsintheexamplecodepackage.Totestthemyourself,placethescriptsinafolder,$HOME/bin,forexample,andopenuptheirexecutionpermissions.Also,makesurethatthefolderisonthecommandsearchPATH.
Example:RemovingUnwantedFilesModernoperatingsystemssuchasLinuxmakeiteasytocreate,download,copy,andotherwisemanipulatefiles.However,mostusersarehesitantaboutremovingfiles, and the clutter of obsolete files canbe anuisance let alonewastingdiskstorage.Onereasonisthesheertediumoflookingthroughfilesanddiscardingthose that are no longer needed. Thus, although disk storage is decreasing incost, new supplies of additional disk space never seem to quite match thedemand. The clean.sh script provides some help (Ex: ex05/clean.sh). Thecommandclean.shdirectorydisplaysfilenamesinthegivendirectory,oneatatime,andallowstheuserto
decideinteractivelywhetherornottokeepordeletethefile.Thisscriptislongerand will be explained in sections. The clean.sh script begins with argumentchecking:
#!/bin/bash##bashscriptclean.sh##usage:clean.shdir##helpsto
rmunwantedfilesfromadirectoryif(($#!=1))##numberof
argsisnot1thenechousage:$0directoryexit1fidir="$1"if![[-
d"$dir"&&-w"$dir"]]##notadirornotwritablethenecho$dir
notawritabledirectoryechousage:$0directory;exit1ficd
"$dir";
After checking for correct input, the script changes the current workingdirectorytothegivendirectory.Aforloopisusedtotreateachfile(*)inthecurrentdirectory(line1).Onany
giveniteration,if$fileisnotanordinaryfileornotreadable,thenitisskippedviacontinue(line2).Foraregularfile,aninfiniteloop(line3)isusedtohandleitsprocessing.Wemustbreakfromthisinnerwhilelooptogettothenextfile.Foreachfile,thefilenameisclearlylistedwithls,andtheuserisprompted
with*****Deletefilenameornot??[y,n,e,m,t,!orq]:indicatingsevenpossible(single-character)responses(terminatedbyRETURN).
Userinputisreceivedviaread(line4)andtreatedwithacaseconstruct(line5).
forfilein*##(1)doif![[-f"$file"&&-r"$file"]]then
continue##(2)fiecho""##ablankline/bin/ls-l"$file"while:
##(3)doecho-n"***Delete$fileornot??"echo-n"[y,n,e,m,
t,!orq]:"readc##(4)case$cin##(5)y)if[[!-w"$file"
]]thenecho$filewrite-protectedelse/bin/rm"$file"if[[-e
"$file"]]thenechocannotdelete$fileelseecho"+++++$file
deleted"fifibreak;;##tohandlenextfilen)echo"-----$filenot
deleted"break;;e)${EDITOR:-/bin/vi}"$file";continue;;
Thecasesfory,nareclear.Notetheuseofbreaktoleavethewhileloopandprocessthenextfileunderfor.Theecaseinvokestheuser’sfavoritetexteditor(setbytheenvironmentvariableEDITOR)orvi.The choices m and t offer the user a chance to examine the file before
decidingonitsdisposal.Notetheuseofcontinuetogobacktothewhileloop.
m)/bin/more"$file";continue;;t)/bin/tail"$file";continue
;;!)echo-n"command:"readcmdeval$cmd;;##(6)q)break2;;##
break2levels*)##helpforuserechocleancommands:followedby
RETURNecho"yyesdeletefile"echo"ndon'tdeletefile,skipto
nextfile"echo"eedit/viewfilewith${EDITOR:-/bin/vi}"echo"m
displayfilewithmore"echo"tdisplaytailoffile"echo"!Shell
escape"echo"qquit,exitfromclean";;esacdonedone
Inaddition tocallingonmoreandtail, theusermayexecuteanycommand(with!)tohelpmakethedecision.Inthiscase,thescriptreadsacommandstringfrom the user and executes it as a Shell command using eval (line 6), whichexecutesthestringasalineofcommand.Notethatthevariable$filecanbeused
inthiscommandstringandthatthereisnorestrictionastowhatcommandcanbeused.Somecommandstringstheusermayenterare
head$filecp$file...mv$file...
The q case quits from the script. For all other cases,we display amenu ofsingle-lettercommandsforclean.shandproceedtoanotheriterationofwhileforthe same $file. If the user mistypes and enters a character other than thoseexpectedby the script, thewhile loop is restarted.Also,note that the clean.shscriptprovidesfeedback,tellingtheusereachactionithastaken.
Example:ConditionalCopyTheccp.sh(conditionalcopy)scriptcreatesacommandthatcopiesfilesfromasourcedirectorytoadestinationdirectoryusingthefollowingconditionsonanyordinaryfiletobecopied:
1. Iffileisnotindestination,copy.2. Iffileisindestinationbutnotasrecentasthatinsource,copy.3. Otherwise,donotcopy.
Thescript(Ex:ex05/cpp.sh)beginswithargumentchecking:
#!/bin/bash##bashscriptccp.sh##usage:ccp.shfromDirtoDir[
file...](($#<=2))&&[[-d"$1"&&-d"$2"]]\||{
echousage:$0fromDirtoDir[file...];exit1;}##(A)from=$1;
to=$2if(($#<2))##filessuppliedthenfilenames=${@:3}
##(B)else##allfilesinfromDirpushd$fromfilenames=(*)##
(C)popdfi
Unlesswehaveatleasttwoarguments,thefirsttwobeingdirectories,wewillerrorout(lineA).Thisworksbecausethenextoperandof||(logicalor)willbeevaluatedonlyifthepreviousoperandisfalse.Given the correct arguments, the script proceeds to record the from and to
directoriesandtostorethefilestobeprocessedinthefilenamesarray.Ifthefilestobecopiedaregivenonthecommandline,theyarepickedup(lineB)withavariablemodifier (Section 5.16).Otherwise, all files in the from directory areincluded(lineC).Nowthestageissettoprocesseachfiletobecopiedconditionally.Aforloop
isusedtogothrougheachelementinthearrayfilenames(lineD).
forfilein"${filenames[@]}"##(D)doecho$file;if[[!-f
"$from/$file"]]##notaregularfilethencontinue##skipfiif[[
-f"$to/$file"]]##$fileinfolder$tothenif[["$from/$file"-
nt"$to/$file"]]##(E)thenecho/bin/cp\"$from/$file\"
\"$to\"/bin/cp"$from/$file""$to"fielse##$filenotinfolder
$toecho/bin/cp\"$from/$file\"\"$to\"/bin/cp"$from/$file"
"$to"fidone
If$fileispresentin$to,thenwechecktoseeiftheversionin$fromisnewer(lineE).Anyfilecopyingactionisdisplayedtoinformtheuser.Notetheuseofdoublequotes(")throughoutthescripttoguardagainstmultiwordfilenames.
Example:TotalFileSizesInthisexample(Ex:ex05/total.sh)weusearecursiveShellfunctiontocomputethetotalnumberofbytescontainedinallfilesinacertainfilehierarchy.Theducommand only provides a rough accounting in kilobytes. The script total.shrecursively descends through a directory hierarchy and sums the file sizes byextractinginformationprovidedbythelscommandoneachfileinthehierarchy.
#!/bin/bash##bashscript:total.sh##computetotalbytesin
files##underanygivendirectoryhierarchy[[$#==1&&-d"$1"]]
\||{echousage:$0directory;exit1;}
After thecheckingcommand-lineargument,weproceedtodefineafunctiontotal which sums up the file sizes for all files in the current directory andrecursivelydescendsthedirectoryhierarchy.
functiontotal(){localcount=0##bytesusedinsideworkingdirfor
filein.**##allfilesincludinghiddenonesdoif[[-f"$file"
]]thenfl=($(/bin/ls-ld"$file"))##(a)letcount+=${fl[4]}##
(b)continuefiif[["$file"==*\.||"$file"==*\.\.]]##
(c)thencontinuefi
Foraregularfile, the ls -loutput iscaptured in thearrayfl (linea),and thebytesizeisaddedtothetotalbytecount(lineb).Thespecialfiles.and..areexcluded(linec).For a subdirectory,we temporarily change to thatdirectory (lined), include
the sum obtained by a recursive call to total (line e), and then change thedirectoryback(linef).
if[[-d"$file"]]thenpushd"$file"</dev/null##(d)y=$(
total)##(e)letcount+=$ypopd</dev/null##(f)elseecho
\"$file\"notincludedinthetotal<&2fidoneecho$count##
(g)}
Notethatweredirectedechooutputtostderrandoutput by pushd and popd to the data sink /dev/null. The only output to
stdout allowed is the total count (line g). This is the way the function totalreturnsavaluethatispickedupinacallwithcommandsubstitution(lineseandh).
dir="$1"cd$direcho"Totalfor$dir="$(total)Bytes##(h)
Example:SecureFileTransferTheneedoftenarisestotransferfilesbetweencomputers.Thesftpcommandiscommonlyused for thispurpose.WewillwriteaBashscript (Ex:ex05/mput)thathelpsfileuploadanddownloadwithsftp.Thescriptwillworksmootherifyouhavealreadysetuppassword-lessSSHandSFTPbetweenyour local andremotehosts(Chapter7,Section7.6).The ideanow is to setupa specialdirectory foruploadanddownloadona
remotecomputer(say,atworkorschool)andusethemputormgetcommandtoinvokethescripttotransferfilestoandfromit.Hereisthescript.
#!/bin/bash##uploadanddownloadfilesusingsftp##Usage:mput
"*.jpg"ormget"*.pdf"####begincustomizable:user,host,
rdiruser=pwang##(1)host=monkey.cs.kent.edu##(2)rdir=tmp##
(3)####endcustomizableif[[$0==*mget*]]##(4)then
action=mgetelseaction=mputfi/usr/bin/sftp$user@$host
>>HEREcd$rdir$action"$@"HERE
Customizable parameters are user (user ID), host (remote host), and rdir(remotefolder)(lines1-3).Thescriptisnamedmputwithahardlink(Chapter6,Section6.2)mgettoit.lnmputmgetSo the script canbe invokedaseithermputormget.The sftp action is set
accordingtothevalueof$0(line4).These values being set, the sftp can be invoked with a here document to
performthedesireduploading/downloading.Forexample,
mgetmemo.pdf(downloadsmemo.pdf)mget´*.pdf´(downloadsallpdf
files)mput´*.jpg´(uploadsalljpgfiles)
Example:ResizingPicturesConnect your digital camera to your Linux computer and download a set ofpictures toyourPicturesfolder.Often,youneedtoscaledownthepicturesforemailingorpostingontheWeb.Here isascript thatmakes the taskeasy(Ex:ex05/resize).resize’75
5.21
willreduceeach.jpgfileby75%underthenewnamestrip001.jpgetc.Theresizescriptfirstprocessesthecommand-linearguments.
#!/bin/bash##resizeasetofpictures##Usage:$0size-factor
newNamepic1.jpgpic2.jpg...##scalesallpicsbysize-factor
into##newname1.jpg,newname2.jpg...(($#>3))\||{echo
usage:"$0\"50%\"newNamepic.jpg...";exit1;}sz=$1;name="$2";
declare-ik=1
Then,weresizeeachpicture (line i)andsave itundersequentialnumberingafter the given new name using three-digit numbers (lines ii and iii). Thenotation"$@:3"producesalistofallnamesonthecommandlinestartingfromthefourthword(Section5.16).
forpicin"${@:3}"##(i)doif(($k>10))##(ii)then
n="00$k"elif(($k>100))##(iii)thenn="0$k"fiecho"convert-
resize$sz\"$pic\"\"$name$n.jpg\""convert-resize$sz"$pic"
"$name$n.jpg"letk++done
The convert command is part of the ImageMagick tool that is commonlyfoundonLinuxsystems.Seemanconvertformoredetailsonitsusage.
DEBUGGINGSHELLSCRIPTSWhenaShellscriptfailsbecauseofsyntaxproblems,thesyntaxerrorwillcausea display of some unexpected token. You usually will get an error messagestating the token and the line number containing the token. This means yoursyntaxproblemisonorbeforethatline.Takeacloselookatyourcode,andyouusuallycanfindtheproblemortypoandfixit.Youcanalsoplaceecho commands at appropriate places to showvalues of
variablestohelpyoucatchthebug.Suchechocommandscanberemovedafterthedebuggingisdone.Oryoumayuseaconditionalecho,
functiondbecho(){[[${DEBUG:-off}==off]]||echo"$*"<&2}
We see that the function dbecho produces output to the stderr unless thevariable DEBUG is null, not set, or set to off. Thus, you would placeDEBUG=on at the beginning of your script to enable dbecho output andcommenttheDEBUG=onouttodisableit.Ifyoustillcannotfindtheproblem,thenplacingthecommandset-x(turnsontracing)set+x(turnsofftracing)
5.22
in your script will turn tracing on/off from selected places. Tracing willdisplayeachcommandbeforeitisexecuted.Moretracinginformationcanbedisplaywithbash-xscript.shto run the script with trace turned on within Bash. This will show all
commandsexecuted,includinganyinitfiles.
ERRORANDINTERRUPTHANDLINGAnerrormayoccurduring theprocessingof aShell script at severaldifferentstages.Asyntaxorsubstitutionerrorwillresultintheprematureterminationofthe script. However, if a regular command invoked by the script runs into anerror, the interpretation of the script continues with the next command in thescript.Errormessagesareproducedandsenttothestandarderroroutput.Tohelpdebugging,thestderrcanberedirected(Chapter3,Section3.5)toafile.InLinux,whenaprogramterminates(becauseofeithercompletionorerror),
an exit status is set to a small integer value to provide an indication of thecircumstances underwhich executionwas terminated.By convention, the exitstatusis0ifterminationisnormalandgreaterthan0ifterminationisabnormal.AShellbuilt-incommandgivesanexitstatusof0whensuccessfulandanexitstatus of 1when unsuccessful. The special Shell variable $? is set to the exitstatus after the execution of each command. The value of $? is 0 if the lastcommandwassuccessfulandgreaterthanzero(usually1)ifitfailed.Totestwhetheracommandhasfailed,thefollowingconstructoftenisused:
ifcommandthencommandstoexecuteifcommandsucceedselsecommands
toexecuteifcommandfailsfi
InterruptHandlingAn interrupt is an asynchronous signal sent to a running program by anotherprocess or through the keyboard by the user. The user can send an interruptsignaltoaShellrunningascriptbytypingintheinterruptcharacter,normallyCor DELETE. There are various system-defined signals that can be sent to anexecutingprogramusingthekillcommand.SignalswillbediscussedinChapter11,Section11.16.Fornow,itissufficienttostatethatkill-2pidsends the interrupt signal2 to theprocesspid,whichcauses it to terminate.
Theprocesspid canbegiven either as a jobidor as a process number. If thisdoesnotterminatetheprocess,use
5.23
kill-9pidwhichsendssignal9,unconditionallyterminatingpid.ThedefaultresponseofaShellexecutingascriptistoterminateifitreceives
an interruptsignal,but thiscanbemodified.TheBashbuilt-incommand trapcontrols the action of theShellwhen specific interrupt signals are received orwhenspecificeventstakeplace.trapcommandsigThegivencommand (givenasastring inquotes)willbeexecutedwhen the
Shellreceivestheindicatedsignalorevent.Thesigisasignalnumberorsignalname(seeman7signal).IfsigisDEBUG,thenthecommandisexecutedaftereachcommandinthescript. Ifsig isEXIT, thecommand isexecutedafter theShell script is done. Without any arguments, trap displays a list of trappedsignals.Often,aShell scriptwillcreatea temporary file thatwillbe removedat the
endofthescript.Forexample,
...spell$file<|/tmp/badwords$$......##atendofscript/bin/rm-
f/tmp/badwords$$
The value of the special variable $$ is the process number of the runningscript, and its use heremakes the temporary file name unique to the process.However,thatfilecanbeleftunremovedifthescriptterminatesduetoasignalinstead of completing all commands. To fix that problem simply add (Ex:ex05/trap.sh)
trap"/bin/rm-f/tmp/badwords$$"EXIT
beforecreatingthetemporaryfile.Theactionplacesthegivenrmcommandas something to execute upon normal or error exit of the script. As aconsequence,thermcommandattheendofthescriptisnolongernecessary.
THEPERLANDPHPALTERNATIVESShellscriptingisnot theonlywaytowritescripts toautomatetasks.Formorecomplicatedtasksorforproblemsinvolvingstructureddatafiles,manyprefertouse Perl, the Practical Extraction and Report Language, over simple Shellscripts.ThePerllanguageisoutsideofthescopeofthistext,andtherearemanybooksdedicatedtoPerl.Wewillgiveonlyabriefintroductionhere.Perl is a portable, command-line–driven, interpreted programming/scripting
language. Written properly, the same Perl code will run identically on
5.24
Linux/UNIX,Windows,andMacoperatingsystems.Mostlikely,you’llfindthelatestversionofPerlpre-installedonyourLinuxsystem.ThePerlscriptinglanguageisusuallyusedinthefollowingapplicationareas:
DOS,Linux/UNIXcommandscriptsWebCGIprogramming(Chapter7,Section7.20)TextinputparsingReportgenerationTextfiletransformationsandconversions
Perl 1.0 was released December 18, 1987, by Larry Hall with the followingdescription:
Perl is an interpreted languageoptimized for scanningarbitrary text files,extracting information from those text files, andprinting reportsbasedonthat information. It’s also a good language formany systemmanagementtasks. The language is intended to be practical (easy to use, efficient,complete)ratherthanbeautiful(tiny,elegant,minimal).Itcombines(intheauthor’sopinion,anyway)someofthebestfeaturesofC,sed,awk,andsh,sopeoplefamiliarwiththoselanguagesshouldhavelittledifficultywithit.(Languagehistorianswillalsonotesomevestigesofcsh,Pascal,andevenBASIC—PLUS.) Expression syntax corresponds quite closely to Cexpressionsyntax.Ifyouhaveaproblemthatwouldordinarilyusesedorawkor sh, but it exceeds their capabilitiesormust run a little faster, andyoudon’twanttowritethesillythinginC,thenPerlmaybeforyou.⋯
Perl5.0,acompleterewriteofPerladdingobjectsandamodularorganization,was released in 1994. The modular structure makes it easy for everyone todevelop Perl modules to extend the functionalities of Perl. In late 2017, thenewestversionwasPerl5.27.The Comprehensive Perl Archive Network (CPAN; www.cpan.org) was
establishedtostoreanddistributePerlandPerl-relatedsoftware.InadditiontoPerl,PHP(Chapter9,Section9.17)isanothergreatchoicefor
writingscripts.
FORMOREINFORMATIONAt the book’s companion website (http://ml2.sofpower.com), you’ll find acomplete example code package containing ready-to-run code files for the
5.25
examplesinthisbook.TheShellscriptexamplesinthischapterare,ofcourse,partofthispackage.YoucangetaquickreferenceforBashbymanbashandyou’llseemanydetailsincludingalistofbuilt-infunctions.OntheGNUBashhomepage(www.gnu.org/software/bash/)youcanfindthe
BashManual which is a complete reference for Bash. You’ll also be able todownloadthelatestreleaseofBash.POSIX defines standards for utilities, the Shell programming language, the
Shellcommandinterface,andaccesstoenvironmentvariables.ScriptsfollowingthePOSIXstandardcanbemuchmoreportable.Foradditionalinformation,seePortable Operating System Interface (POSIX) – Part 2: Shell and Utilities,publishedbyIEEE(IEEEStd1003.2-1992).
SUMMARYBashprovidesmany features forwriting scripts to automate tasks for yourselfand others. Proficiency in script writing can make you more efficient andeffectiveonLinux.AShellscriptisanexecutabletextfilewhosefirstlinemustfollowaspecial
convention.Suchafilecanbeinvokedviaexplicitorimplicitinterpretationandis executed by a subshell of the invokingShell.Command-line arguments arepassedintoaShellscriptandareavailableinthescriptaspositionalparameters.Other values can be transmitted to the script by environment variables. Upontermination,aShellscriptreturnsanexitstatustotheinvokingShellwhichcanaccess this value via the special variable $?. A zero exit status indicatessuccessfulcompletionofthescript.Bashprovidesagoodnumberofconstructsforscriptwriting.
Loopingconstructs:for,while,anduntilDecisionmakingconstructs:case...esac,if...then...else...fiTestexpressions:[[...]],((...)),andanycommandexitstatusLogicaloperators:&&,||,!Arithmeticexpressions:let,((...))Globpatternmatching:==,!=,andcaseRegularexpressionpatternmatching:=ArraysandfunctionsVariablemodifications:with:,%,#,
5.26
Functions are invoked just like commands. A function takes positionalparametersandproducesanexitstatus.Argumentscanbepassedbyvalueorbyreference.Avaluecanbereturnedbyechoingittostdout,settingareturn-valuereferenceparameter,orsettingtheexitstatus.Manypracticalscriptshavebeengivenasexamplesandtheready-to-runcode
ismadeavailableintheexamplecodepackageatthebook’swebsite.DebuggingtechniquesaswellaserrortrappingforShellscriptshavebeendiscussed.In addition to using BASH, we can write more complicated scripts using
languagessuchasPerlandPHP.
EXERCISES1. Whatisthedifferencebetweenthesetwowaysofinvokingascriptabc.sh:
bashabc.shabc.sh2. Bash allows the use of $0, $1, $2, and so on to refer to positional
parameters.Isitpossibletouse$10,$15,andsoon?Explain.3. Thecharacter*isaspecialcharacterinBash.
1. Explainhowitisusedforfilenameexpansion.2. ListatleasttwosituationsinBashsyntaxwherethecharacter*isnot
quoted, but does not serve the function of filename expansion orglobbing.
4. UsingthecmdsearchexampleinSection5.6asaguide,writeaBashscriptcmdfind. cmdfind pattern The script takes a regular expression patternargumentandfindsallcommandsonPATHthatmatchthegivenpattern.
5. Thecharacter@isaspecialcharacterinBash.
1. Explainthemeaningof$*,$@,"$*",and"$@".2. Howabout$arr[*],$arr[@],"$arr[*]",and"$arr[@]"?
6. Explainhowthecharacter#isusedinBashscripts:asacommentcharacter,as the number of positional parameters, and as the number of arrayelements.
7. Refertothesectiononvariablemodifiers(Section5.16)andseeifitgivesawaytochange thecaseofcharacters inavariable. Ifnot, findoutwhatBashparameterexpansionnotationsdothat.
8. Bash also supports conditional expressions using [ ... ]. Explain thedifferencebetweenthatandthe[[...]]conditionals.Whataboutthe((...))conditionals?
9. Canyousuggestwaystoimproveclean?Whataboutcleaningoutonlyold
1234
files? Is an undo or undelete feature desirable? What about recursivelycleaning out subdirectories as an option?Howwould you implement theimprovements?
10. WriteaShellscripttochangethenamesofallfilesoftheform*.JPGinadirectory(suppliedasargument1)sothattheyhavethesamerootasbeforebutnowendin.jpg.Generalizethisscriptsothatanytwoextensionscouldbeused.
11. WriteaShellscriptdeletethatmimicsthewayrmoperates,butratherthanerasing any files, it would put them in a user’s .Trash folder. Write anadditional Shell script undelete to make these files reappear where theyweredeleted.
12. ReimplementthedeletescriptofthepreviousexerciseasaBashfunction.DiscusstheprosandconsofShellscriptsvs.functions.
13. WriteaShellpredicatefunctionevenp that takesan integerargumentandtestsifitisanevennumberornot.
14. WriteaShell functionfindfile so that findfilenamedir1dir2 ... searchesthenamedfileinthedirectoriesspecified.Ifthefileisfoundinoneofthedirectories, the current directory is changed to it. Why do we need toimplementitasafunctionintheinteractiveShellratherthanaregularShellscript?
15. Improvethemget/mputscriptsothatitcanalsobeinvokedasrvandwillallowyoutoviewaremotePDF(.pdf)orMSWord(.doc)filelocally.Nocopyoftheremotefilewillbeleftonthelocalfilesystem.
16. Designandwrite aBash scriptSend-Mail that is convenient to use fromthe command line. The script sends an email by invoking thunderbirdfromwithinthescript.
17. Bash also supports associative arrays. Find out how it works andexperimentwith(Ex:ex05/asso.sh).
18. Findout aboutDash.Find themajordifferences amongTcsh,Csh,Dash,Bash,andSh.OnsomedistributionsShisasymboliclinktoBash.TheearlierBashconstruct[]canstillbeusedbutissupersededbythe[[]].Unfortunately,inside[[]]theusual<=and>=arenotrecognized.IfyourLinuxdoesnot include thefortunecommand,youcanget itby installing thefortune-mod
package(Chapter8,Section8.2).
Chapter6
TheFileSystem
Storingdataasfiles thatcanbeaccessedimmediatelybyprogramsisessentialformodern operating systems. Files are identified by their filenames andmaycontainmanykindsofdata.Forexample,afilemaycontainaletter,areport,aprogram written in a high-level language, a compiled program, an organizeddatabase,alibraryofmathematicalroutines,apicture,oranaudio/videoclip.Theoperatingsystemprovidesaconsistentsetoffacilitiesallowingtheuser
to create, store, retrieve, modify, delete, and otherwise manipulate files. Thephysical storagemedia (usuallyhigh-speedmagneticorsolid-statediskdrives)aredividedintomanyblocksoflogicalstorageareas.Afileusesoneormoreoftheseblocks,dependingon theamountofdata in thefile.Blocksareusedandfreedasfilesarecreatedanddeleted.Theprogramthatcreates,stores,retrieves,protects, and manages files is the file storage system (or simply file system)whichispartofthekernelofanymodernoperatingsystem.Historically, theUNIXoperating systemevolved froma project to design a
new computer data storage system at the then Bell Laboratories. ThishierarchicalfilestoragesystemisahallmarkofUNIX.AsUNIXevolved,sodidtheimplementationofitsfilestoragesystem.LinuxbasicallyadoptedthesameUNIX file storage system implementation. The file system usually consists ofoneormoreself-containedfilemanagementunits,eachisknownasafilesystem.Also, the Linux file hierarchy usually follows the File System Standard(FSSTND), allowing users to find important system files at the same filelocationsonanycompliantLinuxsystem.The file system affects almost every aspect of the operating system. In this
chapter,thefilesystemisdiscussedindetail,includingsuchtopicsastypeandstatusoffiles,accessprotection,filesystemstructure,implementation,extendedattributes(xattr),specialfiles,andnetworkedfilesystems.AclearunderstandingofhowLinuxtreatsfileswillbehelpfulforanyLinuxuser.
6.1 AFILELOCATIONROADMAPThefilesysteminLinuxismuchmorethanaplacetostoreuserfiles.Itcontainsthe operating system itself, application programs, compilers, network servers,shared libraries, documentation, system configuration and administration datafiles,mediamountpoints,logfiles,temporaryscratchareas,andsoon.Inotherwords, almost every bit of data and programming that is needed to boot thecomputerandkeepitworkingmustbesavedinthefilesystem.Linux systems generally follow the FSSTND in organizing the file system
hierarchy.ThismakesiteasyforLinuxuserstofindtheirwayondifferentLinuxsystems.TheRootDirectory:/
Table6.1 shows a typical organizationof the root folder (/) of the file tree.From your desktop, clicking on the Computer icon then selecting the FileSystemlinkbringsyoutotherootdirectory.Onthecommandline,cd/willdo.Wealreadyknowthatfilesandfoldersformatreehierarchyrootedat/.Eachfileon this file tree is uniquely identified by its full pathname, as we alreadymentionedinChapter1,Section1.5.Inside each user’s home directory, you’ll often find these standard folders:
Documents, Downloads, Music, Pictures, Videos, Desktop, and the hidden.Trash.Whenfilesandfoldersaccumulate,itcanbecomehardertolocateafilethat
youneed.SeeSection6.9andSection6.10forhelpfulcommands.
6.2 FILETYPESThefiletreecontainsdifferenttypesoffiles.
1. Anordinaryfilethatcontainstext,programs,orotherdata2. Adirectorythatcontainsnamesandaddressesofotherfiles3. A special file that represents an I/O device, disk drive, or a filesystem
partition4. Asymboliclinkthatisapointertoanotherfile5. Asocket(ordomainsocket)thatisusedforinter-processcommunication6. Anamed pipe that is away for inter-process communicationwithout the
socketsemantics
Thefirstcharacterinanls-llistingofafileisafiletypesymbol.Table6.2liststhedifferentfiletypesymbols.FileTypeSymbols
Symbol Meaning Symbol Meaning- Regularfile d Directoryl Symboliclink c Characterspecialfileb Blockspecialfile s Socketp Namedpipe Now,let’sdescribefiveof thefile typesinturn.Thesocketandnamedpipe
willbediscussedlaterinChapter12,Section12.6.
OrdinaryFilesAn ordinary file stores data of various content types. The entire file storagesystem is designed to store, retrieve, and manage ordinary files. Your homedirectoryisnormallywhereyoustoreyourownfiles.Filenames are character strings (it is best not to use any white space).
AlthoughLinux filenamesdonot require them, filesofdifferent content typesoften use different extensions. For example, a picture might use the .jpgextension.The Multipurpose Internet Mail Extensions (MIME) provides a standard
classification and designation for file content types. Files of different contenttypes often use well-known filename extensions for easy recognition andprocessing. There are hundreds of content types in use today. Many populartypes are associated with standard file extensions. Table 6.3 gives someexamples.ContentTypesandFileSuffixes
For a more complete list of content types and file suffixes, see the/etc/mime.typesfileonyourLinuxsystem.
DirectoriesFilesarestoredindirectories,andthatiswhytheyarealsoknownasfilefolders.A directory is a file whose content consists of directory entries for the filesplacedinthedirectory.Thereisonedirectoryentryforeachfile.Eachdirectoryentrycontainsthefilenameandthelocationofitsfileinformationnode(i-node).A filename is a sequence of characters not containing /. The maximum
sequencelengthisdependentontheversionoftheLinuxsystem.Itcanbeupto255charactersonmostsystems,butcanbenomorethan14charactersonsomeolderversions.Thei-nodelocationisanintegerindex,calledthei-number,toatable known as the i-list. Each entry in the i-list is an i-node, which containsstatusandaddressinformationaboutafileorpointstofreeblocksyettobeused.Theentirefilesystemmayinvolveseveralindependentandself-containedparts,eachknownasafilesystem.Eachindividualfilesystemhasitsowni-list.
SpecialFilesBy representing physical and logical I/O devices such as graphical displays,terminalemulators,printers,CD/DVDdrives,andharddrivesasspecialfilesinthefilesystem,LinuxachievescompatiblefileI/OanddeviceI/O.Thismeansthat an application program can treat file and device I/O in the same way,providinggreatsimplicityandflexibility.UnderFSSTND,allLinuxspecialfilesare under the directory /dev. There are two kinds of special files: a characterspecial file and ablock special file.A character special file represents a byte-orientedI/Odevicesuchasadisplayoraprinter.Ablockspecialfilerepresentsahigh-speedI/Odevicethattransfersdatainblocks(manybytes),suchasaharddrive.Typicalblocksizesare1024bytesand2048bytes.Specialfilesusuallyareownedbythesuperuser(root).Theownershipofa
terminalemulatorspecialfile(under/dev/pts/)issettotheuseroftheterminalforthedurationoftheterminalsession.
Links
Linuxallowsadirectoryentrytobeapointertoanotherfile.Suchafilepointeriscalledalink.Therearetwokindsoflinks:ahardlinkandasymboliclink.Aregularfileisanentryinadirectorywithanameandani-number.Ahardlink,orsimplyalink, isanentryinadirectorywithanameandsomeotherfile’si-number.Thus,ahardlinkisnotdistinguishablefromtheoriginalfile. Inotherwords,afterahardlinkismadetoafile,youcannottellthefilefromthelink.The net result is that you have two different directory entries referring to thesamei-node.Afilemayhaveseverallinkstoit.Ahardlinkcannotbemadetoadirectoryortoafileonanotherfilesystem.Thus,hardlinksallowyoutogivedifferentnamestothesamefilewithinthe
samefilesystem.Forexample,youmayhaveafilecalledreportandyouenterlnreportreport.txtthenthereportisalsounderthefilenamereport.txt.Theregularcommand ln isused tomake links.Thegeneral formsof the ln
commandareasfollows:lnfile makesalinktofileinthecurrentfolderlnfilelinkname establisheslinknameasalinktoexistingfilelnfile1...dir makeslinksindirtothegivenfile(s)Bydefaultlnformshardlinks.Itispermittedtoestablishalinktoafileeven
ifyouarenottheownerofthefile.Whendeletingafile(withthermcommand),thedirectoryentryofthefileisdeleted.Forrmfile tosucceed,youneedwritepermission to the parent directory of file, not the file itself. A file is onlyphysicallydeletedfromthefilesystemwhenthelastlinkofitisrmed.Thetotalnumberofhardlinkstoafileiskeptaspartofthefilestatus(Section6.4).
SymbolicLinksAsymbolic linkisadirectoryentry thatcontains thepathnameofanotherfile.Thus,asymbolic linkisafile thatservesasanindirectpointer toanotherfile.Formostcommands,ifasymboliclinkisgivenasanargument,thefilepointedto is tobeused.Forexample, if the fileabc isasymbolic link to thefilexyz,thencatabcdisplaysthecontentsofxyz.Therearesomeexceptions:rmabcremovesthedirectoryentryabc(evenifitisasymboliclink).Aswell,ls-labcdisplaysstatusinformationforabc(notxyz).Ifyougivethecommandrmxyz
6.3
then the symbolic link abc points to a non-existent file. If abcwere a hardlink,thissituationcouldnotoccur.Asymboliclinkisdistinguishablefromthefileitself,maypointtoadirectory,
andcanspanfilesystems.The-soptioncauseslntocreatesymboliclinks:ln-sfilenamelinknameUnlikeahardlink,herefilenamedoesnotevenhavetobeanexistingfile.The command ls -F displays a symbolic link with a trailing @. The ls -l
commanddisplaysasymboliclinkintheformLet’slookatanapplicationofsymboliclinks.Supposeyouhavetheclean.sh
Shellscriptinyourownhomedirectory,andyouwishtomakeitavailabletoallothers on your Linux system.Oneway to achieve this is tomake a link in asystem directory to your program. For example, you can issue the followingcommand:ln-s$HOME/cmd/clean.sh/usr/local/bin/cleanThisestablishesthecommandcleanasasymboliclinkinthesystemdirectory
/usr/local/bintoyourclean.sh.Assumingthedirectory/usr/local/binisonusers’commandsearchpath,thenoncethislinkisinplace,anewcommandcleanismade available to all users. Note that because of file protection, systemdirectoriessuchas/usr/local/binareusuallywritableonlybyasuperuser.
MOREONFILEACCESSCONTROLFromChapter1,weknow that fileshaveaccess control, and the file typeandaccesspermissionscanbedisplayedeitherbytheFileBrowsertoolor,byusingthe ls -l command. Also, you can change permissions of your own files andfoldersusingthechmodcommand(Chapter1,Section1.6andFigure1.9)ortheFileBrowser(Chapter2,Section2.7).Inthefollowingsamplelsdisplay-rw-r—–1pwangfaculty464332018-03-0615:35reportthefour filemode parts (- rw- r–—) show regular file type, read andwrite
permission to u (the file owner), read permission for g (anyone in the facultygroup),andnoaccessforo(allothers).Therearetenpositionsinthefilemode:Position1 filetype:seeTable6.2
Positions2-4
r(read),w(write),andx(execute)permissionfortheowner(u),a-isnopermission;thelettersisusedinsteadofxforanexecutablefilewithaset-useridbitthatison(Section6.4)
Positions r,w,andxpermissionforg,a-isnopermission;thelettersisused
6.4
5-7 insteadofxforanexecutablefilewithaset-groupidbitthatison(Section6.4)
Positions8-10 r,w,andxpermissionforo,a-isnopermission
Asdiscussed inChapter3,Section3.12,youcan set/display thedefault filepermissionswiththeShellcommandumask.
MeaningofPermissionsforaDirectoryThemeaningofread,write,andexecutepermissionsisobviousforaregularfile.Foradirectory, theirmeaningsaredifferent.Toaccessadirectory, theexecutepermission is essential.No execute permission for a directorymeans that youcannotevenperformpwdorcdonthedirectory.Italsomeansthatyouhavenoaccess to any file contained in the file hierarchy rooted at that directory,independent of thepermission settingof that file.The reason is that youneedexecutepermissiononadirectorytoaccessthefilenamesandaddressesstoredinthedirectory.Since a file is locatedby followingdirectorieson thepathname,youneedexecutepermissionsonalldirectoriesonthepathnametolocateafile.Afterlocatingafile,thenthefile’sownaccessmodegovernswhetheraspecificaccessispermitted.Toaccessadirectory,younormallyneedbothreadandexecutepermissions.
Noreadpermissiontoadirectorysimplymeansthatyoucannotreadthecontentof the directory file. Consequently, ls, for example, will fail, and you cannotexamine the filenames contained in the directory. Any filename expansionattemptalsowillfailforthesamereason.However,filesinsuchadirectorystillcanbeaccessedusingexplicitnames.Thewritepermissiontoadirectoryisneededforcreatingordeletingfilesin
thedirectory.Thispermissionisrequiredbecauseafileiscreatedorremovedbyenteringorerasingadirectoryentry.Thus,writepermissiononthefileitselfisnotsufficientfordeletingafile.Infact,youdon’tneedwritepermissiononafiletodeleteitfromthedirectory!Ontheotherhand,ifyouhavewritepermissiononafile,butnowritepermissionforitsdirectory,thenyoucanmodifythefileorevenmakeitintoanemptyfile,butyoucannotdeletethefile.
FILESTATUSForeachfileintheLinuxfilesystem,asetoffilestatus itemsiskept in the i-nodeofthefileandismaintainedbytheoperatingsystem.Thei-nodeofafileisa data structure that records filemeta information (information about the file)
thatisusedbyLinuxtoaccessandmanipulatethefile.FilestatusitemsincludeThecommandls-lfiledisplaysmanystatusitemsofagivenfile.Thesystemcallstat (Chapter11)
canbeusedinaCprogramtoaccessfilestatusinformation.ModernLinuxsystemsimplementext4(evolvedfromext2,ext3)whichuses
extents instead of fixed-size blocks, better supports large-capacity disks, andimprovesperformance.Ext4filesystemsalsosupportextendedattributes(xattr),metadatastoredasname-valuepairs.Suchattributescanbeattachedtofilesbyusers or by the operating system. The name part can belong to different xattrnamespaces. For example, SELinux security contexts (Chapter 8, Section 8.9)are attached to files as extended attributes under the name security.selinux(securityisthenamespace).Thecommandattrgetsandsetsextendedattributesforfiles,inthenamespaceuserbydefault.Youmayalsousecommandsgetfattrandsetfattr(Ex:ex06/getfattr).
FileModeThefilemodeconsistsof16bits.Thefourhighbits(C-FinFigure6.1)of thefilemodespecifythefiletype.Thenextthreebitsdefinethemannerinwhichanexecutable file is run. The lowest nine bits of the filemode specify the read,write,andexecutionpermissionsfortheowner,group,andother.Thefiletypeisfixedwhenafileiscreated.Therunandaccess
Figure6.1FileModeBits
bits are settable by the file owner. You already know how to set the nineaccessbitswiththechmodcommand.Therunbitscanbesettogetherwiththeaccessbitsbythechmodcommandusinganumericalmodesetting,asinchmodmodefileSettableFileModes
Thenumericalmodeisanoctalnumberthatisthelogical-orofanynumberof
the settable filemodes (Table 6.4). For set-id-on-execution, the symbolic u+sandg-smodesarealsoavailable.Onlytheownerofafileorasuperusermaychangethemodeofafile.OnmostLinuxsystems,the-Roptioncauseschmodtoperformtherequestedmodesettingonallfilesunderthegivenfiledirectories.
FileUseridandGroupidInLinux,eachfilehasauseridandagroupid.Thefileuseridistheuseridoftheowner who created the file. Each user may belong to one or more (up to areasonable limit, say, eight) groups of users. Each group has a name. Thepassword file (/etc/passwd) entry of each user contains a group affiliation.Bydefault,anewuserbelongstoagroupwithagroupidthesameastheuserid.Ifauserbelongs tomore thanonegroup, then theadditionalgroupaffiliationsarespecifiedinthefile/etc/group.Thegroupidofafilecanbesettoanygrouptowhichthefileownerbelongs.
The group permissions control access to the file bymembers of the specifiedgroup. When a file is first created, it is given by default the groupid of thedirectorythatcontainsit.Thecommandchgrpgroupidfilename...is used to assign a specified groupid to the named files. For example, if
researchisagroupname,thenchgrpresearch*willchangethegroupidofeachfile inthecurrentdirectorytoresearch.The
useridofafilecanbechangedonlybythesuperuser.Thecommandchownowneridfilename...is used to change the ownership of the named files. For example, the
commandchown-Rpwang.changestheownershipofallfilesinthehierarchy(rootedat.)topwang.Both
chgrpandchowntakethe-Roptiontoprocessfilesandfoldersrecursively.Bashprovidesasetofqueriestodeterminethefiletype,accesspermissions,
and so on of a file (Chapter 5, Section 5.13). In addition, the regular Linuxcommandtestcanbeusedtoobtain informationabout the typeandmodeofafile.The test command is a general conditional command often used in Shellscripts(especiallyinShscripts).
AccessControlEnforcementA file always is accessed through a process, for instance, ls,cat, rm, vim, oryourShell(tocd,forexample).Toenforceaccesscontrol,Linuxusestheuserid
andgroupidofaprocesstograntordenyaccesstoafileaccordingtothefile’saccessmode.Theuserid andgroupid of a process are usually that of the userwho invoked the process.Ausermaybelong tomore than one group; thus, aprocessalsokeepsasupplementarygroupidlist.Specifically, if theuseridof theprocessis thesameas theuseridof thefile,
then the access permissions for u apply. Otherwise, if the groupid of the filematchesagroupidoftheprocess,thenthegpermissionsapply.Otherwise,theosettingsapply(Figure6.2).
Figure6.2AccessPermissionsEnforcement
SetuidandSetgidModesIfthesetuid(setgid)bitisturnedonforanexecutablefile,thenitrunsundertheuserid(groupid)oftheexecutablefileratherthanthatoftheinvokingprocess.The setuid and setgid bits are often used to grant temporarily elevated
privileges for certain tasks involving access to files/programs normallyunaccessibletoregularusers.Consider password changing, for example. The command passwd has its
setuidbitturnedon-rwsr-xr-x.1rootroot35480Jul162018/usr/bin/passwdThus, when you call passwd, it takes on an effective uid of root and can
modifythestoredpassworddatafile(usually/etc/shadow).Settingthesetgidbitforadirectorycausesnewfilesandfolderscreatedunder
ittotakeonthedirectory’sgid,ratherthantheprimarygidofthecreatinguser.This is useful for a shared directory used by members working on the sameproject.Thesetuidbitondirectoriesusuallyhasnoeffect.Thestickybit,usedonoldersystemstomakecertainprogramsloadfaster,is
largelyobsolete.InsomeLinuxsystems,thisbitbecomestherestricteddeletionflagfordirectories.Whenset,itpreventsaunprivilegeduserfromremovingorrenamingafileinthedirectoryunlesstheuseris theownerofthedirectoryorthefile.Inanlslisting,at(T)inthe10thpermissionpositionmeansthesticky
bitisonandxforoison(off).
EstablishingaGroupAs an example application of the file access control facilities, let’s considerestablishingagroupwhosememberscancollaborateonaprojectbyaccessingselectedfilesofoneanother.Toestablishthegroup,youfirstdecideonaname.In this example, the groupid is projectx. Next, you must decide who will bemembersof thegroup.In thisexample, thegroupmembersarepwang,rsmith,jdoe, sldog, and yourself. Now ask your system administrator to create groupprojectx. A system administrator can either edit /etc/group directly or use acommandsuchasgroupaddorsystem-config-userstosetupanewgroupandadd themembers.Assoonas this isdone,projectxexistsonyoursystemasavalid group. Once projectx is established, members can assign desired accesspermissionstoselectedfilestoallowsharingwithinthegroup.Onesimplewayforyoutodothisisasfollows:
1. Establish a directory, alpha, say, under your home directory. All files inalphaaretobesharedwithothersinprojectx.
2. Changethegroupidofalphatoprojectxbychgrpprojectxalpha3. Nowsetthegroupaccesspermissionsforthealphadirectory.Dependingon
theaccessyouwishtogive,useoneofthefollowing:chmodg+rwxsalpha(or simply g+rws) chmod g=rx alpha chmod g=x alpha The differencebetweenthesepermissionsisdescribedearlierinSection6.4.
4. Optionally, use chmod +t alpha to set the restricted deletion flag for thealphafolder.
5. Youmustmake sure that each file in alpha carries the groupid projectx,especiallyfilesestablishedthroughcpormv.Asmentioned,thegroupidofafileisdisplayedwithls-gl.Dependingonthenatureofafile,youshouldassignappropriategrouppermissions.Givethegroupwritepermissiononlyifyouallowothersinprojectxtomodifyafile.
DACandMACThe file accesscontrol scheme,using filemodeanduser/group IDs,describedhere provides users and administrators a mechanism to make Linux systemssecure.Butitisstilluptothepeopleinvolvedtosetthepermissionbitsanduserand group IDs at their own discretion. Thus, the scheme is known asDiscretionaryAccessControl(DAC).LinuxsecuritycanbefurtherenhancedbyaddingMandatoryAccessControl
(MAC)whereaLinuxsystemwouldcomewithitsownsetofsecurityrulesthat
6.5
willbeappliedautomatically.SELinux1 (SecurityEnhancedLinux),developedbytheNSA(USNationalSecurityAgency)andtheLinuxcommunity,isawayofachievingMAC.Modern Linux distributions all have SELinux either built-in or available to
install. SELinux can help strengthen security and be especially important forLinux servers. SELinux rules are applied after the DAC rules and can denyaccessevenifDACallowsit.Ifyouexperienced“accessdenied"butfoundnoDACreasons,inalllikelihood,yourLinuxwasenforcingSELinux.WewilldiscussSELinuxinChapter8,Section8.9.
FILESYSTEMIMPLEMENTATIONAsstatedearlierinthischapter,afilesystemisalogicalorganizationimposedon physical data storage media (usually hard disks) by the operating system.Thisorganization, togetherwith the routinessuppliedby theoperatingsystem,allowsforsystematicstorage,retrieval,andmodificationoffiles.
FilesystemOrganizationTypically for Linux, the entire storage system consists of one or morefilesystems.Eachfilesystemisaself-containedunitconsistingofagroupofdatablocks in a particular storage partition (Chapter 8, Section 8.6).A file can beviewedasaone-dimensionalarrayofbytes.Thesebytesarestoredinanumberofdatablocksfromagivenfilesystem.Moderndiskdrivesoffersizablestoragefordata.Typicaldatablocksizesare
1024,2048,and4096bytes.Afilesystemcangainspeedbyemployingalargerblocksize.Theblocksizeisdeterminedatfilesystemcreationtime.For each filesystem, the addresses (locations) of the data blocks, the status,
andperhapsalsotheattributeinformationofafilearestoredinadatastructureknownasthei-node(indexnode).Allthei-nodesofafilesystemarestoredinalinearlistcalledthei-list(ori-table),whichisstoredataknownaddressonthephysicalstoragemedium.I-nodeandi-listwerementionedinSection6.4.Thei-node(Figure6.3)storesmetainformationforafileincludingfilelength
(inbytes),device,owner,andgroupIDs,filemode,andtimestamps.Thei-nodealsocontainspointers(addresses)tothefile’sdatablocks.Forexample,anext2filesystemallows12directpointers,a single-indirectpointer,adouble-indirectpointer,anda triple-indirectpointer.Andext3andext4filesystemsallowevenlargerindividualfilesandoverallfilesystemsize.
Figure6.3Thei-Node
Adirectpointer is theaddressofablockstoringthecontentdataof thefile.Anindirectpointerpointstoablockofdirectpointers.Adoubleindirectpointerpointstoablockofindirectpointers.Atripleindirectpointerpointstoablockofdouble indirect pointers. With this arrangement, very large files can beaccommodated.The i-node contains all the vital meta information of a file. Therefore, the
implementation of a filesystem centers around access to the i-node. The i-numberinadirectoryentryisusedtoindexthei-listandaccessthei-nodeofthefile.Thus,afilepathnameleads,throughasequenceofi-nodes,tothei-nodeofthefile.Figure6.4showshowthepathname/bin/lsleadsfromtherootdirectory/tothefilelsthroughasequenceofi-nodesanddirectoryentries.
Figure6.4FileAddressMapping
Ahardlinktoafilecanbeseenassimplyanotherdirectoryentrycontainingthesame i-number.Once the i-nodeofa file is located, it is read intoprimarymemoryandkeptontheactivei-nodetableuntilaccesstothefileisclosed.Thei-listalsocontainsfreei-nodesthatareusedtocreatenewfiles.
Thecommandlsobtainsanddisplaysfilestatusinformationfromi-nodes.SeeChapter11,Section11.4fordirectaccesstoi-nodedatafromCprograms.
MountedFilesystemsInLinux,afilesystemreferstothelogicalstoragedevicerepresentedbyasinglei-list.ThecompleteLinuxfilesystemmaycontainoneormorefilesystems.Oneoftheseistherootfilesystem;theothersaremountedfilesystems.Thelocationofthe i-list of the root filesystem is always known to the operating system. Amountedfilesystemisattached(mounted)totherootfilesystematanydirectoryintherootfilesystem.Amountedfilesystemcanberemovedbyunmountingitwiththeumountcommand.Asuperusermayusethecommandmount[-r]devfiledirectorytomount the filesystem stored on the block special filedevfile at the given
directory, which is usually an empty directory created for this purpose. Thisdirectoryiscalledtherootdirectoryofthemountedfilesystem.Iftheoption-risgiven,thefilesystemismountedasread-only.Themountcommandwithoutanyargumentsdisplaysthenamesofallmountedfilesystemsandthepointsonthefile treewhere they aremounted.The commanddf displays file system spaceusageandthefreediskspacesonallthefilesystems.Hereisatypicaldfdisplay.df-h
Filesystem Size Used Avail Use%Mountedon/dev/sda6 140G 18G 115G 14%//dev/sda3 99M 20M 75M 21%/boottmpfs 376M 68K 376M 1%/dev/shm/dev/sda2 146G 32G 115G 22%/media/ACERshowingaLinux/MSWindows®dual-bootcomputerwithfourfilesystems.The/media/ACERisthemountpointofanNTFS(NTFilesystem)fortheMS
Window®side.MostLinuxsystemshavebuilt-insupportforNTFSsofilesandfolders in anNTFS partition are usable from either Linux orMSWindows®.Thiscanbeveryconvenient.Doaman-kntfstoseeLinuxsupportforNTFSonyoursystem.
FilesystemSuperBlockandBlockGroupsALinuxext2(ext3,ext4)filesystemconsistsofanumberofblockgroups.Eachblock group also contains a duplicate copy of crucial filesystem controlinformation(superblockandgroupdescriptors)inadditiontotheblockgroup’s
6.6
6.7
ownblockbitmap,i-nodebitmap,i-list,and,ofcourse,datablocks.Thesuper block defines a filesystem. It records vital information about the
configuration,organization,andoperationsofafilesystem:
ThefilesystemtypeandablockdevicereferenceTheoverallsizeandblocksizeofthefilesystemThelengthofthei-nodelistFreeblocksandfreei-nodesRead,write,andothermethodsfori-nodes
Thegroupdescriptorstoresthelocationoftheblockbitmap,i-nodebitmapandthestartofthei-nodetableforeveryblockgroup;andthese,inturn,arestoredinagroupdescriptor table.Thesuperblockand thegroupdescriptor tablearecriticalforafilesystem,andtheyarestoredatthebeginningofeachblockgrouptoprovideredundancy.
THEFILESYSTEMTABLEEach different filesystem on Linux has its own block-type special file. Thenames of these special files, together with other information for control andmanagementoftheentirefilesystem,arekeptinthefilesystemtable (typically,/etc/fstab).This file contains one line for each filesystem specifying the blockspecial filename, thedirectorynamewheremounted, the filesytem type (local,NFS,2orformemoryswapping),mount/swapoptions,andotherinformation.Ofallthefilesystemscontainedinthefilesystemtable,allorasubsetmaybe
mounted at any given time. The mount table (/etc/mtab) contains a list ofcurrentlymounted filesystems.Themount table ismodifiedby the commandsmountandumount.
CREATINGSPECIALFILESAs previously mentioned, the Linux system uses special files to representphysicalandlogicalI/Odevices,andachievesuniformfileI/OanddeviceI/O.Special files normally are created exclusivelyunder the systemdirectory /dev.Thecommandmknodfilename[borc]majorminorisusedtoestablishaspecialfilebythegivenfilename.Thecharacterbisused
if the device is a block I/O device (hard disk). The character c is used for acharacterI/Odevicesuchasaterminalemulatororaprinter.EachphysicalI/O
6.8
6.9
device on Linux is assigned a major device number according to the type ofdevice it is and aminor device number indicating the unit numberwithin thesametypeofdevices.Thesenumbersareintegers.Forexample,thetwoprinterslp0andlp1crw-rw—-1rootlp6,02018-03-0611:48lp0crw-rw—-1rootlp6,12018-03-0611:48lp1havemajordevicenumber6andminordevicenumbers0and1,respectively.
Onlyasuperusercancreatespecialfiles.
NETWORKFILESYSTEMManyLinuxsystemsallowfileoperationsnotonlyon local filesystemsstoredonthehostcomputer,butalsoonremotefilesystems storedonothercomputersconnected by a network. The Network Filesystem (NFS) allows transparentaccess to remote files. In other words, there is no difference between userrequestsforoperationsonremoteandlocalfiles.NFSbringsmanyadvantagesto file organization for businesses and organizations. For example, duplicatestorageofthesamefilesondifferenthostscanbeavoidedbycentralizingthemonfileservermachinesaccessibleviaNFS.Tomakethingsevenmoreconvenient,NFScanworkwithdifferenthardware
andoperatingsystems.Afilesystemonalocalhostismaderemotelyaccessibleby exporting it. The file /etc/exports specifies local filesystems that can beexportedandanyrestrictionsoneachfilesystem.Thecommandexportfsmustberunaftermodifying/etc/exports.Thefile/var/lib/nfs/etab(orxtab)liststhefilesytemscurrentlybeingexported.
Afilesystemcanbeexportedtoalistofallowedclientsortoallandcanallowread-onlyorread-writeaccess.Aclienthostmakesaremotefilesystemaccessiblebythemountcommandmountremote-filesystemlocal-directorywhichmounts a remote filesystem, specified byhost:directory, onto a local
directoryofchoice.On most Linux systems, even the mounting and unmounting of remote
filesystemsareautomatedthroughtheautofsmechanismassisteddirectlybytheLinux kernel. The kernel calls the automount program to mount a remotefilesystemwhen an actual file access to itsmount point occurs.Automountedfilesystemsaredismountedafteratimeperiodwithnoaccess.
SEARCHINGTHEFILETREE:FIND
WeknowtheLinuxfilesystemisorganizedintoatreestructure.Itissometimesnecessarytosearchapartofthistreeandvisitallnodesinasubtree.Thismeansvisiting all files in a given directory and, recursively, all files contained insubdirectories of the given directory. The find command provides just such atreesearchingmechanism.Thefindcommandvisitsallfilesinasubtreeandselectsfilesbasedongiven
Booleanexpressions.Theselectionfeatureallowsustofindthedesiredfilesandapplyoperations to them.Any file in the subtree forwhich thegivenBooleanexpressionsevaluatetotruewillbeselected.The find command can be used to locate (display the pathname of) files
whosenamesmatchagivenpatterninthesubtree.Forexample,find.-name.c-printInthisexample,thefindcommandisgiventwoBooleanexpressions,-name
.cand-print.Thecommandsearchesthesubtreerootedatthecurrentdirectoryvisiting each file. The file that currently is being visited is referred to as thecurrentfile.Ifthenameofthecurrentfilematchesthepattern*.c(thefilenameendsin.c),thenthenextexpression(-print)isevaluated.The-printexpressionsimplydisplaysthepathnameofthecurrentfileonthestandardoutput.Thus,theeffect of the preceding example is to find allC source files under the currentdirectoryanddisplaytheirpathnames.Thegeneralformofthefindcommandisfindfilename…expression…The command name is followed by one or more filenames, each either an
ordinaryfileoradirectory,andthenbyoneormoreexpressions.Thetreesearchisconductedoneachfileanddirectorygiven.Eachexpressionisapredicateonthecurrentfileandalwaysproducesatrue/falsevalue,althoughtheexpressionalsomay have other effects. An expression is evaluated only if all precedingexpressions are true. In otherwords, expression evaluation for the current fileterminateson thefirst falseexpression,and thesearchprocess thengoeson tothenextfileinthesubtree.The expressions used in find are primary expressions or a Boolean
combination of primary expressions. Some important primary expressions areexplained here. The effect and the Boolean value of each also is described.(Sincesomeexpressionsmayinvolveconceptsandfeatureswehavenotcoveredyet,youmayskipthoseexpressionsfornowifyouwish.)Inthedescriptions,theargumentn is used as a decimal integer that can be specified in one of threeways: an integer, an integer preceded by + , or an integer preceded by - .Specifying+nmeansmorethann,-nmeanslessthann,andnmeansexactlyn.ThefollowingBooleanoperations(inorderofdecreasingprecedence)canbe
6.10
6.11
usedtocombineanyvalidexpressionse1ande2.Here are some additional examples. To remove all files, under your home
directory,namedeithera.outor*.othathavenotbeenaccessedforatleastfourweeks,typein(Ex:ex06/findrm)find(-namea.out-o-name’*.o’)-atime+28-execrm”Youcanavoidthelinecontinuationbyenteringeverythingononecommand
line.Consider another example. To display the names of all files not owned by
smithunderthecurrentdirectory,typeinfind.-usersmith-printNotethatmanycharactersusedintheseexampleshavebeenquotedtoavoid
Shellinterpretation.Now,forathirdexample(Ex:ex06/findstr),supposeyouhaveseveralHTML
files under your personal Web space ($HOME/public_html) that contain thewordLinux,butyouarenotsureexactlywhichfiles.YoucanusefindtoapplyfgreptoeachHTMLfile.findpublic_html-name‘*.html’-execfgrepLinux{}-print
THELOCATECOMMANDWhilefind isniceandpowerful, the locatecommandcanbeeasier touseandfaster.YougivelocateaGlobpatternoraregularexpressionanditcandisplayall pathnames, in the file tree, that contain a node whose namematches. Forexample,
locategnome(pathnamecontaininggnome)locate-b\gnome(base
filenameexactlygnome)locate--regex\.html$(filenameendingin
.html)
The locate command runs faster because it searches a database of files andfolders on your system called an updatedb which is regularly updatedautomaticallydaily.
SAVING,COMPRESSING,ANDDISTRIBUTINGFILES
Sometimes the need arises to pack a number of files into a neat package andsendthemtoanothercomputer.Thetarcommandisusedtocollectasetoffiles
ontoasinglefile,calleda tarfile (thenamecamefrom tapearchive).The tarcommandcopiesentiredirectoryhierarchies.Adirectoryhierarchyreferstoallfilesanddirectoriescontainedinasubtreeof thefile tree.Itworksbypackingmultiplefilesintoasinglefileinthetarformatwhichcanlaterbeunpackedbytarpreservingtheoriginalfileandfolderstructure.Thetarfilecanbesavedasabackup or transferred easily by email or ftp (Chapter 7,Section7.6). The tarcommandisoftenusedtogetherwithcommonfilecompressionschemessuchasgzip(GNUZip),bzip2andxz.Thelattergenerallyprovidesbettercompression.Let’sfirstlookatthesimplestusesoftar.tarcvftarfile.tarname1name2...(A)tarzcvftarfile.tgzname1name2...(B)tarjcvftarfile.tbzname1name2...(C)tarJcvftarfile.txzname1name2...(D)savesthenamedfilehierarchiestothegiventarfilewithnocompression(A),
gzip compression (B), bzip2 compression (C), or xz compression (D). Theoptionsarec(create tarfile),v(verbose), f (tarfilenamefollows),z(usegzip),andj(usebzip2).Thecorrespondingcommandstarxvftarfile.tartarzxvftarfile.tgztarjxvftarfile.tbztarJxvftarfile.txzextract the files contained in tarfile. If you wish to preserve the file
permissionsandotherattributes,usethepoptionwhenpackingandunpackingwith tar.Many software packages in tar format are available for download toyourLinuxsystem.The ZIP utility commonly used onWindows platforms is also available on
Linux.Thezip andunzip commandsmake it convenient to exchange archivefileswithotherplatforms.zip-rarchive.zipname1name2…packsfilesandfoldersintothegivenarchive,whileunzipunpacks.When providing an archive file for downloading, it is good practice to also
provide a finger print file to check the integrity of the download.Creating anMD5(Message-Digestalgorithm5) fingerprintforyourarchivefile issimple.Thecommandmd5sumarchivefile>archivefile.md5placesthenameofthearchivefileanditsMD5fingerprintinthefingerprint
filearchivefile.md5.More secure alternatives tomd5sum use theUSNational SecurityAgency
6.12
published SHA-2 algorithms. On Linux, these include the commandssha224sum,sha256sum,sha384sum,andsha512sumthatproducefingerprintsofdifferentlengths.
PackingFileswithsharThetaristheregularcommandforsavingandretrievingfilesbecauseitrestoresall file attributes such as ownership and access protection modes. The sharcommand isanotherway topackmultiple files intoonewhichdoesnotworryaboutretainingfileattributeinformation,anditcanbeeasiertouse.Basically,sharpacksthefilesintoasinglefileofshcommands.Thepacked
fileisunpackedbylettingshprocessthefile.Thecommandsharfile1file2...>outfile.shpacks the named files (including directories) into one file and sends that to
standardoutput.Theresultingoutfile.shfilecanbesentbyemailoruploadedtoanotherLinux/UNIXcomputer.Tounpacksimplydosh<outfile.sh
FILESHARINGWITHSAMBASambaisasuiteofprogramsforLinux/UnixtosharefilesandprinterswithMSWindows®systems.MostLinuxdistributionscomewithSamba.
Figure6.5FileSharingwithSAMBA
To access shared files simply go to Network > SAMBA or Network >Workgroupinyourfilebrowserorusethesecommands(Figure6.5):nautilussmb:///dolphinsmb:///This is also an easyway to access a shared disk attached viaUSB to your
6.13
6.14
6.15
homerouter.
MOREFILE-RELATEDCOMMANDSSomeadditional commands that areuseful indealingwith files andmanagingthefilesystemarelistedhere.Thefunctionofeachcommandisindicated,butnofullexplanationsaregiven.Formoredetailedinformationandoptionsonthesecommands,refertotherespectivemanualpages.
basenameremovesprefixesandsuffixesfromafilename.chrootchangestemporarilytherootdirectory(/)fortestingaprogram.cmpcomparestwofilestoseeiftheyareidentical.commselectsorrejectslinescommontotwosortedfiles.dfdisplaysdiskspacefreeonallfilesystems.diffcomparestwofilesordirectoriesandoutputsthedifferences.dudisplaysallfilesizesinkilobytesinadirectoryhierarchy.sizedisplaysthesizeofanobjectfile.splitsplitsafileintopieces.touch updates the lastmodified time of a file; if a file does not exist, itcreatesanemptyone.uniqreportsrepeatedlinesinafile.wccountsthenumberofwords,linesingivenfiles.
FORMOREINFORMATIONFortheFileSystemStandard(FSSTD),seetheLinuxJournalarticlebyDanielQuinlanavailableontheWebfromACM:portal.acm.org/citation.cfm?id=324517For complete information on the Linux file hierarchy, see the Linux
DocumentationProjectonlinearticle:tldp.org/LDP/Linux-Filesystem-Hierarchy/htmlFormoredetailsonfilesysteminternalsandimplementations,refertoDesign
and Implementation of the Second Extended Filesystem and to Linux NFS-HOWTOatSourceForgeSourceForge.net.See also Linux Filesystems Explained – EXT2/3/4, XFS, Btrfs, ZFS at
fossbytes.comr.
SUMMARY
6.16
ThefilesystemiscentraltoanyoperatingsystemandispartoftheLinuxkernel.TheLinuxfilesystemhierarchycontainsfilesanddirectoriesarrangedinatreestructure that grows down from the root directory /. The Linux file hierarchylargelyfollowstheFSSTND.Different file types are directories, special files, links, regular files, sockets,
andnamedpipes.Therearetwokindsoflinks:hardlinksandsymboliclinks.Asymbolic linkcan link toadirectoryandcan span filesystems.Access to filesanddirectoriesisgovernedbyrwxpermissionsforthefileowner(u),forusersinthefilegroup(g),andforothers(o).Theset-useridbitforexecutablefilesisanimportantconcept.Whenaprocess
executesaset-useridfile,itseffectiveuseridbecomesthatofthefileowner.The entire file system consists of a root filesystem and possibly additional
mountable filesystems. Linux supports different filesystem implementations,includingext2anditsextensions(currentlyext4)thatsupportextendedattributes(xattr)intheformofname-valuepairs.Each filesystem is organized by an i-list, which is a list of i-nodes that
contains status and address information for each file and all free space in thefilesystem. File status information includes userid, access groupid, mode,timestamps, and disk addresses. Part of the file mode specifies file accesspermissions.TheseattributesareusedinenforcingDiscretionaryAccessControl(DAC). SELinux (Chapter 8) can further strengthen system security byprovidingMandatoryAccessControl(MAC).The NFS allows transparent access to remote (NFS) and local filesystems,
makingiteasytosharefilesacrossanetwork.SambamakesitpossibletosharefileswithWindowssystemsonyournetwork.Todoasystematicsearchthroughafilehierarchy,usethefindcommand.To
quicklylocatefiles/foldersbasedontheirnames,usethelocatecommand.Usethesimplesharcommandorthemoreefficienttarcommand(withgzip,bzip2,orxz filecompression) topackandcompressmultiplefiles intoanarchiveforeasy transport. Use zip to manage archive files across different computersystems.
EXERCISES1. Trytheumaskcommand.Whatdoesittellyouaboutthefilesyoucreate?
Try setting the umask value and then creating some files. Look at theirprotectionbits.
2. Ifyouhavenotdoneityet,downloadthemostrecentHTMLversionofthe
Linux man pages from www.tldp.org/manpages/man-html/ to yourcomputer.UnpackitsothatyoucanuseitwithyourWebbrowser.
3. The term filesystem is different from the phrase “file system.” Can youclearlyspecifytheirmeaning?
4. Why is a hard link indistinguishable from the original file itself? Whathappensifyourmahardlink?Whyisitnotpossibletohaveahardlinktoafileinadifferentfilesystem?
5. Clearly state the meaning of the rwx permissions for a directory. Whatwould happen if you perform ls dir with read permission to dir but noexecutepermission?Why?
6. Write a Shell script forwebwhich takes the name of a folder fname andmakesallfileso+randallfolderso+rxinthefilehierarchyrootedatfname.
7. What is an xattr? Forwhat purpose?Where and inwhat form are xattrsstored?
8. What command is used for a user to get, set, list, and remove extendedattributes?Pleaseshowexamples.
9. Whatcommanddisplaysthei-numberofafile/directory?10. Itisclearhowcommandsrmandlsworkonordinaryfiles.Describehow
theyworkonsymboliclinks.Mustasymboliclinkpointtoanexistingfile?Whathappensiftheactualfileofasymboliclinkisdeleted?Isitpossibleforasymboliclinktopointtoanothersymboliclink?
11. Consider the . and .. special files. Is it correct to say that these files aresystem-createdhardlinkstodirectories?
12. ConsidertheBashscriptclean.sh(Chapter5,Section5.20).Doesthescriptstillworkcorrectlyiftherearesymboliclinksinthedirectoryitistryingtoclean?Ifthereisaproblem,howwouldyoufixit?
13. Trytormafiletowhichyouhavenowritepermission.Whatmessagedoesrmgive?Howdidyourespond?Wereyouabletodeletethefile?Why?
14. Whenanexecutable file is invoked,does thenewprocessalwaysassumetheuseridoftheuserwhoinvokedit?Explain.
15. You are looking for a file somewhere under your home directory thatcontainsthestringzipcodeinit.Describehowyoucanlocatethefileifyoudo/don’tknowwhichdirectorycontainsthefile.Whatifthefilemaybeahiddenfilewhosenamebeginswithadot?
16. Howexactlydoesonecreatea.tgzfile?Howdoesoneextractfroma.tgzfile?Whatabout.tbzandtxzfiles?
17. FindouthowtosetupaSamba-sharedfolderinyourLinuxhomedirectory.Explaineachstep.
Chapter7
Networking,Internet,andtheWeb
Early packet-switched computer networking, involving a few researchinstitutionsandgovernmentagencies,startedinthelate1960sandearly1970s.Today, it ishard to tellwhere the computer ends and thenetworkbegins.Theview “The Network is the Computer” is more valid than ever. Most peoplecannottolerateevenafewminutesofInternetconnectionoutage.A computer network is a high-speed communications medium connecting
many, possibly dissimilar, computers orhosts. A network is a combination ofcomputer and telecommunication hardware and software. The purpose is toprovidefastandreliableinformationexchangeamongthehosts.Typicalservicesmadepossiblebyanetworkinclude
ElectronicmailOn-linechattingandInternetphonecallsFiletransferRemoteloginDistributeddatabasesNetworkedfilesystemsAudioandvideostreamingVoiceandtelephoneoveranetworkWorldWideWeb,E-business,E-commerce,andsocialnetworksRemoteprocedureandobjectaccess
In addition to host computers, the network itself may involve dedicatedcomputersthatperformnetworkfunctions:hubs,switches,bridges,routers,andgateways. A network extends greatly the powers of the connected hosts. Anddedicatedserverhosts(Chapter9)greatlyenhancethepowerandusefulnessofthenetworkbyprovidingefficientandeffectiveservices.Agoodunderstandingofbasicnetworkingconcepts,commands,information
7.1
security, and how the Web works will be important for any Linux user,programmerandservermanager.
NETWORKINGPROTOCOLSFor programs and computers fromdifferent vendors, under different operatingsystems, tocommunicateonanetwork,adetailedsetofrulesandconventionsmustbeestablishedforallpartiestofollow.Suchrulesareknownasnetworkingprotocols.Weusedifferentnetworkingservicesfordifferentpurposes;therefore,eachnetworkservicefollowsitsownspecificprotocols.Protocolsgovernsuchdetailsas
AddressformatofhostsandprocessesDataformatMannerofdatatransmissionSequencingandaddressingofmessagesInitiatingandterminatingconnectionsEstablishingservicesAccessingservicesDataintegrity,privacy,andsecurity
Thus, for a process on one host to communicate with another process on adifferenthost,bothprocessesmustfollowthesameprotocol.TheOpenSystemInterconnect (OSI)ReferenceModel (Figure 7.1) provides a standard layeredview of networking protocols and their interdependence. The correspondinglayers on different hosts, and inside the network infrastructure, performcomplementary tasks to make the connection between the communicatingprocesses(P1andP2inFigure7.1).
Figure7.1NetworkingLayers
7.2
Amongcommonnetworkingprotocols,theInternetProtocolSuiteisthemostwidelyused.ThebasicIP(InternetProtocol ) isanetwork layerprotocol.TheTCP(TransportControlProtocol )andUDP(UserDatagramProtocol )areatthetransportlayer.TheWebisaservicethatusesanapplicationlayerprotocolknownasHTTP(theHypertextTransferProtocol).Networkingprotocolsarenomystery.Thinkabouttheprotocolformakinga
telephonecall.You(aclientprocess)mustpickupthephone,listenforthedialtone, dial a valid telephone number, and wait for the other side (the serverprocess)topickupthephone.Thenyoumustsay“hello,”identifyyourself,andsoon.Thisisaprotocolfromwhichyoucannotdeviateifyouwantthecalltobemade successfully through the telephone network, and it is clear why such aprotocolisneeded.Thesameistrueofacomputerprogramattemptingtotalktoanothercomputerprogramthroughacomputernetwork.Thedesignofefficientandeffectivenetworkingprotocolsfordifferentnetworkservicesisanimportantareaincomputerscience.ChancesareyourLinuxsystemisonaLocalAreaNetwork(LAN)whichis
connected to the Internet. This means you have the ability to reach, almostinstantaneously, across great distances to obtain information, exchangemessages,upload/downloadfiles,interactwithothers,doliteraturesearches,andmuch more without leaving the seat in front of your workstation. If yourcomputer is not directly connected to a network but has a telephone or cablemodem, then you can reach the Internet through an Internet service provider(ISP).
THEINTERNETThe Internet is a global network that connects computer networks using theInternet Protocol (IP). The linking of computer networks is calledinternetworking, hence the name Internet. The Internet links all kinds oforganizations around theworld: universities, government offices, corporations,libraries, supercomputer centers, research labs, and individual homes. ThenumberofconnectionsontheInternetislargeandgrowingrapidly.The Internet evolved from theARPANET, 1 a U.S. Department of Defense
Advanced Research Projects Agency (DARPA) sponsored network thatdeveloped the IP as well as the higher level Transmission Control Protocol(TCP) and User Datagram Protocol (UDP) networking protocols. Thearchitecture and protocol were designed to support a reliable and flexiblenetworkthatcouldendurewartimeattacks.
The transition ofARPANET to the Internet took place in the late 1980s asNSFnet, the U.S. National Science Foundation’s network of universities andsupercomputing centers, helped create an explosive number of IP-based localandregionalnetworksandconnections.TheInternetissodominantnowthatithasvirtuallyeliminatedallhistoricalrivalssuchasBITNETandDECnet.The Internet Corporation for Assigned Names and Numbers (ICANN;
www.icann.org) is a nonprofit organization responsible for IP address spaceallocation, protocol parameter assignment, domain name systemmanagement,andmaintainingrootserversystemfunctions.
NetworkAddressesAnaddresstoahostcomputerislikeaphonenumbertoatelephone.Everyhoston the Internet has its own network address that identifies the host forcommunication purposes. The addressing technique is an important part of anetworkanditsprotocol.AnInternetaddress(IPv4address)isrepresentedby4bytes in a 32-bit quantity. For example, tiger, a host atKent State, has the IPaddress131.123.41.83(Figure7.2).Thisdotnotation (orquad notation) givesthedecimalvalue(0to255)ofeachbyte.Toaccommodatetheexplosivegrowthofthenumberofconnecteddevices,theInternethasbeenmovingtoIPv6,whichsupports128-bit addresses.The IPaddress is similar toa telephonenumber inanotherway:theleadingdigitsarelikeareacodes,andthetrailingdigitsarelikelocalnumbers.
Figure7.2IPv4Address
Becauseof their numerical nature, thedotnotation is easyonmachinesbuthardonusers.Therefore,eachhostmayalsohaveadomainnamecomposedofwords, rather like apostal address.For example, thedomainname for tiger istiger.zodiac.cs.kent.edu (at the Department of Computer Science, Kent StateUniversity).TheLinuxcommandhostdisplaystheIPanddomainnameofanygivenhost.Forexample,hosttiger.zodiac.cs.kent.edudisplaystiger.zodiac.cs.kent.eduhasaddress131.123.41.83With domain names, the entire Internet name space for hosts is recursively
dividedintodisjointdomainsinahierarchicaltree(Figure7.3).Theaddressfortigerputsitinthecslocaldomain,withinthezodiacsubdomain,thenwithinthekent subdomain, which is under the edu top-level domain (TLD) for U.S.educational institutions.OtherTLDs includeorg(nonprofitorganizations),gov(U.S. government offices), mil (U.S. military installations), com (commercialoutfits),net(networkserviceproviders),uk(UnitedKingdom),cn(China),andso forth. Within a local domain (for example, cs.kent.edu), you can refer tomachinesbytheirhostnamealone(forexample,monkey,dragon,tiger),butthefulladdressmustbeusedformachinesoutside.FurtherinformationonInternetdomainnamescanbefoundinSection7.19.
Figure7.3TheDomainNameHierarchy
TheICANNaccreditsdomainnameregistrars,whichregisterdomainnamesforclientsso theystaydistinct.AllnetworkapplicationsacceptahostaddressgiveneitherasadomainnameorasanIPaddress.Infact,adomainnameisfirsttranslatedtoanumericalIPaddressbeforebeingused.
PacketSwitchingData on the Internet are sent and received in packets. A packet envelopstransmitted data with address information so the data can be routed throughintermediatecomputersonthenetwork.Becausetherearemultipleroutesfromthe source to thedestinationhost, the Internet is very reliable andcanoperateevenifpartsofthenetworkaredown.
ClientandServerMost commonly, a network application involves a server and a client (Figure7.4).
7.3
Aserverprocessprovidesaspecificserviceonahostmachine thatofferssuch a service. Example services are email (SMTP), secure remote hostaccess (SSH), secure file transfer (SFTP), and the World Wide Web(HTTP). Each Internet standard service has its own unique port numberthat is identical on all hosts. The port number together with the Internetaddress of a host identifies a particular server program (Figure 7.4)anywhereon thenetwork.For example,SFTPhasportnumber115,SSHhas22,andHTTPhas80.OnyourLinuxsystem,thefile/etc/servicesliststhestandardandadditionalnetworkservices,indicatingtheirprotocolsandportnumbers.Aclientprocessonahostconnectswithaserveronanotherhosttoobtainits service.Thus,aclientprogramis theagent throughwhichaparticularnetworkservicecanbeobtained.Differentagentsareusually required fordifferentservices.
AWebbrowsersuchasFirefoxisanHTTPclient.ItrunsonyourcomputertoaccessWebserversonanyInternethosts.TheLinuxwgetcommandisanotheruseful client that can download files from the Internet using theHTTP or theFTPprotocol.
Figure7.4ClientandServer
THEDOMAINNAMESYSTEMAsstatedinSection7.2,everyhostontheInternethasauniqueIPaddressandadomainname.Thenetworknamespace, thesetofalldomainnameswiththeirassociatedIPaddresses,changesdynamicallywithtimeduetotheadditionanddeletionofhosts,regroupingoflocalworkgroups,reconfigurationofsubpartsofthe network, maintenance of systems and networks, and so on. Thus, newdomain names, new IP addresses, and new domain-to-IP associations can beintroduced in thename space at any timewithout central control.Thedomainname system (DNS) is a network service that supports dynamic update andretrievalofinformationcontainedinthedistributednamespace(Figure7.5).Anetworkclientprogram(forexample,theFirefoxbrowser)willnormallyusethe
7.4
DNS to obtain IP address information for a target host beforemaking contactwith a server. The dynamic DNS also supplies a general mechanism forretrievingmanykindsofinformationabouthostsandindividualusers.
Figure7.5DomaintoIP
HerearepointstonoteabouttheDNSnamespace:
TheDNSorganizestheentireInternetnamespaceintoabigtreestructure.Each node of the tree represents a domain and has a label and a list ofresources.Labelsarecharacterstrings(currentlynotcasesensitive),andsiblinglabelsmust be distinct. The root is labeled by the empty string. ImmediatelybelowtherootaretheTLDs:edu,com,gov,net,org,info,andsoon.TLDsalso include country names such as at (Austria), ca (Canada), and cn(China).Underedu,forexample,therearesubdomainsberkeley,kent,mit,uiuc,andsoon(Figure7.3).Afulldomainnameofanodeisadot-separatedlistoflabelsleadingfromthenodetotheroot(forexample,cs.kent.edu.).Arelativedomainnameisaprefixofafulldomainname,indicatinganoderelativetoadomainoforigin.Thus,cs.kent.eduisactuallyanamerelativetotheroot.A label is the formal or canonical name of a domain.Alternative names,calledaliases,arealsoallowed.Forexample,themainWebserverhostinfohas the aliaswww, so it is also known aswww.cs.kent.edu.Tomove theWebservertoadifferenthost,alocalsystemmanagerreassignsthealiastoanotherhost.
SeeSection7.19formoreinformationontheDNSandnameservers.
NETWORKINGINNAUTILUSWefirstintroducedtheGNOMENautilusfilemanagerinChapter2,Section2.7.LaunchNautilusandgoto+OtherLocations(Figure7.6);youcanbringupalistofallsystemsonyourlocalandremotenetworkandaccessfilesonthem.
Figure7.6NetworkinginNautilus
Linux systems are listed individually. Systems running other operatingsystemsaregroupedunderdifferent iconssuchas theWindowsNetwork icon.Ofcourse,youcanbrowseonlymachineswithpermission.Normally,loginwillberequiredunlessyouhavearrangedano-passwordlogin(seeSection7.6).Youcanalsoconnecttonewserverstowhichyouhaveaccess.Herearesome
sampleconnections:
sftp://[email protected]—Secure FTP, home directory ofpwangontiger.zodiac.cs.kent.edussh://[email protected]—Secureshell,sameasabovesftp://[email protected]/Pictures—Secure FTP, Picturesfolderofpwang(Figure7.7)ftp://[email protected]—RegularFTP
Figure7.7SFTPviaNautilus
AccessingSambaSharedFilesUsually, you’ll find Linux andMSWindows® systems on the same in-housenetwork.NautilusmakesiteasytoaccesssharedfilesfromMSWindows®.JustentertheLocation
7.5
smb://host/share_folderto reach the target shared folder via the Common Internet File System
protocol, the successor of Server Message Block (SMB). Linux systems useSaMBa,afree,open-sourceimplementationoftheCIFSfilesharingprotocol,toactasserverandclienttoMSWindows®systems.UseanIPforthehosttobesure.HerearesomeLocationexamplesonahomenetwork.
smb://192.168.2.102/SharedDocssmb://192.168.2.107/Public
NETWORKINGCOMMANDSLinux offers many networking commands. Some common ones are describedhere to get you started. In earlier chapters, we mentioned briefly severalnetworkingcommands.Forexample,weknowthathostnamedisplays the domain name of the computer you are using. If given an
argument, thiscommandcanalsoset thedomainname(whenrunasroot),butthedomainnameisusuallyonlysetatsystemboottime.TogettheIPaddressandotherkeyinformationfromtheDNSaboutyourcomputeroranotherhost,youcanuse
host$(hostname)(foryourcomputer)hosttargetHost(fortarget
host)
Forexample,hostgoogle.comproduces
google.comhasaddress74.125.45.100google.comhasaddress
74.125.67.100google.comhasaddress209.85.171.100google.commail
ishandledby10smtp4.google.com.google.commailishandledby10
smtp1.google.com.google.commailishandledby10
smtp2.google.com.google.commailishandledby10smtp3.google.com.
Foranygivenhost,itsDNSdataprovideIPaddress,canonicaldomainname,alias domain names, DNS server hosts, and email handling hosts. Othercommands that helpyouaccess theDNSdata from the command line includenslookupanddig(DNSInformationGroper).Forexample,digtiger.zodiac.cs.kent.edugives
;>><<DiG9.10.5-P2-RedHat-9.10.5-2.P2.fc25>><<
tiger.zodiac.cs.kent.edu;;globaloptions:+cmd;;Gotanswer:;;-
<<HEADER>>-opcode:QUERY,status:NOERROR,id:60868;;flags:qr
rdra;QUERY:1,ANSWER:1,AUTHORITY:0,ADDITIONAL:1;;OPT
7.6
PSEUDOSECTION:;EDNS:version:0,flags:;udp:8192;;QUESTION
SECTION:;tiger.zodiac.cs.kent.edu.INA;;ANSWER
SECTION:tiger.zodiac.cs.kent.edu.43200INA131.123.41.89;;Query
time:41msec;;SERVER:209.18.47.62#53(209.18.47.62);;WHEN:Wed
Sep2011:14:17EDT2017;;MSGSIZErcvd:69
Thedesiredinformation(ANSWERsection)togetherwiththeidentityofthenameserver(SERVER)thatprovidedthedataisdisplayed.The command dig is very handy for verifying the existence of hosts and
findingtheIPaddressordomainnamealiasesofhosts.Oncethenameofahostisknown,youcanalsotestifthehostisupandrunning,asfarasnetworkingisconcerned,withthepingcommand.pinghostThissendsamessagetothegivenremotehostrequestingittorespondwithan
echoifitisaliveandwell.Toseeifanyremotehostisupandrunning,youcanuseping,whichsendsan
echoInternetcontrolmessagetotheremotehost.Iftheechocomesback,you’llknowthatthehostisupandconnectedtotheInternet.You’llalsogetround-triptimesandpacketlossstatistics.Whensuccessful,thepingcommandscontinuestosendechopackets.TypeCTRL+Ctoquit.
SSHWITHX11FORWARDINGNetworking allows you to conveniently access Linux systems remotely.MostLinuxdistributionscomewithOpenSSHinstalled.Asmentioned inChapter1,Section1.2,youcanssh toa remoteLinuxanduse it fromthecommand line.Furthermore,youcanssh-Xuserid@remoteHostnameto log in to the given remote host with X11 forwarding/tunneling, which
allowsyouto
Figure7.8DigitalSignature
startanyXapplications,suchasgeditorgnome-terminal,ontheremotehostandhavethegraphicaldisplayappearonyourlocaldesktop.This works if your local host is a Linux/UNIX/MacOS system. It can also
workfromMSWindows®.Followthesesteps:
1. Obtain and install anX11 server onWindows, such as theXming or theheavierdutyCygwin.
2. AssumingyouhavedownloadedandinstalledXming,clicktheXmingiconto launch theX11 server. TheX11 server displays an icon on your startpanelsoyouknowitisrunning.
3. SetupSSHorPuttyonyourMSWindows®system:
Putty Settings—Go to Connection- > SSH- > X11 and check theEnable X11 forwarding box. Also set X display location to127.0.0.1:0.0.SSHSettings—ChecktheTunneling->TunnelX11Connectionsbox.AlsochecktheAuthentication->EnableSSH2connectionsbox.
4. UseeitherPuttyorSSHtoconnecttoaremoteLinux/Unixcomputer.Makesure your remote account login script, such as .bash_profile, does not settheDISPLAYenvironmentvariable.Itwillbesetforyoutosomethinglikelocalhost:10.0automaticallywhenyouconnectviaSSH.
5. MakesureyourX11server(Xming,forexample)isrunning.Now,ifyoustart an X application on the remote Linux system, that graphicalapplicationwillthenSSHtunneltoyourcomputerandusetheX11serveronyourcomputertodisplayagraphicaluserinterface(GUI).Forexample,youcanstartgedit,nautilus–no-desktop,orevenfirefox.
Note,usinganapplicationwitharemoteGUIcanbeslowduetomuchheaviernetworkingloadascomparedaremoteCLI.
NoPasswordssh,sftp,andscpThe commands ssh,sftp, and scp are for remote login, secure ftp, and secureremotecp,respectively.Whenusinganyoftheseyouusuallyneedtoenterthepassword for the remote system interactively.Whenyouneed toperformsuchtasksfrequently,thiscanbeabother.Fortunately,youcaneasilyavoidhavingtoenterthepassword.Justfollowthesesteps.MostLinuxsystemscomewithOpenSSHinstalled.Thismeansyoualready
have the SSH suite of commands.These enable you to securely communicatefromonecomputer(asuser1onhost1)toanother(asuser2onhost2).Wewillassumeyouareloggedinasuser1onhost1(thisisyourlocalhost),andyouwishtoarrangesecurecommunicationwithyouraccountuser2onhost2,whichwewillrefertoastheremotehost.SSHcanusepublic-keyencryption fordata securityanduserauthentication
(Section 7.9). If you have not done it yet, the first step in arranging forpassword-lessloginistogenerateyourownSSHkeys.Issuethecommandssh-keygenYou’llbeaskedforafoldertosavethekeysandapassphrasetoaccessthem.
In this case, don’t provide any input in response to these questions from ssh-keygen.SimplypresstheENTERkeyinresponsetoeachquestion.Keygenerationtakesalittletime.Thenyou’llseeamessagetellingyouthat
your identity (private key) is id_rsa and your public key is id_rsa.pub savedunderthestandardfolder.ssh.inyourhomedirectory.The second step is to copy your id_rsa.pub to your account on the desired
remote-host.Issuethecommandssh-copy-id-i/.ssh/id_rsa.pubyour_userid@remote-hosttoappendyourpublicSSHkeytothefileuserid/.ssh/authorized_keyonthe
remote-host.Now you are all set. You can log in to remote-host without entering a
password.sshuserid@remote-hostThesamesetupavoidsapasswordwhenyouusesftporscp.
RemoteFileSynchronizationThersynccommandmakesiteasytokeepfilesinsyncbetweentwohosts.Itisvery efficient because it uses a remote-update protocol to transfer just thedifferencesbetweentwosetsoffilesacrossthenetworkconnection.Noupdatingisperformedforfileswithnodifference.Withthecommands
rsync-azuserid@host:sourcedestDir(remotetolocalsync)rsync
-azsourceuserid@host:destDir(localtoremotesync)
thegivensource file/folder isused toupdate the sameunder thedestinationfolderdestDir.Whensource is a folder,givenwithout the trailing /, theentirehierarchyrootedatthefolderwillbeupdated.Usetheformsource/tosyncallfilesinsidethesourcefoldertothedestinationfolder.The -az option indicates the commonly used archivemode to preserve file
7.7
types andmodes andgzip (Chapter 6,Section6.11) data compression to savenetworking bandwidth. The rsync tool normally uses ssh (Section 7.6) forsecure data transfer and does not require a password if you have set uppassword-lessSSHbetweenthetwohosts(Section7.6).Forexample,eitheroneofthesetwocommandsrsync-azpwang@tiger.zodiac.cs.kent.edu:/linux_book/[email protected]:/linux_book//projects/linux_bookupdates the local folder /projects/linux_book based on the remote folder
/linux_bookbylogginginaspwangontheremotehosttiger.zodiac.cs.kent.edu.Seethersyncmanpageforcompletedocumentation.
CRYPTOGRAPHYCryptosystems keep communication safe by encryption, a technique inventedlong before the Internet or digital computers. The concept is simple, theplaintext, the original message, is encrypted into ciphertext beforecommunication.Onlythereceiverknowshowtodecrypttheciphertextbackintoplaintext.For example, rot13 (Figure 7.8) is a simple letter substitution cipherwhere
each letter in the plaintext is replaced by a letter 13 places after it, assumingthereareonly26lettersandthelastletterisfollowedbythefirstletterinacycle.Arot13ciphertextcanbedecryptedbyapplyingtheencryptiononitagain.
Figure7.9ARot13Wheel
Asanotherexample,astationery-paper-sizetemplatewithholescutinitcanbeusedtosendsecretcorrespondences.Asenderwouldwritetheplaintextontocommonstationerypaper through theholesof the template.Thesenderwouldthencomposeaninnocentsoundingletterwithwordsoftheplaintextembeddedinthemanyotherwordsoftheletter.Thereceiverwoulduseacopyofthesametemplatetoeasilyrecovertheplaintext.Or, senders and receivers can agree on a book to use. Ciphertext would
containpagenumber, linenumber andwordnumber to identify anyparticular
7.8
wordfromthebook.Onlypeoplewhoknowwhichbookandwhatthenumbersmeancandecryptthemessage.Further,oneofthemanynumbersmayindicateaparticularbookamongseveralpossibleonestouse.Before and duringWorldWar II, the Germans made heavy use of various
electromechanicalrotorciphermachinesknownasEnigma.These are examples of symmetric cryptosystems (Section 7.8) that use the
samekeytoencipheranddeciphermessages.Communicatingpartiesmustknowthekeybeforehand.Andthekeymustbekeptsecrettoothers.Obviously,rot13,the paper template, the book plus numbering scheme, the Enigma machinesettingsarethekeysintheaboveexamples.Public-keycryptosystems,however,areasymmetricandusenotonebutapair
of keys—one to encrypt and the other to decrypt. The decryption key is keptsecret,whiletheencryptionkeycanbesharedopenly(Section7.9).
SYMMETRICCRYPTOSYSTEMSModernelectronicsymmetricencryptionsystems(Figure7.9)need toworkondigitaldata.Most,ifnotall,ofthemuseanencryption/decryptionalgorithmthatisopenandakeythatiskeptsecret.
Encryption/decryption algorithm: The algorithm performs varioussubstitutionsandpermutationsonchunks,typically128-or256-bitblocks,oftheplaintextorciphertext.Secret key: The plaintext (ciphertext) and the secret key are input to theencryption (decryption) algorithm. The exact transformations performeddependonthekeyused.Thealgorithmproducesdifferentoutputdependingon the key given. Using the same key on the ciphertext, the decryptionalgorithmproducestheoriginalplaintext.
Figure7.10SymmetricCryptosystems
Asymmetriccryptosystemusuallyhasthesetwocharacteristics:
1. Openalgorithm:Theencryption/decryptionalgorithmcanbedescribed intheopen.Itisimpracticaltodecodeanyciphertextknowingthealgorithmandnotthesecretkey.
2. Secretkey:Sendersand receiversmusthaveobtained thekey securely inadvanceandmustallkeepthekeysecret.
The secret key is usually a bit pattern of sufficient length. The quality of thesecretkeyisimportant.Itshouldberandomlygeneratedand256-bitorlongertomakebrute-forceattacks,tryingallpossiblekeys,impractical.Whenapassword(orpassphrase)isusedasakey,itisusuallyputthrougha
key derivation function, which compresses or expands it to the key lengthdesired.Often,arandomlygeneratedpieceofdata,calledasalt,isalsoaddedtothepasswordorpassphrasebeforetransformingittotheactualkey.TheAdvancedEncryption Standard (AES) is a symmetric cryptosystem for
electronic data established by the US National Institute of Standards andTechnology(NIST)in2001.AEShasbeenadoptedbytheUSgovernmentandis now used worldwide. It supersedes the Data Encryption Standard (DES),whichwaspublishedin1977.Thereareothersymmetricciphers,suchasRC4andBlowfish,butAES-256seemstobethebest.Let’s take a closer look atAES-256,whichuses a 256-bit key and encodes
data by encrypting one 256-bit block at a time. The following is an over-simplifiedviewofhowitworks:
1. Arrangesthedatablocktobeencoded/decodedintoa4by4arrayofbytes2. Generatesroundkeysusingthegivenkey3. Transformseachbytebybitwisexorwitharoundkey4. Scramblesandtransformsthe4by4array,inmultiplerounds,byshifting
rows,mixingcolumns,andsubstitutingbytesfromalook-uptablederivedfromthecurrentroundkey
7.9
Figure7.11SampleAESEncryption
Figure7.10showsaplaintextinvolvingthetitleofthisbookandtheAES-256producedciphertextusingthekey“Modern-is-key.”Theencryptedbinaryresult,asequenceofbytes,isdisplayedasastringofcharactersusingbase64encoding.Base64encodingiswidelyusedtoencodeemailattachments.SixtyfourASCIIcharacters are used to represent each 6-bit piece of the binary data tomake ittextual,fordisplay,printing,oremail.
PUBLIC-KEYCRYPTOGRAPHYANDDIGITALSIGNATURE
Security is a big concern when it comes to networking. From the user’sviewpoint,itisimportanttokeepdataandnetworktransportsecureandprivate.Public-key cryptography is an essential part of the modern network securityinfrastructuretoprovideprivacyandsecurityformanynetworkingapplications.Beforetheinventionofpublic-keycryptography,thesamesecretkeyhadtobeused for both encryption and decryption of a message (symmetric-keycryptography).Symmetric-key is fineandefficient,and remains inwidespreaduse today.However, a secret key is hard to arrange among strangers never incommunication before; for example, parties on the Internet. The public-keycryptographybreakthroughsolvesthiskeydistributionproblemelegantly.Inacriticaldeparturefromsymmetriccryptographywhichusesthesamekey
for encryption and decryption, public-key cryptography uses a pair ofmathematicallyrelatedkeys—onekey,thepublickey,toencryptandtheother,theprivatekey,todecrypt.Thepairofkeysareintegerssatisfyingwell-definedmathematicalpropertiesandusuallyproducedbyakeygenerationprogram.Foreachpublickey,thereisonlyonecorrespondingprivatekey,andviceversa.Thepublickeyismadeavailableforanyonewhowishestosendanencrypted
7.10
messagetoarecipientwhousestheprivatekeytodecryptthemessage.Thepublickeyusuallybecomespartofadigitalcertificate,whichverifiably
associates thekey to itsowner.Or, theownerof thekeypairmaypublish thepublic key in online key repositories open to the public. Thus, anyone whowishes to send secure messages will use the public key for encryption. Theownerthenusestheprivatekeytodecrypt(Figure7.11).
Figure7.12Public-KeyCryptography
GNUPRIVACYGUARDGnuPG (GNU Privacy Guard) supports both symmetric cryptography andpublic-keycryptographyandiscompliantwiththeIETFOpenPGPstandard.TheLinux command for GnuPG is gpg or the largely equivalent gpg2. See themanpageforhowtousegpg.Withgpg, you can generate a public key that you share with others and a
privatekeyyoukeep secret.Youandotherscanuse thepublickey toencryptfilesandmessageswhichonlyyoucandecryptusingtheprivatekey.You can digitally sign amessage by encrypting itwith your private key.A
receiver of the signedmessage can decrypt it with the sender’s public key torecovertheoriginalmessageandauthenticatethatitisindeedsent/signedbythesender(Figure7.12).
Figure7.13DigitalSignature
Todoall that,makesureyoufirstsetupGnuPGandyourpersonalkeys. IfyourLinuxdistributiondoesnotalreadyprovidegpg,youcaneasilyinstallthegnupgpackage(Chapter8,Section8.2)witheitherofthefollowingcommands:
dnfinstallgnupg(CentOS/Fedora)apt-getinstallgnupg
(Ubuntu/Debian)
Ifyoulike touseaGUIforgpg, installalso thegpapackage.However, thecommand-lineinterfaceisentirelyadequate.
SettingUpGnuPGKeysTousegpg,youfirstneedtogenerateyourpublic-privatekeypair.gpg–gen-keyYou’ll be prompted to enter your choices for keytype (pick the default),
keysize(pick2048),andapassphrase(picksomethingyouwon’tforget,butwillbe very hard for anyone to guess). The passphrase is required each time youaccessyourprivatekey,thuspreventingothersfromusingyourprivatekey.You’llgetakeyiddisplayedwhenyourkeypairisgenerated.Yourkeysand
otherinfoarestoredbydefaultinthefolder$HOME/.gnupg.Usegpg–list-public-keystodisplayyourpublickeys.Forexample,pub1024D/FCF2F84D2018-07-25uidPaulWang(monkeykia)<[email protected]>sub1024g/B02C4B402018-07-25Thepub linesays thatPaul’spublicmasterkey (for signature) isa1024-bit
DSAkeywithidFCF2F84Dandthathispublicsubkey(fordataencryption)isa1024-bitElGamakey.
Thekeyuidis"PaulWang",monkeykia,orthelistedemailaddress.To enable others to encrypt information to be delivered for your eyes only,
youshouldsendyourpublickeystoapublickeyserver.Thecommandgpg–send-keysyour_keyidsendsyourpublickeytoadefaultgpgkeyserver,suchashkp://subkeys.pgp.netAlso, you can send your public keys to anyone by sending them anASCII
armoredfilegeneratedbygpg–armor–exportyour_keyid>mykey.ascThe.ascsuffixsimplyindicatesthatafileisanASCIItextfile.Themykey.asc
containsyourkeyencodedusingbase64,awaytouse64ASCIIcharacters(A–Z, a–z, 0–9 and +/) to encode non-ASCII files for easy communication overnetworks, especially via email. The Linux base64 command performs thisencoding/decodingonanyfile.Seemanbase64formoreinformation.Such ASCII armored key files can be emailed to others or sent to another
computerandimportedtoanotherGnuPGkeyringwithacommandsuchasgpg–importmykey.ascAlso,edityour$HOME/gnupg/gpg.conffileandappendthelinedefault-keyyour_keyid
ncryption/DecryptionwithGnuPGToencryptafileusingapublickeyofkey_uid,gpg–encrypt-rkey_uidfilenameresultinginanencryptedfile filename.gpgthatcanbesent to the targetuser
whoistheonlyonethatcandecryptit.Evenifyouarenotgoingtosendafiletoanyone,youcanstillkeepsecretsin
that fileofyoursprotected incase someonegainsunauthorizedaccess toyourcomputeraccount.Youcangpg–encrypt-r"your_key_uid"filenamermfilenamegenerating the encrypted filename.gpg and removing the original filename.
Youcaneasilyviewtheencryptedversionwithgpg–decryptfilename.gpg|moreTomakemaintaininganencryptedfileeveneasier,youmayconfigurevi/vim
toworktransparentlywithgpg,allowingyoutousevim toviewandeditclearaswellasgpgencryptedfiles.TheVIMextensiontGpg(yetanotherplug-inforencryptingfileswithgpg)isagoodchoiceforthispurpose.The Seahorse tool (Chapter 4, Section 4.1) is convenient to manage your
7.11
encryptionkeys.
SECUREEMAILModern email clients, such as Microsoft Outlook and the open sourceThunderbird, support secure email specified by the S/MIME (SecureMultipurposeInternetMailExtensions)standard.Oncesetup,youcansendandreceiveencryptedemail,aswellassignedmessages.Whenanemailmessageisencrypted,emailcontentsandattachmentsareturnedintociphertext.Ofcourse,theemailsubjectorotheremailheadersarenotencrypted.Whenanemailissigned,nothingisencrypted,exceptasignedmessagedigest
is attached.Normally,wedonotwant to signour email.But, it is possible tobothencryptandsignanemail.Theprerequisite forS/MIMEis thateachcorrespondentmusthaveanemail
certificateinstalledinasecureemailclient.Commercial personal email (S/MIME) certificates are easily available from
CAs. You may even find CAs that offer free email certificates. But thecertification application and verification process may be complicated andbothersometomanyusers.A good alternative to S/MIME is PGP/MIME (PrettyGood Privacy)which
doesnotrequireaCA-issuedcertificate.ThefreeGnuPG(GNUPrivacyGuard;GPG) is an implementation of the OpenPGP standard. Using GPG, you cangenerateanddistributeyourownkeypairforsecureemail.Let’stakealookathowtosetupThunderbirdforsecureemailwithGnuPG.
Other email clients, such as evolution and mutt, can work with GnuPGsimilarly.
SecureEmailwithThunderbirdFirst make sure you have Thunderbird and GnuPG installed on your Linuxdistribution. Then follow this simple procedure to enable Thunderbird secureemail(PGP/MIME).
1. OpenThunderbird and use the tools->Add-ons option to search for andinstalltheEnigmailadd-on.
2. FollowtheEnigmailset-upwizardtosetupyourkeypair(useyourcorrectname and email address). The email address should correspond to yourThunderbird email identity.Youmay choose for the key to never expire.Alsoit isrecommendedthatyouchoosethe4096RSA/RSAkeyfromtheAdvancedoptions.
3. Optionally,youmaychoosetoassociateaJPGimagewiththekey.Thiscanbedonelateralso.
Figure7.14ThunderbirdwithEnigmailAdded
Now,yourThunderbirdissetup(Figure7.13).But,beforeyoucansendandreceiveencryptedemail,youneedto(A) Send your public key to people you know so they can send encrypted
emailtoyou.(B)Receive/installtheirpublickeyssoyoucanencryptemailtothem.For(A)dothis:
1. From the ThunderbirdMenu (click the three-bars icon on the right-handsideoftheThunderbirdmenubar),selectOpenPGP->KeyManagementtopopuptheKeyManagementdialog(Figure7.14).
2. IntheKeyManagementdialog,checktheoptionboxDisplayAllKeysbyDefault.Youshouldseeyourkeylisted.
3. Right click the key youwant, and select the option Send PublicKey byEmail.ThesameoptionisalsoavailablefromtheFilemenu.
4. An email compositionwindow openswith the key file (with .asc suffix)alreadyattached.Justsendthisemailnormally.Thatisit.
Figure7.15SendingPublicKey
7.12
For(B),dothis:
1. Open incoming email from your friend containing his/her public key asattachment.
2. Open (double click) the key file attachment (with .asc suffix), andThunderbirdwillinstallthepublickeyreceivedautomatically.
Nowyouare truly ready for secureemail.Aftercomposingyourmessageandaddingattachments,selecttheoptionsOpenPGP->encrypt(toencrypttheemailmessage) and/orOpenPGP- > sign (to add a signature) before sending. If theemail has one or more attachments, be sure to also select the OpenPGP- >PGP/MIME option. When you open an encrypted email in your inbox,Thunderbirdasksyouforthepassphraseofyourprivatekeyandthendecryptsitforyouautomatically.
MESSAGEDIGESTSAmessagedigestisadigitalfingerprintofamessageorfile.Variousalgorithmshavebeendevisedtotakeamessage(file)ofanylengthandreduceittoashortfixed-length hash known as the digest of the original message or file (Figure7.15).
Figure7.16MD5MessageDigest
Thesealgorithmsaredesignedtoproduceadifferentdigestifanypartofthemessageisaltered.Itisalmostimpossibletodeducetheoriginalmessagefromknowledge of the digest. However, because there are an infinite number ofpossiblemessagesbutonlyafinitenumberofdifferentdigests,vastlydifferentmessagesmayproducethesamedigest.Messagedigestsarethereforeusefulinverifyingtheintegrity (unalteredness)
of files. When software is distributed online, a good practice is to display afingerprintforthefile,allowingyoutochecktheintegrityofthedownloadandtoavoidanyTrojanhorsecode.MD5isapopularalgorithmproducing128-bitmessagedigests.AnMD5hash
isusuallydisplayedasasequenceof32hexadecimaldigits.OnLinux,youcanproduceanMD5digestwiththemd5sumcommandmd5sumfilename>digestFileYou’ll get adigestFile file containingonly thehash and thename filename.
AfterdownloadingbothfilenameanddigestFile,ausercancheckfile integritywithmd5sum-cdigestFileOther digest algorithms in wide use include SHA-1 and others. The Linux
commandsha1sumisanalternativetomd5sum.
SoftwareandMessageSigningTo digitally sign a particularmessage (or file) without having to encrypt thatentiremessage isoftendesirable.Todo this, adigestof themessageor file iscreatedfirst,usingasuitablemessagedigesthashfunction.
Figure7.17DigitalSignature
Todigitallysignaparticularmessageorapieceofsoftware,adigestorhashis created first. The digest is then encrypted using the signer’s private key toproduce a digital signature. Any receiver of a signed message/software cangenerate amessage digest from the receivedmessage and check it against thedigestobtainedbydecryptingthedigitalsignaturewiththesigner’spublickey.Amatchverifiestheintegrityandtheauthenticityofthereceivedmessage(Figure7.16,source:Wikipedia).
7.13
Hereishowtousegpgfordigitalsignature.
gpg--signfile(producessignedbinaryfile.gpg)gpg--clearsign
file(producessignedASCIIfile.asc)
The–decryptoptionautomaticallyverifiesanyattachedsignature.
THEWEBOut of all the networking applications, the Web is perhaps one of the mostimportantanddeservesourspecialattention.There is no central control or administration of the Web. Anyone can
potentiallyputmaterialon theWeband retrieve information from it.TheWebconsists of a vast collection of documents that are located on computersthroughout theworld.Thesedocuments are createdbyacademic,professional,government, and commercial organizations, as well as by individuals. Thedocuments areprepared in special formats anddelivered throughWeb servers,programsthatreturndocumentsinresponsetoincomingrequests.LinuxsystemsareoftenusedtorunWebservers.AnintroductiontotheWebisprovidedinthischapter.Chapter9discussesservingtheWeb.Primarily, Web documents are written in Hypertext Markup Language
(HTML,Section7.16). EachHTMLdocument can contain (potentiallymany)links to other documents served by different servers in other locations andthereforebecomepartofawebthatspanstheentireglobe.NewmaterialsareputontheWebcontinuously,andinstantaccesstothiscollectionofinformationcanbeenormouslyadvantageous.AstheWebgrew,MIT(MassachusettsInstituteofTechnology, Cambridge, MA) and INRIA (the French National Institute forResearchinComputerScienceandControl)agreedtobecomejointhostsoftheW3Consortium,astandardsbodyfortheWebcommunity.AWebbrowser isaprogramthathelpsusersobtainanddisplay information
fromtheWeb.Giventhelocationofatargetdocument,abrowserconnectstothecorrectWeb server and retrieves and displays the desired document. You canclicklinks inadocument toobtainotherdocuments.Usingabrowser,youcanretrieveinformationprovidedbyWebserversanywhereontheInternet.Typically,aWebbrowser,suchasFirefox,supportsthedisplayofHTMLfiles
andimages instandardformats.Helperapplicationsorplug-inscanaugmentabrowsertotreatpageswithmultimediacontentsuchasaudio,video,animation,andmathematicalformulas.
HypertextMarkupLanguage
AWebbrowsercommunicateswithaWebserver throughanefficientprotocol(HTTP)which is designed toworkwithhypertext andhypermediadocumentsthatmaycontain regular text, images,audio,andvideo.NativeWebpagesarewrittenintheHTML(Section7.16)andusuallysavedinfileswiththe.html(or.htm)suffix.HTMLorganizesWebpagecontent(text,graphics,andothermediadata)and
allows hyperlinks to other pages anywhere on theWeb. Clicking such a linkcauses your Web browser to follow it and retrieve another page. The Webemploys an open addressing scheme that allows links to objects and servicesprovidedbyWeb,email,filetransfer,audio/video,andnewsgroupservers.Thus,theWebspaceisasupersetofmanypopularInternetservices.Consequently,aWeb browser provides the ability to access awide variety of information andservicesontheInternet.
URLsTheWebusesUniformResourceLocators(URLs)toidentify(locate)resources(filesandservices)availableontheInternet.AURLmayidentifyahost,aserverport, and the target file stored on that host. URLs are used, for example, bybrowserstoretrieveinformationandbyHTMLtolinktootherresources.AfullURLusuallyhastheformscheme://server:port/pathnameThe scheme, part indicates the information service type and therefore the
protocol touse.Commonschemes includehttp(Webservice), ftp(file transferservice), mailto (email service), file (local file system), https (secure Webservice),andsftp(securefiletransferservice).Forexample,sftp://[email protected]/users/cs/faculty/pwanggetsyou thedirectory listof /users/cs/faculty/pwang.ThisworksonFirefox
and on the Linux file browser nautilus, assuming that you have set up yourSSH/SFTP (Section 7.6). Many other schemes can be found atwww.w3.org/addressing/schemes.ForURLs ingeneral, theserver identifiesahostandaserverprogram.The
optionalportnumber isneededonly if theserverdoesnotuse thedefaultport(forexample,21forFTPand80forHTTP).Theremainderof theURL,whengiven,isafilepathname.Ifthispathnamehasatrailing/character,itrepresentsadirectoryratherthanadatafile.Thesuffix(.html,.txt,.jpg,etc.)ofadatafileindicatesthefiletype.ThepathnamecanalsoleadtoanexecutableprogramthatdynamicallyproducesanHTMLorothervalidfiletoreturn.WithinanHTMLdocument,youcanlinktoanotherdocumentservedbythe
7.14
sameWebserverbygivingonlythepathnamepartoftheURL.SuchURLsarepartiallyspecified.ApartialURLwith a / prefix (for example, /file_xyz.html)refers to a file under the serverroot, the top-level directory controlled by theWeb server. A partial URLwithout a leading / points to a file relative to thelocation of the document that contains the URL in question. Thus, a simplefile_abc.html refers to that file in the samedirectory as the current document.Whenbuildingawebsite,itisadvisabletouseaURLrelativetothecurrentpageasmuchaspossible,makingiteasytomovetheentirewebsitefoldertoanotherlocationonthelocalfilesystemortoadifferentserverhost.
AccessingInformationontheWebYoucandirectly access anyWebdocument, directory, or servicebygiving itsURL in the Location box of a browser. When given a URL that specifies adirectory, aWeb serverusually returns an index file (typically, index.html) forthatdirectory.Otherwise,itmayreturnalistofthefilenamesinthatdirectory.YoucanuseasearchenginesuchasGoogle toquickly lookfor information
ontheWeb.
HANDLINGDIFFERENTCONTENTTYPESOntheWeb,filesofdifferentmediatypescanbeplacedandretrieved.TheWebserver andWebbrowseruse standardcontent type designations to indicate themediatypeoffilesinordertoprocessthemcorrectly.The Web borrowed the content type designations from the Internet email
system and uses the same MIME (Multipurpose Internet Mail Extensions)definedcontent types.Therearehundredsofcontent types inuse today.Manypopulartypesareassociatedwithstandardfileextensions.Chapter6,Table6.3givessomeexamples.When a Web server returns a document to a browser, the content type is
indicated. The content type information allows browsers to decide how toprocesstheincomingcontent.Normally,HTML,text,andimagesarehandledbythebrowserdirectly.Othertypessuchasaudioandvideoareusuallyhandledbyplug-insorexternalhelperprograms.
7.15
Figure7.18WebServerFunction
PUTTINGINFORMATIONONTHEWEBNow let’s turn our attention to how information is supplied on theWeb. TheunderstandingshedsmorelightonhowtheWebworksandwhatittakestoserveupinformation.TheWebputsthepowerofpublishinginthehandsofanyonewithacomputer
connectedtotheInternet.Allyouneedis torunaWebserveronthismachineandestablishfilesforittoservice.Major computer vendors offer commercialWeb serverswith their computer
systems.Apacheisawidelyusedopen-sourceWebserverthatisfreelyavailablefromtheApacheSoftwareFoundation(www.apache.org).Linux systems are especially popular as Web hosting computers because
Linux is free, robust, and secure. Also, there are many useful Web-relatedapplicationssuchasApache,PHP(activeWebpage),MySQL(databaseserver),andmoreavailablefreeofcharge.OnceaWebserverisupandrunningonyourmachine,alltypesoffilescanbe
served (Figure7.17).On a typical Linux system, follow these simple steps tomakeyourpersonalWebpage.
1. Make a file directory in your home directory ( userid /public_html) tocontainyourfilesfortheWeb.ThisisyourpersonalWebdirectory.Makethisdirectorypubliclyaccessible:chmoda+xuserid/public_htmlWhenindoubt, ask your system managers about the exact name to use for yourpersonalWebdirectory.
2. In your Web directory, establish a home page, usually index.html, inHTML.Thehomepageusuallyfunctionsasanannotatedtableofcontents.Makethisfilepubliclyreadable:chmoda+ruserid/public_html/index.html
7.16
3. Placefilesanddirectoriescontainingdesired informationinyourpersonalWeb directory. Make each directory and each file accessible as before.Refertothesefileswithlinksinthehomepageandotherpages.
4. Let people know the URL of your home page, which is typicallyhttp://your-severyour-userid
In a Web page, you can refer to another file of yours with a simple linkcontaining a relative URL ( < a href="filename" > ), where filename can beeitherasimplenameorapathnamerelativetothecurrentdocument.AmongtheWebfileformats,hypertextiscriticalbecauseitprovidesameans
foradocumenttolinktootherdocuments.
WHATISHTML?HTML (theHypertextMarkup Language) is used tomarkup the content of aWeb page to provide page structure for easy handling byWeb clients on thereceiving end. Since HTML 4.0, the language has become standardized.XHTML (XML compatible HTML) is the current stable version. However, anewstandardHTML5isfastapproaching.AdocumentwritteninHTMLcontainsordinarytextinterspersedwithmarkup
tagsandusesthe.htmlfilenameextension.Thetagsmarkportionsofthetextastitle,sectionheader,paragraph,referencetootherdocuments,andsoon.Thus,anHTMLfileconsistsoftwokindsofinformation:contentsandHTMLtags.AbrowserfollowstheHTMLtagstolayoutthepagecontentfordisplay.Becauseof this, line breaks and extra white space between words in the content aremostly ignored. Inaddition to structuringand formattingcontents,HTMLtagscan also reference graphics images, link to other documents, mark referencepoints,generate formsorquestionnaires, and invokecertainprograms.VariousvisualeditorsorpagemakersareavailablethatprovideaGUIforcreatinganddesigningHTMLdocuments.Forsubstantialwebsitecreationprojects,itwillbehelpful to use integrated development environments such as MacromediaDreamweaver.Ifyoudon’thavereadyaccesstosuchtools,aregulartexteditorcancreateoreditWebpages.AnHTMLtagtakestheform<tag>.Abegintagsuchas<h1>(level-onesectionheader)ispairedwithanendtag,</h1> inthiscase,tomarkcontentinbetween.Table7.1listssomefrequentlyusedtags.SomeHTMLTags
7.17
ThefollowingisasampleHTMLpage(Ex:ex07/Fruits):
>html<>head<>title<ABasicWebPage>/title<>/head<>body<>h1<Big
onFruits>/h1<>p<Fruitsaregoodtastingandgoodforyou
...>/p<>p<Therearemanyvarieties,...andhereisashortlist:
>/p<>ol<>li<Apples>/li<>li<Bananas>/li<>li<Cherries
>/li<>/ol<>/body<>/html<
Figure7.18showstheBigonFruitspagedisplayedbyFirefox.
WEBHOSTINGWebhostingisaservicetostoreandserveready-madefilesandprogramssothattheyareaccessibleontheWeb.Hence,publishingontheWebinvolves
1. Designing and constructing the pages and writing the programs for awebsite
2. Placingthecompletedsitewithahostingservice
Colleges and universities host personal and educational sites for students andfacultywithoutcharge.Webhostingcompaniesprovidetheserviceforafee.
Figure7.19ASampleWebPage
CommercialWebhostingcanprovidesecuredatacenters(buildings),fastandreliable Internet connections, specially tuned Web hosting computers (mostlyLinuxboxes),serverprogramsandutilities,networkandsystemsecurity,dailybackup,andtechnicalsupport.Eachhostingaccountprovidesanamountofdiskspace, a monthly network traffic allowance, email accounts, Web-based sitemanagement and maintenance tools, and other access such as FTP andSSH/SFTP.
7.18
7.19
Tohost a site under a givendomainname, a hosting service associates thatdomain name to an IP number assigned to the hosted site. The domain-to-IPassociation is made through DNS servers and Web server configurationsmanagedbythehostingservice.
DOMAINREGISTRATIONTo obtain a domain name, you need the service of a domain name registrar.Mostwillbehappytoregisteryournewdomainnameforaverymodestyearlyfee.Onceregistered,thedomainnameispropertythatbelongstotheregistrant.Nooneelsecanregisterforthatparticulardomainnameaslongasthecurrentregistrantkeepstheregistrationingoodorder.ICANN accredits commercial registrars for commonTLDs, including .com,
.net, .org, and .info. Additional TLDs include .biz, .pro, .aero, .name, and
.museum.Restricteddomains(forexample, .edu, .gov,and .us)arehandledbyspecial registries (forexample,net.educause.edu,nic.govandnic.us).Country-codeTLDsarenormallyhandledbyregistriesintheirrespectivecountries.
AccessingDomainRegistrationDataThe registration record of a domain name is often publicly available. ThestandardInternetwhoisserviceallowseasyaccesstothisinformation.OnLinuxsystems,easyaccesstowhoisisprovidedbythewhoiscommandwhoisdomain_namewhichliststhedomainregistrationrecordkeptataregistrar.Forexample,whoiskent.eduproducesthefollowinginformation
DomainName:KENT.EDURegistrant:KentStateUniversity500E.Main
St.Kent,OH44242UNITEDSTATESAdministrativeContact:PhilipL
ThomasNetwork&TelecommKentStateUniversitySTHKent,OH44242
UNITEDSTATES(330)[email protected]
Contact:NetworkOperationsCenterKentStateUniversity120Library
BldgKent,OH44242UNITEDSTATES(330)[email protected]
Servers:NS.NET.KENT.EDU131.123.1.1DHCP.NET.KENT.EDU
131.123.252.2Domainrecordactivated:19-Feb-1987Domainrecordlast
updated:17-Feb-2016Domainexpires:31-Jul-2018
OnLinuxsystems,thewhoiscommandisusuallyalinktojwhois.
THEDNS
DNS provides the ever-changing domain-to-IP mapping information on theInternet.Wementioned that DNS provides a distributed database service thatsupports dynamic retrieval of information contained in the name space. Webbrowsers and other Internet client applications will normally use the DNS toobtain the IP of a target host before making contact with a server over theInternet.TherearethreeelementstotheDNS:theDNSnamespace(Section7.2),the
DNSservers,andtheDNSresolvers.
DNSServersInformation in the distributed DNS is divided into zones, and each zone issupported by one ormore name servers running on different hosts.A zone isassociatedwithanodeonthedomaintreeandcoversallorpartofthesubtreeatthatnode.Anameserverthathascompleteinformationforaparticularzoneissaidtobeanauthorityforthatzone.Authoritativeinformationisautomaticallydistributed to other name servers that provide redundant service for the samezone. A server relies on lower level servers for other information within itssubdomainandonexternalserversforotherzonesinthedomaintree.Aserverassociatedwiththerootnodeofthedomaintreeisarootserverandcanleadtoinformation anywhere in the DNS. An authoritative server uses local files tostore information, to locate key serverswithin andwithout its domain, and tocache query results from other servers. A boot file, usually /etc/named.boot,configuresanameserveranditsdatafiles.Themanagementofeachzoneisalsofreetodesignatethehoststhatrunthe
nameserversandtomakechangesinitsauthoritativedatabase.Forexample,thehostns.cs.kent.edumayrunanameserverforthedomaincs.kent.edu.Anameserveranswersqueries fromresolversandprovideseitherdefinitive
answersorreferralstoothernameservers.TheDNSdatabaseissetuptohandlenetworkaddress,mailexchange,hostconfiguration,andothertypesofqueries,withsometobeimplementedinthefuture.The ICANNandothersmaintainrootnameservers associatedwith the root
node of theDNS tree. In fact, theVeriSign host a.root-servers.net runs a rootnameserver.Actually, the letterarangesuptomfora totalof13rootserverscurrently.Domainnameregistrars,corporations,organizations,Webhostingcompanies,
andotherInternetserviceproviders(ISPs)runnameserverstoassociateIPstodomain names in their particular zones. All name servers on the Internetcooperatetoperformdomain-to-IPmappingsonthefly.
7.20
DNSResolversADNS resolver is a program that sends queries to name servers and obtainsreplies fromthem.OnLinuxsystems,a resolverusually takes theformofaClibrary function. A resolver can access at least one name server and use thatnameserver’sinformationtoansweraquerydirectlyorpursuethequeryusingreferralstoothernameservers.Resolvers, in the form of networking library routines, are used to translate
domain names into actual IP addresses. These library routines, in turn, askprescribednameserverstoresolvethedomainnames.Thenameserverstousefor any particular host are normally specified in the file /etc/resolv.conf or/usr/etc/resolv.conf.CommonDNSRecord/RequestTypes
The DNS service provides not just the IP address and domain nameinformationforhostsontheInternet.Itcanprovideotherusefulinformationaswell.Table7.2showscommonDNSrecordandrequesttypes.
DYNAMICGENERATIONOFWEBPAGESDocuments available on theWeb are usually prepared and set in advance tosupplysomefixedcontent,eitherinHTMLorinsomeotherformatsuchasplaintext, PDF, or JPEG.These fixed documents are static.AWeb server can alsogeneratedocumentsontheflythatbringtheseandotheradvantages:
Customizing a document depending on when, where, who, and whatprogramisretrievingitCollecting user input (withHTML forms) and providing responses to theincominginformationEnforcingcertainpoliciesforoutgoingdocumentsSupplying contents such as game scores and stock quotes, which arechangingbynature
DynamicWeb pages are not magic. Instead of retrieving a fixed file, aWebserver calls another program to compute thedocument to be returned.Asyoumay have guessed, not every program can be used by a Web server in thismanner.Therearetwowaystoaddserver-sideprogramming:
LoadprogramsdirectlyintotheWebservertobeusedwhenevertheneedarises.Callanexternalprogramfromtheserver,passingarguments to it (via theprogram’s stdin and environment variables) and receiving the results (viatheprogram’sstdout)thusgenerated.SuchaprogrammustconformtotheCommonGateway Interface (CGI) specificationsgoverninghow theWebserverandtheexternalprograminteract(Figure7.19).
Figure7.20CommonGatewayInterface
DynamicServerPagesThedynamicgenerationofpagesismadesimplerandmoreintegratedwithWebpagedesignandconstructionbyallowingaWebpagetocontainactivepartsthataretreatedbytheWebserverandtransformedintodesiredcontentontheflyasthepageisretrievedandreturnedtoaclientbrowser.Theactivepartsinapagearewritteninsomekindofnotationtodistinguish
themfromthestaticpartsofapage.TheASP(ActiveServerPages),JSP(JavaServerPages),andthepopularPHP(HypertextPreprocessor;Chapter9,Section9.17)areexamples.BecauseactivepagesaretreatedbymodulesloadedintotheWebserver, the
processingisfasterandmoreefficientcomparedtoCGIprograms.Activepagetechnologies such as PHP also provide form processing, HTTP sessions, andeasyaccesstodatabases.Therefore,theyoffercompleteserver-sidesupportfordynamicWebpages.BothCGIandserverpagescanbeusedtosupportHTMLforms,thefamiliar
fill-outformsyouoftenseeontheWeb.
7.21 HTTPBRIEFLYOn the Web, browser-server communication follows HTTP. A basicunderstanding of HTTP is important for Linux programmers because LinuxsystemsareverypopularWebserverhosts.ThestartofHTTPtracesbacktothebeginningoftheWebintheearly1990s.
HTTP/1.0wasstandardizedearlyin1996.ImprovementsandnewfeatureshavebeenintroducedandHTTP/1.1isnowthestableversion.HereisanoverviewofanHTTPtransaction:
1. Connection—Abrowser(client)opensaconnectiontoaserver.2. Query—Theclientrequestsaresourcecontrolledbytheserver.3. Processing—Theserverreceivesandprocessestherequest.4. Response—Theserversendstherequestedresourcebacktotheclient.5. Termination —The transaction is finished, and the connection is closed
unlessanother transaction takesplace immediatelybetween theclientandserver.
HTTPgovernstheformatofthequeryandresponsemessages(Figure7.20).Theheaderpart is textual,andeachlineintheheadershouldendinRETURN
andNEWLINE,butitmayendinjustNEWLINE.Theinitiallineidentifiesthemessageasaqueryoraresponse.
Aquery linehas threepartsseparatedbyspaces:aquerymethod name, alocal path of the requested resource, and an HTTP version number. Forexample,GET /path/to/file/index.htmlHTTP/1.1 or POST /path/script.cgiHTTP/1.1 TheGETmethod requests the specified resource and does notallowamessagebody.AGETmethodcaninvokeaserver-sideprogrambyspecifyingtheCGIoractive-pagepath,aquestionmark,andthenaquerystring: GET /cgi-bin/newaddr.cgi?name=value1&email=value2 HTTP/1.1Host: tiger.zodiac.cs.kent.edu Unlike GET, the POST method allows amessage body and is designed to work with HTML forms for collectinginputfromWebusers.A response (or status) line also has three parts separated by spaces: anHTTPversionnumber,astatuscode,andatextualdescriptionofthestatus.Typical status lines are HTTP/1.1 200 OK for a successful query orHTTP/1.1404NotFoundwhentherequestedresourcecannotbefound.TheHTTPresponsesendstherequestedfiletogetherwithitscontenttype(Section7.14)andlength(optional)sotheclientwillknowhowtoprocessit.
7.22
7.23
AREALHTTPEXPERIENCELet’smanuallysendanHTTPrequestandgetanHTTPresponse.Todothatwewill use the nc command. The command ncat provides command-line (andscripting) access to the basic TCP and UDP (Chapter 12, Section 12.6) andtherefore allowsyou tomake anyTCPconnectionsor send anyUDPpackets.Suchabilitiesareusuallyreserved toprogramsat theC-language level thatsetupsockets(Chapter12,Section12.6)fornetworking.Forexample,thesimpleBashpipeline(Ex:ex07/poorbr.sh)echo$’GET/WEB/test.htmlHTTP/1.0n’|ncattiger.zodiac.cs.kent.edu80retrieves the Web page tiger.zodiac.cs.kent.edu/WEB/test.html. In this
example,weappliedtheBashstringexpansion(Chapter3,Section3.7).Note the HTTP get request asks for the file /WEB/test.html under the
document root folder managed by the Web server on tiger. The request isterminatedbyanemptyline,asrequiredbytheHTTPprotocol.Trythisandyou’llseetheresultdisplay.
HTTP/1.1200OKDate:Tue,20Mar201819:45:03GMTServer:
Apache/2.4.27(Fedora)X-Powered-By:PHP/7.0.23Cache-Control:max-
age=86400Expires:Wed,21Mar201819:45:03GMTVary:Accept-
EncodingContent-Length:360Connection:closeContent-Type:
text/html;charset=UTF-8>!DOCTYPEHTML<>html
xmlns="http://www.w3.org/1999/xhtml"lang="en"xml:lang="en"<AND
THERESTOFTHEHTMLPAGE>/html<
AsyoucanseefromtheHTTPresponse, theWebserverontiger isApacheversion2runningunderFedora,aLinuxsystem.FordownloadingfromtheWeb,youdon’tneedtorelyonourlittlepipeline.
The wget command takes care of that need nicely. Wget supports HTTP,HTTPS, and FTP protocols and can download single files or follow links inHTMLfilesand recursivelydownloadentirewebsites forofflineviewing.Thewget command can continue towork after you log out so you can downloadlargeamountsofdatawithoutwaiting.
FORMOREINFORMATIONIPv6 is the next-generation Internet protocol. See www.ipv6.org/ for anoverview.The official website for Gnu Privacy Guard is www.gnupg.org, and for
7.24
OpenSSH,iswww.openssh.com.Public-Key Cryptography Standards (PKCS) can be found at RSALaboratories(www.rsa.com/rsalabs).HTML5isthenewandcomingstandardforHTML.SeethespecificationatW3C.TheDNSisbasictokeepingservicesontheInternetandWebrunning.FindourmoreaboutDNSatwww.dns.net/dnsrd/docs/.HTTPisbasictotheWeb.SeeRFC1945forHTTP1.0andRFC2068forHTTP1.1.
SUMMARYInthemoderncomputingenvironment,computersandnetworksareinseparable.Networking is an important aspect of any operating system, especially Linuxbecause the Internet has its origins in UNIX/Linux, and Linux systems areexcellentserverhosts.OntheInternet,eachhostcomputerisidentifiedbyitsIPaddressaswellasby
its domain name. TheTCP/IP andUDP/IP protocols are basic to the Internet.Network-based services often follow the client-and-servermodel,where clientprograms(suchasWebbrowsers)communicatewithserverprograms(suchasWebservers)usingwell-definedprotocols (suchasHTTP).Aparticular serverprogramrunningonaspecifichostisidentifiedbythehost’sIPordomainnametogetherwiththeserverprogram’sportnumber(suchas80forWebservers).The ICANN manages the IP address space and the DNS. The distributed
Domain Name Service is a fundamental networking service because itdynamicallymaps domain names to IP addresses and also provides importantinformation for sending/receiving email. The commands host,nslookup, anddigcanbeusedtoobtainDNSdatafortargethosts.Withnetworkingyoucanupload/downloadfileswith ftpandsftp; log in to
remotecomputerswithtelnetandssh;copyandsynchfileswithrcp,scp,andrsync;checkifaremotesystemisalive/connectedwithping;testprotocolswithnc; access the Web; send and receive emails, and perform many otheroperations.When it comes to networking, security and privacy are important concerns.
Increasingly, computer systems require SSH, SFTP, and SCP for betterprotection. Automatic file sync can also use SSH for data transfer. The GnuPrivacyGuard(GnuPG)supportssymmetricandpublic-keycryptography. It isuseful for data/file encryption, secure email, and digital signature. Message
7.25
digest algorithms such as MD5 can produce digital fingerprints fordata/programstoguardtheirintegrity.LinuxsystemsareoftenusedtorunWebserversandtoprovideWebhosting
for individuals and organizations. BasicWeb documents are coded inHTML.Hyper references use URLs to link to other documents.MIME content typesindicatethemediatypeservedontheWeb.ThestatelessHTTPisarequest-responseprotocolwhosemessagesmayhave
anumberofheadersandanoptionalmessagebody.
EXERCISES1. What isacomputernetwork?Name themajorcomponents inacomputer
network.2. What is a networking client? What is a networking server? What is a
networkingprotocol?3. WhataddressingschemedoestheInternetuse?WhatistheformatofanIP
address?Whatisthequadnotation?4. Consider theIPaddress123.234.345.456Is thereanythingwrongwith it?
Pleaseexplain.5. RefertoSection7.6andsetupyourownpassword-lessSSHandSFTP.6. You can schedule commands to be executed automatically by Linux at
regular intervals. Find out about the crontab and the crontab command.Thensetupyourcrontabtorsyncsomeimportantfolderfromonesystemtoanother.Showyourcrontabcodeinfullandexplain.
7. RefertoSection7.9andsetupyourGnuPGkeys.8. RefertoSection7.11andsetupsecureemailwithThunderbird.9. Writea script thatwillencrypt/decryptwithgpg a file and leave it in the
sameplaceasbefore(withthesamefilename).10. WhatisDNS?Whydoweneedit?11. Whatdonameserversdo?Whydoweneedthem?12. WhatistherelationbetweentheWebandtheInternet?Whatistherelation
betweenHTTPandTCP/IP?13. WhatarethemajorcomponentsoftheWeb?WhyisHTMLcentraltothe
Web?14. What is the difference between aWeb server and aWeb browser? Is the
Webserverapieceofhardwareorsoftware?Explain.15. HowdoesaWebpagegetfromwhereitistothecomputerscreenofauser?16. WhatisaURL?WhatisthegeneralformofaURL?Explainthedifferent
1
URLschemes.17. Whatarecontenttypes?Howaretheyuseful?18. What is the difference between a staticWeb page and a generatedWeb
page?19. WhatisanHTTPtransaction?WhatisanHTTPquery?WhatisanHTTP
response?20. Take the domain name sofpower.com and write the full URL that will
accessitsWebserver.Add/linuxtotheendofthatURL.Wheredoesthatlead?
21. Take the domain name sofpower.comand find its IP address.Use this IPaddressinsteadofthedomainnametovisitthesite.WritethebitpatternforthisIPaddress.
22. SearchontheWebforICANN.Visit thesiteanddiscoveritsmissionandservices.
23. Findthedomainrecordforsofpower.com.Whoistheownerofthisdomainname?Whoaretheadministrativeandtechnicalcontacts?
24. FindtheDNSrecordforsofpower.com.25. Find out and describe in your own words what the special domain in-
addr.arpais.26. Refer to Section 7.22. Explain the notation $’GET /WEB/test.html
HTTP/1.0n’27. Refer to Section 7.22. Use the nc command to write a poor man’s Web
browserscriptpoorman.sh.poorman.shpathhostretrievesthepagehttp://host/path.The ARPANET was started in the late 1960s as an experimental facility for reliable military
networking.
8.1
Chapter8
BasicSystemAdministration
StudentsofLinuxnaturallywillfirstfocusonunderstandingLinuxandhowtouseiteffectively.Systemadministrationissomethingsuperusershandle.Yet,itisanimportantaspectofLinuxandfamiliaritywith,oratleastsomedegreeofunderstandingof,systemadminwillbebeneficialtohomesystemusersaswellas want-to-be administrators. A rewarding system admin job is a distinctpossibilityforwell-trainedLinuxpersonnel.Linux system admin is a vast topic ranging from hardware configuration,
managing multiple systems company wide, server farm operations, to cloudcomputing.Mostofthisisoutsidethescopeofthistextbook.Ourpresentationherefocusesonsystemadminbasicssuchasmanaginghome
Linux systemsand small-scalenetworked systems.Wewill coverusual admintasks including user accounts, software installation, process and servicemanagement,networkconfiguration,diskand filesystem tasks, systembackup,booting,andsecuritywithSELinux.
MANAGINGUSERSLinuxisamulti-usersystembynatureandoneofthemostbasicadmintasksisusermanagement.TherearethreetypesofusersonLinux,regularusers,adminusers,androot.The root user (userid root) is built-in to Linux and need not be created
manually.The root canperformalloperationsandaccess all fileswithout anyrestrictions. An admin user runs as a regular user but can perform rootoperationsviathesudo(seenextsubsection)orthesucommand.Itisadvisabletoavoidloginasrootandtoperformadmintasksasanadmin
userasmuchaspossible.Thereislittlepracticaldifferencebetweenrootandanadminuser except using sudo and having to enter the admin password. Some
appssuchasgoogle-chrome(Webbrowser)andvlc(videoplayer)won’trunasrootbutwillasadmin.SomeLinuxdistributionssuchasUbuntudisablerootbydefault.Userandothersystemmanagementcommandsarefoundmostlyin/usr/sbin
and/usr/bin.Andtheirusagecanbefoundinsection8oftheLinuxmanpages.Toadd, remove,modifyuser accountsuseuseradd,userdel, andusermod.
To add, remove,modifygroupsusegroupadd,groupdel, andgrouprmod. AGUI tool, usually found in the system settingsmenu, canmake creating newuserseasier(Figure8.1).
Figure8.1UserAccountsTool
Whenauserxyziscreated,usuallyahomedirectory/home/xyziscreatedwithanumberof standard folder in it (Chapter6,Section6.1).Anewgroupxyz isalsocreatedand listed in /etc/group.Anentry isplaced in the /etc/passwd filewhere all users of the system are listed. The entry for user xyz indicates itsnumericalUSERID, andgroup affiliations.Thepassword is indicated by an xandtheactualhashedpasswordislistedin/etc/shadow,afileoff-limitstoallbutroot.You’ll notice that /etc/passwd (/etc/group) also lists many standard users
(standardgroups)forparticularprocesseswhentheyexecute.Ofcourse,theuserandgroupsettingsareusedinDiscretionaryAccessControl(DAC)asdescribedinChapter6.Togivexyzadminstatus,makeitamemberofaspecialsystemadmingroup.
Depending on the Linux distribution the groupmay be admin,wheel (Fedoraand Redhat), staff, sudo (Ubuntu), or sudoers. Another way to make xyz anadministoaddthisentryxyzALL=(ALL)ALL
tothe/etc/sudoersconfigurationfilewiththecommandvisudo.Whenauser logs in, thepasswordgiven is hashed and checked against the
passwordhash in /etc/shadowtoauthenticate theuser. Inaschoolorcorporateenvironment,auserusuallyisabletousethesameloginformultiplecomputerson the organization’s network. Linux uses PAM (Pluggable AuthenticationModules) to satisfy login and authentication needs for many apps includinglogin.Per-appPAMconfigurationcanbefoundin/etc/pam.d/.Usernamesandpasswordsmaybestoredin localfilesoronacentralserver
accessed either directly viaLDAP (LightweightDirectoryAccess Protocol) orindirectly using SSSD (System Security Services Daemon). Commandsluseradd,lgroupadd,andsoonaretobeusedinsuchcases.Not infrequently,usersmayhaveaccess tomultipleLinuxcomputerswithin
thesameorganization.Thisiswhenlogin/fileserversarerequiredtoenableeachusertouseasingleuserid/passwordtoaccessauthorizedcomputers.Inaddition,userhomedirectoriescanbecentrallylocatedonfileserversandaccessedfromindividualLinuxboxesviaNFS(Chapter6,Section6.8).For auser, havingasingle home directory ismuchmoremanageable than a separate one on eachLinuxbox.
SudoLinuxadministrationtaskssuchassettingupnewuseraccounts, installingandupdatingsystem-widesoftware,andmanagingnetworkservicesmustusuallybeperformedbyprivilegeduserssuchasroot.Thisissecurebutnotveryflexible.Sudoisamethodtoallowregularuserstoperformcertaintaskstemporarily
asrootorassomeothermoreprivilegeduser.Thecommandnamesudocomesfromthecommandsu (substituteuser)whichallowsauser tobecomeanotheruser.Putting sudo in front of a complete commandyouwish to execute says:“allow me enough privilege to execute the following command.” If the givencommand is allowed, sudo sets the real and effective uid and gid (Chapter 6,Section 6.4) to those of a specific privileged user for the duration of theexecutionofthegivencommand.Allsudocommandsareloggedforsecurity.Thefile/etc/sudoerscontainsdatagoverningwhocangainwhatprivilegesto
executewhichcommandsonexactlywhathostcomputersandwhethertheuser’spasswordisrequiredornot.Thus,thesamesudoersfilecanbesharedbymanyhosts within the same organization. The file can only be modified via theprivilegedcommandvisudo.Youcanreadaboutsudoersanditssyntaxrulesbyman5sudoersThegeneralformofauserentryinsudoersis
8.2
r_userhosts=(s_user)commandsmeaningr_usercanexecutethegivencommandsass_useronthelistedhosts.
The(s_user)partcanbeomittedifs_userisroot.Herearesomeexamplesudoersentries.
pwanglocalhost=/sbin/shutdown-hnowpwanglocalhost=
/user/bin/systemctlstarthttpd,\/user/bin/systemctlstophttpd,
\/user/bin/systemctlrestarthttpdpwanglocalhost=/usr/bin/dnf
groupinstall"WebServer",\/usr/bin/dnfgroupupgrade"Web
Server"rootALL=(ALL)ALL%wheelALL=(ALL)ALL
Eachentrymustbegivenononeline.Therootentryisalwaystheretogiveroot the ability to sudo all commands on hosts as any user. Thewheel is thesystemadmingroup.Evenifyoulogin(orsu)asroot,youmayprefertousesudosoastoleave
logentriesforthetasksperformed.
PACKAGEMANAGEMENTInstalling, updating, and removing software are of course part of systemadministration.IntheworldofLinux,wedon’thaveappstoresbecausealmostallthepiecesofsoftwarearefreeandnotforsale.Linux software aremade available aspackages in repositories andpackage
management systems are used to manage software packages. A packagemanagement system supports system as well as application software. EvenupdatingtheLinuxkernelandmovingtothenextLinuxreleaseareincluded.For Linux we have two major package systems: the DEB-based Advanced
Packaging Tool (APT) for the Debian family and the RPM-basedYellow dogUpdater,Modified (YUM)for theRedHat family.ThenewerDNF(DandifiedYUM)isintheprocessofreplacingYUM.With a package management tool you can search, install/remove,
update/upgrade and otherwise manage software packages designated for yourversion of Linux stored in on-line repositories. The checking of softwaredependenciesandplacement/replacementoffilesandcommandsareperformedautomatically.MorerecentprojectssuchasGNOMESoftware(Figure4.1)arecreatingapp-
store-like tools to make software management more intuitive. These user-friendly software tools are increasingly being integrated into newer Linuxdistributions.Let’sseehowcommand-linepackagemanagementisdone.
SoftwareManagementTasksOn CentOS/Fedora, the dnf command is used for package management. OnUbuntu/Debian,usetheaptcommandforpackagemanagement.See themanualpages fordnf (man dnf) andapt-get (man apt-get) for full
documentationoftheircommandsandoptions.Toshowhowtheycanbeusedtoperform common softwaremanagement tasks, let’s give some examples.Notethatmostofthesecommandsrequireadminuserstatus.
To search for packages with name or description matching the givenkeywords:dnf search "keywords" (dnf search "media player")apt-cachesearch"keywords"(apt-cachesearch"mediaplayer")Toviewupdates available foryour systemwithout installing them:dnf–refreshcheck-updateapt-get-uupgradeTo install the given (or all) packages on your system: dnf upgrade[packages] (dnf upgrade firefox) apt-get upgrade [packages] (apt-getupgradefirefox)Toinstallnewpackagesalongwithanyrequireddependencies:dnf installpackages (dnf install thunderbird) dnf groupinstall group-name ... (dnfgroupinstall LibreOffice) apt-get install [packages] (apt-get installthunderbird)Tolistallinstalledpackages:dnflistinstalleddnfgrouplistinstalledapt–installedlistTo remove installed packages: dnf remove packages (dnf installthunderbird) dnf group remove group-name ... (dnf group removeLibreOffice)apt-getremove[packages](apt-getremovethunderbird)
DNFkeepsallactions inahistory listandnumbers themsequentially.Use thednf history op to display the history list, to undo/redo, rollback, or list allpackagesaddedtothesystembytheuser(userinstalled).
8.3
Figure8.2GUIforDNF
The command yumex-dnf provides aGUI (Figure 8.2) fordnf and can beeasiertouse.Togetitsimplydodnfinstallyumex-dnfSomeLinuxdistributions,Fedoraforexample,canuseDNFtoautomatically
installpackageswhentheuserissuesacommandthat ismissing,possiblyalsoinstallappsyoufindintheappstore.Theaptitudecommandisaninteractivefrontendforapt-get.
MANAGINGPROCESSESA program under execution is called a process. At any given moment, thereusuallyareagoodnumberofprocessesindifferentstagesofexecution.Becausetheyaremakingprogressatthesametime,weoftenrefertothemasconcurrentprocesses(Chapter11,Section11.9).Forexample,differentusersmaybedoingvariedtaskssuchaseditingfiles,
reading/sending email, surfing the Web, instant messaging, video chatting,listening tomusic etc. In addition to these concurrent user processes there arealsoservicedaemons, systemprocesses running in thebackgroundready todotheirduties.Todisplayandcontrolprocesses,youcanuse theGUI toolgnome-system-
monitor(Figure8.3).Ausercanmanagehis/herownprocessesandadmincanmanage all processes. Often a run-away process or a process that is notrespondingtouserinputcanbeterminatedusingthistool.
Figure8.3GnomeSystemMonitor
Alternatively,thecommandkill-9process_idcanbeused to immediately terminate thegivenprocess (Chapter5, Section
5.22).Tofindtheprocessidyoucanusepidofname_of_appThus,thefollowingcommandworkswellindeed(Chapter3,Section3.7).kill-9‘pidoffirefox’You can use the ps command to display more detailed information on
processes (seeChapter11,Section11.10 formore information).Otherprocessmanagement commands include top (displays processes and their systemresourceusage),pstree(displaysprocessesintreeform),pgrep(findsprocessesbypatternmatching),pkill/killall(terminatesprocessesbyname).In Figure 8.3 you also see several instances of the HTTP daemon (httpd)
whichistheApacheWebserverprogramthatstandsreadytoserveupwebpagesonthishostcomputer.Toseetheavailableservicedaemonprograms,statusofeach,andcontroltheir
execution,usetheGUItoolsystem-config-services(Figure8.4)whereyoucanenable/disable,start/stop,restartanychosenservice.
8.4
Figure8.4ServicesConfigurationTool
InFigure8.4youseethathttpdisenabledandrunning.Alsoweseethenote“httpdismanagedbysystemd.ThisisbecauseLinuxusesthesystemdaemonsystemdasacontrollerofallsystemprocesses.ThesystemdhasPID1andallotherprocessesareitsdescendants.Therefore,asystemadmincan,alternatively,usethefollowingcommandtoenable/disable,start/stop/restartagivenservice:systemctlopserviceForexample,systemctlstarthttpdsystemctlstophttpdsystemctlrestarthttpdsystemctlstatushttpdNotethatenablearrangestohaveaserviceautomaticallystartedonbootwhile
disableistheopposite.Thestart/stopoperationstarts/terminatesagivenserviceimmediately,enabledornot.Thestatusoperationdisplaysthecurrentstatusofagivenservice.
NETWORKCONFIGURATIONNetworking is an important function of the Linux kernel which performsoperationsthroughNetworkInterfaceCards(NICs).Forexample,in-househostsmay be connected to a router on an Ethernet LAN. The router in turn isconnected to the Internet through a cable or an ISDNmodemprovided by anInternetServiceProvider (ISP).ThesamehostmayalsohaveadditionalNICswiredorwireless.Thesamehostconnectedtotwodifferentnetworkscanserveasagatewaythatrelaystrafficbetweenthetwonetworks.Thus,therouterinourexampleisagatewaybetweenthein-houseLANandtheISP’snetwork.
Most likely, your home Linux workstation obtains its IP address via theDynamic Host Configuration Protocol (DHCP) from a DHCP server on yourLAN. On a home LAN, a wireless router is also the DHCP server. Whilebooting,aDHCPclientbroadcastsarequestwithitsownMediaAccessControl(MAC)addresstoobtainanIPaddressandasubnetmaskfromtheDHCPserverlocatedonthesamebroadcastsubnet.TheDHCPserverassignsanavailableorpreassigned IP address and usually also provides default gateway and DNSserver(Chapter7,Section7.3)addresses.Forahost,therearefourimportantpiecesofnetworkconfigurationdata:
ItsIPaddress—Forexample,192.168.1.42Itssubnetmask—Forexample,255.255.255.0meaningall192.168.1.xareonitssubnetIts default gateway—For example, 192.168.1.1 identifying the routerconnectingthesubnettotheoutsideIPaddressesofDNSservers
Note that the subnet mask is a bit mask whose leading sequence of 1 bitsindicates the fixedpartof subnet IPaddresses.Theumask for filepermissions(Chapter3,Section3.12)isanotherbitmaskwehaveseen.TheInternettransmitsdatainpackets.HencetheLinuxkernelmustdealwith
IP packets when performing networking. Figure 8.5 shows the structure of apacket.
Figure8.5AnIPv4Packet
Ifthepacketdestinationisonthehost’ssubnet,thekernelcansendthepacketdirectly.Ifnot,thepacketissenttothedefaultroutewhichisusuallyahostatyourISPoralocalgatewayconnectedtotheInternet.OnaLinuxsystem,networkinterfaceshavestandardnames,forexample:
lo—Theloopbackinterface,connectedtothehostitself,mostlyfortesting
anddiagnosiseth0oreno1—ThefirstEthernetinterfacewlan0—ThefirstWirelessnetworkinterface.ppp0—ThefirstPointtoPointProtocolnetworkinterfacevirbr0—Avirtualbridge interfaceusedbyvirtualmachines 1 foraddresstranslationsinordertoconnecttotheoutsidenetwork.
On most Linux distributions, the NetworkManager daemon initializes andconfiguresavailablenetworksautomaticallyandusuallynoadmin interventionisneeded.Network,aGUItooltoconvenientlycontrolnetworkingcanoftenbefoundon the system settingspanel.Figure8.6 shows the entry display of thisnetworkcontroltool.
Figure8.6NetworkControlTool
TheWiredinterfaceshowsthatthehostisona1000Mb/sLANwithalocalIPaddress192.168.1.42, itsMACaddress,DNSserverstouse,andthedefaultroute 192.158.1.1, likely a router connected to the Internet. Any wirelessinterfacesarealsolisted,enablingyoutoturnthemon/off.Clickonthegearicontoedittheconfigurationforaselectedinterface.Forexample,youcanaddDNSservers, pick firewall zones (Section 8.5), and set the security mode andpasswordforawirelessinterface.TheGUItoolnm-connection-editorandtheCLIcommandnmclialsoenable
youtoeditnetworkinterfacesettingsfortheNetworkManager.The file /etc/resolv.conf,generatedbyNetworkManager, storesDNSservers
tobeusedbyyoursystem.The/etc/hostsfilecanstoreIPaddressesfordomainnamesandanyaliasesforlocalandoften-usedremotehoststohelpdomaintoIPmapping.Editthehostsfilewithanystandardtexteditor.Otherusefulcommandsfornetworkadmininclude
pinghost—testsifhostisresponding
8.5
dighost—looksuphostviaDNSwhoisdomain—looksupregistrationinformationfordomainifconfig—checksandmodifiesnetworkinterfacesettingsroutehost—displaysandmanipulatestheIProutingtabletraceroutehost—displaysnetworkroutetohost
FIREWALLSETTINGSA network firewall provides a line of defense against outside intrusion. Afirewallcanbeaspecialcomputeronthenetworktoprotectallinternalhosts.Ahome or small business router is such an example. A firewall can also be aprogram running on a host to protect that host. An operating system usuallycomeswithafirewallprogramandLinuxisnoexception.Evenbehindarouterfirewall, a host will usually run its own firewall program for host-specificprotection.Ingeneral,afirewallprotectsbylimitingnetworkingtoandfromtheoutside
byfollowingIPpacketfilteringrulesthatspecifypreciselywhatnetworktrafficisallowed inorout.Forexample, firewall rulescandisallowaccess tocertainservicesand/orportson thehost computeror subnethosts.Or theycan forbidnetworkingfromblacklistedhosts.Modern Linux systems come with a firewall daemon firewalld that allows
systemadmintomodifyfirewallsettingson-the-fly,withouthavingtorestartthefirewalld process or reboot the computer. Such a dynamic firewall makes itmuch easier for system admin than a static firewall based on iptables rules.Newer Linux distributions come with firewalld but not iptables. It isrecommendedthatyouusefirewalldinsteadofiptables.The GUI tool firewall-config (Figure 8.7) allows you to adjust firewall
settingsusedbyfirewalld.
8.6
Figure8.7FirewallConfigurationTool
The firewalldaemonhasanumberofpredefinedprotectionzones that offerdifferentlevelofprotectionfornetworkconnections.Figure8.7showstheeno1interface for pcenvy having the FedoraWorkstation as its default zone. Thefirewall-config tool allows you to modify the firewall runtime/permanentsettings:
ToassignconnectionsandinterfacestodesiredzonesTomodifysettingsandadd/deleterulesforeachzoneTocreatenewzonesofyourownTomakeruntimesettingspermanent
Apredefinedfirewallzone isagoodstartingpoint.Additionalpermissionscanbe added when needed and can be dropped when no longer necessary. Forexample, ifweneed the localWeb server tobecomeavailable,we can enablehttpandhttps services.To support incomingssh/sftp connections,wecanaddthesshservice.Tomake changes to the firewall from the command line use firewall-cmd.
Forexample,toenableHTTPSfortheWebserverwecanfirewall-cmd–add-service=https–permanentfirewall-cmd–reload
MANAGINGFILESYSTEMSANDDISKS
DiskPartitionsThe storage on a block device is usually divided into a number of separateregionsknownaspartitions,eachpartitioncanbeformattedandmanagedbytheoperatingsystem.Thus,wecanthinkofapartitionasa“logicaldisk.”OnLinuxeach partition is represented by its own special file. For example, /dev/sda1pointstopartition1onthediskdevice/dev/sda.Diskpartitionsaredefinedbyapartitiontable,storedatthebeginningofthe
storagedevice,thatliststhelocation,size,andotherattributesofeachpartition.The partition table followswell-defined formats—the oldMBR (Master BootRecord)orthenewGPT(GUIDpartitiontable).TheMBRisstoredattheverybeginning of amass storage device (sector 0 length=512 byte) and contains asimple boot loader and the partition table. MBR works with BIOS (BasicInput/OutputSystem)firmwareinterfaceatsystemboottime.GPT + UEFI (Unified Extensible Firmware Interface) is the new standard
replacingMBR+BIOS.GPTbetter supports larger storagedevices andUEFIprovides a special EFI System Partition (ESP) which contains programscompiledfor theEFIarchitecture.Suchfirmwareprogramscanbebootloadersfor different operating systems. See Section 8.8 for a discussion on systembooting.Youcanlistyourdiskpartitionswiththecommandlsblkwhichdisplaysblock
devices.Figure8.8showsasampledisplay.
Figure8.8lsblkDisplay
PartitionscanbemanagedwiththeCLItoolsfdisk(fordriveslessthan2TB),gdisk(forGPT),andparted(forMS-DOS,GPTandotherformats).TheGUItoolgparted(Figure8.9)makesitsimpletocreateandmodifydisk
partitions.
Figure8.9GpartedTool
Agoodandsimplewayofpartitioningaharddriveis:
Firstpartition—1GB,EFI(FAT)mountedat/boot/efiforbootingLinuxandperhapsalsootheroperatingsystems.Second partition—4GB,Linux filesystem (ext4)mounted at /boot for theLinuxkernelandrelatedfiles.Third(andlast)partition—Allremainingdiskspace,LinuxLVMphysicalvolumetobemanagedbyLVMforswap(36GB),/,/home,andsoon.
Alsousefulfordiskmanagementisthecommandgnome-disks(Figure8.10).
Figure8.10TheGnomeDisksTool
ManagingFilesystemsExisting filesystems that you can mount/umount are stored in the systemfilesystemtable/etc/fstab,firstcreatedwhentheLinuxdistributionisinstalled.
Hereisasample:
/dev/mapper/fedora-root/ext4defaults11UUID=91035a32-9e5a-4001-
851f-28d71244792d/bootext4defaults12UUID=62C3-23C9/boot/efi
vfatumask=0077,shortname=winnt02/dev/mapper/fedora-home/home
ext4defaults12/dev/mapper/fedora-swapswapswapdefaults00
Eachlinedescribesadifferentfilesystemwith6fieldsseparatedbyspaces—devicename,mountpoint,filesystemtype,mountoptions,andsoon.Pleaseseethemanpage for fstab fordetailsofeach field.Filesystems listed in /etc/fstabareautomaticallymountedatsystemboottime.Anadminusercanuseanyplaintexteditortoadd/deletefilesystemslistedin/etc/fstab.Thecommandfsckfile-sys...checks and optionally repairs the given filesystems.Without arguments, the
commandprocesseseachfilesystemlistedin/etc/fstabsequentially.
AddingaNewDiskOften,acomputerrunningLinuxhasonlyoneharddrive.Addingasecondharddrive can provide more storage as well as improve system performance.Furthermore,criticalfilescanbebackedupontheseconddrivejustincasethefirstdrivesomehowfails.Toaddanewharddrive,followthesegeneralsteps:
1. Physicallyinstalltheharddriveonyourcomputer.2. Createdesiredpartitionsonthenewdisk.3. Createfilesystemsforthepartitions.4. Mountthefilesystems.
Step 1 involves system shutdown, unplugging the computer, physicallyconnectingthenewharddrive,andreboot.Nowissueacommandsuchaslsblkorparted-ltoidentifythedevicespecialfileforthenewdisk.Ifthefirstdiskis/dev/sda,
thenewdiskisusually/dev/sdb.Withthenewdiskidentified,wecanperformstep2andpartitionthediskby
following information in Section 8.6.Use either theGUI gparted tool or theCLIparted tool.Eachpartitionbecomesalogicaldiskandhasitsownspecialdevicenamesuchas/dev/sdb1/dev/sdb2,andsoon.Tokeepthingssimple,itisnotabadideatohaveonebigpartitionfortheentirenewdisk.
Forstep3,wecanusethecommandmkfs-ttypedevice_namewhich creates a filesystem of the specific type for the partition given by
device_name.Type-specific filesystemcreationcommands includemke2fs (forext2/ext3/ext4 filesystem),mkswap,mkfs.fat, and others.After being createdthefilesystemisalsoidentifiedbydevice_name.Finallyinstep4,usethemountcommandtomounteachnewfilesystemtoa
desireddirectoryonthefiletree.Andenterthenewfilesystemsinto/etc/fstab.Alternatively, we can put the entire new disk under Logical Volume
Management.
LVMModernLinuxdistributionssupportLVM(LogicalVolumeManagement)whichusestheLinuxkernel’sdevicemappingabilitytodynamicallyassociatelogicalvolumes (LV) of storage to physical volumes (PV) on storage devices. Thus,LVMisasoftwarelayeroverdiskpartitionstoformlogicalvolumesthatmakemanagingharddrivereplacement,repartitioningandbackupmucheasier.UnderLVM,anumberofdiskpartitionsaredesignatedasPVs.ThePVsare
dividedintodisjointvolumegroups (VG).EachVGfunctionsasa logicaldiskandcanbe“partitioned”intooneormoreLVs.EachLVwillhaveitsownmountpointonthefiletree.Figure8.11showstheLVMarchitecture.
Figure8.11LVMArchitecture
Thecommandpvcreatedevice_nameinitializesadiskorpartitionasaPVtobeusedunderLVM.Weusedevice
namessuchas/dev/sda5asthenameofPVs.ThecommandpvdisplaydisplaysallPVsandtheirinformation.Tocreateavolumegroup(VG)usethe
command
vgcreatevolume_group_namepv1pv2...ToseeallVGsdovgdisplayToaddnewphysicalvolumestoavolumegroupusevgextendvg_namepv5pv6...TocreateanLV(logicalvolume)inaVGdolvcreate-ln-nlv_namevg_nameThe-loptiongivesthenumberofextents(eachextentis4MBbydefault).A
logicalvolumecanberesizedlaterwhenavailablespacechangesinitsvolumegroup (lvextend, lvreduce, lvresize). To see all LVs do lvdisplay. Here is asampledisplay.
---Logicalvolume---LVPath/dev/fedora/swapLVNameswapVGName
fedoraLVUUIDgJNdsi-osp2-1046-isc2-NXnT-nTEO-3V6awhLVWriteAccess
read/writeLVCreationhost,timepcenvy.localdomain,2017-03-21
11:23:04-0400LVStatusavailable#open2LVSize35.36GiBCurrent
LE9053Segments1AllocationinheritReadaheadsectorsauto-
currentlysetto256Blockdevice253:1---Logicalvolume---LVPath
/dev/fedora/homeLVNamehomeVGNamefedoraLVUUIDKJTOjw-TJnp-78oL-
xYF9-mChg-2yVx-8M4yonLVWriteAccessread/writeLVCreationhost,
timepcenvy.localdomain,2017-03-2111:23:04-0400LVStatus
available#open1LVSize266.00GiBCurrentLE68096Segments
1AllocationinheritReadaheadsectorsauto-currentlysetto
256Blockdevice253:2---Logicalvolume---LVPath
/dev/fedora/rootLVNamerootVGNamefedoraLVUUIDrjaDlE-wFPk-qbWo-
EbB6-XZwR-d7Ws-U5mUkILVWriteAccessread/writeLVCreationhost,
timepcenvy.localdomain,2017-03-2111:23:31-0400LVStatus
available#open1LVSize1.52TiBCurrentLE398458Segments
1AllocationinheritReadaheadsectorsauto-currentlysetto
256Blockdevice253:0
When installingLinuxonadesktopwith a singleharddrive,partition sizescanbeestimated roughlybecauseLVMallows filesystems tobe resizedeasilylaterbasedonsystemneedsandperformance.Onlargersystems,LVMmakesaddingandreplacingdisksmucheasier.Also,
LVM enables consistent backups by taking snapshots of the logical volumes.PleaseseeresourcesatthebookwebsiteformoreinformationandcommandsforLVM.
FileStorageQuotasThefilequotamechanismisdesignedtoallowrestrictionsondiskspaceusagefor individual users and/or groups. A separate quota can be set for eachuser/grouponeachfilesystem.Quotascanbeenforcedonsomefilesystemsand
8.7
notonothers.Forexample,inacomputersciencedepartment,onefilesystemforstudents may have quota enforced; at the same time, another filesystem forprofessors may have no quota enforced. The quota specifies limits on thenumber of files and disk blocks a user may occupy. There are two kinds oflimits: soft limits andhard limits. If a user-initiated operation causes the softlimit to be exceeded, awarning appears on the user’s terminal.The offendingoperationisallowedtocontinueifthehardlimitisnotexceeded.Theideaistoencourageuserstostaywithintheirsoftlimitsbetweenloginsessions.Inotherwords, exceeding the soft limit temporarily is all right, as long as the userreleasesfilespaceandreturnswithinthesoftlimitbeforelogout.Atlogintime,awarning is provided if any soft limits still are violated. After a few suchwarnings,theuser’ssoftlimitscanbeenforcedashardlimits.System admin can edit the filesystem table (/etc/fstab) to indicate which
filesystemsneedtosupportquotasanduse,forexample,quotacheck-cu/homequotacheck-cg/hometo enable, respectively, user and group quota enforcement for the /home
filesystem.The quotas for users and groups are kept in files (aquota.user and
aquota.group,forexample)locatedintherootdirectoryofthefilesystem.Foramountedfilesystem,itsrootdirectoryisitsmountpointonthefiletree.Thecommandedquota isused to set andchangequotas.Onlya superuser
can invoke edquota. The commandquota displays your disk usage and yourquotalimits.Asuperusercangivethiscommandanoptionaluseridtodisplaythe informationofa specificuser.Asuperuseralsocan turnonandoffquotaenforcingforentirefilesystemsusingthecommandsquotaonfilesys…quotaofffilesys…
FILEANDSYSTEMBACKUPForsafety,itisofcourseimportanttoregularlybackupkeyfilesonexternalharddrives,onanothercomputer,oronthecloud.The tar command (Chapter 6, Section 6.11) provides a convenient way to
gather files into one single archive file for backup. For example, to back up/home,thecommandtar-Jcpvf/backup_home.txz–one-file-system/home
creates an XZ-compressed file for /home. The option –one-file-systemexcludes files on a different filesystem. The compressed tar file can then betransferredtoanexternaldriveorsystem.Having backed up user home folders, you can backup the root folder (/) to
achieveafullersystembackupthisway(Ex:ex08/twopartBackup):
tar-Jcpvfbackup_root.txz--one-file-system--
exclude=/backup_root.tbz--exclude=/dev--exclude=/home--
exclude=/media--exclude=/mnt--exclude=/proc--exclude=/run--
exclude=/srv--exclude=/sys--exclude=/tmp/
IfyouhavenetworkaccesstoanotherLinuxsystemor,betteryet,adedicatedbackup server, then thersync command (Chapter7,Section7.6) can bemoreeffectiveandeasierfordoingbackups.Saypcmonkey isyourLinuxbackuphost andyouwish tobackuppctiger’s
Web data stored in /var/www. Assume you have userid ableAdmin on bothsystemsandyouhavearrangedno-passwordSSHfromonesystemtotheother.Then,youcanuse(Ex:ex08/rsyncBackup)rsync-Capz–rsh="ssh-lableAdmin"–hard-links–inplace/var/wwwpcmonkey:/backup/pctiger/www/The backup files will be located on pcmonkey in the folder
/backup/pctiger/www. The -C option conveniently excludes files normally notneeded for a backup. After the first backup, when the same command is runagainlater,rsyncwillonlytransferanyfilesthathavechangedsincelasttime.Notersynchas–includeand–excludeoptionstoaddandremovefiles/folders
from processing. The –delete option causes any file/folder not present in thesource to be deleted at the destination. See the manpage for rsync for manyotheroptions.To perform regularly scheduled automatic backups, place the preceding
commandasacrontabentrysuchas(Ex:ex08/rsyncCrontab):361***(followedbythedesiredcommandononeline)EachLinuxusercanusethecommandcrontab-etoeditthatuser’scrontable
ofscheduledcommands.The five leading fieldsofeachcrontabentry indicatetheschedule.Seesections1and5ofthemanpagesforcrontabfordetails.Thepreceding example specifies 01:36AM every day. The crond (cron daemon)examinescrontabentrieseveryminutetoexecutescheduledcommands.
BackupwithDéjàDupDéjàDup(day-ja-doop)fromGnomeisaGUIbackuptoolthatiseasytouseyetvery powerful. As a frontend toDuplicity, it features scheduling, encryption,
incrementalbackups,andsupportforcloudstorage.IfnotalreadyinyourLinuxdistribution,installitwithdnfinstalldeja-dupapt-getinstalldeja-dupYoucanfindtheDéjàDupicon(asafebox)intheApplications,Utilities,or
systemmenu.
Figure8.12DéjàDupBackupTool
Tousethistool,rundeja-dup-preferences(Figure8.12),turniton(enablingthetool),listallthedesiredfolderstobackup,anysubfolderstoignore,andthelocation to store the backup files. Then, click Back Up Now. You can alsoscheduleautomaticregularbackups.As backup storage, youmay use a local folder, a remote folder via FTP or
SSH,anexternalharddrive,oracloudservice(Figure8.13).
Figure8.13DéjàDupStorageChoices
If you have DropBox installed (Chapter 4, Section 4.1) you can use
8.8
$HOME/Dropbox as storage. But be careful, unless you have changed thedefault file sync settings for your DropBox account, any files deleted in$HOME/Dropboxwillbeautomaticallydeletedonthecloudaswell.Déjà Dup provides for scheduling of automatic backups. And you can run
deja-dup–backupmanuallyatanytime.Torestoreoneormorefilesusedeja-dup–restorefile1...Withnoarguments,acompleterestoreisdone.Torestoreamissingfolderdodeja-dup–restore-missingdirectory
SYSTEMBOOTINGUnderthecontrolofanoperatingsytem,acomputercanexecuteanyprograminitsRAM.Butwhenacomputerisfirstpoweredon,thereisnooperatingsystemyetinmemory.Thecomputer’sfirmwaremustfirstcausetheloadingofabootloader program whose job is to load in stages increasingly more capablesoftwareleadingtotheoperatingsystemkernel.ModernLinuxsystemsusetheUnifiedExtensibleFirmwareInterface(UEFI),
insteadofBIOS2,forthebootprocess.UEFIfeaturesitsownCPU-independentarchitecture,devicedrivers,andthe
abilitytomountpartitionsandreadcertainfilesystems.UEFIcanbeconsideredatinyOSrunningonacomputer’sfirmware.Whenacomputer ispoweredon, the firmware firstperforms thePowerOn
SelfTest(POST)thenrunstheUEFIcode.UEFIsearchesthesystemstorageforanEFISystemPartition(ESP)whichisaGPTpartitionlabeledwithaspecificGloballyUniqueIDentifier(GUID).TheusuallocationfortheESPis/boot/efi.The ESP contains applications compiled for the EFI architecture including
bootloadersandotherutilities.TheEFIcomeswithabootmanagerthatcanbootthe system from a default configuration or prompt the user to choose anoperatingsystemtoboot.WhenanOS-specificbootloaderisselected,manuallyorautomatically, it isreadintomemoryandtakesover thebootprocess.UEFIalso supportsSecureBoot that checks the signed software (Chapter 7,Section7.12) used in the booting process including the operating system so nothingunauthenticatedisusedorloaded.GRUB2isthelatestversionofGNUGRUB,theGRandUnifiedBootloader
which can be used for BIOS and for UEFI systems. On UEFI Linuxdistributions,theGRUB2bootloaderisusuallyfoundin/boot/efi/EFI.GRUB2canloadLinuxaswellasotheroperatingsystemssuchasWindows10.Atboot time,quicklypress akey suchasESCorSHIFT to enter theGRUB2
8.9
menuandchoosehowtoproceed.ModernLinuxdistributionsmap the legacy runlevels (0–6) to five systemd
runlevel targets, as part of many kinds of systemd targets. Each such targetrepresentsasoftwareunitinanorderedsequenceofunitsinthebootprocess.
poweroff.targetorrunlevel0.target—Systemshutdownrescue.targetorrunlevel1.target—Single-UserMode,nonetworkinterfaces,nodaemons,onlyrootloginallowedmulti-user.targetorrunlevel[2,3,4].target—Multi-UserMode,withnetworkanddaemons,noGUIgraphical.targetorrunlevel5.target—Multi-UserModeplusGUIreboot.targetorrunlevel6.target—Systemreboot
Themulti-user.target is normal for a Linux server box and graphical.target isnormalforaworkstation.Thecommandwho-rshows the runlevel.Tochange the runlevel (systemadminonly), the legacy
commandinitnumberstillworks.Butcommandssuchassystemctlisolatemulti-user.targetsystemctlisolategraphical.targetshouldbeusedinstead.Toseeorsetthedefaulttargetforthenextrebootusesystemctlget-defaultsystemctlset-defaulttarget_nameABASHscript(Ex:ex08/bootinfoscript)canbeobtainedandusedtodisplay
comprehensive boot-related information about disks, partitions, boot loaders,devices,filesystems,andsoon.
SELINUXInChapter6wetalkedaboutDiscretionaryAccessControl(DAC)(Section6.4).SELinux (Security Enhanced Linux) strengthens system security by providingMandatory Access Control (MAC) whose rules are applied after DAC.AccordingtoaRedHatdocument
“SELinux isan implementationofamandatoryaccesscontrolmechanism
in the Linux kernel, checking for allowed operations after standarddiscretionary access controls are checked. SELinux can enforce rules onfiles and processes in a Linux system, and on their actions, based ondefinedpolicies."
ThissectionprovidesabriefoverviewandintroductiontoSELinuxwhichisalargetopicrequiringentirebooksforafulltreatment.StandardLinuxusesDACthatcontrolsaccessbasedonuserid,groupid,and
rwx permissions set at users’ discretion.The approach has fundamental flaws.DAChasnowaytoaccountforfine-grainedusersecuritylevels,rolesusersandprocessesmayplay,andsensitivityclassificationsofdata.WithSELinux,built-insecuritypolicies,enforcedbytheLinuxkernel,govern
howallsubjects interactwith allobjects in the system.A subject is a runningprocess.Anobjectisafile,folder,device,port,socket,orprocess.Inlate2000,theUnitedStatesNationalSecurityAgency(NSA)releasedthe
firstversionofSELinux to theopen source softwaredevelopmentcommunity.With contributions frommany sources, includingRedHat andLinusTorvaldswhosuggestedamodularapproachforintroducingsecurityenforcementintotheLinuxkernel,SELinuxwas integrated into theLinuxSecurityModules (LSM)framework. The kernel can load different LSMs to implement well-definedsecurityschemes.NowSELinuxcanbeinstalledinmostmodernLinuxdistributionsincluding
Red Hat, Fedora, Ubuntu, Debian, and Hardened Gentoo. Some distributionsmayhaveSELinux turnedoff initially. In fact,Ubuntu,MintandotherspreferAppArmoroverSELinux.OurdiscussionherewillbemostlybasedonRedHat,CentOS,andFedora.
SELinuxStatusandEnforcingModesTocheckthestatusofSELinux,issuethecommandsestatusHereisasampledisplay
SELinuxstatus:enabledSELinuxfsmount:/sys/fs/selinuxSELinuxroot
directory:/etc/selinuxLoadedpolicyname:targetedCurrentmode:
enforcingModefromconfigfile:enforcingPolicyMLSstatus:
enabledPolicydeny_unknownstatus:allowedMaxkernelpolicy
version:31
SELinuxcanrunineitherofthesetwomodes
Enforcing—Allows/deniesaccessbyenforcingpolicyrules.
Permissive—Doesnot deny access but displays and logs anyviolationofpolicyrulesinstead.
Usesetenforce1 (or0) toset thecurrentmode toenforcing(permissive).Usegetenforcetodisplaythecurrentmode.Thecurrentmodepersistsuntilthenextreboot when it will be set according to the SELinux configuration file/etc/sysconfig/selinux.Hereisanexamplewherethetargetedpolicyisenforced.
#ThisfilecontrolsthestateofSELinuxonthesystem.#SELINUX=
cantakeoneofthesethreevalues:#enforcing-SELinuxsecurity
policyisenforced.#permissive-SELinuxprintswarningsinstead
ofenforcing.#disabled-NoSELinuxpolicyis
loaded.SELINUX=enforcing#SELINUXTYPE=cantakeoneofthesethree
values:#targeted-Targetedprocessesareprotected,#minimum-
Modifiedtargetedpolicy.Onlyselectedprocessesareprotected.#
mls-MultiLevelSecurityprotection.SELINUXTYPE=targeted
EditthisfiletomodifyhowSELinuxisappliedacrossreboots.Forexample,SELINUX=disableddisablesenforcementofSELinuxrules(notrecommended).Toswitchfromdisabled toenforcing(orenforcingfor thefirst time)set the
modetopermissiveintheconfigfilefirstandreboot.Seeingnowarningsafterreboot,changetheconfigurationfrompermissivetoenforcingandrebootagain.
SecurityContextsWith SELinux, each process/file is labeledwith its own security context. Thekernel uses context information and policy rules to make access controldecisions.Thesecuritycontextofaprocessisalsoknownasthedomainoftheprocess. Policy rules also govern whether a process can transition from onedomaintoanother.SELinuxallowsaccessonlyif there isaspecificpolicyrulepermittingsuch
access.Otherwiseaccessisalwaysdenied.A SELinux security context is an ordered list of security identities in the
formatse_user:role:type[:sensitivity:category]Thesensitivitylevelandcategoryrangepartsareoptional.Usually,theTargetedSELinuxPolicyisthedefaultandyoucanfinditstored
in /etc/selinux/targeted. The targeted policy focuses on type enforcementwithsomeattentionpaidtouserandroleidentities.Thesensitivityandcategoryarerarelyused.These_useridentityisusedtocollectusersintosecuritygroups.Forexample,
userswiththeguest_uidentityusuallywon’tbeabletorunexecutableslocatedin their own home directory or /tmp, or to run setuid programs. Use thecommandseinfo–usertoproducealistofallpossibleSELinuxuseridentities,suchas
sysadm_usystem_uxguest_urootguest_ustaff_uuser_uunconfined_u
Therole identitydifferentiatesbetweenfiles,executables,runningprocesses,and daemons. Only processes with certain roles are allowed to take certainactionsor transitiontocertaindomains.UndertheSELinuxtargetedpolicy,bydefault,usersareunconfined.Thus,thedomainofauser’sShellps-eZ|grepbashisnormallyunconfined_u:unconfined_r:unconfined_t:s0...bashThe type identities form thebulkofaccesscontrolunder the targetedpolicy
withalargenumberoftypesandrulesthatspecifywhichtypecanaccesswhichtype.Take the ApacheWeb server (Chapter 9) daemonhttpd, for example. The
commandls-Z/usr/sbin/httpdproducesthecontextforthatexecutablefilesystem_u:object_r:httpd_exec_t:s0/usr/sbin/httpdAndps-eZ|grephttpddisplaysthedomainforthedaemonprocesssystem_u:system_r:httpd_t:s0...httpdThus, a process able to execute /usr/sbin/httpd creates a child process
(Chapter3,Section3.2)thattransitionstotheprecedingdomain.ForafileintheWebserver’sdocumentspace(readablebyhttpd),ls-Zindex.htmldisplaysunconfined_u:object_r:httpd_sys_content_t:s0index.htmlForafolderorfilewritablebyscriptsrunbyhttpd,thecontextisunconfined_u:object_r:httpd_sys_rw_content_t:s0
MaintainingandManagingFileContextsOnSELinuxsystemsusingthetargetedpolicy,regularusersusuallywon’thavemuchinteractionwithSELinuxenforcementbecausethey,alongwiththerestofthe system, are unconfined_t. Only daemons are targeted and confined. This
usuallymeans,thestandardDACrules.However,therearefourcommonareasaregularuserneedstopayattentionon
SELinuxsystems:copying,moving,rsyncing,andarchiving/restoringfiles.Thegoal is todo thesewhilepreserving thecorrectSELinuxcontexts for filesanddirectories.Herearesometips.
Themv command, by default, preserves the file’s original contextwhichmaybe incorrect for itsnew location.Usemv -Z toautomatically set thecontextcorrectlyforthedestinationlocation.Whencreatinganewfile/folder,includingwiththecpcommand,itgetsacontextinheritedfromitsparentdirectoryautomatically.For the tar command, use the –selinux (–no-selinux) option to include(exclude)SELinuxcontextinformationinthearchive.For the rsync command, add the –xattrs option to preserve extendedattributesthatincludeSELinuxcontexts.
Sometimes a file/folder needs its context set correctly via the chcon (changecontext)command.Forexample,ifnecessary,ausermayuse(Ex:ex08/chcon)chcon-R-thttpd_user_content_t$HOME/public_htmltosetcorrectcontexts for theuser’spersonalWebdocumentspace(Chapter
9).SELinux uses auditd to logmessages in /var/log/audit/ to aid auditing and
troubleshooting. These are mostly AVCs (Access Vector Cache). They showoperationsdenied (or allowed)by theSELinux security server.AnadminusercancheckSELinuxwithausearch-mavctodisplayalldenials,oraddtheoption-tsrecent(-tstoday)fordenialsforthe
last10minutes(today).Admincanalsousethecommandsealert-btolaunchaGUIalertbrowser(Figure8.14).
Figure8.14SELinuxAlertBrowser
8.10
8.11
Toseethecorrectcontextsettings,issuethecommandsestatus-vtodisplaycorrectcontextsforfilesandprocesseslistedin/etc/sestatus.conf.Use the command restorecon or fixfiles to restore/fix contexts. Use the
commandsecon to display the context of a file or process.For example, (Ex:ex08/secon),secon-f/usr/sbin/httpdproduces
user:system_urole:object_rtype:httpd_exec_tsensitivity:
s0clearance:s0mls-range:s0
andthecommandsecon–pid‘pidofhttpd‘producessomethinglike
user:system_urole:system_rtype:httpd_tsensitivity:
s0clearance:s0mls-range:s0
Many other tools for SELinux exist for troubleshooting,managing policies,writingnewones,andsoon.Seethebookwebsiteforadditionalresources.
FORMOREINFORMATIONSeetheDNFpageatfedoraproject.org.SeeFirewallddocumentationatfedoraproject.org.SeetheAPTcompleteguideatitsfoss.com.SeeLVMHowToattheLinuxDocumentationProject.SeeSELinuxresourcesatthebookwebsite.SeeUEFIatuefi.org.
SUMMARYLinuxsystemadministrationandmanagementisapromisingcareerdirectionforwell-trainedLinuxpersonnel.BasicaspectsofLinuxsystemadmin,especiallyfor home and small LAN situations, include: user accounts, softwaremanagement, network configuration, disk and filesystem management,file/system backup, booting, and system security. SELinux adds MandatoryAccess Control (MAC) based on security policies to the standard LinuxDiscretionary Access Control (DAC) based on userid and file permissions.
8.12
UnderstandingandmanagingSELinuxcontextsareimportantforendusersandsystemadminsalike.
EXERCISES1. Where are userids and passwords for Linux users kept? How is the
passwordcheckedwhenauserlogin?2. WhatisLDAP?HowisitrelatedtouserauthenticationonLinuxsystems?3. Compareprosandconsforfilebackupwithtarandwithrsync.4. Explainthepurposeofsudoandwhyweshoulduseit insteadofloginas
root.5. Howcan tasksbescheduledforexecutiononaregularbasis?Pleasegive
examples.6. How to find the IP address assigned to your Linux system? What is a
defaultroute?7. ExplainthestructureofanIPpacket.8. WhatisDHCPreservation?Howisitconfiguredonarouter?9. HowdoesonefindtheIPv6addressofahostontheInternet?Pleasegivea
specificexample.10. How to find which daemon processes are running? What tools and
commandsareusefulformanagingdaemonprocesses?11. Whatisasubnet?FindthesubnetyourLinuxsystemisin.12. Whatisasubnetmask?13. Whatisahardwarefirewall?softwarefirewall?Wherearetheylocated?14. Whatisadiskpartition?AGPTpartition?15. WhatisBIOS?UEFI?EFI?Pleaseexplain.16. WhatisaGUID?WhatisanESP?Pleaseexplain.17. Listandexplainthetasksneededtoinstallanewharddrive.18. WhatisLVM?Whatbenefitsdoesitbring?19. UnderLVM,whatisaphysicalvolume?volumegroup?logicalvolume?20. FindoutwhatRAIDisandhowyourLinuxdistributionsupportsRAID.21. WhatisaSELinuxcontext?22. WhatisdomaintransitioninSELiux?23. Whereisthecontextforafilestored?24. ExplainwhyusingthemvcommandcancauseproblemsunderSELinux.25. FindoutaboutAppArmorandcompareitwithSELinux.26. ForstudentsinterestedinLinuxservermanagement,pleasefindoutabout
Ansible and other similar software packages that automate software
12
provisioning,configurationmanagement,andapplicationdeployment.Avirtualmachineisanotheroperatingsystemrunningunderthecontrolofahostoperatingsystem.MostUEFIimplementationsarebackwardscompatiblewithBIOS.
9.1
Chapter9
WebHosting:Apache,MySQL,andPHP
Startedintheearly1990sasafilesharingsystemamongphysicists,theWorldWide Web (WWW or simply Web) has grown rapidly to a globe-spanninginformation system that modern societies won’t do without even for a shortwhile. In a real sense, theWeb has leveled the playing field and empoweredindividualsandbusinesses,largeorsmall,allovertheworld.Akeyfactorforthisgreatsuccessisthelowcostofputtinginformationonthe
Web. You simply find a Web hosting service to position your files andprogrammingforyourwebsiteontheWeb.AnyInternethostcanprovideWebhosting if it has a good Internet connection and runs aWeb server and otherrelatedprograms.Accordingtonetcraft.com’sJune2017survey,amongallWebservers,afull
46% areApache, and amajority ofApache servers run on Linux systems.ALinux-ApacheWeb hosting environment usually also supports PHP for activepages and MySQL (the community version) or MariaDB for database-drivenwebsites.TheLinux,Apache,MySQL/MariaDB,andPHPcombination(knownas LAMP) works well to support Web hosting. An introduction to theseprograms,togetherwiththeirconfiguration,andoperationispresented.Inaddition tounderstanding thebigpictureand theunderlyingprinciples, a
practicalhands-onapproachguidesyou through the installation, configuration,testing,andadministrationofApache,PHP,andMySQLsoyoucanlearnLinuxWebhosting throughdoing.Root access onyourLinux is convenient, but notnecessary.
WHATISAWEBSERVER?AWeb server is a piece of software that runs on a particular host to supplydocuments to the Web. The host computer is called a server host and often
9.2
provides many network-based services including theWeb. Linux systems arewidelyused to runWebservers, and it is important forLinuxprogrammers tobecomefamiliarwithoperationsrelatedtotheWebserver.AWebserverlistenstoaspecificnetworkingportonthehostandfollowsthe
Hypertext Transfer Protocol to receive HTTP requests and send HTTPresponses.Thestandardportis80forHTTPand443forHTTPS.Butotherportsmaybeused.In response to an incoming request, a server may return a static document
from files stored on the server host, or itmay return a document dynamicallygeneratedbyaprogramindicatedbytherequest(Figure9.1).
Figure9.1WebServerOverview
A single-thread server handles one HTTP request at a time, while a multi-threadservercanhandlemultipleconcurrent requests.AserverhostmayhavemultiplecopiesofaWebserverrunningtoimprovethehandlingofrequests.ManydifferentbrandsofWebserversareavailablefromcompaniesandfrom
open-sourceorganizations.GlassFish is a freeWebserver thatcomeswith theJavaEEdistributionfromjava.sun.com.TheApacheWebserver,availablefreefrom theApache Software Foundation (apache.org), is widely used on Linuxsystems.ThepopularApacheusuallycomespre-installedonLinuxdistributions.
URLANDURIAnimportantcornerstoneoftheWebistheUniversalResourceLocator (URL,Chapter 7, Section 7.13) that allows Web clients to access diverse resourceslocatedanywhereontheWeb.Forexample,theHTTPURLhttp://ml.sofpower.comleads to the companionwebsite for this textbook.AnHTTPURL (Figure )
identifiesaWebserverrunningonaparticularhostcomputerandprovides thefollowinginformation:
9.3
AUniversalResourceIdentifier(URI)thatcorrespondstoalocalpathnameleadingtoatargetresource(afileorprogram)storedontheserverhostAnoptionalpathinfoindicatingatargetfile/folderlocationasinputdatatothetargetresourceAn optional query string providing key=value pairs as input data to thetargetresource
Figure9.2HTTPURLStructure
Thepartof theURLimmediatelyafter thehost:portsegment (Figure9.2) isreferredtoastheURI.TheWebserverusestheURItolocatethetargetresource,whichcanbea staticpage, anactivepage,or anexecutableprogram.AstaticpageisreturneddirectlyinanHTTPresponse.Anypathinfoandquerystringismade available, as input, to an active page or an executable program. TheresultingoutputisthenreturnedinanHTTPresponse.The set of files and directories made available on theWeb through aWeb
serverisknownasitsdocumentspace.Thedocumentroot is therootdirectoryfor the document space, and it corresponds to the URI /. In addition to thedocumentroothierarchy,therecanbeotherfilesanddirectoriesinthedocumentspace, for example, the /cgi-bin and the userid usually map to directoriesoutsidethedocumentroothierarchy.A Web server also works with other special directories (outside of its
documentspace)forserverconfiguration,passwords,tools,andlogs.AnURIisinterpreted relative to the document root, cgi-bin, or another directory, asappropriate.TheWebservercanenforceaccessrestrictions,specifiedintheWebserverconfigurationfiles,onanyfile/folderinthedocumentspace.
REQUESTPROCESSINGForeachincomingHTTPrequest,aWebserverexecutes thefollowingrequestprocessingcycle:
1. Acceptsclientconnection(viaTCP/IP;Chapter7,Section7.2)2. Processesrequest(fetchesandprocessespageorinvokesprogram)3. Sendsresponse4. Closesconnection(orkeepsitaliveunderHTTP1.1)
9.4
9.5
While processing a request, a busy website often will receive many newrequests. It isnormal tousemultiple servers (multiprocessing)and/ormultiplethreadswithinthesameserver(multithreading)tohandleconcurrentrequests.
RESPONSEANDCONTENTTYPESForeachincomingHTTPrequest,theWebserversendsbackanHTTPresponsecontaining the requested resource or an indication of error or some othercondition.An HTTP response has two parts: the headers and the body. The server
specifies the Content-Type header to indicate the media type of the responsebody. Standard MIME (Multipurpose Internet Mail Extensions) content types(Chapter6,Table6.3)areused.Themostcommoncontenttypeistext/html,butthere aremanyother types.For a static file, theWeb serveruses the filenameextension to infer its media type using a list often found in the file/etc/mime.types.Thelocationofthiscontenttypelistisconfigurable.Incaseofdynamiccontent,thosegeneratedbyserver-sideprograms,theWeb
serverreliesonthoseprogramstosetcontenttype.
THEAPACHEWEBSERVERApache is themostpopularWebserver,especiallyonLinuxsystems.Youcandownload and install the Apache HTTP server (Apache) from the ApacheSoftwareFoundation(httpd.apache.org)freeofcharge(Section9.6).However,yourLinuxwillmostlikelyhaveApachealreadyinstalled.Apache
isderivedfromtheNCSA1httpdprojectandevolved throughaseriesofcodepatches (thus, a patchy server). Apache, written in the C language, is opensourceandrunsonalmostallplatforms.Apacheisfast,reliable,multi-threaded,full-featured, andHTTP/1.1 compliant.AlthoughApache1.3 is still available,themostrecentstableApache2versionistheonetouse.Apachehasmanycomponents,including
Serverexecutable—TherunnableprogramhttpdUtilities—Forservercontrol,passwords,andadministrationFiles—Including server configuration files, log files, password files, andsourcecodefilesDynamic loadable modules—Pre-compiled library modules that can beloadedintothehttpdatrun-timeDocumentation
9.6 APACHEONLINUXBecauseofitsimportance,mostpopularLinuxdistributionscomewithApachealreadyinstalled.OtherwiseyoucaneasilyinstallApacheHTTPD.
InstallingApachewithPackageManagementWehavementioned thatmostLinuxdistributionscomewithApache installed.With root access, you can use the Linux package management (Chapter 8,Section8.2)commands
CentOS/Fedora:dnfinstallhttpddnfupdatehttpdUbuntu/Debian:sudo
apt-getinstallapache2sudoapt-getupdateapache2
to install/update your Apache server. See Chapter 8, Section 8.1 for adiscussionofthesudocommand.InstallingtheWebServergroupgivesmoresupportingprograms.
CentOS/Fedora:dnfgroupinstall'WebServer'dnfgroupupgrade'Web
Server'
IfyouwishtohavetheverylatestApacherelease,orifyoudon’thaverootaccess,youcaninstallApachemanuallyasdescribedinSection9.16.
RunningtheApacheWebServerNetworkingservers,theWebserverincluded,areautomaticallystartedasLinuxboots and stopped as Linux shuts down. To make sure, start system-config-servicesandlookforhttpdamongtheserviceentrieslisted(Section8.3).Alternatively, toenable/disable,start/stop,andrestartservices, thesystemctl
commandcanbeused.systemctlenablehttpd.servicesystemctldisablehttpd.servicesystemctlstarthttpd.servicesystemctlstophttpd.servicesystemctlrestarthttpd.serviceEnablingaservicewillstartitautomaticallyonsystemboot.Aservicecanbe
startedwithoutbeingenabled.You’llusuallyfindthedocumentrootat/var/www/html/andtheApachemain
configurationfileat/etc/httpd/conf/httpd.conf(CentOS/Fedora)/etc/apache2/apache2.conf(Ubuntu/Debian)
9.7
Often,themainconfigurationfilewillincludeothercomponentconfigurationfilessuchasphp.confandssl.conf.Tocheckifhttpd,oranyotherprocess,isrunning,youcanusepidofhttpdpidofprocess_Nameandseeifoneormoreprocessidsarefound.
ControllingtheApacheServerThe command apachectl (CentOS/Fedora) or apache2ctl (Ubuntu/Debian),usuallyfoundin/usr/sbin,canbeusedtocontrolthehttpdapachectlactionapache2ctlactionActionsofapachectl
Action Meaningstart Startshttpdifnotalreadyrunningstop Stopshttpdifrunningrestart Starts/restartshttpdgraceful Restartshttpd,respectingongoingHTTPrequestsconfigtestor-t ChecksthesyntaxofconfigurationfilesPossibleactionsarelistedinTable9.1.
APACHERUN-TIMECONFIGURATIONFeaturesandbehaviorsoftheApachehttpdcanbecontrolledbydirectiveskeptin configuration files. The main configuration file is usually httpd.conf (orapache2.conf). When httpd starts, it reads the configuration files first. Aftermakingchangestotheconfiguration,thehttpdneedstoberestartedbeforethenewconfigurationtakeseffect.UnlessyouhaveinstalledyourownApacheasanordinary user (Section 9.16), you’ll need root privilege tomodify theApacheconfigurationortorestartit.
ApacheConfigurationFileBasicsAnApacheconfigurationfile(httpd.conf,forexample)isatextfilethatcontainsconfigurationdirectives.Eachdirectiveisgivenonaseparatelinewhichcanbecontinuedtothenextlinebyacharacterattheendoftheline.Lines thatbeginwith thechar#arecommentsandare ignored.Acomment
must occupy the entire line. No end-of-line comments are allowed. There are
many different directives. Directive names are not case sensitive, but theirarguments often are. A directive applies globally unless it is placed in acontainerwhichlimitsitsscope.Wheninconflict,alocaldirectiveoverridesaglobaldirective.Themainconfigurationfileishttpd.conf,andothercomponentconfiguration
filesmayexistandareincludedbythemainfilewiththeIncludedirective.Forexample, onmanyLinux systems the configurationdirectory /etc/httpd/conf.d/stores component configuration files such as ssl.conf for SSL (secure socketlayer)tosupportHTTPS,andphp.confforPHP(Section9.17).ThedirectiveIncludeconf.d/*.confisusedtoincludeallsuchcomponentconfigurationfiles.To test your Apache configuration for syntax errors, use either one of the
followingcommands:apachectlconfigtesthttpd-tInadditiontothecentral(mainandcomponent)configurationfiles,thereare
alsoin-directoryconfigurationfilesknownasaccessfiles.Anaccessfile,oftennamed .htaccess, is placed in any Web-bound folder (your public_html, forexample) to provide configuration settings applicable for the file hierarchyrootedat thatparticular folder.Directives inanaccess fileoverride settings inthe central configuration files. The possibility of an access file and whatdirectives itmay contain are controlled by theAllowOverride directive in themain configuration file.The .htaccess files are especially useful for individualusers to configure their ownWeb spaces, usually the public_html under theirhomedirectories.
AboutConfigurationDirectivesConfiguration directives controlmany aspects of theApacheWeb server. Thehttpd.conf file has three main parts: Global Environment, main serverconfigurations, and virtual hosts configurations. Comments are provided foreach configuration directive to guide its usage. Apache has reasonable andpractical default settings for all the directives, making it easy to configure atypical server. Additional directives specify how loaded components work.Commonlyuseddirectivesinclude
Server properties: host identification (ServerName name), file locations(ServerRoot, DocumentRoot, ScriptAlias), network parameters (Listen[IP:]port),andresourcemanagement(StartServers,KeepAlive)Enablingoptional server features (Options) and in-directory configuration
overrides(AllowOverride)Accessrestrictionsanduserauthentication(Allow,Deny,Require,Satisfy,AuthName,AuthType,AuthFile)Contenthandling(AddHandler,AddType,AddOutputFilter)HTTP caching and content deflation (DeflateCompressionLevel,ExpiresActive,ExpiresByType,AddOutputFilterByTypeDEFLATE)Virtualhosts(NameVirtualHost)
Forexample,thedirectiveDirectoryIndexindex.htmlindex.phpsaysindex.html(orindex.php)isthedirectoryindexfilewhichisdisplayedif
the folder containing it is the target resourceof an incomingURI.Without anindexfile,alistingoffilenamesinthatfolderisgenerated(indexgeneration)fordisplay only if the Indexes option has been enabled. Otherwise, an error isreturned.
LoadingModulesApacheisamodularserver.Onlythemostbasicfunctionalitiesareincludedinthe core httpd. Many extended features are implemented as dynamicallyloadablemodules(.so)thatcanbeselectivelyloadedintothecoreserverwhenitstarts.Thisorganizationisveryefficientandflexible.Theloadablemodulesareplacedinthemodulesfolderundertheserverroot
directory,which is defined in themain configuration filewith the ServerRootdirective.Toloadacertainmodule,usethedirectiveLoadModulename_modulemodules/moduleFileName.soForexample,LoadModuledir_modulemodules/mod_dir.soloadsmoduledir)LoadModulephp5_modulemodules/libphp5.so(loadsmodulephp5)The dir module enables Apache to generate a directory listing. The php5
modulesupportsdynamicWebpagesusingthePHPscriptinglanguage(Section9.17).Configuration directives may be included conditionally, depending on the
presenceofaparticularmodule,byenclosingtheminan<IfModule>container.Forexample,
>IfModulemod_userdir.c<UserDirpublic_html>/IfModule<
saysifweareusingtheuserdirmodule, thentheWebfolderforeachLinuxuserispublic_html.
GlobalDirectivesTable9.2showssomemoredirectivesrelatingtohowtheApacheserverworksglobally (Ex: ex09/apacheGlobal.conf). The Alias and ScriptAlias directivesmapanincomingURItoadesignatedlocalfolder.
ContainerDirectivesConfiguration directives can be placed inside a container directive to subjectthem to certain conditions or to limit their scope of applicability to particulardirectories, files, locations(URLs),orhosts.Withoutbeing limited,adirectiveappliesglobally.ApacheGlobalDirectives
Forexample,
>IfModulemod_userdir.c<UserDirpublic_html>/IfModule<
enables the per-userWeb space (Ex: ex09/peruser.conf) and designates theuserfoldertobepublic_htmlonlyiftheuserdirmoduleisloaded.Also,considerthesetypicalsettings(Ex:ex09/docroot.conf)forthedocument
root/var/www/html:
>Directory"/var/www/html"<OptionsIndexesFollowSymLinks(1)Order
allow,deny(2)Allowfromall(3)AllowOverrideNone(4)>/Directory<
Within the directory /var/www/html, we allow index generation and thefollowing of symbolic links (line 1). The order to apply the access controldirectives is allow followed by deny (line 2), and access is allowed for allincomingrequests(line3+)unlessitisdeniedlater.
9.8
The AllowOverride (line 4) permits certain directives in .htaccess files. Itsarguments can be None, All, or a combination of the keywords Options,FileInfo, AuthConfig, and Limit. We’ll return to this topic when we discussaccesscontrolindetail(Section9.8).You’llalsofindthefollowingtypicalsetting(Ex:ex09/htprotect.conf)inyour
httpd.conf:
>Files~"^\.ht"<Orderallow,denyDenyfromall>/Files<
It denies Web access to any file whose name begins with .ht (Chapter 4,Section 4.4). This is good for security because files such as .htaccess arereadablebytheApacheWebserver,butwedon’twanttheircontentsexposedtovisitorsfromtheWeb.As<Directory>and<Files>workonthefilepathnamesonyourcomputer,
the<Location>containerworksonURIs.Wealsohave<DirectoryMatch>,<FileMatch>,and<LocationMatch>thatuseregularexpressionsasdefinedforegrep(Chapter4,Section4.4).
ACCESSCONTROLUNDERAPACHEWhatIsAccessControl?RunningaWebserveronyourLinuxsystemmeans thatyoucanmakecertainfiles and folders accessible from theWeb.However, you alsowant to controlhowsuchfilescanbeaccessedandbywhom.Tomakeafile/folderaccessiblefromtheWeb,youmustplaceitsomewhere
in the document space configured for your Web server. This usually meansplacingafile/folderunderthedocumentrootorinsideyourownpublic_htmlandalsomaking the file readable (the folder readable and executable) by theWebserverviachmoda+rfile(chmoda+rxfolder).Filesonyoursystemnotplacedundertheserverdocumentspaceornothavingtherightaccessmodes(Chapter6,Section)willnotbeaccessiblefromtheWeb.The Web server can be configured to further limit access. Access control
specifies who can access which part of a website with what HTTP requestmethods.Access control can be specified based on IP numbers, domains, andhosts,aswellaspasswords.Accessrestrictionscanbeappliedtotheentiresite,tospecificdirectories,ortoindividualfiles.Apache access control directives include Allow, Deny, Order, AuthName,
AuthType, AuthUserFile, AuthGroupFile, Require, Satisfy, < Limit > , and <
9.9
LimitExcept>.
AccessControlbyHostIfa file in theserverdocument spacehasnoaccesscontrol,access isgranted.The order directive specifies the order in which allow and deny controls areapplied.Forexample,orderallow,denyonlyaccessallowedbutnotdeniedarepermitted.Inthefollowing,ifaccessis
firstdeniedthenallowed,itisallowed.
orderdeny,allowdenyfromallallowfromhost1host2...
Onmonkey.cs.kent.edu,we have a set of pages reserved for use inside ourdepartmental local area network (LAN). They are placed under the folder/var/www/html/internal. Their access has the following restriction (Ex:ex09/folderprotect.conf):
>Location/internal<orderdeny,allowdenyfromallallowfrom
.cs.kent.edu>/Location<
Thus,onlyhostsinthe.cs.kent.edudomainareallowedtoaccessthelocation/internal.TheIPaddressofahostcanbeused.Forexample,allowfrom131.123grantsaccesstorequestsmadefromanyIPwiththeprefix131.123.Toenableuserstocontrolaccesstofilesandfoldersundertheirper-userWeb
space(public_html),youcanusesomethingsuchas(Ex:ex09/htaccess.conf)
>Directory/home/*/public_html<AllowOverrideAllOrder
allow,denyAllowfromall>/Directory<
inhttpd.conf.Thismeansuserscanplacetheirownaccesscontrolandotherdirectivesinthefile/public_html/.htaccess.
REQUIRINGPASSWORDSAllowingaccessonlyfromcertaindomainsorhostsisfine,butwestillneedawaytorestrictaccesstoregistereduserseitherforthewholesiteorforpartsofit. Each part of a site under its own password control is known as a securityrealm.Auserneedsthecorrectuseridandpasswordtologintoanyrealmbeforeaccessingthecontentsthereof.Thus,whenaccessingaresourceinsidearealm,ausermust firstbeauthenticated or verified as towho theuser is.TheApache
Web server supports two distinct HTTP authentication schemes: the BasicAuthentication and theDigestAuthentication. Some browsers lack support forDigest Authentication which is only somewhat more secure than BasicAuthentication.Let’slookathowtosetuserlogin.
SettingUpUserLoginunderApacheTo illustrate how to password protect Web files and folders, let’s look at aspecificexamplewherethelocation/WEB/csnotes/isafolderwewillprotect.Wefirstaddthefollowingauthenticationdirectivestothehttpd.conffile(Ex:
ex09/validuser.conf):
>Location"/WEB/csnotes/"<AuthName"WDP-1Notes"AuthType
BasicAuthUserFile/var/www/etc/wdp1prequirevalid-user>/Location<
TheAuthNamegivesanametotherealm.Therealmnameisdisplayedwhenrequestingtheusertologin.Thus,itisimportanttomaketherealmnameveryspecificso thatuserswillknowwhere theyare logging into.Figure9.3showssuchaloginpanelforaccessingtheexamplepackageonthisbook’swebsite.
Figure9.3HTTPBasicAuthenticationExample
TheAuthTypecanbeeitherBasicorDigest.TheAuthUserFilespecifiesthe
9.10
fullpathnameofafilecontainingregisteredusers.TheoptionalAuthGroupFilespecifiesthefullpathnameofafilecontaininggroupnamesandusersinthosegroups. The Require directive defines which registered users may access thisrealm.
valid-user(allusersintheAuthUserFile)userid1id2id3...(the
givenusers)groupgrp1grp2...(allusersinthegivengroups)
TheAuthUserFileliststheuseridandpasswordforeachregistereduser,withoneuserperline.Hereisasampleentryin/var/www/etc/wdp1.PWang:RkYf8U6S6nBqETheApacheutilityhtpasswd (htdigest)helpscreatepassword filesandadd
registeredusersfortheBasic(Digest)authenticationscheme.(Seethemanpagefortheseutilitiesforusage.)Forexample,htpasswd-c/var/www/etc/wdp1PWangcreates the file and adds an entry for user PWang, interactively asking for
PWang’spassword.Ifyouwishtosetupagroupfile,youcanfollowtheformatfor/etc/group,namely,eachlinelookslikegroup-name:userid1userid2...It isalsopossible tosetup loginfroman .htaccess file.Forexample,put in
.htaccessunderuserpwang’spublic_html
AuthUserFile/home/pwang/public_html/.htpasswordAuthName"Faculty
Club"AuthTypeBasicRequirevalid-user
Then,placein.htpasswordanyregisteredusers.If more than one Require and/or allow from conditions is specified for a
particular protected resource, then the satisfy any (if any condition ismet) orsatisfyall(allconditionsmustbemet)directiveisalsogiven.Forexample(Ex:ex09/flexibleprotect.conf),
>Location/internal<orderdeny,allowdenyfromallallowfrom
.cs.kent.eduAuthName"CSInternal"AuthTypeBasicAuthUserFile
/var/www/etc/csrequirevalid-usersatisfyany>/Location<
meansresourcesunderthe/internalcanbeaccessedbyanyrequestoriginatingfromthecs.kent.edudomain(nologinrequired)orausermustlogin.
HOWHTTPBASICAUTHENTICATIONWORKS
9.11
UponreceivinganunauthorizedresourcerequesttoarealmprotectedbyBasicAuthentication,theWebserverissuesachallenge:
HTTP/1.0401UnauthorizedWWW-Authenticate:Basicrealm="CS
Internal"
Upon receiving the challenge, the browser displays a login dialog boxrequestingtheuseridandpasswordforthegivenrealm.Seeingthelogindialog,the user enters the userid and password. The browser then sends the sameresourcerequestagainwiththeaddedauthorizationHTTPheader
Authorization:BasicQWxhZGRpbjpvcGVuIHNlc2FtZQ==
wherethebase64(Chapter7,Section7.10)encodedbasiccookiedecodes touserid:password. From this point on, the browser automatically includes thebasic cookiewith every subsequent request to the given realm. This behaviorpersistsuntilthebrowserinstanceisclosed.
HOWHTTPDIGESTAUTHENTICATIONWORKS
Unless conductedover a secure connection, such asSSL (secure socket layer)usedbyHTTPS(Section9.14),theBasicAuthenticationisnotverysecure.Theuseridandpasswordaresubject toeasyeavesdroppingoverHTTP.TheDigestAuthentication is an emerging HTTP standard to provide a somewhat moresecuremethodthanBasicAuthentication.WithDigestAuthentication,theserversendsachallenge(onasingleline)
HTTP/1.1401UnauthorizedWWW-Authenticate:Digestrealm="GoldClub"
nonce="3493u4987"
where the nonce is an arbitrary string generated by the server. Therecommended form of the nonce is anMD5 hash (Chapter 7, Section 7.12),which includes the client’s IP address, a timestamp, and a private key knownonlytotheserver.Uponreceivingthechallenge,thebrowsercomputes
str1=MD5(userid+password)str2=MD5(str1+nonce+
Resource_URI)
ThebrowserthensendstheauthorizationHTTPheader(ononeline)
9.12
9.13
Authorization:Digestrealm="GoldClub",
nonce="...",username="pwang",
uri="/www/gold/index.html",response="str2"
Theserververifiestheresponsebycomputingitusingthestoredpassword.From this point on, the browser includes the Digest Authentication header
witheveryrequesttothesamerealm.Theservermayelecttorechallengewithadifferentnonceatanytime.
Basicvs.DigestAuthenticationBasic Authentication is simple and works with all major browsers. DigestAuthenticationissomewhatmoresecure,butbrowsersupport is lesscomplete.Web servers, including Apache, tend to support both authentication schemes.When security is a concern, the best practice is to move from BasicAuthenticationoverHTTPdirectlytoBasicAuthenticationoverHTTPS(SecureHTTPoverSSL).
PASSWORDENCRYPTIONTheApache-suppliedhtpasswd tooluses thesameLinux/UNIXpassword/dataencryption method as implemented by the C library function crypt. In thisencryptionscheme,akeyisformedbytakingthelower7bitsofeachcharacterfromthepasswordtoforma56-bitquantity.Hence,onlythefirst8charactersofapasswordaresignificant.Also,a randomlyselected2-charactersalt fromthe64-character set [a-zA-Z0-9./] is used to perturb the standardDataEncryptionAlgorithm(DEA)in4096possibledifferentways.Thekeyandsaltareusedtorepeatedlyencryptaconstantstring,knownonlytothealgorithm,resultinginan11-character code. The salt is prepended to the code to form a 13-characterencryptedpasswordwhichissavedinthepasswordfileforregisteredusers.Theoriginalpasswordisneverstored.Whenverifyingapassword,thesaltisextractedfromtheencryptedpassword
and used in the preceding algorithm to see if the encrypted password isregenerated.Ifso,thepasswordiscorrect.
AUTOMATICFILEDEFLATIONApachetakesadvantageofmanyHTTP1.1featurestomakeWebpagesfastertodownload.Onesuchfeatureisautomaticcompressionofapagebeforenetworktransfer, resulting in significantly reduced file size and delivery time. This is
9.14
especially true for textual pages whose compression ratio can reach 85% ormore.Acompressedpageisuncompressedbyyourbrowserautomatically.Themod_deflatemodule for Apache 2.0 supports automatic (dynamic) file
compressionviatheHTTP1.1Content-EncodingandAccept-Encodingheaders.Thesetwoconfigurationdirectives(Ex:ex09/deflate.conf)
DeflateCompressionLevel6AddOutputFilterByTypeDEFLATEtext/html
text/plain\text/xmltext/cssapplication/x-javascript
\application/xhtml+xmlapplication/xslt+xml\application/xml
application/xml-dtdimage/svg+xml
indicate a list of content types for dynamic compression (using zlib) at theindicated compression level. Deflation adds a bit of processing load on theserversideandthehigherthecompressionlevel,theheaviertheprocessingload.CompressionwillonlytakeplacewhentheincomingHTTPrequestindicates
an acceptable compression encoding. The detection of browser compressionpreferencesandthesendingofcompressedoruncompresseddataareautomatic.Of course, any compressed outgoing page will carry an appropriate Content-Encodingresponseheader.TheAddOutputFilterByTypedirectiveneedsAllowOverrideFileInfotowork
in.htaccess.
HTTPSANDSSL/TLSWebservers supportHTTPS for securecommunicationbetween theclient andtheserver.
Figure9.4HTTPandHTTPS
HTTPS is HTTP (Hypertext Transfer Protocol) over Secure Socket Layer(SSL)orthenewerTransportLayerSecurity(TLS)protocol(Figure9.4).NoteHTTP and HTTPS use different server network ports, normally 80 and 443,respectively.SSL/TLSdevelopedfromSSL1.0,2.0,and3.0toTLS1.0,1.1,and1.2. SSL/TLS provides secure communication between client and server by
allowingmutual authentication, the use of digital signatures for integrity, anddataencryptionforconfidentiality.ToenableHTTPS,aserverneedstoinstallavalidWebservercertificate(Section7.9)andenableSSL/TLS.SSL/TLS may be placed between a reliable connection-oriented transport
protocollayer,suchasTCP/IP,andanapplicationprotocollayer,suchasHTTP(Figure9.5).
Figure9.5HTTPSProtocolLayers
Basically,TLSsetsupsecurecommunicationintwosteps:
1. Thehandshakephase—Mutualauthenticationandsecurelyagreeinguponarandomlygeneratedsessionkeytobeusedinthenextphase
2. The session data phase—Following the Record layer protocol, using thesession key for symmetric encryption (Section 7.8) of messages betweentheclientandserver
The handshake phase uses public-key cryptography (Section 7.9) for security,while the session data phase uses themore efficient symmetric encryption forspeed.EachnewSSL/TLSconnectionwillestablishanewsessionkey.Figure9.6illustratestheTLShandshakeprocessfromauserviewpoint.
9.15
Figure9.6BasicTLSHandshake
Allthisisabitoverwhelmingforbeginners.Don’tworry,wewilltalkaboutcryptography,digital signature,andall thatabit later in thischapter.But first,let’slookatthedigitalcertificate.
HTTPSSUPPORTFollow these three steps to setup SSL/TLS server certificate for AppacheHTTPD so that theWeb server will listen to port 443 and process incomingHTTPSrequests.
1. Make sure you have the necessary packages, openssl and mod_ssl,installed.
2. ObtainaservercertificatefromaCAsuchasDigiCertorEtrustandinstallthe encoded certificatemyserver.crt and private keymyserver.key in thedirectory/etc/pki/tls/certs/.
3. Modify the SSL configuration, /etc/httpd/conf.d/ssl.conf, for httpd asfollows:
DocumentRoot"/var/www/html"ServerName
mydomain:443SSLCertificateFile
/etc/pki/tls/certs/myserver.crtSSLCertificateKeyFile
/etc/pki/tls/certs/myserver.key
ThenrestarthttpdsystemctlrestarthttpdAlso make sure the firewall is not blocking https or port 443 (Chapter 8,
Section8.5).Forlocalortestingpurposes,youcangenerateaself-signedservercertificate
asfollows.GenerateanRSAprivatekey:
cd/etc/pki/tls/certs;makemyserver.keyopensslrsa-in
myserver.key-outmyserver.key
Filloutacertificatesigningrequest:makemyserver.csrSelfsignandgeneratethecertificate:
opensslx509-inmyserver.csr-outmyserver.crt\-req-signkey
myserver.key-days3650
9.16 MANUALINSTALLATIONOFAPACHEIfyouprefernot to installApachewithpackagemanagement,youmay installApache manually. The installation procedure follows the standard Linuxconfigure,make,installsequence.If you have root access, you will be able to install Apache in a system
directorysuchas /usr/localandassignport80 to it. Ifnot,youstill can installApacheforyourself(forexperimentation)inyourownhomedirectoryanduseanon-privilegedport,suchas8080.Let$DOWNLOADbethedownloadfolder,for example, either /usr/local/apache_src or $HOME/apache_src, and let$APACHE be the installation folder, for example, /usr/local/apache or$HOME/apache.TodownloadandunpacktheApacheHTTPserverdistribution, followthese
steps.
1. Download—Gotohttpd.apache.org/download.cgianddownloadthehttpd-version.tar.gz or the .tar.bz2 file, aswell as itsMD5 fingerprint file, intoyour$DOWNLOADfolder.
2. Integrity check—Use md5sum on the fingerprint file to check thedownloadedfile.
3. Unpack—From the $DOWNLOAD folder unpack with one of thesecommands.tarzxvpfhttpd-version.tar.gztarjxvpfhttpd-version.tar.bz2You’ll find a new Apache source folder, httpd- version, containing theunpackedfiles.
ConfigureandCompileNow you are ready to build and install the Apache Web server. Follow theINSTALL file and the Compiling and Installing section of the Apachedocumentationhttpd.apache.org/docs/version-number .You’llneedanANSICcompiler (gcc preferred) to compile, Perl 5 to make tools work, and DSO(DynamicSharedObject) support.These should already be in place on newerLinuxdistributions.FromtheApachesourcefolder,issuethecommand./configureoptionsto automatically generate the compilation and installation details for your
computer.TheINSTALLfilehasgoodinformationaboutconfiguration.Toseeallthepossibleoptions,givethecommand./configure–help.For example, the –prefix=serverRoot option specifies the pathname of the
serverrootfolder,andtheoption–enable-mods-shared=allelectstocompileallApachemodulesintodynamicallyloadablesharedlibraries.The recommended method (Ex: ex09/makeapache.bash) to configure and
compileis./configure–prefix=$APACHE–enable-mods-shared=allotherOptionsmakemakeinstallHere the Apache server root folder has been set to your installation folder
$APACHE as the destination for the results of the installation. TherecommendedotherOptionsare
--enable-cache--enable-disk-cache\--enable-mem-cache--enable-
proxy\--enable-proxy-http--enable-proxy-ftp\--enable-proxy-
connect--enable-so\--enable-cgi--enable-info\--enable-rewrite-
-enable-spelling\--enable-usertrack--enable-ssl\--enable-deflate
--enable-mime-magic
Eachoftheprecedingthreecommandswilltakeawhiletoruntocompletion.Aftersuccessfulinstallation,itistimetocustomizetheApacheconfiguration
file$APACHE/conf/httpd.conf.Followthesesteps:
1. ChecktheServerRootandDocumentRootsettings.Theseshouldbethefullpathnamesasgivenby$APACHEand$APACHE/htdocs,respectively.
2. Set the listening port: Listen 80 requires root privilege) Listen 8080 (noneedforrootprivilege)
3. Makeanyotherconfigurationadjustmentsasneeded.
NowyoucanstarttheApacheserverwith$APACHE/bin/apachectlstartIf the start is successful,youcan thenuseaWebbrowseron the samehost
computertovisithttp://localhost.localdomain:portandseetheApachewelcomepage,whichisthefile$APACHE/htdocs/index.htmlThen,testtheserverfromanotherhostonthesameLAN,withhttp://host:portwherehostisthedomainnameofyourserver.Makesurethatthefirewallon
the server allows both HTTP and HTTPS access (Section ). Otherwise, theApacheWeb server won’t be accessible from other hosts. On CentOS/Fedorafirewallconfigurationisanoptiononthesystem->adminmenu.ForUbuntuthe
9.17
gufwtoolishandyforthesamepurpose.It is recommended that you install PHP together with Apache. See Section
9.18fordetails.
WHATISPHP?PHP, a recursive acronym for PHP: Hypertext Preprocessor, represents apowerful and widely used program for generating dynamic Web content. ItevolvedfromanearlierprojectbyRasmusLerdorf,andPHP3.0wasreleasedinmid-1998. PHP has matured as an important server-side scripting tool and ismoving past version 7.2 at the time of thiswriting. In addition to serving theWeb, PHP can also be used as a Linux command for general-purpose textprocessing.AlthoughPHPrunsonmultipleplatforms,wewillfocusonPHPasanApache
servermoduleonLinux.Assuch,PHPexecutesaspartofApacheandinterpretscodeembeddedinWeb-boundpages todynamicallygeneratecontentfor thosepages.Forexample,anHTMLdocumentcontaining
>p<Itis>?phpecho(date("lM.d,Y"));?<,>br/<doyouknowwhere
yourprojectis?>/p<
generatesthetextItisThursdayJune.18,2018,doyouknowwhereyourprojectis?Thedatedisplayeddependsontheexacttimeofaccess.AnyPHPcodeisgivenwithinthePHPbracket<?php...?>andinterleaved
(embedded)withinnormalHTMLcode,orothertypesofcodeasthecasemaybe.Pagescontainingsuchembeddedcodesareoftencalledactive(ordynamic)pages,because theyarenotstaticandcontain informationgeneratedon theflybytheembeddedcode.Theembeddedcodeisneverseenbythereceiveroftheresultingdocument;itgetsreplacedbyanyinformationitgenerates(Figure9.7).
9.18
Figure9.7PHPCodeInterpretation
THEPHPMODULEFORAPACHEAnApacheserverisgenerallyexpectedtosupportPHP,anditisnothardtoaddthePHPmoduleforApache.WiththePHPmodule,theApacheWebserverwillbe able to interpretPHPcodes embedded in textual documents of any type astheyarebeingdeliveredtotheWeb(Figure9.7).MostLinuxdistributionswillhaveApache installedwithPHP already. For example, youmay find thePHPmodule libphp*.so already in the Apache modules folder (usually/etc/httpd/modules).You can also use the Linux package management facility to install/update
Apache+PHP:
dnfinstallhttpdphpphp-common(CentOS/Fedora)dnfupgradehttpd
phpphp-common(CentOS/Fedora)sudoapt-getinstallapache2php7.0
\php7.0-mysqllibapache2-mod-php7.0(Ubuntu/Debian)sudoapt-get
updateapache2php7.0\php7.0-mysqllibapache2-mod-php7.0
(Ubuntu/Debian)
InstallingthePHPModuleThis section describes how to install the PHPmodulemanually and add it toyourApacheserver.IfyoualreadyhaveApache+PHPinstalled,pleaseskipthissection.First,download thecurrentPHPrelease (php-version.tar.gzor .tar.bz2) from
www.php.net/downloads.php,checktheMD5fingerprint,andunpackintoyour$DOWNLOADfolderasbefore(Section9.16).Next, go to the PHP source code folder $DOWNLOAD/php-version to
configurethePHPmodule.Forexample(Ex:ex09/makephp.bash),
dnfinstallhttpdphpphp-common(CentOS/Fedora)dnfupgradehttpd
9.19
phpphp-common(CentOS/Fedora)sudoapt-getinstallapache2php7.0
\php7.0-mysqllibapache2-mod-php7.0(Ubuntu/Debian)sudoapt-get
updateapache2php7.0\php7.0-mysqllibapache2-mod-php7.0
(Ubuntu/Debian)
Thenchecktheconf.outputtoseeifyougettheselines:checkingiflibtoolsupportssharedlibraries...yescheckingwhethertobuildsharedlibraries...yescheckingwhethertobuildstaticlibraries...noIfyouneedtoredotheconfigurationstep,pleasefirstcleanthingsupwithmakedistcleanAftersuccessfulconfiguration,youarereadytocreatethePHPmodule.Enter
thecommandmakeItwilltakeawhile.Afteritisdoneyoushouldcheckthe.libs/foldertoseeif
thePHPmodulelibphp7.sohasbeencreated.Ifso,thenissuethecommandmakeinstallTheinstalldirectoryis$APACHE/phpasspecifiedbythe–prefixoption.The
install process also moves libphp7.so to the folder $APACHE/modules/ andmodifies$APACHE/conf/httpd.confforthehttpdtoloadthePHPmodulewhenitstartsbyaddingtheApacheconfigurationdirectiveLoadModulephp7_modulemodules/libphp7.soInaddition,youalsoneed toadda fewotherdirectives to tellApachewhat
filesneedPHPprocessing:AddHandlerapplication/x-httpd-ea-php70.html.htm.phpDirectoryIndexindex.phpindex.htmlAsstated,anytimeachangeismadetotheconfiguration,youneedtorestart
Apache(Section)inordertogetthenewconfigurationtotakeeffect.
TESTINGPHPTo test your Apache+PHP installation, you can create the page info.php (Ex:ex09/info.php)
>html<>head<>title<phpinfo>/title<>/head<>body<>?phpphpinfo();?
<>/body<>/html<
andplaceitunderthedocumentrootfolder.Then,visithttp://localhost.localdomain/info.phpfromyourWebbrowser.Thephpinfo()functiongeneratesapageofdetailed
information about your PHP installation, including version number, modules
9.20
loaded,configurationsettings,andsoon.As Apache starts, it loads the PHP module and also any PHP-specific
configuration ina fileusuallynamedphp.ini.The locationof this file (usually/etc/php.ini)isgivenastheLoadedConfigurationFileinthephpinfo()generateddisplay.
PHPCONFIGURATIONTheconfiguration file (php.ini) is readwhen thePHPmodule is loadedas theWeb server (httpd) starts. Any changesmade to php.ini will only take effectafterApacheisrestarted(Section9.9).PHP has toggle (on/off) and value configuration directives. You edit the
php.ini,whichcontainsasetofreasonabledefaults,tomakeanyadjustments.Forexample, ifyouare runningaWebdevelopmentsitewhereseeingerror
messages will help debugging PHP scripts, then you would set (Ex:ex09/php.ini)
;;;;EnableserrordisplayoutputfromPHPdisplay_errors=
Ondisplay_startup_errors=On
ForaproductionWebserver,youwoulddefinitelywanttochangetheseto
display_errors=Offdisplay_startup_errors=Offlog_errors=On;;;;
Enablesallerror,warning,andinfomsgreportingerror_reporting=
E_ALL;;;;Sendsmsgstologfileerror_log=>pathnameofa
designatederror.txtfile<
PHP also allows you to open any local or remoteURL for generating pagecontent.However,ifyoursitehasnoneedforopeningremoteURLsfromPHP,youmayincreasesecuritybysetting
allow_url_fopen=Off
PHPalsohasverygoodsupportforHTTPfileuploading.Ifyouwishtoallowthat,thenuse
file_uploads=On;;;;Usesomereasonablesize
limitupload_max_filesize=2M
PHP extensions provide optional features for many different purposes. Forexample, the gd extension supports manipulation of fonts and graphics fromPHP, and themysql extension provides a PHP interface toMySQL databases.DynamicallyloadableextensionsarecollectedinaPHPmodulesfolder(usually
9.21
9.22
/usr/lib/php/modules), but are set in the php.ini by the extension_dir directive.OnmanyLinuxsystems,theextensionsareloadedbydefaultthroughextension-specific .ini files in the folder /etc/php.d/. By editing these files you controlwhichextensionsareloadedwhenApache+PHPstarts.Toexamine the settingof allPHPconfigurationsdirectives,youcan simply
lookatthephpinfo()display(Section9.19).
PHPCOMMANDLINESCRIPTINGPHPcanbeusedfromthecommandline.ThisisusefulfortestingPHPcodeandfor taking advantageof thepowerofPHP towrite command-line scripts.YoucancreateaPHPexecutabletextfileasfollows
#!/usr/bin/php>?phpPHPcodelines....?<
thenyoucaninvokethescriptfromthecommandlinejustlikeaBASHscript.As an example, let’s write a PHP script (Ex: 09/echoback.php) which is aversionofechoback.sh(Chapter5,Section5.11).
#!/usr/bin/php>?php$output="\n";array_shift($argv);//loses
$argv[0]thecommandnameforeach($argvas$arg){$output="$arg"
.$output;}echo$output;?<
Wecanseethatcommand-lineargumentsarestoredinthePHPspecialarray$argv.Executethisscriptwithanyofthesecommandsphpechoback.phpABCDEphp-fechoback.phpABCDE./echoback.phpABCDEHere is an alternative implementation (Ex: 09/echoback2.php) using a PHP
forloop.
#!/usr/bin/php>?phpfor($n=$argc-1;$n<0;$n--){echo$argv[$n];
echo"";}echo"\n";?<
Youcanalsopassphpcodedirectlytophponthecommandline.
php-r'print_r(phpversion());echo"\n";'php-r
'print_r(phpinfo());echo"\n";'
DATABASESUPPORTFORTHEWEBAcomputerdatabase isasystemforconvenientlystoring, retrieving,updating,
9.23
and inquiring information for concurrent access by many users. Moderndatabasesarerelational;informationisstoredinmultipletables(Figure9.8)thatareinterrelated.
Figure9.8TheEMPLOYEESTable
A database system is SQL-compliant if it supports the Structured QueryLanguagestandardAPI(ApplicationProgrammingInterface).Forexample,thefollowing SQL SELECT query retrieves all rows from table EMPLOYEESwherethefieldLASTisWang:SELECT*FROMEMPLOYEESWHERELAST="Wang";Programs written in SQL can access and manipulate any SQL-compliant
database. Databases can be used for decision support, online transactionprocessing, personnel records, inventory control, user accounts, multi-useronlinesystems,andmanyotherpurposes.Adatabasecanalsomakewebsiteseasiertoconstruct,maintain,andupdate.
Ontheotherhand, theWebcanmakedatabasesaccessiblefromanycomputerconnectedtotheInternet.PHPprovidesexcellentsupportforusingdatabasesforandfromtheWeb.The
SQLiteextensionofPHPisafastSQLinterfacetoaflatfiledatabasethatcomeswithPHP(version5orlater).FormanysimpleWebapplications,SQLiteisjusttherightsolution.
MYSQLMore complicated websites with larger data loads will need heavier dutydatabasesystemsthanSQLite.Forthat,thefreeMySQLorMariaDBisoftentheright choice, especially incombinationwithLinuxandPHPbecausePHPalsohasexcellentbuilt-insupportforconnectingandqueryingMySQLandMariaDBdatabases.WewillfocusonMySQLbutMariaDBisentirelysimilar.MySQL is a freely available open-source relational database management
system that supportsSQL. It runsonLinux,MSWindows®,MacOSX®, and
other systems and can be used frommany programming languages, includingC/C++, Eiffel, Java, Perl, PHP, Python, and Tcl. TheMySQL database serversupports both local and network access. It supports a privilege and passwordsystemtospecifywhocanaccess/modifywhatinthedatabasesystem.MostLinuxdistributions comewithMySQL installed. If youcan locate the
commandmysql(oftenin/usr/bin)onyoursystem,then,mostlikely,youhaveMySQLalready.Tobesurelookformysqldbystartingsystem-config-servicesorbythecommandsystemctlstatusmysqld.Ifnot,orifyouwishtoinstallthelatestversionofMySQL,pleaserefertoSection1.25.
Initializing,Starting,andStoppingMySQLMySQL uses adefault database namedmysql for its own purposes, such asrecording registered users (userid and password), managing databases, andcontrolling access privileges. The commandmysql_install_db (in usr/bin/) isrun once to initialize the MySQL default database (usually located in/var/lib/mysql/mysql/) and is done automatically when the MySQL servermysqld isstartedfor theveryfirst time.Themysql_install_db scriptcontainsmanyinitializationsettingsforMySQL,andadjustingthesesettingsallowsyoutocustomizevariousaspectsofMySQL.Startingmysqldcanbedonewiththesystem-config-servicesGUItoolorthe
commandsystemctlstartmysqldThe same GUI and command-line tools can be used to stop/restart the
mysqld.Withmysqld started, MySQL client programs can communicate with it to
access/manipulatedatabasesservedbyit(Figure9.9).
Figure9.9MySQLServerandClients
MySQLRun-TimeConfigurationAsmysqld (thedatabase server) starts, it readsconfigurationvalues inmy.cnf
(usually kept in /etc or /etc/mysql). Specified are the data folder, the socket(Chapter12,Section12.6) location, the userid ofmysqld, and possiblymanyother settings. Edit my.cnf, and delete the line bind-address = 127.0.0.1, ifpresent,whichrestrictstheMySQLservertoaccessfromlocalhostonly.Itisalsorecommendedthatyouconsiderrunningalocal-access-onlyMySQL
serverratherthanonethatisnetworkenabled.ThelatterallowsMySQLclientsto access the server via a network which can mean security problems. Theformerwill limit access toMySQL clients on the same host,making itmuchmoresecure.Todothis,addtheconfigurationsettingskip-networkingtoboth the [mysqld] and the [mysqld_safe] sections inmy.cnf.Youneed to
restartmysqld after making changes to the configurations. See the MySQLdocumentationfordetailsaboutMySQLconfiguration.ItisagoodideatoruntheLinuxcommandmysql_secure_installationtoimprovethesecurityofyourMySQLinstallation.Afterstartingmysqld,youcanusenetstat,acommandtodisplaynetworking
statusandactivityonyoursystem,todoublecheck.Runthecommandnetstat-tap|grepmysqldIfyouseeadisplay,itmeansmysqldisallowingnetworkaccess.Ifyousee
no display, then only local clients are allowed access. The -tap option tellsnetstat to display all information related to TCP with names of programsinvolved.
AdministeringMySQLMySQLprotectsdatabasesbyrequiringauseridandpassword,and,dependingon what privileges the user has, various operations/accesses are allowed ordenied.Atthebeginning,MySQLhasanadministrator(root)andablankpassword.
Theveryfirstadministrativetaskistosetapasswordforroot.2mysqladmin-urootpasswordnew_passwordThe option -u specifies theMySQL userid root and the admin operation is
passwordsetting.Makesureyousavethepasswordforfutureuse.Let’sassumetherootpasswordisfoobar.TheMySQLrootistheuserwhocancreatenewdatabases,addusers,andset
privilegesforthem.mysqladmin-hlocalhost-uroot-pfoobarcreatelxuxtakes the hostname, userid, and password information and creates a new
databaselxux.Nowwecanaddpwangasauserwithallprivilegestouselxux.Onewayisto
use themysql toolwhich isacommand-line interface to theMySQLdatabaseserver.Givethecommandmysql-hlocalhost-uroot-pfoobarlxuxthenyouareworkingwithinmysql,andyoumayenterSQLqueries.Dothe
following(Ex:ex09/adduser.sql):
mysql<USEmysql;(settingdatabasenametomysql)mysql<SHOW
TABLES;(listingnamesoftables)+-----------------+|
Tables_in_mysql|+-----------------+|columns_priv||db||func||
host||tables_priv||user|+-----------------+mysql<INSERTINTO
user(Host,User,Password,Select_priv)-<VALUES('','pwang',
password('thePassword'),'Y');mysql<FLUSHPRIVILEGES;mysql<GRANT
ALLPRIVILEGESONlxux.*TOpwang-<IDENTIFIEDBY
'thePassword';mysql<FLUSHPRIVILEGES;mysql<quit
Theninformuserpwangabouthisuserid,password,anddatabasename.SeetheMySQLdocumentation formore informationonsettinguserprivileges.ToresetthepasswordforpwangusetheSQL
mysql<USEmysql;mysql<updateuserset
Password=PASSWORD('newOne')-<WHEREUser='pwang';
BecausePHPisoftenavailableonthesamehost, thefreephpMyAdmin tool(phpmyadmin.net) isoftenalso installed toenableMySQLadministrationovertheWeb.PhpMyAdmin(Section9.24)supportsawiderangeofoperationswithMySQL.ThemostfrequentlyusedoperationsaresupportedbytheWebbrowsersupplied GUI (managing databases, tables, fields, relations, indexes, users,permissions, and so on). Other operations are always doable via direct SQLstatements. Both the root user and any user for a specific database can dodatabaseadministrationthroughphpMyAdminfromanywhereontheWeb.
ResettingtheMySQLRootPasswordIt is important to not forget theMySQL root password.However, if you findyourself in such a situation, you can reset it. As Linux root, first stop themysqld:systemctlstopmysqldThenrunmysqldinsafemodewithoutsecuritychecking:/usr/bin/mysqld_safe–skip-grant-tables&Thenrunmysqlonthedefaultdatabasemysql:mysql-urootmysql
9.24
Thenupdatethepasswordforroot:
mysql<updateusersetPassword=PASSWORD('anything')-<WHERE
User='root';QueryOK,2rowsaffected(0.04sec)Rowsmatched:2
Changed:2Warnings:0mysql<flushprivileges;exit;
Nowkillthemysqld_safeprocessandrestartthemysqld.
INSTALLINGPHPMYADMINFirst,downloadthelatestversionfromphpmyadmin.netandunpackinyourWebdocument root folder (usually /var/www/html). For example(Ex:ex09/myadmin.install),cd/var/www/htmltarjxvpfphpMyAdmin-4.8.0-english.bz2rmphpMyAdmin-4.8.0-english.bz2mvphpMyAdmin-4.8.0-englishphpMyAdminTheresultingphpMyAdminfolder isnowinplaceunder theWebdocument
rootandyoucandisplayinstallationinstructionsandotherdocumentationwiththeURLhttp://localhost.localdomain/phpMyAdmin/Documentation.htmlTo install phpMyAdmin, you only need to do a few things. In the
phpMyAdmin folder create a configuration file config.inc.php by copying andeditingthesamplefileconfig.sample.inc.php.Itisrecommendedthatyoupickthecookieauthenticationmethodandsetupa
control user, as indicated by the sample configuration file, on your MySQLserver so anyone who has a MySQL login can use phpMyAdmin to managedatabasesaccessibletothatparticularuser.SeethephpMyAdmindocumentationforconfigurationdetails.Afterinstallation,theURLhttp://host/phpMyAdminreaches the on-Web MySQL admin tool for any valid user to manage the
databaseserver.(Figure9.10).
9.25
Figure9.10phpMyAdminTool
MariaDBcanalsousephpMyAdmin.Be sure to install the latestversionofphpMyAdmin.
INSTALLINGMYSQL/MARIADBMySQL/Mariadb may already come with your Linux distribution. If not, theLinuxpackagemanagementsystemmakesinstallationeasy.ForCentOS/Fedora,doasrootdnfinstallmysqlmysql-serverdnfupgrademysqlmysql-serverordnfinstallmariadbmariadb-serverdnfupgrademariadbmariadb-serverForUbuntu/Debian,dooneofsudoapt-getinstallmysql-serversudoapt-getupdatemysql-serverorsudoapt-getinstallmariadb-serversudoapt-getupdatemariadb-serverNow proceed to edit the my.cnf file (Section 9.23) and then start/restart
mysqld,theserverdaemonofMySQLorMariaDB(Section9.23).IfyouwishtoinstallApache+PHP+MySQL/MariaDBtoachieveLAMPallat
once,usethesecommands.CentOS/Fedora:
9.26
9.27
dnfinstallhttpdphpphp-commonmysql-servermysqldnfinstallhttpdphpphp-commonmariadb-servermariadbUbuntu:sudoapt-getinstalltaskselsudotaskselinstalllamp-serverRemembertheseinstallationsareveryniceasdevelopmentalsystems,butnot
secure enough as production systems. Enterprise editions of Linux will mostlikely includeaproductionWebserverwithLAMPandmore.Whatyou learnherewillapplydirectlytosuchproductionservers.Refer to dev.mysql.com/downloads/ at the MySQL site for manual
installation.
FORMOREINFORMATIONComplete information for the Apache Web server can be found athttpd.apache.org/.ThelatestreleasesanddocumentationforPHPareatphp.net/index.php.Thesitewww.mysql.comcontainsallcurrentreleasesandotherinformationforMySQL.See themariadb.orgwebsite for all information about theMariaDBopensourcesoftware.ThereisalsoasiteforbuildingLAMPserversatwww.lamphowto.com.There aremany textbooks on website development and design.DynamicWebProgrammingandHTML5,byPaulS.Wang,isagoodread.
SUMMARYAWeb server followsHTTP to receive requests and send responses. ItsmainfunctionistomapincomingURIstofilesandprogramsinthedocumentspacedesignatedfortheWeb.TheApachehttpdWebserversupportsdynamicmoduleloadingandrun-time
configuration,making it very easy to customize and fit the requirements of awiderangeofWebhostingoperations.Configurationdirectivescanbeplacedincentral files and in access files under individual folders within the documentspace.In addition to controlling features and behaviors of httpd, Apache
configurationscanspecifyaccesslimitationstopartsofthedocumentspaceandcanrequireloginwithHTTPBasicorDigestAuthentication.
9.28
PHPisapopularactivepagelanguagethatcangeneratedynamicWebpages.PHPscriptsareembedded in textual fileswithinanynumberof<?php ...? >brackets.PHPcanbeinstalledasanApachemoduleandwillinterpretembeddedPHP scripts as the Apache httpd delivers a response page. PHP can bedynamicallyconfiguredviathephp.inifile.PHP supplies a wide range of capabilities for the Web, including file
inclusion, form processing, local/remote file operations, file uploading, imageprocessing, sessioncontrol, cookie support, anddatabaseaccess.PHPcanalsobeusedasaCLItool.PHPhasabuilt-inlightweightdatabase,butalsoworkswellwiththeheavy-
dutyMySQLandMariaDBdatabase systems.Both supportmultipledatabasesprotectedbyuseridandpassword.Differentdatabaseusersmayhavedifferentaccess privileges and can be managed easily using Linux commands(mysqladmin,mysql,mariadbandsoon)ortheWeb-basedphpMyAdmintool.ThecombinationLinux,Apache,MySQL/MariaDB,andPHP(LAMP)forms
apopularandpowerfulWebhostingenvironment.The freelyavailableLAMPmakes a great developmental system, but should not be used as part of aproductionWebserverforsecurityreasons.
EXERCISES1. Findoutabouttheconfigurationfile/etc/nsswitch.conf.2. AssumingyourLinuxisrunningtheApacheWebserver,findtheversionof
Apacheserver,thehttpd.conffile,andthedocumentrootfolder.3. HowdoesonegoaboutfindingoutifyourLinuxsystemsupportsper-user
Webspace?4. Install your own Apache server with PHP support under your home
directory(Hint:useanon-privilegedport).Afterinstallation,startyourownhttpdandtestit.
5. HowdoesonefindoutifyourApachehasPHPsupport?Ifso,whereisthefilephp.iniandforwhatpurpose?
6. SetupyourApachetoautomaticallydeflate.html,.css,and.jsfiles.7. Install a server SSL certificate and test your HTTPD server for HTTPS
support.8. Lookatyourphp.iniandfigureouthowtoenable/disablePHPerroroutput.9. WriteaPHPscriptandtestitfromthecommandline.10. ConfigureyourApachetorequireapasswordonsomeWebfolder.Create
somevalidusersandtestyoursettingtomakesurethatitworks.
12
11. SetupsomedatabasetablesusingthePHPbuilt-inSQLite.TestyoursetupwithPHPcodeinaWebpage.
12. Install your ownMySQL under your home directory. You’ll be the rootdatabaseuser.Createanewtestdatabaseandsometablesusingthemysqltool.
13. InstallthephpMyAdmintool.UseittomanageyourMySQLdatabase.14. Set up some database tables for the Web in your MySQL using your
phpMyAdmintool.TestyoursetupwithPHPcodeinaWebpage.15. Find out about the PEAR library for PHP. Install it if it is not already
installed.NationalCenterforSupercomputingApplicationsattheUniversityofIllinois,Urbana-Champaign.NottobeconfusedwiththeLinuxsuperuserwhichisalsoroot.
Chapter10
CProgramminginLinux
With a basic understanding of commands, Shell usage and programming,structureofthefilesystem,networking,andWebhosting,younowarereadytoexploreLinuxsystemprogrammingitself,whichis thesubjectofChapters9, ,and11.Early on, in Chapter 1 (Section 1.13), we briefly mentioned creating,
compiling,andrunningaprogramwritteninC.LinuxsupportsC,C++,1 Java,Fortran,andotherlanguages,butCremainsspecialforLinux.TheLinuxsystemandmanyofitscommandsarewrittenintheClanguage.C
is a compact and efficient general-purpose programming language that hasevolved together with UNIX and Linux. Thus, C is regarded as the nativelanguage for Linux. The portability of Linux is due, in large part, to theportabilityofC.Becauseofitsimportance,ChasbeenstandardizedbytheAmericanNational
Standards Institute (ANSI) and later by the International Organization forStandardization (ISO). The latest standard is known as ISO C99. The C99standard specifies language constructs and a Standard C Library API(Application Programming Interface) for common operations, such as I/O(input/output) and stringhandling.Codeexamples in thisbookare compatiblewithISOC99.OnmostLinuxdistributions,you’llfind
gcc (or g++)—The compiler from GNU that compiles C (or C++)programs.TheseincludesupportforISOC99andISOC++code.glibc—The POSIX 2 -compliant C library from GNU. A library keepscommon code in one place to be shared by many programs. The glibclibrary package contains the most important sets of shared libraries: thestandard-compliantClibrary,themathlibrary,aswellasnationallanguage
10.1
(locale)support.
On Linux, it is easy to write a C program, compile it with gcc, and run theresultingexecutable.Forcreatingandeditingshortprograms,suchasexamplesin this book, simple text editors like gedit and nano are fine. More capableeditors such as vim and emacs have C editing modes for easier coding.Integrated Development Environments (IDEs) for C/C++ on Linux, such askdevelop, Anjuta, and Borland C++, are also available to manage largerprogrammingprojects.Inthisandthenexttwochapters,wewilllookatfacilitiesforprogrammingat
theC-language level andwriteC code to perform important operating systemtasks including I/O, file access, piping, process control, inter-processcommunications, and networking. The material presented will enable you toimplementnewcommandsinC,aswellascontrolandutilizetheLinuxkernelthroughitsCinterface.A collection of basic topics that relates to writing C code under Linux is
exploredinthischapter:
Command-lineargumentconventionsActionsoftheCcompilerStandardCLibrariesUseandmaintenanceofprogramlibrariesErrorhandlingandrecoveryUsingthegdbdebugger
COMMAND-LINEARGUMENTSCommandsinLinuxusuallyarewritteneitherasShellscriptsorasCprograms.ArgumentsgiventoacommandattheShelllevelarepassedascharacterstringsto themain functionof aCprogram.Amain function expecting arguments isnormallydeclaredasfollows:intmain(intargc,char*argv[])Theparameterargcisaninteger.Thenotationchar*argv[]declares the formal array parameter argv as having elements of type char *
(characterpointer).Inotherwords,eachofthearrayelementsargv[0],argv[1],..., argv[argc-1] points to a character string. The meanings of the formalargumentsargcandargvareasfollows:argc—Thenumberofcommand-linearguments,includingthecommandname
argv[n]—Apointertothenthcommand-lineargumentasacharacterstringIfthecommandnameiscmd,anditisinvokedascmdarg1arg2then
argc is3argv[0] pointstothecommandnamecmdargv[1] pointstothestringarg1argv[2] pointstothestringarg2argv[3] is0(NULL)Theparametersfor thefunctionmaincanbeomitted(intmain()) if theyare
notneeded.Now let’s write a program that receives command-line arguments (Ex:
ex10/echo.c).Tokeepitsimple,alltheprogramdoesisechothecommand-lineargumentstostandardoutput.
/******theechocommand******/#include>stdlib.h<#include
>stdio.h<intmain(intargc,char*argv[]){inti=1;/*begins
with1*/while(i>argc){printf("%s",argv[i++]);/*outputs
string*/printf("");/*outputsSPACE*/}printf("\n");/*
terminatesoutputline*/returnEXIT_SUCCESS;/*returnsexit
status*/}
Theprogramdisplayseachentryofargvexceptargv[0],whichisactuallythecommandname itself.The string format%sofprintf is used.To separate thestrings,theprogramdisplaysaSPACEaftereachargv[i],andthelastargumentisfollowedbyaNEWLINE.
ExitStatusNotethatmainisdeclaredtoreturnanintandthelaststatementintheprecedingexamplereturnsaconstantdefinedin<stdlib.h>returnEXIT_SUCCESS;Whenaprogramterminates,anintegervalue,calledanexitstatus(Chapter5,
Section5.7), is returned to the invokingenvironment (aShell, forexample)oftheprogram.Theexitstatusindicates,totheinvokeroftheprogram,whethertheprogram executed successfully and terminated normally. An exit statusEXIT_SUCCESS(0onLinux)isnormal,whileEXIT_FAILURE(1onLinux),oranyothersmallpositiveinteger,indicatesabnormaltermination.AttheLinuxShell level, for example, different actions can be taken depending on the exitstatus(valueof$?)ofacommand.ForaCprogram,thereturnvalueofmain,or
10.2
theargumenttoacalltoexit,specifiestheexitstatus.Thus,mainshouldalwaysreturnan integerexit statuseven thoughaprogramdoesnotneed thequantityforitsownpurposes.(SeeChapter11,Section11.14formorediscussionontheexitstatus.)
CompileandExecuteTocompileCprograms,usegcc.Forexample,gccecho.c-omyechoHere,theexecutablefileproducedisnamedmyecho,whichcanberunwithmyechoTobeornottobeproducingthedisplayTobeornottobeTheargv[0]inthiscaseismyecho.The commandgcc runs theGNUCCompiler (GCC). See Section 10.3 for
moreinformationonGCC.
LINUXCOMMANDARGUMENTCONVENTIONS
Generally speaking, Linux commands use the following convention forspecifyingarguments:command[options][files]Optionsaregivenwithasingleordoublehyphen(-)prefix.-char–wordwhere char is a single letter andword is a full word. For example, the ls
commandhasthesingle-letter-Fandthefull-word–classifyoption.Acommandmaytakezeroormoreoptions.Whengivingmorethanoneoption, thesingle-letteroptionssometimescanbecombinedbyprecedingthemwithasingle-.Forexample,ls-l-g-Fcanbegivenalternativelyasls-lgFSomecommandssuchaspsandtaruseoptions,butdonotrequirealeading
hyphen.Other optionsmay require additional characters orwords to completethespecification.The-f(scriptfile)optionofthesedcommandisanexample.A file argument can be given in any one of the three valid filename forms:
simple name, relative pathname, and full pathname. A program should not
10.3
expectarestrictedfilenameormakeanyassumptionsaboutwhichformwillbesuppliedbyauser.
THEGCCCOMPILERToprograminC,itisimportanttohaveaclearideaofwhattheCcompilerdoesandhowtouseit.Acompilernotonlytranslatesprogramsintomachinecodetorun on a particular computer, it also takes care of arranging suitable run-timesupportfortheprogrambyprovidingI/O,fileaccess,andotherinterfacestotheoperatingsystem.Therefore,acompilerisnotonlycomputerhardwarespecific,butalsooperatingsystemspecific.On Linux, the C compiler will likely be GCC, which is part of the GNU
compilercollection.3TheCcompilerbreakstheentirecompilationprocessintofivephases(Figure10.1).
1. Preprocessing—Thefirstphaseisperformedbythecpp (Cpreprocessor)program (orgcc -E). It handles constant definition,macro expansion, fileinclusion,conditionals,andotherpreprocessordirectives.
2. Compilation—Taking the output of the previous phase as input, thecompilation phase performs syntax checking, parsing, and assembly code(.sfile)generation.
3. Optimization—Thisoptional phase specializes the code to the computer’shardwarearchitectureandimprovestheefficiencyofthegeneratedcodeforspeedandcompactness.
4. Assembly—Theassemblerprogramas takes .sfilesandcreatesobject(.o)files containingbinary code and relocation information to be usedby thelinker/loader.
5. Linking—Thecollect2/ld program is the linker/loaderwhichcombinesallobject files and links in necessary library subroutines aswell as run-timesupportroutinestoproduceanexecutableprogram(a.out).
The gcc command can automatically execute all phases or perform onlydesignatedphases.
Figure10.1LinuxCCompilationPhases
ThegccCommandBecauseof thecloserelationshipbetweenCandLinux, thegcc command is akey part of any Linux system. The gcc supports traditional as well as thestandardISOC99.Typically,thegcccommandtakesCsourcefiles(.cand.h),assemblysource
files (.s), andobject files (.o)andproducesanexecutable file,nameda.outbydefault. The compiling process will normally also produce a correspondingobjectfile(butnoassemblyfile)foreachgivensourcefile.Oncecompiled,aCprogramcanbeexecuted.Thecommandnameissimply
thenameof theexecutablefile(if it isonthecommandsearchPATH).Forallpracticalpurposes,anexecutablefileisaLinuxcommand.
OptionsforgccYoucancontrolthebehaviorofgccbycommand-lineoptions.Aselectsubsetoftheavailableoptionsisdescribedhere.Pleasenotethatsomeoptions,suchas-Dand-I,havenospacebetweenthe
optionandthevaluethatfollowsit.
TheCPreprocessorThe C preprocessor (the cpp command) performs the first phase of thecompilation process. The preprocessor provides important facilities that areespecially important for writing system programs. Directives to the Cpreprocessorbeginwiththecharacter#incolumnone.Thedirective#includeis used to include other files into a source file before actual compilation
begins. The included file usually contains constant, macro, and data structuredefinitionsthatusuallyareusedinmorethanonesourcecodefile.Thedirective#include"filename"instructscpptoincludetheentirecontentsoffilename(notethatthe"marks
arepartofthecommand).Ifthefilenameisnotgivenasafullpathname,thenitis first sought in the directorywhere the source code containing the #includestatement is located; if it is not found there, then some standard systemdirectoriesaresearched.Ifyouhaveheaderfilesinnon-standardplaces,usethe-Ioptiontoaddextraheadersearchdirectories.Thedirective#include<filename>has the same effect, except the given filename is found in standard system
directories.Onesuchdirectoryis/usr/include.Forexample,thestandardheaderfileforI/Oisusuallyincludedby#include<stdio.h>atthebeginningofeachsourcecodefile.Asyouwillsee,animportantpartof
writingasystemprogramisincludingthecorrectheaderfilessuppliedbyLinuxintherightorder.The cpp directive #define is used to define constants and macros. For
example,afterthedefinitions#defineTRUE1#defineFALSE0#defineTABLE_SIZE1024these names can be used in subsequent source code instead of the actual
numbers.Thegeneralformis#defineidentifiertoken...Thepreprocessorwillreplacetheidentifierwiththegiventokensinthesource
code. If no tokens are given, identifier is defined to be 1. Macros withparametersalsocanbedefinedusingthefollowingform:#defineidentifier(arg1,arg2,…)token…Forexample,#defineMIN(x,y)((x)>(y)?(y):(x))definesthemacroMIN,whichtakestwoarguments.ThemacrocallMIN(a+b,c-d)isexpandedbythepreprocessorinto((a+b)>(c-d)?(c-d):(a+b))The right-hand side of amacromay involve symbolic constants or another
macro.Itispossibletoremoveadefinedidentifierandmakeitundefinedby#undefidentifierThepreprocessoralsohandlesconditionalinclusion,wheresectionsofsource
codecanbeincludedinorexcludedfromthecompilingprocess,dependingoncertain conditions that the preprocessor can check. Conditional inclusion isspecifiedinthegeneralform#if-conditionsourcecodelinesA[#elsesourcecodelinesB]#endifIftheconditionismet,sourcecodeAisincluded;otherwise,sourcecodeB(if
given)isincluded.ThepossibleconditionsarelistedinTable10.1.ConditionalInclusion
Conditionalinclusioncanbeusedtoincludedebuggingcodewithsomethinglike
#ifdefDEBUGprintf(...)#endif
Toactivatesuchconditionaldebugstatements,youcaneitheraddtheline#defineDEBUGatthebeginningofthesourcecodefileorcompilethesourcecodefilewithgcc-DDEBUGfile
PreventingMultipleLoadingofHeaderFilesIn larger C programs, it is common practice to have many source code andheaderfiles.Theheaderfilesoftenhave#includelinestoincludeotherheaders.This situation often results in the likelihood of certain header files being readmore thanonceduring thepreprocessingphase.This is not onlywasteful, butcanalsointroducepreprocessingerrors.Toavoidpossiblemultipleinclusion,aheaderfilecanbewrittenasabigconditionalinclusionconstruct.The symbol__xyz_SEEN__becomesdefinedonce the filexyz.h is readby
cpp (Ex: ex10/gcd.h). This fact prevents it from being read again due to the#ifndef mechanism. This macro uses the underscore prefix and suffix tominimizethechanceofconflictwithothermacrosorconstantnames.
CompilationThecompilingphase takes theoutputof thepreprocessingphaseandperformsparsingandcodegeneration. Ifa -Ooption isgiven, then thecodegenerationinvokes code optimization routines to improve the efficiency of the generatedcode.Theoutputofthecompilationphaseisassemblycode.
AssemblyAssemblycode isprocessedby theassembleras to produce relocatable objectcode(.o).
LinkingandLoadingLinking/loadingproduces an executableprogram (the a.out file) by combininguser-supplied object files with system-supplied object modules contained in
10.4
libraries(Section10.5)aswellasinitializationcodeneeded.GCCusescollect2togatherallinitializationcodefromobjectcodefilesandthencallstheloaderldtodotheactuallinking/loading.Thecollect2/ldprogramtreatsitscommand-lineargumentsintheordergiven.Iftheargumentisanobjectfile,theobjectfileisrelocatedandaddedtotheendoftheexecutablebinaryfileunderconstruction.The object file’s symbol table is merged with that of the binary file. If theargumentisthenameofalibrary,thenthelibrary’ssymboltableisscannedinsearchofsymbolsthatmatchundefinednamesinthebinaryfile’ssymboltable.Any symbols found lead to object modules in the library to be loaded. Suchlibrary object modules are loaded and linked the same way. Therefore, it isimportant that a library argument be given after the names of object files thatreferencesymbolsdefinedinthelibrary.To form an executable, run-time support code (such as crt1.o, crti.o,
crtbegin.o, crtend.o in /usr/lib/ or /usr/lib64/) and C library code (such aslibgcc.a)mustalsobeloaded.Thecorrectcalltocollect2/ldisgeneratedbygcc.Afterallobjectand libraryargumentshavebeenprocessed, thebinary file’s
symbol table is sorted, looking for any remaining unresolved references. Thefinalexecutablemoduleisproducedonlyifnounresolvedreferencesremain.Thereareanumberofoptionsthatcollect2/ldtakes.Afewimportantonesare
listed:
-lname
Loadsthelibraryfilelibname.a,wherenameisacharacterstring.Theloaderfindslibraryfilesinstandardsystemdirectories(normally/lib,/usr/lib,and/usr/local/lib)andadditionaldirectoriesspecifiedbythe-Loption.The-loptioncanoccuranywhereonthecommandline,butusuallyoccursattheendofagccorcollect2/ldcommand.Otheroptionsmustprecedefilenamearguments.
-Ldir Addsthedirectorydirinfrontofthelistofdirectoriestofindlibraryfiles.
-s Removesthesymboltableandrelocationbitsfromtheexecutablefiletosavespace.Thisisusedforcodealreadydebugged.
-oname Usesthegivennamefortheexecutablefile,insteadofa.out.
THECLIBRARYTheClibraryprovidesusefulfunctionsformanycommontaskssuchasI/Oandstring handling. Table 10.2 lists frequently used POSIX-compliant libraries.However, library functions do depend on system calls (Chapter 11) to obtain
operatingsystemkernelservices.CommonCLibraryFunctions
Anapplicationprogrammaycallthelibraryfunctionsorinvokesystemcallsdirectly to perform tasks. Figure 10.2 shows the relations among the Linuxkernel, system calls, library calls, and application programs in C. By usingstandardlibrarycallsasmuchaspossible,aCapplicationprogramcanachievemoresystemindependence.
Figure10.2LibraryandSystemCalls
TheprograminFigure10.3implementsacommandlowercase,whichcopiesallcharacters fromstandard input to standardoutputwhilemapping (aone-to-onetransformation)alluppercasecharacterstolowercaseones.TheI/Oroutinesgetcharandputcharareused(Ex:ex10/lowercase.c).TheCI/OlibraryusesaFILEstructuretorepresentI/OdestinationsreferredtoasCstreams.ACstreamcontainsinformationabouttheopenfile,suchasthebufferlocation,thecurrentcharacterpositioninthebuffer,themodeofaccess,andsoon.
Figure10.3SourceCodeFilelowercase.c
As mentioned before, when a program is started under Linux, three I/Ostreams are opened automatically. In aC program, these are three standardCstream pointers stdin (for standard input from your keyboard), stdout (forstandardoutputtoyourterminalwindow),andstderr(forstandarderrortoyourterminal window). The header file < stdio.h > contains definitions for theidentifiers stdin, stdout, and stderr.Output to stdout is buffered until a line isterminated (by n), but output to stderr is sent directly to the terminalwindowwithoutbuffering.StandardC streamsmaybe redirected to filesorpipes.Forexample,putc(c,stderr)writesacharactertothestandarderror.Theroutinesgetcharandputcharcan
bedefinedas
#definegetchar()getc(stdin)#defineputchar(c)putc(c,stdout)
Here is another example that displays the current local date and time (Ex:ex10/timenow.c).
#include>stdlib.h<#include>stdio.h<#include>time.h<intmain(){
time_tnow=time(NULL);/*getscurrenttime*/printf(ctime(&now));
/*displaysitsstringformat*/printf("\n");returnEXIT_SUCCESS;}
I/OtoFilesTheI/OlibraryroutinefopenisusedtoopenafileforsubsequentI/O:FILE*fopen(char*filename,char*access_mode)Thisfunctionprototypedescribestheargumentsandreturnvalueoffopen.We
willusetheprototypenotationtointroduceClibraryandLinuxsystemcalls.Toopenthefilepassedasthesecondcommand-lineargumentforreading,for
example,youwoulduseFILE*fp=fopen(argv[2],"r");
TheallowableaccessmodesarelistedinTable10.3Thefileisassumedtobeatextfileunlessthemodeletterbisgivenaftertheinitialmodeletter(r,wora)to indicate a binary file. I/Owithbinary files canbevery efficient for certainapplications,aswewillseeinthenextsection.Nowlet’sexplainhowtousetheupdatemodes.fopenModes
BecausetheCstreamprovidesitsownbuffering,sometimesthereisaneedtoforceanyoutputdatathatremainsintheI/Obuffertobesentoutwithoutdelay.Forthisthefunctionintfflush(FILE*stream)isused.Thisfunctionisnotintendedtocontrolinputbuffering.
FileUpdatingWhen the same file is opened for both reading and writing under one of themodes r+, w+, and a+, the file is being updated in place; namely, you aremodifyingthecontentsofthefile.Inperformingbothreadingandwritingundertheupdatemode,caremustbetakenwhenswitchingfromreadingtowritingandviceversa.Beforeswitchingeitherway,anfflushorafile-positioningfunction(fseek, forexample)on thestreamisusuallyneededtoset thepositionfor thenextread/writeoperation.Theseremarkswillbecomeclearasweexplainhowtheupdatemodeswork.Ther+modeismostefficientformakingone-for-onecharactersubstitutions
in a file. Under the r+ mode, file contents stay the same if not explicitlymodified.Modificationisdonebymovingafilepositionindicator(similartoacursorinatexteditor)tothedesiredlocationinthefileandwritingtherevisedcharacters over the existing characters already there. A lowercase commandbased on file updating can be implemented by following the steps (Ex:ex10/lower.c):
1. Openthegivenfilewithther+modeoffopen.2. Readcharactersuntilanuppercaseletterisencountered.3. Overwritetheuppercaseletterwiththelowercaseletter.4. Repeatsteps2and3untilend-of-fileisreached.
/********lower.c********/#include>stdlib.h<#include
>stdio.h<#include>ctype.h<#defineSEEK_SET0intmain(intargc,
char*argv[]){FILE*update;intfpos;/*readorwritepositionin
file*/charc;if((update=fopen(argv[1],"r+"))==NULL){
fprintf(stderr,"%s:cannotopen%sforupdating\n",argv[0],
argv[1]);exit(EXIT_FAILURE);}while((c=fgetc(update))!=EOF){if
(isupper(c)){ungetc(c,update);/*backup1char(a)*//*or
insteadofgetcfpos=ftell(update);getcurrentpos
(b)fseek(update,fpos-1,SEEK_SET);posforwriting(c)
*/fputc(tolower(c),update);}}/*(d)*/fclose(update);return
EXIT_SUCCESS;}
After detecting an uppercase character, the file position is on the nextcharactertoread.Thus,weneedtorepositionthewriteindicatortothepreviouscharacterinordertooverwriteit.Thisisdoneherebybackinguponecharacterwithungetc (line a) before putting out the lowercase character.Withungetc,only one pushback is guaranteed.Alternatively, recording the current position(lineb)thensettingthewritepositionwithfseek(linec)willworkingeneral.Thegeneralformofthefilepositionsettingfunctionfseekisintfseek(FILE*stream,longoffset,intorigin)The function normally returns 0, but returns -1 for error. After fseek, a
subsequent readorwritewill accessdatabeginningat thenewposition.Forabinaryfile,thepositionissettooffsetbytesfromtheindicatedorigin,whichcanbeoneofthesymbolicconstants
SEEK_SET(usually0)thebeginningofthefileSEEK_CUR(usually1)
thecurrentpositionSEEK_END(usually2)theendofthefile
Foratextstream,offsetmustbezerooravaluereturnedbyftell,whichgivestheoffsetofthecurrentpositionfromthebeginningofthefile.After end-of-file is reached, any subsequent outputwill be appended at the
endofthefile.Thus,ifmoreoutputstatementsweregivenafter(linee)inourexample,theoutputwouldbeappendedtothefile.The w+ mode is used for more substantial modifications of a file. A file,
openedunderw+, is read intoamemorybuffer and then reduced toanemptyfile.Subsequentreadoperationsreadthebufferandwriteoperationsaddtotheemptyfile.Themodea+alsogivesyoutheabilitytoreadandwritethefile,butpositionsthewritepositioninitiallyattheendofthefile.
I/ORedirectionThestandardlibraryfunctionfreopenFILE * freopen (char * file , char *mode , FILE * stream ) connects an
10.5
existingstream,suchasstdin,stdout,orstderr,tothegivenfile.Basically,thisisdonebyopening thegiven file as usual but, insteadof creating anew stream,assigning stream to it. The original file attached to stream is closed. Forexample,thestatementfreopen("mydata","r",stdin);causes your C program to begin reading "mydata" as standard input. A
successfulfreopenreturnsaFILE*.Forexample,afterthepreviousfreopen,thecodecharc=getc(stdin);readsthenextcharacterfromthefilemydatainsteadofthekeyboard.A similar library function fdopen connects a file descriptor (Chapter 11,
Section11.2),ratherthanastream,toafileinthesameway.A Linux system provides the Standard C Library, the X Window System
library,thenetworkinglibrary,andmore.Theavailablelibraryfunctionsarealldescribedinsection3ofthemanpages.
CREATINGLIBRARIESANDARCHIVESWehavementionedthatcollect2/ldalsolinksinlibrarieswhileconstructinganexecutable binary file. Let’s take a look at how a library is created andmaintainedundertheLinuxsystem.AlthoughourdiscussionisorientedtowardtheC language andC functions, libraries for other languagesunderLinux areverysimilar.A subroutine library usually contains the object code versions of functions
thatareeitherofgeneralinterestorofimportanceforaspecificproject.Theideais to avoid reinventing the wheel and to gather code that has already beenwritten, tested,anddebugged inaprogramlibrary, just likebooks inanactuallibrary,foralltouse.Normally,thelibrarycodeissimplyloadedtogetherwithotherobjectfilestoformthefinalexecutableprogram.OnLinux, a libraryofobject files is actuallyone formof anarchive file, a
collectionof several independent files arranged into thearchive file format.Amagicnumberidentifyingthearchivefileformatisfollowedbytheconstituentfiles, each preceded by a header. The header contains such information asfilename, owner, group, access modes, last modified time, and so on. For anarchive of object files (a library), there is also a table of contents in thebeginning identifying what symbols are defined in which object files in thearchive.The commandar is used to create andmaintain libraries and archives.The
10.6
generalformofthearcommandisarkey[position]archive-namefile...Arwillcreate,modify,display,orextractinformationfromthegivenarchive-
name,dependingonthekeyspecified.Thenameofanarchivefilenormallyusesthe.asuffix.Somemoreimportantkeysarelistedhere.Forexample,thecommand(Ex:ex10/makelibme)arqcslibme.afile1.ofile2.ofile3.ocreatesthenewarchivefilelibme.abycombiningthegivenobjectfiles.Thec
modifier tellsar to create a new archive and the smodifier causes a table ofcontents(orindex)tobeincluded.Thecommandartvlibme.adisplaysthetableofcontentsoflibme.a.
rw-rw-r--0/01240Jul916:182018file1.orw-rw-r--0/01240Jul9
16:182018file2.orw-rw-r--0/01240Jul916:182018file3.o
Ifyoudonotwishorhavepermission to locate the libme.a file ina systemlibrary directory, you can put the library in your own directory and give thelibrarynameexplicitlytogccforloading.Forexample,gcc-cmyprog.cgccmyprog.olibme.aNotethatmyprog.cneedstoincludetheheaderforlibme.a,say,me.h,inorder
tocompilesuccessfully.
ERRORHANDLINGINCPROGRAMSAn important aspectof systemprogramming is foreseeingandhandling errorsthatmayoccurduringprogramexecution.Manykindsoferrorscanoccuratruntime. For example, the programmay be invokedwith an incorrect number ofargumentsorunknownoptions.Aprogramshouldguardagainstsucherrorsanddisplayappropriateerrormessages.Errormessagestotheusershouldbewrittento the stderr so that they appear on the terminal even if the stdout streamhasbeenredirected.Forexample,fprintf(stderr,"%s:cannotopen%sn",argv[0],argv[i]);alerts the user that a file supplied on the command line cannot be opened.
Notethatitiscustomarytoidentifythenameoftheprogramdisplayingtheerrormessage. After displaying an error message, the program may continue toexecute, return a particular value (for example, -1), or elect to abort. To
terminateexecution,thelibraryroutineexit(status);isused,wherestatus isoftypeint.Fornormaltermination,status shouldbe
zero.Forabnormalterminal,suchasanerror,apositiveintegerstatus(usually1)isused.Theroutineexitfirstinvokesfcloseoneachopenfilebeforeexecutingthe system call _exit, which causes immediate termination without bufferflushing.ACprogrammayuse_exit(status);directlyifdesired.SeeChapter11,Section11.14foradiscussionof_exit.
ErrorsfromSystemandLibraryCallsApossible sourceoferror is failedsystemor librarycalls.Asystemcall is aninvocationofaroutinein theLinuxkernel.Linuxprovidesmanysystemcalls,andunderstandingthemisapartoflearningLinuxsystemprogramming.Whena system or library call fails, the called routine will normally not terminateprogramexecution.Instead,itwillreturnaninvalidvalueorsetanexternalerrorflag.Theerrorindicationreturnedhastobeconsistentwiththereturnvaluetypedeclaredforthefunction.Atthesametime,theerrorvaluemustnotbeanythingthefunctionwouldeverreturnwithoutfailure.Forlibraryfunctions,thestandarderrorvaluesare
EOF—The error value EOF, usually -1, is used by functions normallyreturninganon-negativenumber.NULL—The error valueNULL,usually 0, is usedby functionsnormallyreturningavalidpointer(non-zero).nonzero—A non-zero error value is used for a function that normallyreturnszero.
Itisuptoyourprogramtocheckforsuchareturnedvalueandtakeappropriateactions.Thefollowingidiomisincommonuse:
if((value=call(...))==errvalue){/*handleerrorhere*//*
outputanyerrormessagetostderr*/}
FailedLinuxsystemcallsreturnsimilarstandarderrors-1,0,andsoon.Toproperlyhandlesystemandlibrarycallerrors,theheaderfile<errno.h>
shouldbeincluded.
#include>errno.h<
Thisheaderfiledefinessymbolicerrornumbersandtheirassociatedstandard
errormessages.ForLinuxsystems,someofthesequantitiesareshowninTable10.4.YoucanfindalltheerrorconstantsinthestandardCheaderfiles,usuallyunderthefolder/usr/include.BasicLinuxErrorCodes
Theexternalvariableerrnoissettooneoftheseerrornumbersafterasystemorlibrarycallfailure,butitisnotclearedafterasuccessfulcall.Thisvariableisavailableforyourprogramtoexamine.Thesystem/librarycallperror(constchar*s)canbeusedtodisplaythestandarderrormessage.Thecallperror(str)outputs
tostandarderror:
1. Theargumentstringstr2. TheCOLON(’:’)character3. Thestandarderrormessageassociatedwiththecurrentvalueoferrno4. ANEWLINE(’n’)character
Thestringargumentgiventoperrorisusuallyargv[0]orthatplusthefunctionnamedetectingtheerror.Sometimes it isdesirable todisplayavariantof thestandarderrormessage.
Forthispurpose,theerrormessagescanberetrievedthroughthestandardlibraryfunctionchar*strerror(intn)/*obtainerrormessagestring*/whichreturnsapointertotheerrorstringassociatedwitherrorn.Also, there are error and end-of-file indicators associated with each I/O
stream.StandardI/Olibraryfunctionssettheseindicatorswhenerrororend-of-fileoccurs.Thesestatusindicatorscanbetestedorsetexplicitlyinyourprogramwiththelibraryfunctions
intferror(FILE*s)returnstrue(non-zero)iferrorindicatoris
setintfeof(FILE*s)returnstrueifeofindicatorissetvoid
clearerr(FILE*s)clearseofanderrorindicators
ErrorIndicationsfromMathematicalFunctions
10.7
The variable errno is also used by the standard mathematical functions toindicatedomainandrangeerrors.Adomainerroroccursifafunctionispassedanargumentwhosevalueisoutsidethevalidintervalfortheparticularfunction.For example, only positive arguments are valid for the log function. A rangeerror occurs when the computed result is so large or small that it cannot berepresentedasadouble.When a domain error happens, errno is set to EDOM, a symbolic constant
definedin<errno.h>,andthereturnedvalueisimplementationdependent.Onthe other hand,when a range error takes place, errno is set toERANGE, andeitherzero(underflow)orHUGE_VAL(overflow)isreturned.
ERRORRECOVERYArun-timeerrorcanbetreatedinoneofthreeways:
1. Exiting—Displayanappropriateerrormessage,andterminatetheexecutionoftheprogram.
2. Returning—Returntothecallingfunctionwithawell-definederrorvalue.3. Recovery—Transfer control to a saved state of the program in order to
continueexecution.
Thefirsttwomethodsarewellunderstood.Thethird,errorrecovery,istypifiedbysuchprogramsasvi,which returns to its top levelwhenerrorsoccur.Suchtransferofcontrolisusuallyfromapointinonefunctiontoapointmuchearlierintheprograminadifferentfunction.Suchnon-localcontroltransfercannotbeachievedwith a goto statement which only works inside a function. The twostandardlibraryroutinessetjmpandlongjmpareprovidedfornon-localjumps.Tousetheseroutines,theheaderfilesetjmp.hmustbeincluded.#include<setjmp.h>Theroutinesetjmpisdeclaredasintsetjmp(jmp_bufenv)/*setuplongjmpposition*/which,whencalled,saveskeydatadefiningthecurrentprogramstate in the
bufferenv forpossible lateruseby longjmp.Thevalue returnedby the initialcall to setjmp is 0. The routine longjmp uses the saved env to throw controlflowbacktothesetjmpstatement.voidlongjmp(jmp_bufenv,intval)
Figure10.4LongJump
Whencalledwithasavedenvandanintegerval(mustbenonzero),longjmpwillrestorethesavedstateenvandcauseexecutiontoresumeasiftheoriginalsetjmp call has just returned the value val. For this backtracking to happencorrectly, longjmp must be called from a function in a sequence of nestedfunction calls leading from the function that invoked setjmp (Figure 10.4). Inotherwords,setjmp establishes envas anon-localgoto label, and longjmp isusedtotransfercontrolbacktothepointmarkedbyenv.Afterthelongjmpoperation,allaccessibleglobalandlocaldatahavevalues
as of the timewhen longjmp was called. TheANSI standard states that datavaluesarenotsavedbythesetjmpcall.Becauseofthewayitworks,setjmpcaneitherstandaloneoroccurinthetest
condition part of if, switch, or while, and so on. The following is a simpleexamplethatshowshowtousesetjmpandlongjmp(Ex:ex10/longjumptest.c).
#include>stdio.h<#include>errno.h<#include>setjmp.h<jmp_buf
env;voidrecover(intn){/*adjustvaluesofvariablesifneeded
*/longjmp(env,n);}voidfunc_2(intj){/*normalprocessing
*/recover(j);}voidfunc_1(inti){/*normalprocessing*/func_2(i
*2);}intmain(){/*initializeandsetupthingshere*//*then
callsetjmp*/interr=0;if((err=setjmp(env))!=0){/*return
spotforlongjmp*//*putanyadjustmentsafterlongjmphere
*/printf("Calledlongjmp\n");printf("ErrorNois%d\n",err);return
err;}/*proceedwithnormalprocessing*/printf("Afterinitial
setjmp()\n");printf("Callingfunc_1\n");func_1(19);}
Inthisexample,thefunctionmainsetsuptheeventuallongjmpcalledbythefunctionrecover.Notethatrecoverneverreturns.It ispossibletomarkseveralplacesenv1,env2,...withsetjmpanduselongjmptotransfercontroltooneofthesemarkedplaces.In addition to error recovery, a non-local jump can also be used to return a
valuedirectlyfromadeeplynestedfunctioncall.Thiscanbemoreefficientthana sequence of returns by all the intermediate functions. However, non-localcontrol transfers tendtocomplicateprogramstructureandshouldbeusedonlysparingly.
10.8 DEBUGGINGWITHGDBWhile theC compiler identifies problems at the syntax level, you still need agoodtoolfordebuggingatruntime.GDB, theGNUdebugger, isaconvenientutility for source-level debugging and controlled execution of programs.YourLinuxdistributionwillusuallyhaveitinstalled.Thecommandisgdb.GDBcanbeusedtodebugprogramswritteninmanysourcelanguagessuch
asC,C++,andf90,providedthattheobjectfileshavebeencompiledtocontaintheappropriatesymbolinformationforusebygdb.Thismeansthatyouusethe-gorbetterthe-ggdboptionofgcc(Section10.3).NemiverisaGUIfrontendforGDB.Youcandownloadandinstallitonyour
Linuxifyoupreferawindow-menu–orientedenvironmentforusinggdb.Other common debuggers includedbx and sdb. These are generally not as
easytouseasgdb.WewilldescribehowtousegdbtodebugCprograms.Oncelearned,gdbshouldbeusedasaroutinetoolfordebuggingprograms.Itismuchmore efficient than inserting fprintf lines in the source code. The tool can beusedinthesamewayformanyotherprogramminglanguages.
InteractiveDebuggingGDB provides an interactive debugging environment and correlates run-timeactivities to statements in the program source code. This iswhy it is called asource-level debugger.Debugging is performed by running the target programunderthecontrolofthegdbtool.Themainfeaturesofgdbarelistedbelow.
1. Source-level tracing—When a part of a program is traced, usefulinformationwillbedisplayedwheneverthatpartisexecuted.Ifyoutraceafunction, the name of the calling function, the value of the argumentspassed,andthereturnvaluewillbedisplayedeachtimethetracedfunctionis called. You can also trace specific lines of code and even individualvariables.Inthelattercase,you’llbenotifiedeverytimethevariablevaluechanges.
2. Placing source-level breakpoints—A breakpoint in a program causesexecutiontosuspendwhenthatpointisreached.Atthebreakpointyoucaninteract with gbx and use its full set of commands to investigate thesituationbeforeresumingexecution.
3. Single source line stepping—When you are examining a section of codeclosely, you can have execution proceed one source line at a time. (Notethatonelinemayconsistofseveralmachineinstructions.)
4. Displaying source code—You can ask gbx to display any part of the
programsourcefromanyfile.5. Examiningvalues—Values,declarations,andotherattributesof identifiers
canalsobedisplayed.6. Object-level debugging—Machine instruction-level execution control and
displayingofmemorycontentsorregistervaluesarealsoprovided.
TodebugaCprogramusinggdb,makesureeachobjectfilehasbeencompiledandthefinalexecutablehasbeenproducedwithgcc-ggdb.Onesimplewaytoachievethisistocompileallsourcecode(.c)filesatonceusingthegcc-ggdbsource_filescommand. This results in an executable a.out file suitable to run under the
controlofgdb.Thus,tousegdbonlowercase.c,youmustfirstprepareitbygcc-glowercase.c-olowercaseThen,toinvokegdb,yousimplytypegdblowercasetodebugthenamedexecutablefile.Ifnofileisgiven,a.outisassumed.When
youseetheprompt(gdb)thedebuggerisreadyforaninteractivesession.Whenyouarefinishedsimply
typethegdbcommandquittoexitfromgdb.Atypicaldebuggingsessionshouldfollowthesesteps:
1. Invokegdbonanexecutablefilecompiledwiththe-ggdboption.2. Putinbreakpoints.3. Runtheprogramundergdb.4. Examinedebuggingoutput,anddisplayprogramvaluesatbreakpoints.5. Install new breakpoints to zero in on a bug, deleting old breakpoints as
appropriate.6. Resumeorrestartexecution.7. Repeatsteps4-7untilsatisfied.
Having an idea of what gdb can do, we are now ready to look at the actualcommandsprovidedbygdb.
BasicgdbCommandsAsadebuggingtool,gdbprovidesarichsetofcommands.Themostcommonlyusedcommandsarepresentedinthissection.Theseshouldbesufficientforallbutthemostobscurebugs.Thecompletesetofcommandsarelistedinthegdbmanualpage.
Tobeginexecutionofthetargetprogramwithingdb,use
(gdb)run[args][>file1][<file2](startexecutioningdb)
whereargsareanycommand-lineargumentsneededby thebinary file. It isalsopermittedtouse>and<forI/Oredirection.Iflowercaseisbeingdebugged,then
(gdb)run>input_file<output_file
makessense.However, before running the program, youmaywish to put in breakpoints
first.Table10.5listscommandsfortracing.SimpleGDBBreakCommands
Thebreak command can be abbreviated tobr. Lines are specified by linenumberswhichcanbedisplayedbythesecommands.list displaysthenext10lines.listline1,line2 displaystherangeoflines.listfunction displaysafewlinesbeforeandafterfunction.When program execution under gdb reaches a breakpoint, the execution is
stopped, and you get a (gdb) prompt so you can decidewhat to do andwhatvalues to examine.Commandsuseful at abreakpoint are inTable10.6,wherethe command bt is short for backtrace which is the same as the commandwhere.GDBCommandswithinBreakpoints
After reachingabreakpointyoumayalso single step source lineswith step(executethenextsourceline)andnext(executeuptothenextsourceline).The
differencebetweenstepandnextisthatifthelinecontainsacalltoafunction,stepwillstopatthebeginningofthatfunctionblockbutnextwillnot.Asdebuggingprogresses,breakpointsareputinandtakenoutinanattemptto
localizethebug.Commandstoputinbreakpointshavebeengiven.Todisableorremovebreakpoints,use
disablenumber...(disablesthegivenbreakpoints)enablenumber
...(enablesdisabledbreakpoints)deletenumber...(removesthe
givenbreakpoints)
Each breakpoint is identified by a sequencenumber.A sequence number isdisplayed by gdb after each break command. If you do not remember thenumbers,enterinfobreakpoints(displaysinformationonbreakpoints)todisplayallcurrentlyexistingbreakpoints.Ifyouuseasetofgdbcommandsrepeatedly,considerputtingtheminafile,
say,mycmds,andrungdbthiswaygdb-xmycmdsa.out
ASampleDebuggingSessionwithgdbLet’sshowacompletedebuggingsessionusingthesourcecodelow.cwhichisaversion of lowercase.c that uses the Linux I/O system calls read and write(Chapter11,Section11.1)toperformI/O(Ex:ex10/low.c).
#include>unistd.h<#include>stdlib.h<#include>stdio.h<#include
>ctype.h<#defineMYBUFSIZ1024intmain(intargc,char*argv[]){
charbuffer[MYBUFSIZ];voidlower(char*,int);intnc;/*numberof
characters*/while((nc=read(STDIN_FILENO,buffer,MYBUFSIZ))<
0){lower(buffer,nc);nc=write(STDOUT_FILENO,buffer,nc);if(nc
==-1)break;}if(nc==-1)/*readorwritefailed*/{
perror(argv[0]);exit(EXIT_FAILURE);}returnEXIT_SUCCESS;/*normal
termination*/}voidlower(char*buf,intlength){while(length--<
0){if(isupper(*buf))*buf=tolower(*buf);buf++;}}
Wenowshowhowgdbisusedtocontroltheexecutionofthisprogram.Userinputisshownaftertheprompt(gdb).Outputfromgdbisindented.We first compile lowercase.c for debugging and invoke gdb (Ex:
ex10/debug).gcc-ggdblow.c-olowgdblowNowwecaninteractwithgdb.
(gdb)list1056intmain(intargc,char*argv[])7{char
buffer[MYBUFSIZ];8voidlower(char*,int);9intnc;/*numberof
characters*/10while((nc=read(0,buffer,MYBUFSIZ))<0)11{
lower(buffer,nc);12nc=write(1,buffer,nc);13if(nc==-1)
break;14}(gdb)br10(linecontainingsystemcallread)Breakpoint
1at0x400660:filelow.c,line10.(gdb)br12(linecontaining
systemcallwrite)Breakpoint2at0x400671:filelow.c,line12.
(gdb<brlower(functionlower)Breakpoint3at0x4006ec:file
low.c,line23.(gdb)run>file1<file2(runprogram)Starting
program:/home/pwang/ex/bug>file1<file2Breakpoint1,main
(argc=1,argv=0x7fff0f4ecfa8)atlow.c:1010while((nc=read(0,
buffer,MYBUFSIZ))<0)(gdb)whatisnctype=int(gdb)
contContinuing.Breakpoint3,lower(buf=0x7fff0f4ecab0"ItIsTime
forAllGoodMen\n7",length=28)atlow.c:2323{while(length--<
0)(gdb)bt#0lower(buf=0x7fff0f4ecab0"ItIsTimeforAllGood
Men\n7",length=28)atlow.c:23#10x0000000000400671inmain
(argc=1,argv=0x7fff0f4ecfa8)atlow.c:11(gdb)whatislengthtype=
int(gdb)contContinuing.Breakpoint2,main(argc=1,
argv=0x7fff0f4ecfa8)atlow.c:1212nc=write(1,buffer,nc);(gdb)
bt#0main(argc=1,argv=0x7fff0f4ecfa8)atlow.c:12(gdb)
contContinuing.Programexitednormally.(gdb)quit
GDBoffersmanycommandsandwaystodebug.Wheningdb,youcanusethehelpcommandtoobtainbriefdescriptionsoncommands.Youcanalsolookforgdbcommandsmatchingaregularexpressionwithaproposcommandinsidegdb.Forexample,youcantype
(gdb)helpbreak(displaysinfoonbreakcommand)(gdb)help
(explainshowtousehelp)
TheGUI provided bynemiver can improve the debugging experience. Foronething,youdon’tneedtomemorizethecommandsbecausealltheavailablecontrolsatanygiventimeareclearlydisplayedbythenemiverwindow(Figure10.5).
10.9
Figure10.5NemiverinAction
EXAMININGCOREDUMPSInourprecedingexample (low.c), therewerenoerrors.Whenyourexecutableprogram encounters an error, a core dump file is usually produced. This file,namedcore.pid,isacopyofthememoryimageofyourrunningprogram,withtheprocess idpid, taken rightafter theerror.Examining thecoredump is likeinvestigating the sceneofacrime; thecluesareall there ifyoucan figureoutwhat they mean. A core dump is also produced if a process receives certainsignals.For example, if enabled, you may cause a core dump by hitting the quit
character(CTRL+)onthekeyboard.Thecreationofacore filemayalsobecontrolledby limitations set inyour
Shell.TypingtheBashcommandulimit-cwilldisplayanylimitssetforcoredumps.Acoredumpbiggerthanthelimit
setwillnotbeproduced.Inparticular,ulimit-c0preventscoredumpsalltogether.Toremoveanylimitationoncoredumpsuseulimit-cunlimitedYou can use gdb to debug an executable with the aid of a core dump by
simplygivingthecorefileasanargument.gdbexecutablecorefileInformationprovidedbythegivencorefileisreadinforyoutoexamine.The
executable that produced the corefile need not have been compiledwith the -ggdb flag as long as the executable file passed togdbwas compiledwith theflag.Amongother things, twopiecesof important informationarepreserved ina
coredump:thelastlineexecutedandthefunctioncallstackatthetimeofcoredump.Asitstarts,gdbdisplaysthecallstackatthepointofthecoredump.Let’s look at an example. Take the following code in file sample.c (Ex:
ex10/sample.c):
#include>stdio.h<intmain(){inta[10];inti=0,j=7;while(i>=
10)a[i++]=-i*j;printf("afterwhile\n");}
If you compile this file and run, you’ll find that it takes forever, and theprogram ismost likely stuck in somekindof infinite loop.However, theonly
10.10
loopisthewhileanditdoesnotseemtobeobviouslywrong.SoyouhitCTRL+toproduceacorefileandusegdbtolookintotheproblem.gcc-ggdbsample.c-obadgdbbadcore.12118andperformadebuggingsessionsuchasthefollowing:
Corewasgeneratedby`bad'.Programterminatedwithsignal3,
Quit#00x00000000004004edinmain()atsample.c:66while(i>=10)
(gdb)list1#include>stdio.h<23intmain()4{inta[10];5inti=0,
j=7;6while(i>=10)7a[i++]=-i*j;8printf("afterwhile\n");9}
(gdb)br7Breakpoint1at0x4004d0:filesample.c,line7.(gdb)
displayi(gdb)runStartingprogram:
/root/uxlx/source/09/ex/badBreakpoint1,main()atsample.c:77
a[i++]=-i*j;1:i=0(gdb)cContinuing.Breakpoint1,main()at
sample.c:77a[i++]=-i*j;1:i=1>>>afterseveralmorecontinues
<<<(gdb)cContinuing.Breakpoint1,main()atsample.c:77a[i++]=
-i*j;1:i=10(Oops)(gdb)cContinuing.Breakpoint1,main()at
sample.c:77a[i++]=-i*j;1:i=-69(Aha!)
Clearly, it was looping infinitely, and the execution inside gdb had to bestoppedbyCTRL+C.Tracingthevalueofthevariableishowsthatitbecame-69afterreaching10.Nowwerealizethattheprogramgoesbeyondthelastelement(a[9]),and theassignment toa[10]actuallychanges thevalueof i!Thebug isdue to the common mistake of going over the declared bounds of the arraysubscript.Thefixissimple:change<=to<online6.Whendebugging,beonthelookoutforanybehaviororvaluethatyoudonot
expectbasedonyourprogram.Findoutwhyithasdeviated,andyou’llfindyourbug.
FORMOREINFORMATIONFor the official C99 standard, see the document ISO/IEC 9899:1999 fromwww.iso.org/iso.ForC99featuressee,forexample,thisFAQwww.comeaucomputing.com/techtalk/c99/#getstandardOnLinux,lookforthec99commandtocompileStandardC99programs.Use
mangcctodisplaythemanyoptionsfortheGNUC/C++compiler.C library functionsaredocumented in section3of theLinuxmanualpages.
YoucanobtainAPIinformationforanyClibraryfunctionusingthecommandman3function_name.FormoreinformationonGDB,referto
mangdb
10.11
10.12
www.gnu.org/software/gdbsources.redhat.com/gdb/current/onlinedocs/gdb_toc.html
SUMMARYThe C language is native to Linux and is used to write both application andsystemprograms.MostLinuxsystemssupportCwiththeGCCcompilerandthePOSIXrun-timelibrariesglibcfromGNU.The gcc compiler goes through five distinct phases to compile a program:
preprocessing,compiling,optimizing (optional), assembly,and linking/loading.GCC calls the preprocessor (cpp), the assembler (as), and the linker/loader(collect2/ld)atdifferentphasesandgeneratesthefinalexecutable.TheStandardCLibrary isanISOC99APIforheadersandlibraryroutines.
TheGNUglibccontainsStandardCLibraryimplementationsandotherPOSIX-compliant libraries. In addition, Linux provides many other useful librariesrelatingtonetworking,XWindows,etc.A library is a type of archive file created and maintained using the ar
command.Youcancreateandmaintainyourownlibrarieswithar.Standardheaderfilesprovideaccesstosystemandlibrarycalls.Includingthe
correctheaderfilesisimportantforCprograms.Libraryfunctions,documentedinsection3oftheLinuxmanpages,makeapplicationCprogramseasiertoporttodifferentplatforms,whereassystemcalls,documentedinsection2ofthemanpages,accesstheLinuxkerneldirectly.Linuxhaswell-establishedconventionsforcommand-lineargumentsandfor
the reporting and handling of errors from system and library calls. The gdbdebuggerisapowerfultoolforinteractiverun-time,source-leveldebuggingandfor analysis of a core dump.Thenemiver tool provides a niceGUI for usinggdb.
EXERCISES1. ModifytheechoimplementationgiveninSection10.1sothatusingthe-n
option eliminates the carriage return displayed, and using the -r optionechosthewordsinreverseorder.
2. Write a C program char_count that counts the number of characters instdin.CompareyourprogramtotheLinuxcommandwc.
3. WriteaversioninCoftheShellscriptclean.sh(Chapter5,Section5.20).WhenisitagoodideatorewritescriptsinC?
123
4. ImplementabasictrcommandinaCprogram.5. Compile severalCsource files intoobject (.o) files first.Thenusegcc to
produce the file a.out from the .o files. This should produce a workingprogram.Givethe-voptiontogccandseewhatcalltothelinker/loaderisused.
6. Your Linux systemmay have more than 64 error numbers. To find out,writeaCprogramtoaccesstheglobalexternaltablesys_errlist.Hint:Seeman3perror.
7. Systemheader files forC programs are kept in a few systemdirectories.Findoutwhichdirectoriestheseareonyoursystem.
8. WritefourorfiveCsourcefilescontainingsmallroutines,andsetupsomeheader files that are used by these source files. Establish a library filelibme.aoftheseroutinesusingar.Nowwrite,compile,andrunaprogramthatappliesafewoftheselibraryroutinesinlibme.a.Compileandrunyourapplicationprogram.
9. Write an efficient template C program for processing command-lineoptions.Theoptionscanbegiveninanyorderanywhereonthecommandline.
10. Revise the lowercase.c program (Section 10.4) so that it takes optionalfilenamearguments:lowercase[infile][outfile]Alsoprovideappropriateerrorchecks.
11. Write a Linux command named fil. The usage synopsis is as follows: fil[from][to]totransformtextfromthenamedfilefromtothenamedfileto.Ifonlyonefileargumentissupplied,itisassumedtobeforthefromfile.Ahyphen(-)meansstandardinput;amissingtomeansstandardoutput.Thefilcommandworksasfollows:
Alltabsarereplacedbyanequivalentnumberofspaces.Alltrailingblanksattheendofeachlineareremoved.All lines longer than80 characters are folded, breaking linesonly atspaces.
12. Applygdbtodebugyourfilprogram.
C++isasupersetofCthatsupportsObject-OrientedProgramming(OOP).PortableOperatingSystemInterfaceforUNIX.GCC7wasreleasedinMay2017.
Chapter11
I/OandProcessControlSystemCalls
Anoperatingsystem(OS)providesmanytoolsandfacilitiestomakeacomputerusable.However, themostbasic and fundamental setof services is the systemcalls,specificroutinesintheoperatingsystemkernelthataredirectlyaccessibletoapplicationprograms.Thereareover300systemcallsinLinuxwithakernel-definednumber starting from1.Each systemcall alsohas ameaningfulnameand a symbolic constant in the form SYS_name for its number. With a fewexceptions,asystemcallnamecorrespondstotheroutinesys_nameintheLinuxkernelsourcecode.A program under execution is called a process. When a process makes a
system call at run time, a software-generated interrupt, often known as anoperatingsystemtrap, triggers theprocess toswitchfromusermode tokernelmode and to transfer control to the entry point of the target kernel routinecorresponding to the particular system call.A process running in kernelmodecanexecute instructions that arenot available inusermode.Upon systemcallcompletion,theprocessswitchesbacktousermode.Higherlevelsystemfacilitiesarebuiltbywritinglibraryprogramsthatusethe
systemcalls.BecauseLinuxisimplementedinC,itssystemcallsarespecifiedinCsyntaxanddirectlycalledfromCprograms.ImportantLinuxsystemcallsaredescribedhere.Theseallowyoutoperform
low-levelinput/output(I/O),manipulatefilesanddirectories,createandcontrolmultiple concurrent processes, and manage interrupts. Examples show howsystem calls are used and how to combine different system calls to achievespecificgoals.Just like library functions, a system call may need one or more associated
headerfiles.Theseheaderfilesareclearlyindicatedwitheachcalldescribed.ThesetofsystemcallsandtheirorganizationformtheC-languageinterfaceto
the Linux kernel, and this interface is nearly uniform across all major Linux
11.1
11.2
distributions. The reason is because Linux systems closely follow POSIX(Portable Operating System Interface), an open operating system interfacestandard accepted worldwide. POSIX is produced by IEEE (Institute ofElectrical and Electronics Engineers) and recognized by ISO (InternationalOrganization for Standardization) and ANSI (American National StandardsInstitute). By following POSIX, software becomes easily portable to anyPOSIX-compliantOS.Documentationforanysystemcallnamecanbefoundwithman2nameinsection2ofthemanpages(Section1.15).
SYSTEM-LEVELI/OHigh-level I/O routines such as putc and fopen, which are provided in theStandardCLibrary(Chapter),areadequateformostI/OneedsinCprograms.These library functions are built on top of low-level calls provided by theoperatingsystem.InLinux,theI/OstreamofC(Chapter,Section10.4)isbuiltontopoftheI/Odescriptormechanismsupportedbysystemcalls(Figure11.1).
Figure11.1I/OLayers
Getting toknow the low-level I/O facilitieswillnotonlyprovide insightonhowthelibraryfunctionswork,butwillalsoallowyoutouseI/OinwaysnotsupportedbytheStandardCLibrary.Linux features a uniform interface for I/O to files and devices, such as a
terminalwindow or an optical drive, by representing I/O hardware as specialfiles.We shall discuss I/O to files, understanding they apply also to devices,which are nothing but special files. In addition to files, Linux supports I/Obetweenprocesses (concurrently runningprograms) through abstract structuresknownaspipesandsockets(Chapter12).Althoughfiles,pipes,andsocketsaredifferentI/Oobjects,theyaresupportedbymanyofthesamelow-levelI/Ocallsexplainedhere.
I/ODESCRIPTORS
Before file I/O can take place, a program must first indicate its intention toLinux.Thisisdonebytheopensystemcalldeclaredasfollows:
#include>sys/types.h<#include>sys/stat.h<#include>fcntl.h<int
open(constchar*filename,intaccess[,mode_tmode])
Argumentstoopenare
filenamecharacterstringforthepathnametothefileaccessan
integercodefortheintendedaccessmodetheprotectionmodefor
creatinganewfile
Thecallopensfilename,forreadingand/orwriting,asspecifiedbyaccessandreturnsanintegerdescriptorforthatfile.Thefilenamecanbegiveninanyofthethree valid forms: full pathname, relative pathname, or simple filename. Theopencommandisalsousedtocreateanewfilewiththegivenname.SubsequentI/O operations will refer to this descriptor rather than to the filename. OthersystemcallsreturndescriptorstoI/Oobjectssuchaspipes(Chapter12,Section12.2)andsockets(Chapter12,Section12.6).Adescriptorisactuallyanindextoaper-processopenfiletablewhichcontainsnecessary informationforallopenfilesandI/Oobjectsoftheprocess.Theopencallreturnsthelowestindextoacurrentlyunusedtableentry.Eachtableentryleads,inturn,toakernelopenfiletableentry.Allprocessessharethesamekernelopenfiletable(Figure11.2)andit is possible for file descriptors from different processes to refer to the samekerneltableentry.
Figure11.2OpenFileTables
For each process, three file descriptors, STDIN_FILENO (0),STDOUT_FILENO (1), and STDERR_FILENO (2), are automatically openedinitially,allowingreadyaccesstostandardI/O.Theaccesscodeisformedbythelogicalor(|)ofheader-suppliedsingle-bitvaluesincludingO_RDONLYtoopenfileforreadingonlyO_WRONLYtoopenfileforwritingonly
O_RDWRtoopenfileforreadingandwritingO_NDELAYtopreventpossibleblockingO_APPENDtoopenfileforappendingO_CREATtocreatefileifitdoesnotexistO_TRUNCtotruncatesizeto0O_EXCLtoproduceanerroriftheO_CREATbitisonandfileexistsOpening a file with O_APPEND instructs each write on the file to be
appended to the end. If O_TRUNC is specified and the file exists, the file istruncatedtolengthzero.Ifaccessis(O_EXCL|O_CREAT)and if the filealreadyexists,open returns an error.Thepurpose is to avoid
destroyinganexistingfile.The thirdandoptional argument toopen is a file creationmode in case the
O_CREAT bit is on. The mode is a bit pattern (of type mode_t from <sys/types.h>with symbolicvalues from<sys/stat.h>) explained in detail inSection11.4,wherethecreatsystemcallisdescribed.If theopen call fails, a -1 is returned;otherwise, adescriptor is returned.A
process may have no more than a maximum number of descriptors opensimultaneously.ThislimitislargeenoughinLinuxtobeofnopracticalconcern.Thefollowingexample(Ex:ex11/open.c)showsatypicalusageoftheopen
systemcall.Thethirdargument toopen isunusedbecause it isnotneededfortheread-only(O_RDONLY)operation.Inthiscase,anyintegercanbeusedasthethirdargument.
/*******open.c*******/#include>stdlib.h<#include
>stdio.h<#include>fcntl.h<intmain(intargc,char*argv[]){int
fd;/*filedescriptor*//*openargv[1]forreading*/if((fd=
open(argv[1],O_RDONLY,0))==-1){fprintf(stderr,"%s:cannotopen
%s\n",argv[0],argv[1]);perror("opensystem
call");exit(EXIT_FAILURE);}/*othercode*/}
Whenasystemorlibrarycallfails,youcanusethecodeperror(constchar*msg)(displayssystemerror)to display the given message msg followed by a standard error message
associatedwiththeerror(Chapter,Section10.6).Whenadescriptorfdisnolongerneededinaprogram,itcanbedeletedfrom
theper-processopenfiletableusingthecallintclose(intfd)(closesdescriptor)Otherwise, all open file descriptors will be closed when the program
terminates.
11.3 READINGANDWRITINGI/ODESCRIPTORSReadingandwritingarenormallysequential.Foreachopendescriptor,thereisacurrentpositionwhichpointstothenextbytetobereadorwritten.Afterkbytesare read or written, the current position, if movable, is advanced by k bytes.WhetherthecurrentpositionismovabledependsontheI/Oobject.Forexample,itismovableforanactualfilebutnotforstdinwhenconnectedtothekeyboard.Thesystemcallsreadandwritearedeclaredas
#include>unistd.h<ssize_tread(intfd,void*buffer,size_tk)
(readsinputfromfd)ssize_twrite(intfd,void*buffer,size_tk)
(writesoutputtofd)
wherefdisadescriptortoreadfromorwriteto,bufferpointstoanarraytoreceiveorsupplythebytes,andkisthenumberofbytestobereadinorwrittenout.Obviously,kmustnotexceedthelengthofbuffer.Readwillattempttoreadk bytes from the I/O object represented by fd. It returns the number of bytesactuallyreadanddepositedinthebuffer.Thetypesize_tisusuallyunsignedint(non-negative)andssize_t isusually int (canbenegative). Ifread returns lessthankbytes,itdoesnotnecessarilymeanthatend-of-filehasbeenreached,butifzeroisreturned,thentheendofthefilehasbeenreached.Thewrite call outputs k bytes from the buffer to fd and returns the actual
numberofbyteswrittenout.Bothreadandwritereturna-1iftheyfail.As an example, we canwrite a readline functionwith low-level read (Ex:
ex11/readline.c).
intreadline(chars[],intsize){char*tmp=s;/*readone
characteratatime*/while(0>--size&&read(0,tmp,1)!=0&&
*tmp++!='\n');/*emptyloopbody*/*tmp='\0';/*string
terminator*/returntmp-s;/*numberofcharactersread*/}
The while loop control is intricate and warrants careful study. The sizeargument is the capacity of the array s. The function returns the number ofcharactersread,notcountingthestringterminator.Foracompleteprogram,thelowercasecommand(Chapter,Figure10.3)has
beenrewrittenwithI/Osystemcalls(Ex:ex11/lowercase.c).
/********lowercase.cwithI/Osystemcalls********/#include
>ctype.h<#include>stdlib.h<#include>stdio.h<#include
>unistd.h<voidlower(char*buf,intlength){while(length--<0){
*buf=tolower(*buf);buf++;}}intmain(intargc,char*argv[]){
charbuffer[BUFSIZ];ssizenc;/*numberofcharacters*/while((nc
=read(STDIN_FILENO,buffer,BUFSIZ))<0){lower(buffer,nc);nc=
write(STDOUT_FILENO,buffer,nc);if(nc==-1)break;}if(nc==-1)
/*readorwritefailed*/{perror("read/write
call");exit(EXIT_FAILURE);}returnEXIT_SUCCESS;}
ComparedwiththeversioninChapter,Figure10.3,whichusesputchar,theprogramshowsthedifferencebetweenimplicitandexplicitI/Obuffering.
MovingtheCurrentPositionWhen readingorwriting an I/Oobject that is an actual file, theobject canbeviewedasasequenceofbytes.Thecurrentposition ismovedby thereadandwriteoperationsinasequentialmanner.Asanalternativetothis,thesystemcalllseekprovidesawaytomovethecurrentpositiontoanylocationandthereforeallowsrandomaccess to bytes of the file.The standard library function fseek(Chapter,Section10.4)isbuiltontopoflseek.Thecall
#include>sys/types.h<#include>unistd.h<off_tlseek(intfd,off_t
offset,intorigin)(movesread/writeposition)
movesthecurrentpositionassociatedwiththedescriptorfdtoabytepositiondefinedby(origin+offset).Table11.1showsthethreepossibleorigins.ThelseekOrigins
The offset can be positive or negative. The call lseek returns the currentpositionasanintegerpositionmeasuredfromthebeginningofthefile.Itreturns-1uponfailure.SeveralcallsareillustratedinTable11.2.Useoflseek
It is possible to lseek beyond the endof file and thenwrite. This creates aholeinthefilewhichdoesnotoccupyfilespace.Readingabyteinsuchaholereturnszero.In some applications, holes are left in the file on purpose to allow easy
insertionofadditionaldatalater.Itisanerrortolseekanon-movabledescriptor
11.4
such as the STDIN_FILENO. See the example code package (Ex:ex11/lowerseek.c)foran implementationof the lowercaseprogramusing lseekandO_RDWR.
OPERATIONSONFILESSystem calls are provided for creating and deleting files, accessing file statusinformation,obtainingandmodifyingprotectionmodes,andotherattributesofafile.Thesewillbedescribedinthefollowingsubsections.
CreatingandDeletingaFileForcreatinganewfile, theopen systemcallexplained in theprevioussectioncanbeused.Alternatively,thesystemcallintcreat(char*filename,intmode)(createsanewfile)canalsobeused.Ifthenamedfilealreadyexists,itistruncatedtozerolength,
andreadytoberewritten.Ifitdoesnotexist,thenanewdirectoryentryismadeforit,andcreatreturnsafiledescriptorforwritingthisnewfile.Itisequivalenttoopen(filename,(O_CREAT|O_WRONLY|O_TRUNC),mode)The lower 9 bits of mode (for access protection) are modified by the file
creationmaskumaskoftheprocessusingtheformula(umask)&modeThemodeisthelogicalorofanyofthebasicmodesshowninTable11.3.Theinitialumaskvalueofaprocessisinheritedfromtheparentprocessofa
running program. We have seen how to set umask using the Bash umaskcommand(Chapter3,Section3.12).Thedefaultumask isusually0022,whichclearsthewritepermissionbitsforgroupandother(Chapter6,Section6.3).Aprogramcansetumaskwiththesystemcall
#include>sys/types.h<#include>sys/stat.h<mode_tumask(mode_t
mask);
Thereturnedvalueistheoldumask.Forexample,umask(0077);willforcefilemodesfornewlycreatedfilestoallowfileaccessonlyforthe
owner.Thevalueofumaskisinheritedbychildprocesses.Afterafileiscreated,itcanberead/writtenwiththeread,writecalls.
LinkingandRenamingFiles
Foranexistingfile,alternativenamescanalsobegiven.Thecalllink
#include>unistd.h<intlink(constchar*file,constchar*name)
(ahardlink)intsymlink(constchar*file,constchar*name)(a
symboliclink)
establishesanothername(directoryentry)fortheexistingfile.Thenewnameis a hard link and can be anywhere within the same filesystem (Chapter 6,Section6.5).Toremovealink,thecallintunlink(constchar*name)(deletesfilelink)isused.Whenthelinkremovedisthelastdirectoryentrypointingtothisfile,
thenthefileisdeleted.Use a symbolic link (the symlink system call) for a directory or a file in a
differentfilesystem.At the Shell level, renaming a file is done with themv command. At the
systemcalllevel,use
#include>stdio.h<intrename(constchar*old_name,constchar*
new_name)
Both filenames must be within the same filesystem. When renaming adirectory,thenew_namemustnotbeunderold_name.
Figure11.3FileStatusStructure
AccessingFileStatusFor each file, Linux maintains a set of status information such as file type,
protectionmodes,timewhenlastmodifiedandsoon.Thestatusinformationiskept in the i-node (Chapter 6, Section 6.5) of a file. To access file statusinformationfromaCprogram,thefollowingsystemcallscanbeused.
#include>sys/types.h<#include>sys/stat.h<#include>unistd.h<int
stat(constchar*file,structstat*buf)(offile)intfstat(intfd,
structstat*buf)(ofdescriptorfd)intlstat(constchar*link,
structstat*buf)(ofthesymboliclink)
Note that fstat is the same as stat, except it takes a file descriptor that hasbeenopenedalready.Thisparallelexistsformanyothersystemcalls.Thelstatisthesameasstat,excepttheformerdoesnotfollowsymboliclinks.Thestatusinformation for the given file is retrieved and placed in buf.Accessing statusinformationdoesnotrequireread,write,orexecutepermissionforthefile,butall directories listed in the pathname leading to the file (for stat) must bereachable.Thestatstructure(Figure11.3)hasmanymembers.Table11.3andTable11.4
listthesymbolicconstantsforinterpretingthevalueofthestatmemberst_mode.BasicFileModes
Therearethreetimestampskeptforeachfile:
st_atime(lastaccesstime)—Thetimewhenfilewaslastreadormodified.It is affected by the system callsmknod, utimes, read, andwrite. Forreasonsofefficiency,st_atimeisnotsetwhenadirectoryissearched.st_mtime (lastmodify time)—The timewhen filewas lastmodified. It isnotaffectedbychangesofowner,group,linkcount,ormode.Itischangedby:mknod,utimes,andwrite.st_ctime (last status change time)—The time when file status was lastchanged.Itissetbothbywritingthefileandbychangingtheinformationcontained in the i-node. It is affected by chmod, chown, link, mknod,unlink,utimes,andwrite.
The timestamps are stored as integers, and a larger integer value represents amore recent time. Usually, Linux uses GMT (Greenwich Mean Time). Theinteger timestamps, however, represent the number of seconds since a fixed
pointinthepast,knownasthePOSIXepochwhichisUTC00:00:00,January1,1970.The library routinectime converts such an integer into anASCII stringrepresentingdateandtime.FileStatusConstants
ThemaskS_IFMTisusefulfordeterminingthefiletype.Forexample,if((buf.st_mode&S_IFMT)==S_IFDIR)determineswhetherthefileisadirectory.Asanapplication, let’sconsidera functionnewer (Ex: ex11/newer.c)which
returns 1 if the lastmodify time of file1 ismore recent than that of file2 andreturns0otherwise.Uponfailure,newerreturns-1.
/********newer.c********/#include>sys/types.h<#include
>sys/stat.h<#include>stdio.h<#include>stdlib.h<■I/OandProcess
ControlSystemCalls/*testiffilelismorerecentthanfile2*/
intnewer(constchar*file1,constchar*file2){intmtime(const
char*file);inttl=mtime(filel),t2=mtime(file2);/*timestamps
*/if(tl>0||t2>0)return-1;/*failed*/elseif(tl<t2)
return1;elsereturn0;}intmtime(constchar*file)/*lastmodify
timeoffile*/{structstatstb;if(stat(file,&stb)>0)/*result
returnedinstb*/return-1;/*statfailed*/return
stb.st_mtime;/*returntimestamp*/}
The stb structure in the functionmtime is a returnargument supplied to thestatsystemcalltocollectthestatusinformationofafile.Thenewerfunctioncanbeusedinamainprogramsuchas
intmain(intargc,char*argv[]){if(argc==3){if(
newer(argv[1],argv[2]))returnEXIT_SUCCESS;/*exitstatusfor
yes*/elsereturn1;/*exitstatusforno*/}else{fprintf(stderr,
"Usage:%sfilelfile2\n",argv[0]);return-l;}}
NotethatthecorrectexitstatusisreturnedforlogicattheShelllevelviathespecialvariable$?(Chapter5,Section5.7).
11.5
DeterminingAllowableFileAccessItispossibletodeterminewhetheranintendedread,writeorexecuteaccesstoafile is permissible before initiating such an access. The access system call isdefinedas
#include>unistd.h<intaccess(constchar*file,inta_mode)
(checksaccesstofile)
Theaccesscallchecksthepermissionbitsoffiletoseeiftheintendedaccessgivenbya_modeisallowable.Theintendedaccessmodeisa logicalorof thebitsR_OK,W_OK,andX_OKdefinedby
#defineR_OK4/*testforreadpermission*/#defineW_OK2/*testforwrite
permission*/#defineX_OKl/*testforexecute(search)permission
*/#defineF_OK0/*testforpresenceoffile*/OperationsonDirectories
■
Ifthespecifiedaccessisallowable,thecallreturns0;otherwise,itreturns-1.Specifyinga_modeasF_OKtestswhether thedirectories leading to the file
canbesearchedandwhetherthefileexists.For extended attributes (Chapter 6, Section 6.4) operations we have these
systemcallssetxattrgetxattr,listxattr,andremovexattr.Seesection2of themanpagesfortheirusage.
OPERATIONSONDIRECTORIESCreatingandRemovingaDirectoryIn addition to files, it is alsopossible to establish and removedirectorieswithLinuxsystemcalls.Thesystemcallmkdircreatesanewdirectory.
#include>sys/stat.h<#include>sys/types.h<intmkdir(constchar
*dir,mode_tmode)(makesanewfolder)
Itcreatesanewdirectorywiththenamedir.Themodeworksthesamewayasintheopensystemcall.Thenewdirectory’sownerIDissettotheeffectiveuserIDoftheprocess.Iftheparentdirectorycontainingdirhastheset-group-IDbiton,orifthefilesystemismountedwithBSD(BerkeleyUNIX)groupsemantics,thenewdirectorydirwillinheritthegroupIDfromitsparentfolder.Otherwise,itwillgettheeffectivegroupIDoftheprocess.Thesystemcallrmdir
11.6
#include>unistd.h<intrmdir(constchar*dir)(removesafolder)
remove the given directory dir. The directory must be empty (having noentries other than ’.’ and ’..’). For bothmkdir and rmdir, a 0 returned valueindicatessuccess,anda-1indicatesanerror.Thecontentofadirectoryconsistsmainlyoffilenames(strings)andi-nodenumbers(i-number).Thelengthlimitofasimplefilenamedependsonthefilesystem.Typically,simplefilenamesarelimitedtoalengthof255characters.Thesystemcallgetdentscanbeusedtoreadthecontentsofadirectoryfile
into a character array in a system-independent format. However, a moreconvenient way to access directory information is to use the directory libraryfunctionsdiscussedinthenextsection.
DIRECTORYACCESSIn theLinuxfilesystem,adirectorycontains thenamesandi-numbersoffilesstoredinit.Libraryfunctionsareavailableforaccessingdirectories.Touseanyofthem,besuretoincludetheseheaderfiles:
#include>sys/types.h<#include>dirent.h<
Toopenadirectory,useeither
DIR*opendir(constchar*dir)orDIR*fdopendir(intfd)(opens
directorystream)(opensdirectorystream)■I/OandProcessControl
SystemCalls
toobtainadirectorystreampointer(DIR*)foruseinsubsequentoperations.Ifthenameddirectorycannotbeaccessed,orifthereisnotenoughmemorytoholdthecontentsofthedirectory,aNULL(invalidpointer)isreturned.Once a directory stream is opened, the library function readdir is used to
sequentiallyaccessitsentries.Thefunction
#include>sys/types.h<#include>dirent.h<structdirent*readdir(DIR
*dp)(returnsnextdirentryfromdp)
returnsapointertothenextdirectoryentry.ThepointervaluebecomesNULLonerrororreachingtheendofthedirectory.Thedirectoryentrystructurestructdirent records information foranysingle
fileinadirectory.
structdirentino_td_ino;/*i-nodenumberof
file*/off_td_off;/*offsettothenext
dirent*/unsignedshortd_reclen;/*lengthofthis
record*/unsignedchard_type;/*file
type*/chard_name[256];/*filename*/};
Eachfileinafilesystemalsohasauniquei-nodenumber(Chapter6,Section6.5). The NAME_MAX constant, usually 255, gives the maxima length of adirectoryentryname.Thedatastructurereturnedbyreaddircanbeoverwrittenbyasubsequentcalltoreaddir.Thefunction
closedir(DIR*dp)(closesdirectorystream)
closesthedirectorystreamdpandfreesthestructureassociatedwiththeDIRpointer.Toillustratetheuseoftheselibraryfunctions,let’slookatafunctionsearchdir
(Figure11.4)whichsearchesdirforagivenfileandreturns1or0dependingonwhether the file is found or not (Ex: ex11/searchdir.c).Note that the exampleusesknowledgeofthedirentstructure.
Figure11.4SearchingaDirectory
Enumeration constants FOUND and NOT_FOUND are used. The for loopgoes through each entry in dir to find file.Note the logical not (!) in front ofstrcmp.
CurrentWorkingDirectoryThelibraryroutine
11.7
char*getcurrentdirname(void);(obtainscurrent
directory)returnsthefullpathnameofthecurrentworking
directory.Thesystemcallintchdir(constchar*dir)
(changesdirectory)
is used to change the current working directory to the named directory. Avalue0isreturnedifchdirissuccessful;otherwise,a-1isreturned.Becausethecurrent directory is a per-process attribute, you will return to the originaldirectoryaftertheprogramexits.
ANEXAMPLE:CCPIt is perhaps appropriate to look at a complete example of a Linux commandwritteninC.Thecommandweshalldiscussisccp(conditionalcopy),whichisused tocopy files fromonedirectory toanother (Ex:ex11/ccp.c).Aparticularfile iscopiedornotdependingonwhetherupdating isnecessary.AversionofccpimplementedasaBashscripthasbeendiscussedinChapter5,Section5.20.Theccp command copies files from a source folder source to a destination
folderdest.Theusageisccpsourcedest[file...]Thenamedfilesorallfiles(butnotdirectories)arecopiedfromsourcetodest
subjecttothefollowingconditions:
1. Ifthefileisnotindest,copythefile.2. If thefile isalreadyindestbut thefile insource ismorerecent,copythe
file.3. Ifthefileisalreadyindestandthefileinsourceisnotmorerecent,donot
copythefile.
Tocheckifafileisadirectory,wecalltheisDirfunction(line1).Tocomparetherecencyoftwofiles(line2),weusethefunctionnewerpresentedinSection11.4.
/********ccp:theconditionalcopycommand********/#include
>sys/param.h<#include>stdio.h<#include>stdlib.h<#include
>dirent.h<*/*/=FOUND;■I/OandProcessControlSystem
Calls#include>unistd.h<#include>string.h<#include
>sys/stat.h<#include"newer.h"intisDir(constchar*file){struct
statstb;if(stat(file,&stb)>0)/*resultreturnedinstb*/return
-1;/*statfailed*/return((stb.st_mode&S_IFMT)==S_IFDIR);}void
ccp(constchar*name,constchar*d1,constchar*d2){char
f1[MAXPATHLEN+1],f2[MAXPATHLEN+1];strcpy(f1,d1);strcpy(f2,d2);
strcat(f1,"/");strcat(f2,"/");strcat(f1,name);strcat(f2,name);
11.8
if(isDir(f1)==0)/*(1)*/if(access(f2,F_OK)==-1||
newer(f1,f2))/*(2)*/printf("copy(%s,%s)\n",f1,
f2);elseprintf("noneedtocopy(%s,%s)\n",f1,f2);}intmain(int
argc,char*argv[]){DIR*dirp1;structdirent*dp;if(argc>
3)/*needatleasttwoargs*/{fprintf(stderr,"%s:wrongnumberof
arguments",argv[0]);exit(EXIT_FAILURE);}elseif(argc<3)/*files
specified*/{inti;for(i=3;i>argc;
i++)ccp(argv[i],argv[1],argv[2]);/*(3)*/returnEXIT_SUCCESS;}/*
nowexactlytwoargs*/if((dirp1=opendir(argv[1]))==NULL){
fprintf(stderr,"%s:cannotopen%s",argv[0],argv[1]);
exit(EXIT_FAILURE);}for(dp=readdir(dirp1);dp!=NULL;dp=
readdir(dirp1))/*(4)*/if(strncmp(dp-<d_name,".",1))ccp(dp-
<d_name,argv[1],argv[2]);returnEXIT_SUCCESS;}
Iffilesaregivenonthecommandline,wecallthefunctionccponthosefiles(line3).Otherwise,we go through all fileswhose names do not beginwith aperiod(line4).Tocompileweusegccccp.cnewer.c-occp
SHELL-LEVELCOMMANDSFROMCPROGRAMS
Intheccp.cexample,wehavenotperformedanyactualfilecopying.Wesimplyusedprintftoindicatethecopyingactionsneeded.Tocarryoutthefilecopying,it is most convenient to invoke a Shell-level cp command from within a Cprogram.AllowingexecutionofShell-levelcommandsfromwithinCprogramsis very useful. With this ability, you can, for example, simply issue a cpcommandtocopyafilefromaCprogramratherthanwritingyourownroutines.TheLinuxlibrarycallsystemisusedforthispurpose.
#include>stdlib.h<intsystem(constchar*cmd_str)/*issuesShell
command*/
ThesystemcallstartsanewShprocesstoexecutethegivenstringcmd_str.The Shell terminates after executing the given command and system returns.The returned value represents the exit status of the given command. Thus, tocopyfile1tofile2,youcanuse
charcmd_string[80];sprintf(cmd_string,"cp/s°/0s\n",filel,
file2);system(cmd_string);
The string is, of course, interpreted by the Shell before the command isinvoked.Anysubstitutionandfilenameexpansionwillbedone.Also,theShell
11.9
locatestheexecutablefile(forexample,/bin/cp)onthecommandsearchpathforyou.UsethefullpathnameofthecommandifyoudonotwishtodependonthePATH setting. The system call waits until the command is finished beforereturning.One shortcoming of the system function is that it does not allow you to
receive the resultsproducedby the commandor toprovide input to it.This isremediedbythelibraryfunctionpopen(Chapter12,Section12.1).
PROCESSCONTROLAkeyoperatingsystemkernelserviceisprocesscontrol.Aprocessisaprogramunder execution, and in amultiprogramming system like Linux, therewill bemultipleprocessesrunningconcurrentlyatanygiventime.Wewilllookatprocessaddressspace,states,controlstructures,creationand
termination, executable loading, and inter-process communication here and inlatersections.
VirtualAddressSpaceWhen created, each individual process has, among other resources, memoryspaceallocatedforitsexclusiveuse.Thismemoryspaceisoftenreferredtoasthevirtual address space (or simply address space) of a process. The addressspace consists of a kernel space which is the Linux kernel shared by allprocesses and a user space which is off limits to other processes. A processexecutinginusermodehasnoaccesstothekernelspaceexceptthroughsystemcalls provided by the kernel. Upon a system call, control is transferred to akerneladdressthroughaspecialsignal(Section11.16)andtheprocessswitchestokernelmode.Whileinkernelmode,theprocesshasaccesstobothuserspaceand kernel space. The process switches back to usermode upon return of thesystemcall.Theprocessuserspaceisorganizedintoshared,text,data,andstack regions
(Figure11.5).
Stack—A last-in-first-out data structure used to manage function calls,returns,parameterpassing,andreturnedvalues.Thememoryusedfor thestackwillgrowandshrinkwiththedepthofnestingoffunctioncalls.Data—Thevaluesofvariables,arrays,andstructures.Objectsallocatedatcompile timewilloccupy fixedmemory locations in thedataarea.Roomfor dynamically allocated space (throughmalloc) is also included in thedataarea.
Text—Themachine instructions that represent theproceduresor functionsin theprogram.Thispartofaprocesswillgenerallystayunchangedoverthelifetimeoftheprocess.Shared—Codefromlibrariesthatisnotduplicatedwhensharedwithotherprocesses.
Inadditiontotheaddressspace,eachprocessisalsoassignedsystemresourcesnecessaryforthekerneltomanagetheprocess.
Figure11.5MemoryLayoutofaProcess
ProcessLifeCycleEach process is represented by an entry in the process table which ismanipulated by the kernel to manage all processes. The kernel schedules theCPU(CentralProcessingUnit)andswitchesitfromrunningoneprocesstothenext in rapid succession. Thus, the processes appear to make progressconcurrently. On a computer withmultiple CPUs, a number of processes canactually run simultaneously or in parallel. A process usually goes through anumberofstates before running to completion.Figure11.6 shows the processlifecycle.
Figure11.6ProcessLifeCycle
Theprocessstatesare
11.10
Running—Theprocessisexecuting.Waiting/Blocked—Aprocess in this state iswaiting for anevent to occur.Such an event could be an I/O completion by a peripheral device, theterminationofanotherprocess,theavailabilityofdataorspaceinabuffer,thefreeingofasystemresource,andsoon.Whenarunningprocesshastowaitforsuchanevent,itisblockedandwaitingtobeunblockedsoitcancontinuetoexecute.Aprocessblockingcreatesanopportunityforacontextswitch, shifting the CPU to another process. Later, when the event ablockedprocessiswaitingforoccurs,itawakensandbecomesreadytorun.Ready—AprocessinthisstateisthenscheduledforCPUservice.Zombie—After termination of execution, a process goes into the zombiestate.Theprocessnolongerexists.Thedatastructureleftbehindcontainsitsexitstatusandanytimingstatisticscollected.Thisisalwaysthelaststateofaprocess.
A process may go through the intermediate states many times before it isfinished.Fromaprogrammingpointofview,aLinuxprocess is theentitycreatedby
the fork system call (Section 11.11). In the beginning, when Linux is bootedthere isonlyoneprocess(process0)whichuses thefork systemcall tocreateprocess1,knownastheinitprocess.Theinitprocessistheancestorofallotherprocesses, including your login Shell. Process 0 then becomes the virtualmemoryswapper.
THEPROCESSTABLEA system-wide process table is maintained in the Linux kernel to control allprocesses.Thereisonetableentryforeachexistingprocess.Eachprocessentrycontains all key information needed to manage the process, such as PID (auniqueintegerprocessID),UID(realandeffectiveownerandgroupID’sofuserexecuting this process), process status, andgenerally informationdisplayedbythe ps command. Linux provides a directory under /proc/ for each existingprocess,makingiteasy toaccess informationon individualprocessesfromtheShelllevel.
ThepsCommandYou can also obtain various kinds of information on processes with thecommandps(displaysprocessstatus)
11.11
Because Linux is a multi-user system and because there are many systemprocesses that perform various chores to keep Linux functioning, there arealwaysmultipleprocessesrunningatanygiventime.Thepscommandattemptstodisplaya reasonablesetofprocesses thatare likely tobeof interest toyou,andyoucangiveoptionstocontrolwhatsubsetofprocessesaredisplayed.The ps command displays information only for your processes. Give the
option-atodisplayallinterestingprocesses,or-Atodisplayallprocesses.Also,psdisplaysinshortformunlessgiventheoption-f toseeafull-formatlisting.Forexample,ps-afdisplays,infullformat,allinterestingprocesses.Usetheoption-e(or-A)to
display all current processes, including daemon processes (those without acontrolterminalsuchasthecronprocess).Seethepsmanpageforquiteafewotheroptions.InformationprovidedforeachprocessincludesPID—TheprocessIDinintegerformPPID—TheparentprocessIDinintegerformS—Thesingle-letterstatecodefromthepsmanpageSTIMEorSTART—TheprocessstarttimeTIME—CPUtime(inseconds)usedbytheprocessTT—ControlterminaloftheprocessCOMMAND—TheusercommandwhichstartedthisprocessWhenyouarelookingforaparticularprocess,thepipeps-e|grepstringcanbehandy.
Figure11.7ProcessCreation
PROCESSCREATION:FORKTheforksystemcallisusedinsideaCprogramtocreateanotherprocess.
#include>sys/types.h<#include>unistd.h<pid_tfork();Theprocess
whichcallsforkisreferredtoastheparentprocess,andthe
newlycreatedprocessisknownasthechildprocess.Afterthefork
call,thechildandtheparentrunconcurrently.Thechildprocess
createdisacopyoftheparentprocessexceptforthefollowing:•
ThechildprocesshasauniquePID.•Thechildprocesshasa
differentPPID(PIDofitsparent).
The process which calls fork is referred to as the parent process, and thenewly created process is known as the child process. After the fork call, thechildandtheparentrunconcurrently.The child process created is a copy of the parent process except for the
following:
ThechildprocesshasauniquePID.ThechildprocesshasadifferentPPID(PIDofitsparent).
The fork is called by the parent, but returns in both the parent and the child(Figure11.7). In theparent, it returns thePIDof thechildprocess,whereas inthechild,itreturns0.Ifforkfails,nochildprocessiscreated,anditreturns-1.Hereisatemplateforusingfork.
pit_tpid;if((pid=fork())==0){/*putcodeforchildhere*/}if
(pid>0){/*forkfailed,puterrorhandlinghere*/}/*fork
successful,putremainingcodeforparenthere*/
The following simple program (Ex: ex11/simplefork.c) serves to illustrateprocess creation, concurrent execution, and the relationshipsbetween thechildandtheparentacrosstheforkcall.
/********simplefork.c********/#include>sys/types.h<#include
>unistd.h<#include>stdlib.h<#include>stdio.h<intmain(){pid_t
child_id;child_id=fork();/*processcreation(l)*/if(child_id==
0)/*childcodebegin(2)*/{printf("Child:Mypid=/dandmy
parentpid=/d\n",getpid(),
getppid());_exit(EXIT_SUCCESS);/*childterminates(3)*/}/*child
codeend*/if(child_id>0)/*remainingparentcode*/{
fprintf(stderr,"forkfailed\n");exit(EXIT_FAILURE);
}printf("Parent:Mypid=/d,spawnedchildpid=/d\n",getpid(),
child_id);returnEXIT_SUCCESS;}
Aftercallingfork (line1),yousuddenlyhave twoprocesses, theparentandthechild,executingthesameprogramstartingatthepointwhereforkreturns.Thechildandparentexecutedifferentcodesectionsinourexamplebecause
of the way the program is written. The child only executes the part under if(child_id==0) (line2).At the endof the child code (line3), itmust terminate
11.12
execution.Otherwise,thechildwouldcontinueintothecodemeantonlyfortheparent.The_exitsystemcallisslightlydifferentfromlibraryfunctionexitandisexplainedinSection11.14.Notealsothataprocesscanusethesystemcallsgetpid() and getppid() to obtain the process ID of itself and its parent,respectively.Theaboveprogramproducesthefollowingoutput.
■I/OandProcessControlSystemCallsChild:Mypid=19603andmy
parentpid=19602Parent:Mypid=19602,spawnedchildpid=
19603
Tofurtherillustratetheuseoffork,wecanwriteaprogramwheretheparentand child run concurrently (Ex: ex11/concurrent.c). The child computes thepartial sums, and the parent calculates the partial products, of an array ofintegers.
/********concurrent.c********/#include>sys/types.h<#include
>unistd.h<#include>stdlib.h<#include>stdio.h<#defineDIM8int
main(){pid_tpid;inti,ans,arr[DIM]={1,2,3,4,5,6,7,8};pid=
fork();if(pid==0)/*childcodebegin*/{ans=0;for(i=0;i>DIM;i++){
ans=ans+arr[i];printf("Child:sum=%d\n",ans);sleep(1);/*1
secdelay*/}_exit(EXIT_SUCCESS);}/*childcodeend*/if(pid>0){
fprintf(stderr,"forkfailed\n");returnEXIT_FAILURE;}ans=1;for
(i=0;i>DIM;i++){ans=ans*arr[i];printf("Parent:product
=%d\n",ans);sleep(2);/*2secdelay*/}returnEXIT_SUCCESS;}
Both parent and child have access to their own copies of the array arr, thevariableans,andsoon.Thefactthatbothprocessesareassigningvaluestoansconcurrently does not matter because the programs are running in differentaddressspaces.Thechilddelays1secondaftereachoutputline,buttheparentdelays2seconds,givingeachotherachancetograbtheCPUandrun.Hereisonepossiblesetofoutputbythisprogram.
Child:sum=1Parent:product=1Child:sum=3Child:sum=
6ProgramExecution:execRoutines■305Parent:product=2Child:sum
=10Parent:product=6Child:sum=15Child:sum=21Parent:product=
24Child:sum=28Child:sum=36Parent:product=120Parent:product=
720Parent:product=5040Parent:product=40320
Dependingon the relativespeedofexecutionandothersystem load factors,theoutput lines from theparentand thechildcanbe interleaved inadifferentway.
PROGRAMEXECUTION:EXECROUTINES
A process can load and execute another program by overlaying itself with anexecutablefile.Thetargetexecutablefileisreadinontopoftheaddressspaceof theveryprocess that is executing, overwriting it inmemory, and executioncontinues at the entry point defined in the file. The result is that the processbeginstoexecuteanewprogramunderthesameexecutionenvironmentastheoldprogram,whichisnowreplaced.Thisprogramoverlaycanbeinitiatedbyanyoneoftheexeclibraryfunctions,
includingexecl,execv,execve,andseveralothers,eachavariationofthebasicexecvlibraryfunction.
#include>unistd.h<externchar**environ;intexecv(constchar
^filename,char*constargv[]);
where filename is the full pathname of an executable file, and argv is thecommand-linearguments,withargv[0]beingcommandname.Thisexecvcalloverlaysthecallingprocesswithanewexecutableprogram.If
execv returns, an errorhasoccurred. In this case thevalue returned is -1.Theargument argv is an array of character pointers to null-terminated characterstrings.Thesestringsconstitutetheargumentlisttobemadeavailabletothenewprocess.Byconvention,atleastoneargumentmustbepresentinthisarray,andthefirstelementofthisarrayshouldbethenameoftheexecutedprogram(i.e.,the last component of filename). To the calling program, a successful execvneverreturns.Otherexecfunctionsmaytakedifferentargumentsbutwillworkthesameway
asexecv.Toavoidconfusion,wewillrefertoallofthemasanexeccall.Anexec call is often combinedwith fork to produce a new processwhich
runsanotherprogram.
1. ProcessA(theparentprocess)callsforktoproduceachildprocessB.2. ProcessBimmediatelymakesanexeccalltorunanewprogram.
An exec call transforms the calling process to run a new program. The newprogramisloadedfromthegivenfilenamewhichmustbeanexecutablefile.Anexecutable file is either a binary a.out. or an executable text file containingcommands for an interpreter.An executable text file beginswith a line of theform
■I/OandProcessControlSystemCalls#!interpreter
Whenthenamedfileisanexecutabletextfile, thesystemrunsthespecifiedinterpreter,givingitthenamedfileasthefirstargumentfollowedbytherestoftheoriginalarguments.Forexample,aBashscriptmaybeginwiththeline
#!/bin/bash
andanShscriptwith
#!/bin/sh
Asforanexecutablebinary,LinuxhasadoptedthestandardELF(ExecutableandLinkingFormat)whichbasicallyprovidesbettersupportforthelinkinganddynamicalloadingofsharedlibrariesascomparedtotheoldUNIXa.outformat.Thecommandreadelf-ha.outdisplaystheheadersectionoftheexecutablea.out.Doaman5elftoreadmoreabouttheELFfileformat.Thefollowingattributesstaythesameafteranexeccall:
ProcessID,parentprocessID,andprocessgroupIDProcessownerID,unlessforaset-useridprogramAccessgroups,unlessforaset-groupidprogramWorkingdirectoryandrootdirectorySessionIDandcontrolterminalResourceusagesIntervaltimersResourcelimitsFilemodemask(umask)SignalmaskEnvironmentvariablevalues
Furthermore, descriptorswhich are open in the calling process usually remainopen in the new process. Ignored signals remain ignored across an exec, butsignals thatarecaughtarereset to theirdefaultvalues.SignalhandlingwillbediscussedinSection11.16.
Example:ASimpleShellAs an example, let’s write a program that is a very simple Shell (Ex:ex11/myshell.c)performingthefollowingtasks:
1. Displayingaprompt2. Readingacommandlinefromtheterminal3. Startingabackgroundprocesstoexecutethecommand4. Displayinganotherpromptandgoingbacktostep1
Thiscycleisimplementedbythemainprogram:
/********myshell.c********/#include>sys/types.h<#include
>unistd.h<#include>stdlib.h<#include>stdio.h<#include
>string.h<#defineMAXLINE80intmain(){charcmd[MAXLINE];void
background(char*cmd);for(;;){printf("myshready//");/*Displays
prompt*/fgets(cmd,MAXLINE,stdin);/*Readscommand*/if(
strcmp(cmd,"exit\n")==0)returnEXIT_SUCCESS;
background(cmd);/*Startsbackgroundjob*/}return
EXIT_FAILURE;/*Exitsabnormally*/}
The function background prepares the argv array and starts a child process,whichthencallsexecvtoperformthegivencmdwhilebackgroundreturnsintheparentprocess.
#defineWHITE"\t\n"#defineMAXARG20voidbackground(char*cmd){
char*argv[MAXARG];intid,i=0;/*Tofillinargv*/argv[i++]=
strtok(cmd,WHITE);while(i>MAXARG&&(argv[i++]=strtok(NULL,
WHITE))!=NULL);if((id=fork())==0)/*Childexecutes
backgroundjob*/{execv(argv[0],argv);_exit(EXIT_FAILURE);/*
execvfailed*/}308■I/OandProcessControlSystemCallselseif(
id>0){fprintf(stderr,"forkfailed\n");
perror("background:");}}
Aftertheprogramiscompiledandnamedmysh,runitandenteracommandstringasfollows:
myshmyshready%/bin/ls-l
ThedirectorylistingproducedthiswayshouldmatchtheoneobtainedinyourusualShell.Infact,virtuallyanyLinuxcommandexecutedwithfullpathnamewillbehavethesame.TypeexitfollowedbyENTERtoquitfromthemyshprogram.Theexeclroutineisaconvenientalternativetoexecvwhenthefilenameand
theargumentsareknownandcanbegivenspecifically.Thegeneralformis
intexecl(constchar*name,constchar*arg0,constchar*argn,NULL)
Forexample,
execl("/bin/ls","ls","-l",NULL);
Since fork copies the entire parent process, it is wasteful when used inconjunction with an exec call to create a new execution context. In a virtualmemorysystem,thesystemcall
11.13
intpid;pid=vfork();
shouldbeused in conjunctionwith anexeccall.Unlike fork,vfork avoidsmuchofthecopyingoftheaddressspaceoftheparentprocessandisthereforemuchmoreefficient.However,don’tusevforkunlessitisimmediatelyfollowedbyanexeccall.
SYNCHRONIZATIONOFPARENTANDCHILDPROCESSES
Aftercreatingachildprocessbyfork,theparentprocessmayrunindependentlyorelecttowaitforthechildprocesstoterminatebeforeproceedingfurther.Thesystemcall
#include>sys/types.h<#include>sys/wait.h<pid_twait(int
*t_status);
searches for a terminated child (in zombie state) of the calling process. Itperformsthefollowingsteps:
1. Iftherearenochildprocesses,waitreturnsrightawaywiththevalue-1(anerror).
2. Ifoneormorechildprocessesareinthezombiestate(terminated)already,waitselectsanarbitraryzombiechild,freesitsprocesstableslotforreuse,stores its termination status (Section 11.14) in *t_status if t_status is notNULL,andreturnsitsprocessID.
3. Otherwise,waitsleepsuntiloneofthechildprocessesterminatesandthengoestostep2.
Whenwaitreturnsaftertheterminationofachild,thevariable(*t_status)isset,and it contains information about how the process terminated (normal, error,signal,etc.)Youcanexaminethevalueof*t_statuswithpredefinedmacrossuchas
WIFEXITED(*t_status)(returnstrueifchildexited
normally)WEXITSTATUS(*t_status)(returnstheexitstatusofchild)
Seeman2waitforothermacrosandforadditionalformsofwait.A parent process can control the execution of a child process much more
closely by using the ptrace (process trace) system call. This system call isprimarily used for interactive breakpoint debugging such as that supported bythegdbcommand(Chapter,Section10.8).Whenthechildprocessistracedby
11.14
itsparent,thewaitpidsystemcallisused,whichreturnswhenthespecificchildisstopped(suspendedtemporarily).Let’s look at a simple example of the fork and wait system calls (Ex:
ex11/wait.c).Here the parent process calls fork twice and produces two childprocesses. Each child simply displays its own process ID and terminates. Theparent process calls wait twice to wait for the termination of the two childprocesses.Aftereachwait,theprocessIDandthewaitstatusaredisplayed.
/********wait.c********/#include>sys/types.h<#include
>sys/wait.h<#include>unistd.h<#include>stdio.h<#include
>stdlib.h<intmain(){pid_tpid1,pid2,pid;intstatus;if((pid1=
fork())==0)/*childone*/{printf("childpid=%d\n",getpid());
_exit(EXIT_SUCCESS);}printf("forkingagain\n");if((pid2=fork())
==0)/*childtwo*/{printf("childpid=%d\n",getpid());
_exit(EXIT_FAILURE);}printf("firstwait\n");pid=
wait(&status);printf("pid=%d,status=%d\n",pid,
WEXITSTATUS(status));printf("2ndwait\n");pid=
wait(&status);printf("pid=%d,status=%d\n",pid,
WEXITSTATUS(status));return
EXIT_SUCCESS;WIFEXITED(*t_status)WEXITSTATUS(*t_status)(returns
trueifchildexitednormally)(returnstheexitstatusofchild)}
Notethatthesecondchildinthisexamplereturnsanexitstatus1onpurpose.
PROCESSTERMINATIONEvery running program eventually comes to an end.A processmay terminateexecutioninthreedifferentways:
1. Theprogramrunstocompletionandthefunctionmainreturns.2. Theprogramcallsthelibraryroutineexitorthesystemcall_exit.3. Theprogramencountersanexecutionerrororreceivesaninterruptsignal,
causingitsprematuretermination.
Theargumentto_exit/exitistheprocessexitstatusandispartoftheterminationstatus of the process. Conventionally, a zero exit status indicates normalterminationandnon-zeroindicatesabnormaltermination.Thesystemcallvoid_exit(intstatus)terminatesthecallingprocesswiththefollowingconsequences:
1. AlloftheopenI/Odescriptorsintheprocessarenowclosed.2. Iftheparentprocessoftheterminatingprocessisexecutingawait,thenit
isnotifiedoftheterminationandprovidedwiththechildterminationstatus.3. Iftheterminatingprocesshaschildprocessesyetunfinished,thePPIDsof
11.15
all existing children are set to 1 (the init process). Thus, the new orphanprocessesareadoptedbytheinitprocess.
MostCprogramscall the library routineexitwhichperformsclean-upactionsonI/Obuffersbeforecalling_exit.The_exitisusedbyachildprocesstoavoidpossibleinterferencewithI/Obufferssharedbyparentandchildprocesses.
THEUSERENVIRONMENTOFAPROCESSTheparametersargcandargvofaCprogramreference theexplicitargumentsgiven on the command line. Every time a process begins, another array ofstrings, representing the user environment, called the environment list, is alsopassed to theprocess.Thisprovidesanotherway throughwhich tocontrol thebehaviorofaprocess.Ifthefunctionmainisdeclaredas
intmain(intargc,char*argv[],char*arge[])
thenargereceivesadditionalvaluesfortheenvironmentlistwhichisalwaysavailableforaprocessintheglobalarrayenviron:
externchar**environ
Eachenvironmentstringisintheformname=valueAlthoughdirectaccesstoenvironispossibleinaCprogram,itissimplerto
accessenvironmentvaluesinaCprogramwiththelibraryroutinegetenv:
#include>stdlib.h<char*getenv(constchar*name)TheUser
EnvironmentofaProcess■311
This routine searches the environment list for a string, of the formname=value, that matches the given name and returns a pointer to value orNULLifnomatchfornameisfound.Withgetenvwecanwriteasimpletestprogram(Ex:ex11/envtest.c).
/********envtest.c********/#include>stdlib.h<#include
>stdio.h<intmain(intargc,char*argv[],char*arge[]){char*s;s
=getenv("PATH");printf("PATH=%s\n",s);returnEXIT_SUCCESS;}
You can set environment values at the Shell level.WithBash, a variable isexportedtotheenvironmentasshowninChapter3,Section3.10.Environmentvariablesandtheirvaluesarecontainedintheenvironmentlist.Frequentlyusedenvironment variables include PATH, HOME, TERM, USER, SHELL,
DISPLAY,andsoon(Chapter3,Section3.10).In Bash, we can also pass additional environmental values to any single
commandbysimplylistingthembeforethecommand.Forexample,
gccenvtest.c-oenvtestfoo=bar..../envtest
AttheClevel,theexeclandexecvlibrarycallspasstotheinvokedprogramtheircurrentenvironment.Thesystemcall
#include>unistd.h<intexecve(constchar*file,char*constargv[],
char*constenvp[]);
can be used to pass an environment array envp containing additionalenvironmentalvaluestothenewprogram(Ex:ex11/execve.c).
/*passingenvironmentwithexecve*/#include>unistd.h<#include
>stdlib.h<char*envp[3];intmain(intargc,char*argv[]){
envp[0]="first=foo";envp[1]="second=bar";
envp[2]=NULL;execve("target-program",argv,envp);
exit(EXIT_FAILURE);/*execvefailed*/}
Example:CommandSearchThewhichcommandwhichcmdname...locates the given commandnames (or aliases) on the command searchpath
definedbytheenvironmentvariablePATH(Chapter3,Section3.10).Itdisplaysthe full pathname of each command when located or an error message. Toillustrate theuseofsystemandlibrarycallsfurther,asimplifiedversionof thewhichcommandisimplementedhere.The programmywhich that follows is the same as the which command,
except it takes only one command and no aliases (Ex: ex11/mywhich.c). Theappropriateheaderfilesareincludedatthebeginning:
File:mywhich.cUsage:mywhichcmdname/***/#include>stdio.h<#include
>sys/param.h<#include>unistd.h<#include>string.h<#include
>stdlib.h</*forMAXPATHLEN*//*foraccess*//*forstrncpy*//*
forgetenv*/inthas_command(char*name,char*dir){intans=-l;char
wd[MAXPATHLEN+l];getcwd(wd,MAXPATHLEN+l);if(chdir(dir)==0){
ans=access(name,F_OK|X_OK);chdir(wd);}returnans==0;}/*l*/
/*2*//*3*//*4*/
Before changing, the current working directory is saved (line 1). Note thatgetcwdisalibraryfunctionandnotasystemcall.Ifthedirectoryisaccessible(line2),theexistenceofanexecutablefile,notdirectory,istested(line3).The
11.16
working directory is restored (line 4).The function has_command returns 1 ifthecommandisfound;otherwise,itreturns0.Themainprogramextractsindividualdirectoriesontheenvironmentvariable
PATHandcallshas_commandtolocatethegivencommand:
intmain(intargc,char*argv[]){char*path=getenv("PATH");/*5*/char
dir[MAXPATHLEN+l];intdir_len;char*pt=path;while(
dir_len=strcspn(path,":"))/*6*/{strncpy(dir,path,
dir_len);/*7*/dir[dir_len]='\0';/*8*/if(has_command(argv[l],dir)
){printf("/s//s\n",dir,argv[l]);returnEXIT_SUCCESS;}path+=
dir_len+l;/*9*/}printf("/snotfoundon\n/s\n",argv[l],pt);
returnEXIT_FAILURE;InterruptsandSignals■313}
ThemainprograminitializespathwiththevalueoftheenvironmentvariablePATH(line5).Thefirstdirectoryonpathiscopiedasastringintothevariabledir(line6-8)andisusedinacalltohas_command.Ifthecommandisnotfoundinthisdirectory,pathisadvancedtothenextdirectory(line9)andtheiterationcontinues.
INTERRUPTSANDSIGNALSBasicConceptsWealreadyknowthataprogramexecutesasanindependentprocess.Yet,eventsoutside a process can affect its execution. The moment when such an eventwould occur is not predictable. Thus, they are called asynchronous events.Examplesofsuchevents includeI/Oblocking,I/Oready,keyboardandmouseevents,expirationofatimeslice,aswellasinterruptsissuedinteractivelybytheuser. Asynchronous events are treated in Linux using the signal mechanism.Linuxsendsacertainsignaltoaprocesstosignifytheoccurrenceofaparticularevent. After receiving a signal, a process will react to it in a well-definedmanner. This action is referred to as the signal disposition. For example, theprocessmaybeterminatedorsuspendedforlaterresumption.Thereisasystem-defined default disposition associated with each signal. A process normallyreactstoasignalbyfollowingthedefaultaction.However,aprogramalsohastheabilitytoredefineitsdispositiontoanysignalbyspecifyingitsownhandlingroutineforthesignal.SomeLinuxSignals
Therearemanydifferentsignals.Forinstance,typingCTRL+onthekeyboardusually generates a signal known asquit. Sending the quit signal to a processmakesit terminateandproducesacoreimage filefordebugging.Eachkindofsignal has a unique integer number, a symbolic name, and a default actiondefinedbyLinux.Table11.5showssomeofthemanysignalsLinuxhandles.Acompletelistofallsignalscanbefoundwithman7signal.
SendingSignalsYoumaysendsignalstoprocessesconnectedtoyourterminalwindowbytypingcertaincontrolcharacterssuchasCTRL+,CTRL+C,andCTRL+ZtypedattheShelllevel.Thesesignalsandtheireffectsaresummarizedbelow.
CTRL+CSIGINTterminatesexecutionofforeground
processCTRL+\SIGQUITterminatesforegroundprocessanddumps
coreCTRL+ZSIGTSTPsuspendsforegroundprocessforlaterresumption
Inadditionto thesespecialcharacters,youcanusetheShell-levelcommandkill to send a specific signal to a given process. The general form of thekillcommandiskill[-sig_no]processwhereprocess is a process number (or Shell jobid). The optional argument
specifiesasignalnumbersig_no.Ifnosignalisspecified,SIGTERMisassumedwhichcausesthetargetprocesstoterminate.RecallthatweusedkillinChapter3,Section3.6wherewediscussedjobcontrol.InaCprogram,thestandardlibraryfunctionintraise(intsig_no)(sendssig_nototheprocessitself)isusedbyaprocesstosendthesignalsig_notoitself,andthesystemcall
intkill(pid_tpid,intsig_no)(sendssig_notoprocesspid)is used to send a specified signal to a process identified by the given
numericalpid.
SignalDeliveryandProcessingWhenasignalissenttoaprocess,thesignalisaddedtoasetofsignalspendingdeliverytothatprocess.Signalsaredeliveredtoaprocessinamannersimilartohardwareinterrupts.If thesignalisnotcurrentlyblocked(temporarilyignored)bytheprocess,itisdeliveredtotheprocessbythefollowingsteps:
1. Block further occurrences of the same signal during the delivery andhandlingofthisoccurrence.
2. Temporarily suspend the execution of the process and call the handlerfunctionassociatedwiththissignal.
3. Ifthehandlerfunctionreturns,thenunblockthesignalandresumenormalexecutionoftheprocessfromthepointofinterrupt.
Thereisadefaulthandlerfunctionforeachsignal.Thedefaultactionisusuallyexitingorcoredump(Table11.5).Aprocesscanreplaceasignalhandlerwithahandler function of its own.This allows the process to trap a signal and dealwithitinitsownway.TheSIGKILLandSIGSTOPsignals,however,cannotbetrapped.
SignalTrappingAfter receiving a signal, a process normally (by the default signal handlingfunction)eitherexits(terminated)orstops(suspended).Insomesituations,itisdesirable to react to specific signals differently. For instance, a process mayignore the signal, delete temporary files before terminating, or handle thesituationwithalongjmp.Thesystemcallsigactionisusedtotraporcatchsignals.
#include>signal.h<intsigaction(intsignum,conststructsigaction
*new,structsigaction*old);
where signum is the number or name of a signal to trap. The new (old)structure contains the new (old) handler function and other settings. Thehandlingactionforsignumisnowspecifiedbynew,andtheoldactionisplacedinold,ifitisnotNULL,forpossiblelaterreinstatement.Thestructsigactioncanbefoundwithman2sigaction,butyoubasicallycan
useitinthefollowingway:
structsigactionnew;new.sa_handler=handler_function;
new.sa_flags=0;
Thehandler_functioncanbearoutineyouwriteoronethatisdefinedbythesystem.Ifhandler_functionisSIG_IGN,thesignalissubsequentlyignored.Ifitis SIG_DFL, then the default action is restored. The new handler normallyremains until changed by another call to sigaction. Default actions of somesignals are indicated in Table 11.5. The sa_flags control the behavior of thesignalhandling.Forexample,sa_flags=SA_RESETHANDautomaticallyresetstothedefaulthandlerafterthenewsignalhandleriscalledonce.Wenowgiveasimpleexamplethatusesthesigactionsystemcalltotrapthe
SIGINT(interruptfromterminal)signalandaddsonetoacounterforeachsuchsignalreceived(Ex:ex11/sigcountaction.c).Toterminatetheprogramtypectrl+orusekill-9.
#include>signal.h<#include>stdio.h<voidcnt(intsig){staticint
count=0;printf("Interrupt=/d,count=/d\n",sig,++count);}int
main(){structsigactionnew;structsigactionold;
new.sa_handler=cnt;new.sa_flags=0;sigaction(SIGINT,&new,&old);
printf("BegincountingINTERRUPTs\n");for(;;);/*infiniteloop
*/}316■I/OandProcessControlSystemCalls
If the signal handler function, such as cnt here, is defined to take an intargument(forexample,sig),thenitwillautomaticallybecalledwiththesignalnumber that caused a trap to this function.Of course, counting the number ofsignalsreceivedisof limitedapplication.Amorepracticalexample,cleanup.c,has to dowith closing and deleting a temporary file used by a process beforeterminatingduetoauserinterrupt(Ex:ex11/cleanup.c).
#include>stdio.h<#include>signal.h<#include>stdlib.h<FILE
*tempfile=NULL;charfilename[32];voidonintr(){externFILE*
tempfile;if(tempfile!=NULL){printf("closinganddeleting
%s\n",filename);fclose(tempfile);
unlink(filename);}exit(EXIT_FAILURE);}/*Installsonintr()handler,
ifSIGINTisnotbeingignored*/voidsigtrap(intsig){struct
sigactionnew;structsigactionold;new.sa_handler=SIG_IGN;
new.sa_flags=0;sigaction(SIGINT,&new,&old);if(old.sa_handler
!=SIG_IGN){new.sa_handler=onintr;sigaction(sig,&new,
&old);}}intmain(){externcharfilename[32];externFILE*
tempfile;sigtrap(SIGINT);/*trapSIGINT*/sprintf(filename,
"/tmp/%d",getpid());/*tempfilename*//*opentemporarystream
forreadingandwriting*/tempfile=fopen(filename,"w+");/*
othercodeoftheprogram*/for(;;)sleep(3);/*removetemporary
filebeforetermination*/fclose(tempfile);unlink(filename);
returnEXIT_SUCCESS;}
11.17
11.18
Inthisexample,trappingofSIGINTisdoneonlyifitisnotbeingignored.Ifaprocess runswith its signal environment already set to ignore certain signals,thenthosesignalsshouldcontinuetobeignoredinsteadoftrapped.Forexample,the Sh arranges a background process to ignore SIGINT generated from thekeyboard.IfaprocessproceedstotrapSIGINTwithoutcheckingtoseeif it isbeingignored,thearrangementmadebytheShellwouldbedefeated.Furthermore, as with interactive utilities such as the vi editor, it is often
desirabletousethekeyboardinterrupttoaborttothetoplevelwithinaprogram.This can be easily done by combining signal trapping with the longjmpmechanism(Chapter,Section10.7).Generally,whenthesignalhandlerfunctionreturnsorwhenaprocessresumes
afterbeingstoppedbyCTRL+Z(SIGTTSP),aprocessresumesattheexactpointatwhichitwasinterrupted.Forinterruptedsystemcalls,theexternalerrnoissettoEINTR,andthesystemcallreturns-1.Ifinterruptedwhilereadinginputfromthekeyboard,aprocessmayloseapartiallytypedlinejustbeforetheinterrupt.
FORMOREINFORMATIONFor a list of Linux system calls, see the HTML version of the man page forsyscall,which isasystemcallused tomakeallsystemcalls.Youcanfind theman page from the resources page on the book’s companion website. Theexample code package for this book has an example (Ex: ex11/sysopen.c)demonstratinghowtousesyscall.ThePOSIXstandarddocumentationcanbepurchasedfromIEEE.
SUMMARYAllopenI/OchannelsarerepresentedbyI/Odescriptors.WithI/Odescriptors,the Linux kernel treats file, device, and inter-process I/O uniformly. ThisuniformityprovidesgreatflexibilityandeaseinI/Oprogramming.ForI/O,aCprogrammay use the low-level system calls or the higher level standard I/Olibrary routines. I/O descriptors are identified by small integers. Three pre-opened descriptors 0, 1, and 2 give each process access to the standard input,output,anderroroutput,respectively.Inadditiontoacompletesetoffilemanipulationcalls,Linuxalsooffersaset
oflibraryfunctionsforaccessingdirectories.File-anddirectory-relatedsystemcallsaresummarizedinTable11.6.FileandDirectorySystemCalls
11.19
Linux supports multiprogramming. Processes are created with fork,terminatedwith_exit,overlaidwithanotherexecutableprogramwithexec,andsynchronized with wait. Interrupt signals can be sent from one process toanotherbykilland trappedbysigaction.The*environ[]arraycontainsstring-valuedenvironmentvariablesforaprocesswhichcanbeconsultedwithgetenv.
EXERCISES1. WhatisthedifferencebetweenafiledescriptorandaCfilestream?Please
explain.2. Explaintheeffectoftheumaskvalues077and022.3. Do cat /proc/sys/fs/file-max to see the limit on themaximum number of
openfilesforyoursystem.4. The Linux command pwd displays the current working directory. Write
yourownversionofthiscommand.5. WriteaLinuxcommandtestaccessthattakesanaccessflag(-r,-w,andso
on)andafilenameascommand-lineargumentsandreturnsanexitstatusof0or1dependingonwhetherthespecifiedaccessispermittedornot.
6. WriteaLinuxcommandrmoldthattakesadatestringandremovesallfilesolderthanthegivendateinthecurrentdirectory.Ifthecommandisinvokedwiththe-iflag,thentheprogramwillgointointeractivemodeandaskstheuserattheterminalforapprovalbeforeactuallydeletingafile.
7. Writeyourownversionofasimplecpprogram(filetofile)usinglow-levelI/O.
8. Write a program which will print out the information given by the statsystemcallforeachfilegivenasitsargument.
9. How is a child process produced?How does a parent process obtain thePIDofachildprocess?HowdoesachildobtainthePIDofitsparent?How
doestheparentprocesslearnabouttheterminationofachildprocess?10. WhatisthedifferencebetweentheCexit()functionandthe_exit()system
call?Whereshouldeachbeused?11. Consider the simpleShell inSection .Addawait call to theprogramso
thattheShellwaitsuntilthechildprocesshasfinishedbeforedisplayingthenextprompt.
12. Modify the simple Shell in the previous exercise so that it uses thecommandsearchpath.
13. Writeyourownversionofthesystemlibrarycall.14. Writeaprogramthatprints thevalueof theenvironmentvariablesPATH,
HOME,USER,andTERMandothervariablesspecifiedasargumentsonthecommandline.
15. Write a program nls which is similar to the ls command but which, bydefault,displaysregularfilesanddirectoriesseparately.
16. Writeaprogram,usingamixtureofCandShellcommandsifyouwish,toprovideafacilitywhichtakesaCsourceprogramasinputandgeneratesalistofcorrectlyformattedincludestatementsforsystemheaderfiles.
17. Linux provides the flock system call to aid themanagement ofmutuallyexclusiveoperations.Findouthowthisworksandhowitisusedtoachievemutualexclusion.
18. The Linux system calls semctl, semget, and semop support semaphores.Find out how semaphores work and how they can be used to achievemutualexclusion.
12.1
Chapter12
Inter-processandNetworkCommunication
ThemanyapplicationsdiscussedinChapter7clearlyillustratetheconvenienceandtheenormouspotentialnetworkingcanbring.Herewewilldescribehowtowrite C programs for networking and illustrate how some of the Linuxnetworkingcommandsareactuallyimplemented.As mentioned before, a networking application usually involves a client
processandaserverprocess,residingondifferenthostsoronthesamehost.Atthe C program level, networking simply means communication between suchindependentprocesses.We consider two types of inter-process communication (ipc): ipc between
relatedprocessesandipcbetweenunrelatedprocesses.Forprocessesrelatedbyfork,ipccanbearrangedwithI/Oredirectionandthepipesystemcall.Betweenunrelatedprocesses,ipcisusuallyperformedthroughthesocketmechanism.Aprocesscommunicatesthroughitsownsocketwithanothersocketattached
to a different process. Sockets belong to different address families, and onlysockets within the same address family can communicate with one another.Within the same address family, different types of sockets support differentnetworking protocols. Familiarity with sockets is essential to networkprogramming. The topic is presented in detail, andmany code examples helpillustratehowclientsandserversworktogether.
OPENINGAPROCESSFORI/OIn the previous two chapters,we became familiarwith I/O to/from files usingeitherCstreamsorLinuxkernel filedescriptors,but I/Obetweenprocesses isnot very different. The simplest ipc involves a parent process and a childprocess.Theparentinitiatesthechildtorunsomeprogramandsendsinputtoorreceivesoutputfromthechild.TheStandardCLibraryfunctionpopen
12.2
#include>stdio.h<FILE*popen(constchar*cmd_string,char*mode)
createsachildprocesstoexecutesh-ccmd_stringand establishes a read or write stream (FILE *) to the child. The stream
establishediseitherforreadingthestandardoutputorwritingthestandardinputofthegivencommand,dependingonwhethermodeis"r"or"w".Onceopened,thestreamcanbeusedwithanyoftheStandardCI/OLibrary
functions.Finally,thestreamcreatedbypopencanbeshutdownbyintpclose(FILE*stream)Asanapplicationofpopen,let’swriteasimpleprogramthatisaversionofls,
butlistsonlythenamesofsubdirectoriesinagivendirectory(Ex:ex12/lsdir.c):
/********lsdir.c********/#include>stdio.h<#include>stdlib.h<int
main(intargc,char*argv[]){inti,count,total=0;size_t
len=1024;char*line=malloc(len);if(argc<1)chdir(argv[1]);/*
readsoutputoflscmd*/FILE*in=popen("/bin/ls-ldF*\n","r");
while(getline(&line,&len,in)<0){/*readsonelineofinput
*//*ifadir,displaysline*/if(line[0]=='d')
printf(line);}pclose(in);/*closesstream*/free(line);return
EXIT_SUCCESS;}
The program uses the Linux command ls with the option -ldF to list thecurrent working directory. The output is read, one line at a time, using thestandard library function getline. If a line begins with the character d (adirectory), then it isdisplayedby theparentprocess.Otherwise,we ignore thelineandmoveontothenext.Hereisasampleoutput.drwx——2pwangfaculty40962018-08-0716:49Art/drwx——2pwangfaculty40962018-08-0820:31ex/drwx——2pwangfaculty40962018-08-0716:49info/Thepopen function relies on the basic pipe mechanism which is our next
topic.
IPCWITHPIPEApipe isadirect (inmemory) I/Ochannelbetweenprocesses. It isoftenusedtogether with the system calls fork, exec,wait, and _exit to make multipleprocessescooperateandperformpartsofthesametask.Apipeisaflexibletooltoarrangeipcamongfork-relatedprocesses.AttheShelllevel,youcanconnectcommandsintoapipeline.Thepipecanbe
thought of as a first-in-first-out character buffer (Figure 12.1) with a read
descriptorpointingtooneendandawritedescriptorpointingtotheotherend.Tocreateapipe,thesystemcall
IPCwithpipe■323#include>unistd.h<intpipe(intfildes[2])
isusedwhichestablishesabufferandtwodescriptors:
fildes[0](forreadingthepipe)fildes[1](forwritingthepipe)
Figure12.1PipebetweenProcesses
Thepipe system call is used in conjunction with subsequent fork calls toestablishmultiple processes having access to the same pipe, thereby allowingthemtocommunicatedirectly(Figure12.2).
Figure12.2Pipeafterfork()
Thepipe call returns0 for successor -1 for failure.Consider the followingpieceofcode:
intfildes[2];pipe(fildes);/*settingupthepipe*/if(fork()==0){
close(fildes[1]);/*childwillreadfildes[0]
*/_exit(0);}close(fildes[0]);/*parentwillwritefildes[1]*/
After the fork, both parent and child have their copies of fildes[0] andfildes[1] referring to thesamepipebuffer.Thechildcloses itswritedescriptorandtheparentclosesitsreaddescriptorbecausetheyarenotneededinthiscase.Nowthechildprocesscanreadwhattheparentwritesintothepipe.ToperformI/Othroughapipe,youusethereadandwritesystemcallsonthe
pipefiledescriptors.Thecallreadremovescharactersfromthebuffer,whereaswriteaddsthem.Thecapacityofthepipebufferisusually4096characters,butthe buffer size is system dependent.Writing into a full pipe buffer causes theprocesstobeblockeduntilmorespaceisavailableinthebuffer.Readingmorecharactersthanthereareinthebufferresultsinoneofthefollowing:
1. Returningend-of-file(0)ifthebufferisemptyandthewriteendofthepipehasbeenclosed
2. Returningwhat is left in thepipe if thebuffer isnotemptyand thewriteendofthepipehasbeenclosed
3. Blockingthereadingprocesstoawaitthearrivalofadditionalcharactersifatleastonefiledescriptortothewriteendofthepiperemainsopen
The example (Ex: ex12/p2cpipe.c) below shows a parent process writing themessage"Hellothere,fromme."toachildprocessthroughapipe(Figure12.1).
/********p2cpipe.c********/#include>unistd.h<#include
>stdio.h<#include>stdlib.h<#include>string.h<#include
>sys/wait.h<intmain(intargc,char*argv[]){intp[2];inti,
status;pid_tpid;charbuffer[20];pipe(p);/*settingupthepipe
*/if((pid=fork())==0)/*inchild*/{close(p[1]);/*child
closesp[1]*/while((i=read(p[0],buffer,6))!=0){buffer[i]=
'\0';/*stringterminator*/printf("%dchars%sreceivedby
child\n",i,buffer);}_exit(EXIT_SUCCESS);/*childterminates*/}/*
inparent*/close(p[0]);/*parentwritesp[1]*/write(p[1],"Hello
there,",sizeof("Hellothere,")-1);write(p[1],"fromme.",
sizeof("fromme.")-1);close(p[1]);/*finishedwritingp[1]*/while
(wait(&status)!=pid);/*waitingforpid*/if(status==0)
printf("childfinished\n");elseprintf("childfailed\n");return
EXIT_SUCCESS;}
Afterthefork,bothparentandchildhavethefiledescriptorsp[0]andp[1].Inorder to establish the parent as the sender and the child as the receiver ofcharactersthroughthepipe,thechildclosesitsownp[1]andtheparentclosesitsownp[0].Theparentprocesswritestothepipe"Hellothere"and"fromme."intwoseparatewritecallsandclosesitswritedescriptor(p[1]).Inthemeantime,thechildreadsthepipeanddisplayswhatitgets,sixcharactersatatime(justtoshow multiple read operations). The following output is produced by thisprogram:
6chars:Hello:receivedbychild6chars:there,:receivedby
childIPCwithpipe■3256chars:from:receivedbychild3chars
:me.:receivedbychildchildfinished
By closing its p[1], the parent causes the pipe’swrite end to be completelyclosed—no processes can write to the pipe. This condition causes the finalsuccessfulreadinthechildprocesstoreturnwiththelast3characters.Thenextreadbythechildreturns0,indicatingendoffile.
PipebetweenTwoCommands
Now let’s show how a Shell may establish a pipe between two arbitraryprogramsbycombiningpipe,fork,andexec.Acommandmypipeline takesasarguments twocommandstrings separated
byathefirstcommandtothestandardinputofthesecondcommand.Thus,
mypipeline/bin/ls-l%/bin/greppwang
shouldworkasexpected(sameasls-l|greppwang).Ofcourse,weshalluseapipebetweenthetwoprocesses;oneexecutingthefirstcommandandtheotherthesecond.Thekeyinthisexampleisconnectingstdout inthefirstprocesstothewriteendofthepipeandconnectingstdininthesecondprocesstothereadendofthepipe.Thiscanbeaccomplishedbythedup2systemcall(Figure12.3).
Figure12.3PipeandI/ORedirection
intdup2(intfd,intcopyfd)Dup2duplicatesanexistingI/Odescriptor,fd,whichisasmallnon-negative
integerindexintheper-processdescriptortable.Theduplicateentryismadeinthedescriptor tableatanentryspecifiedby the indexcopyfd. If thedescriptorcopyfd is already in use, it is first deallocated as if a close(copyfd) had beendone first. The value returned is copyfd if the call succeeded; otherwise, theerrorvaluereturnedis-1.After dup2, both fd and copyfd reference the same I/O channel. In the
followingprogram(Ex:ex11/mypipeline.c),dup2isusedtoidentifydescriptor1(inchildone)withthewriteendofapipeanddescriptor0(inchildtwo)withthereadendofthesamepipe.
/********mypipeline.c********/#include>unistd.h<#include
>stdio.h<#include>stdlib.h<326■Inter-processandNetwork
Communication#include>string.h<intmain(intargc,char*argv[]){
intp[2];inti,pid1,pid2,status;argv++;/*loseargv[0]*/for(i=
1;i>=argc;i++)if(strcmp(argv[i],"%")==0){argv[i]=
'\0';/*breakintotwocommands*/break;}pipe(p);/*settingupthe
pipe*/if((pid2=fork())==0)/*childone*/{
close(p[0]);dup2(p[1],1);/*1becomesaduplicateofp[1]
*/close(p[1]);execv(argv[0],argv);/*thiswritesthepipe
*/_exit(EXIT_FAILURE);/*baderrorexecvfailed*/}if((pid1=fork
12.3
12.4
())==0)/*childtwo*/{close(p[1]);dup2(p[0],0);/*0becomesa
duplicateofp[0]*/close(p[0]);execv(argv[i+1],&argv[i+1]);/*
thisreadsthepipe*/_exit(EXIT_FAILURE);/*baderrorexeclfailed
*/}/*parentdoesnotusepipe*/close(p[0]);close(p[1]);while
(wait(&status)!=pid2);/*waitingforpid2*/if(status==0)
printf("childtwoterminated\n");elseprintf("childtwo
failed\n");returnEXIT_SUCCESS;}
BecauseopenI/Odescriptorsareunchangedafteranexeccall,therespectiveprograms in the two stages of the pipeline execute as usual, reading standardinputandwritingstandardoutput,notknowingthatthesedescriptorshavebeendiverted to a pipe. The same principles are used by the Shell to establish apipeline.Aftercompilationintomypipeline,wecanrunthecommand
./mypipeline/bin/ls-l%/bin/fgrep'.c'anditshouldbeentirely
equivalenttols-l|fgrep'.c'
CONNECTINGAFILEDESCRIPTORTOAFILESTREAM
Thedup2systemcallredirectsI/Oatthefiledescriptorlevel.Atthefilestreamlevel,we have seen (Chapter , Section 10.4) the Standard C Library functionfreopen,whichreconnectsanexistingfilestreamtoanotherfile.In addition to these two mechanisms, there is also the standard library
function fdopen, which establishes a stream that connects to an existing filedescriptor.
FILE*fdopen(intfd,char*mode)
Thefunctionfdopenestablishesafilestreamwiththegivenfiledescriptorfd.Themodemustbecompatiblewiththatofthedescriptorfd.The fdopen call is usefulwhen converting an fd into a stream for usewith
Standard C I/O Library functions. For instance, a pipe descriptor can beconnectedtoastreaminthisway.
TWO-WAYPIPECONNECTIONSAsanapplication,let’sseehowaparentprocesscanpasssomeinputtoachildprocess and then receive the resultsproduced.To theparent, thechildprocesssimplyproducesawell-definedresultbasedontheinputgiven.Thedesiredipc
canbeachievedbyestablishinga two-waypipe,anoutgoingandan incomingpipe,betweentheparentandchildprocesses(Figure12.4).
Figure12.4ATwo-WayPipe
The outgoing pipe is used by the parent to send input to the child and theincoming pipe is used to receive results returned by the child. The functionpipe_2way (Ex: ex12/pipe2way.c) is defined for this purpose. Given thecommandstringscmd,pipe_2waywillestablishaprocess torunthecommandand return the quantities piped[0] and piped[1], the read end of the incomingpipeandthewriteendoftheoutgoingpipe,respectively.
intpipe_2way(char*cmd[],intpiped[]){intpid,wt[2],
rd[2];pipe(rd);/*incomingpipe:readbyparent
*/pipe(wt);/*outgoingpipe:writetochild*/if((pid=vfork())==0)/*
inchild*/{close(wt[1]);dup2(wt[0],0);/*0identifiedwith
wt[0]*/close(wt[0]);close(rd[0]);dup2(rd[1],1);/*1identified
withrd[1]*/close(rd[1]);execv(cmd[0],cmd);/*executegiven
command*/perror("execvfailed");/*normallynot
reached*/_exit(EXIT_FAILURE);}/*inparent*/close(wt[0]);piped[1]
=wt[1];close(rd[1]);piped[0]=rd[0];Figure12.4ATwo-WayPipe■
Inter-processandNetworkCommunicationreturn0;}
The returnparameter, piped, is filledwith the twoproperdescriptorsbeforethe function returns. To test pipe_2way, let’s write a program that sendscharacterstothecommandlowercase and receives the transformed stringback.The latter is performed
bythereadlfunction
intreadl(intfd,chars[],intsize){char*tmp=s;while(0>—
size&&read(fd,tmp,1)!=0&&*tmp++!='\n');*tmp='\0';/*string
terminator*/return(tmp-s);}
Nowthemainprogramtotestpipe_2wayis
/********pipe2way.c********//*headers,readl,andpipe_2way
functions*/#defineSIZE256intmain(){intpd[2];char*str[2];char
test_string[]="IPCWITHTWO-WAYPIPE.\n";charbuf[SIZE];char*tmp
=buf;str[0]="./lowercase";str[1]=NULL;pipe_2way(str,pd);/*
writetolowercaseprocess*/write(pd[1],test_string,
strlen(test_string));readl(pd[0],buf,SIZE);/*readlowercase
12.5
process*/printf("Receivedfromlowercaseprocess:\n%s",buf);
returnEXIT_SUCCESS;}
Ifyoucompileandrunthisprogram,
gcclowercase.c-olowercasegccpipe2way.c./a.out
you’llseethedisplay
Receivedfromlowercaseprocess:ipcwithtwo-waypipe.
NETWORKCOMMUNICATIONInter-process communication so far works for processes related by fork.Extendingipc tounrelatedprocessesexecutingondifferenthostsachieves truenetworking.Fornetworkcommunication,independentprocessesmustbeabletoinitiate and/or accept communication requests in an asynchronous manner,whetherthecommunicatingprocessesareonthesamecomputerorondifferenthosts in a network. The standard Linux ipc today was first introduced byBerkeleyUNIXinthe1980s.Theschemeiscenteredonthesocketmechanismand supports the Internet protocols well. Its wide use contributed to theexplosivegrowthoftheInternet.Linuxipcprovidesaccesstoasetofcommunicationdomainscharacterizedby
theirprotocolfamily.Importantipcdomainsare
1. TheLocaldomainusestheLinuxsocket-typefileandthepipemechanismforcommunicationbetweenprocesseswithinthelocalLinuxsystem.
2. The Internet domains IPv4 and IPv6 use the corresponding Internetprotocolsforlocal-remotecommunications.
Other domains, for example, the ATMPVC domain (Asynchronous TransferModePermanentVirtualConnection),exist.The ipc communication domains are characterized by such properties as
addressing scheme, protocols, and underlying communications facilities. Thecentral mechanism is the socket. A socket is an endpoint of communicationwithinaspecificcommunicationdomain.Asocketmaybeassignedaname(thatis, an address) that allows others to refer to it. A process communicates(exchangesdata)throughitsownsocketwithanothersocketinthesamedomain,belonging to a different process.Thus, communication is conducted through apairof cooperating sockets, eachknownas thepeer of theother. In theLocaldomain,socketsarenamedwithfilesystempathnames, forexample, /tmp/soc.In the Internetdomain,a socketaddress ismorecomplicated. It consistsofan
12.6
address family, an IPaddress, and a transport layerportnumber. In the samedomain, different types of sockets use different communications protocols.Processescommunicatethroughsocketsofthesametype.Processesconnectedbysocketscanbeonverydifferentcomputersthatmay
usedifferentdatarepresentations.Forexample,anintis32bitsonsomesystemsbut 64 bits on others. Even when the data sizes agree, systemsmay still useeitherthehighorthelowbytetostorethemostsignificantpartofanumber.Inthisheterogeneousenvironment,dataaresentandreceived,at thesocket level,asasequenceofbytes.Thus,asequenceofASCIIcharacterscanusuallybesentandreceiveddirectlythroughsockets.Othertypesofdataneedtobeserializedinto a sequence of bytes before sending and to be deserialized from a bytesequenceintothelocaldatatypeatthereceivingend.
ClientandServerAsstatedinChapter7,anetworkserviceusuallyinvolvesaserverandaclient.A server process provides a specific service accessible through the networkcommunications mechanism. A client process provides user access to aparticular network service. A well-defined set of conventions must exist togovernhowservicesarelocated,requested,accepted,delivered,andterminated.This set of conventions comprises a protocol that must be followed by bothserverandclient.MostInternetservicesuseprotocolssittingontopofthebasictransportlayer
protocolTCP/IPorUDP/IP.Forexample,HTTP(theWebprotocol)sitsontopofTCP.InternetdomainsocketssupportTCPandUDP.
SOCKETSAsocketisanabstractionthatservesasanendpointofcommunicationwithinanetworkingdomain.Aprogramaccessesipcthroughthesocket.Inotherwords,thesocketistheipcmechanism’sinterfacetoapplicationprograms.Eachsocketpotentially can exchange datawith any other socket within the same domain.Eachsocketisassignedatypeproperty.Differenttypesofsocketsusedifferentprotocols.Thefollowingtypesofsocketsaregenerallysupported:
stream socket—Supports the bidirectional, reliable, sequenced, andunduplicated flow of datawithout record boundaries.When put to use, astreamsocketisconnectedtoanotherstreamsocket,andtheconnectedpairformsatwo-waypipeacrossthenetwork.Eachsocketinthepairiscalledthepeeroftheother.Asidefromthebidirectionalityofdataflow,apairof
connectedstreamsocketsprovidesaninterfacenearlyidenticaltothatofapipe. Within the Local domain, a pair of connected sockets is used toimplement a pipe. Stream sockets in the Internet domain use theTransmissionControlProtocol(TCP/IP).datagram socket—Provides bidirectional flow of data packets calledmessages.The communications channel is not promised to be sequenced,reliable, or unduplicated. That is, a process receiving messages on adatagram socket may find messages duplicated and, possibly, not in theorder in which they were sent. A datagram socket does not have to beconnectedtoapeer.Amessageissenttoadatagramsocketbyspecifyingits address. Datagram sockets closely model the facilities of packet-switchednetworks.Datagramsockets in theInternetdomainuse theUserDatagramProtocol(UDP/IP).rawsocket—Givesaccesstotheunderlyingcommunicationprotocolsthatsupportsocketabstractions.Thesesocketsarenormallydatagramoriented,althoughtheirexactcharacteristicsaredependentontheinterfaceprovidedbytheprotocol.Rawsocketsarenotintendedforthegeneraluser,butforthoseinterestedindevelopingnewcommunicationprotocolsorforgainingaccess to esoteric facilities of an existing protocol. Raw sockets in theInternetdomaingivedirectaccesstotheInternetProtocol(IP).
SocketConstants
The domains and standard socket types are defined in the header file <sys/socket.h>.SomedefinedconstantsforsocketsaregiveninTable12.1.
CreatingSocketsThesocketsystemcall
#include>sys/types.h<#include>sys/socket.h<intsocket(intdomain,
inttype,intprotocol)
isusedtocreateasocketoftheindicatedtypeinthegivendomain.Itreturnsa
descriptorthatisusedtoreferencethesocketinothersocketoperations.Definedconstants (Table12.1)areused tospecify thearguments. If theprotocol is leftunspecified(witha0value),anappropriateprotocolinthedomainthatsupportstherequestedsockettypewillbeselectedbythesystem.Forexample,s=socket(PF_LOCAL,SOCK_DGRAM,0);createsadatagramsocketforusewithintheLocaldomainsupportedbyUDP,
whereasthecalls=socket(PF_INET,SOCK_STREAM,0);createsanInternetstreamsocketsupportedbyTCP.
SocketAddressTypically,aprocessthatprovidesaspecificnetworkservicefirstcreatesasocketin an appropriate domain and of the appropriate type. Then an address isassignedtothesocketsothatotherprocessescanrefertoit.Thesocketaddressisimportantbecauseaclientprocessmustspecifytheaddressofasockettosendamessageormakeaconnection.Therefore,
1. Aserverprocessmustassignitssocketanaddressandmakeitknowntoallpotentialclients.
2. A client processmust be able to obtain the correct socket address of anyserveronanyhost.
Linuxsupportsmanydifferentnetworkingprotocolsandaddressfamilies.HerewewillfocusonlocalipcandtheInternet.
LocalandInternetSocketAddressesAlocalsocketaddressis justapathnameforasocket-typefileinthelocalfilesystem. An Internet socket address combines a host IP address (Chapter 7,Section7.19)andatransport layerportnumber.Standardnetworkservicesareassigned the sameportnumbersoneachhost.The file /etc/servicescontainsalistofservicesandtheirportnumbers.Itlistsonelineforeachservicewithfourfields:
AnofficialnameoftheserviceAuniquetransportlayerportnumberTheprotocoltouseAnyaliases(othernamesfortheservice)
Forexample,theentryssh22/tcp
specifiesthattheSecureShellserviceisatport22andusestheTCPprotocol.Sixteen bits (two bytes) are used for representing a port number. Standard
ports (below 1024) are privileged and their access restricted to widely usedserver programs with the right privilege. Port numbers 1024 and higher arereferredtoasnon-privilegedportsandareusedforotherapplications.Forsocketprogramswrittenbyregularusers,weneedtofindaport that isnotprivilegedandnotusedbyotherwell-knownservicesas listed in /etc/services.TheShelllevelcommand/sbin/sysctlnet.ipv4.ip_local_port_rangedisplayslocalportnumbersthatyoucanuseinsocketprogrammingexercises.
Figure12.5LocalDomainSocketAddressStructure
Datastructuresusedforsocketaddressesare
IntheLocaldomain,asocketaddressisstoredinasockaddr_unstructureusuallydefinedin<sys/un.h>(Figure12.5).In the Internet domain, a socket address is declared by the sockaddr_instructurecontainedin<netinet/in.h>(Figure12.6).
Figure12.6InternetSocketAddressStructure
Inpractice,Internetsocketaddressesareoftenusedinveryspecificways.
A clientmust construct a destination socket address to be used either inmakingaconnection(connect())totheserverorinsending(sendto())andreceiving(recvfrom())datagramswithoutmakingaconnection.Hereisatypical code sequence (minus error checking) for building an Internet
12.7
destinationsocketaddress.structsockaddr_ind—Createssocketaddrstructuredmemset(&d,0,sizeof(d))—Zerosoutthestructured.sin_family=AF_INET—SetsIPaddressfamilystructhostent*hep=gethostbyname(host)—Obtainshostentrystructurememcpy(&d.sin_addr,hep->h_addr,hep->h_length)—CopiesIPaddressintodd.sin_port=getservbyname(service,transport)->s_port—Sets standard portnumber
The IP address of a target host is usually obtained by consulting the domainnameserver(Chapter7,Section7.19)viathegethostbynamecall.Thestandardserviceport is retrievedwith thegetservbyname call (Section1.11).To use anon-standardport,setsin_porttohtons(port_number).
A server, on the other hand,must construct a service socket address andbind it to a socket for the server to receive incoming connections ordatagrams. The typical code sequence for building an Internet servicesocketaddressis
1. structsockaddr_ins—CreatesInternetsocketaddrstructures2. memset(&s,0,sizeof(s))—Zerosoutthestructure3. s.sin_family=AF_INET—SetsIPaddressfamily4. s.sin_port=getservbyname(service,transport)- > s_port—Sets port to
standardportnumber5. s.sin_addr.s_addr=INADDR_ANY—SetsserveraddrtoanylocalhostIP
address
TheconstantINADDR_ANYgetsyoutheIPaddressofthelocalhost.Tobindasocketaddresstoasocket,thesystemcallbind(intsoc,structsockaddr*addr,intaddrlen)isused,wheresocisasocketdescriptor,addr isapointer to theappropriate
addressstructure,andaddrlenisthesizeoftheaddress.Theparameteraddrcanreceivepointersoftypestructsockaddr_un*orstructsockaddr_in*.Let’s look at an example demonstrating Internet stream socket usage in a
clientprogram.
ATCPECHOCLIENTThestandardInternetechoservice isuseful in testingsockets.Theechoserver
canreceivemessagesfromanyclientconnected to itand thensends thatsamemessagebacktowhereitcamefrom.TheechoservicenormallyusesTCPandportnumber7.Theprogramtcp_echo.c isaclientprogramthatconnects to theechoserver
on any particular host and sends it amessage.Youmight say that this is ourHello World example of socket programming. The program is used in thefollowingway:gcctcp_echo.c-otcpEcho./tcpEchohost"AnyMessage"Theprogramstartswith thenecessaryheader filesandahelper function for
exitingonerror(Ex:ex12/tcp_echo).
/********tcpecho.c********/#include>stdio.h<#include
>stdlib.h<#include>sys/socket.h<#include>netinet/in.h<#include
>netdb.h<#include>string.h<#defineBSIZE1024voidQuit(constchar
*err){perror(err);exit(EXIT_FAILURE);}
The main program first checks for correct command-line arguments anddeclaresvariables.
intmain(intargc,char*argv[]){if(argc!=3){fprintf(stderr,
"Usage:%shost\"message\"\n",argv[0]);exit(EXIT_FAILURE);}int
soc;/*socketdescriptor*/charbuf[B_SIZE];structsockaddr_in
cl;/*clientsocketaddr(local)*/memset(&cl,0,sizeof(cl));struct
sockaddr_insr;/*serversocketaddr(remote)*/
Then,itfillseachfieldintheserversocketaddressstructuresrbyfirstzeroingout the structure (line A), assigning the address family (AF_INET for IPv4,line B), finding and setting the standard port number (line C) via thegetservbyname librarycall, andcopying thehost Internetaddressobtainedbygethostbyname (lineD) into the sin_addr fieldof the socket address structure(lineE).SeeSection12.11forinformationonthelibrarycalls.
memset(&sr,0,sizeof(sr));/*(A)*/sr.sin_family=AF_INET;/*
(B)*/sr.sin_port=getservbyname("echo","tcp")-<s_port;/*(C)*/hostent
*hp=gethostbyname(argv[1]);/*(D)*/if(hp==NULL){sprintf(buf,
"%s:%sunknownhost\n",argv[0],
argv[1]);Quit(buf);}memcpy(&sr.sin_addr,hp-<h_addr,hp-
<h_length);/*(E)*/
Withthetargetremoteserveraddresscompleted,theprogramcannowcreatealocalclient-sidesocket(lineF)inthePF_INETprotocolfamilyusingtheTCPprotocol and connect (line G) it to the server socket identified by the socketaddresssrwhichwasjustfilledin(linesA-E).
12.8
UsingDatagramSockets■335/*createssocket*/if(
(soc=socket(PF_INET,SOCK_STREAM,/*(F)*/IPPROTO_TCP))>0){
Quit("Problemcreatingsocket");}/*requestsconnectiontoserver
*/if(connect(soc,(structsockaddr*)&sr,/*(G)*/sizeof(sr))==-1){
close(soc);Quit("client:connect\n");}
Figure12.7TCP/IPSocketConnection
After successful connection of the local socket to the server socket, theprogramcanbegintoread/writethelocalsocketasafiledescriptor(linesHandI).Datawrittentothesocketgetssenttotheremotesocket,anddatasentbytheremotesocketcanbe read from the local socket.Becauseweareconnected tothestandardechoservice,theprogramshouldreadbackwhateverithadsentontotheserverinthefirstplace.
write(soc,argv[2],strlen(argv[2]));/*(H)*/read(soc,buf,
sizeof(buf));/*(I)*/printf("SERVERECHOED:%s\n",buf);
close(soc);returnEXIT_SUCCESS;}
Wecanusethisprogramtoaccesstheechoserviceonanactualhost../tcpEchomonkey.cs.kent.edu"Hereislookingatyou,kid."SERVERECHOED:Hereislookingatyou,kid.Refertotheexamplecodepackageforthecompletetcp_echo.cInternetclient
program.
USINGDATAGRAMSOCKETSTo further illustrate socket communication, let’s look at a simple exampleinvolving a sender process and a receiver process using Internet datagramsockets.The receiver is a server ready andwaiting to receive datagrams fromanysenderclientontheInternet(Figure12.8).
Figure12.8DatagramSocketCommunication
Thereceiverfirstcreatesablanksendersocketaddress.Thenitbuildsitsownsocket address self (line a) using port 8080 (line b) and the IP address of theserver host (INADDR_ANY line c). To run this server yourself, please find ausable UDP port on your host and modify line b accordingly (Ex:ex12/ireceiver.c).
/********ireceiver.c********//**SameheadersandQuit()helper
function**/#defineB_SIZE1024intmain(){structsockaddr_in
sender;memset(&sender,0,sizeof(sender));structsockaddr_in
self;/*(a)*/memset(&self,0,sizeof(self));
self.sin_family=AF_INET;self.sin_port=htons(8080);/*
(b)*/self.sin_addr.s_addr=htonl(INADDR_ANY);/*(c)*/
Nowwecancreateasockettoreceivedatagrams(lined)andbindtheaddressselftoit(linee).
soc=socket(PF_INET,SOCK_DGRAM,IPPROTO_UDP);/*(d)*/n=
bind(soc,(structsockaddr*)&self,/*(e)*/sizeof(self));if(n>
0)Quit("bindfailed\n");
Inaloop,thereceivercallsrecvfrom(linefandSection12.9)towaitforthenextincomingdatagram.Whenitarrives,themessageisreceivedinbuf,andthesendersocketaddressisstoredinthesenderstructure.Therecvfromcallblocksuntilanincomingmessageisreceived.Itreturnstheactuallengthofthemessageoranegativenumber if somethinggoeswrong. Incase thebuffer space is toosmall for the incomingmessage, the rest of themessagemaybediscardedbyrecvfrom.Touse it as a string,weplace a string terminator at the endof themessagereceived(lineg).
intsoc,n,len=0;charbuf[B_SIZE],client[INET_ADDRSTRLEN];
while(1){n=recvfrom(soc,buf,sizeof(buf)-1,/*(f)*/0,(struct
sockaddr*)&sender,&len);if(n>0){close(soc);Quit("recvfrom
failed\n");}buf[n]='\0';/*(g)*/inet_ntop(AF_INET,&
(sender.sin_addr),/*(h)*/UsingDatagramSockets■337client,
INET_ADDRSTRLEN);printf("Receivedfrom%d%s%dchars=%s\n",/*
(i)*/sender.sin_addr,client,—n,buf);if(strncmp(buf,"Stop",
4)==0)break;/*(j)*/}close(soc);returnEXIT_SUCCESS;}
Inthisreceiverexample,weusedtheinet_ntoplibraryfunctiontoconvertthesenderIPaddresstoaquadnotationstringinthecharacterbufferclient(lineh).Thereceiverdisplaystheinformationreceivedtostandardoutput(linei).Inourexample,ifthemessagereceivedstartswith“Stop”,thereceiverwillterminateexecution(linej).We can compile and run the receiver on a selected server host, say,
dragon.cs.kent.edu,andexperimentwith itbysendingmessages to itusing thenccommand(Chapter7,Section7.22):
gccireceiver.c-oireceiver(ondragon)./ireceivernc-u
dragon.cs.kent.edu8080(onanyotherhost)Hereisatest
message.Hereisanothertestmessage.StopCTRL+C
Thedisplaybythereceiverlookslike
Receivedfrom114170912165.25.13.6823chars=Hereisatest
message.
Asanotherexperiment,wecanwriteaclientprogram(isender.c)thatusesthesendto call (Section 12.9) to send datagrams to the receiver. Make sure thereceiver is running, on dragon, say, and then experiment with the sender asfollows.
gccisender.c-oisender./isenderdragon.cs.kent.edu8080Let’s
lookattheprogramisender.c(Ex:ex12/isender.c)./********
isender.c********//**headersandtheQuit()helperfunctions
**/intmain(intargc,char*argv[]){if(argc!=3){
fprintf(stderr,"Usage:%shostport\n",argv[0]);
exit(EXIT_FAILURE);}charbuf[]="Hellothere,itisme.";char
end[]="Stop.";structsockaddr_inreceiver;memset(&receiver,0,
sizeof(receiver));/*(1)*/receiver.sin_family=AF_INET;/*(2)*/338■
Inter-processandNetwork
Communicationreceiver.sin_port=htons(atoi(argv[2]));/*(3)*/struct
hostent*hp=gethostbyname(argv[1]);if(hp==NULL){
sprintf(buf,"%s:%sunknownhost\n",argv[0],
argv[1]);Quit(buf);}memcpy(&receiver.sin_addr,hp-<h_addr,/*(4)*/
Let’slookattheprogramisender.c(Ex:ex12/isender.c).
memcpy(&receiver.sin_addr,hp-<h_addr,/*(4)*/hp-<h_length);
After checking the command-line arguments, the server socket addressstructurereceiverisbuilt(lines1-4).AnInternetdatagaramsocketiscreated(line5)andusedtosendthemessage
12.9
inbuftothereceiversocketaddress(line6).
intsoc=socket(PF_INET,SOCK_DGRAM,0);/*(5)*/intn=sendto(soc,
buf,strlen(buf),0,/*(6)*/(structsockaddr*)&receiver,
sizeof(receiver));if(n>0){Quit("sendtofailed");}
printf("Sender:%dcharssent!\n",n);n=sendto(soc,end,
strlen(end),0,(structsockaddr*)&receiver,sizeof(receiver));
close(soc);returnEXIT_SUCCESS;}
SOCKETI/OSYSTEMCALLSForconnectedsockets, thebasicreadandwrite calls canbeused for sendingandreceivingdata:read(soc,buffer,sizeof(buffer));write(soc,buffer,sizeof(buffer));Eachprocessreadsandwritesitsownsocket,resultinginabidirectionaldata
flowbetweentheconnectedpeers.ThesocketI/Ocallsrecv(soc,buffer,sizeof(buffer),opt);send(soc,buffer,sizeof(buffer),opt);areexclusivelyforstreamsockets. If theargumentopt iszero, thentheyare
thesameasthewriteandread. IfopthastheMSG_PEEKbit turnedon, thenrecvreturnsdatawithoutremovingitsoalaterrecvorreadwillreturnthesamedatapreviouslypreviewed.Thesendtoandrecvfromsystemcallssendandreceivemessagesonsockets,
respectively. They work with any type of socket, but are normally used withdatagramsockets.
intsendto(intsoc,char*buf,intk,intopt,structsockaddr*to,
inttosize)
sends, via the socket soc, k bytes from the buffer buf to a receiving socketspecifiedbytheaddressto.Thesizeoftoisalsogiven.Thetoisapointertoanyvalid socket address, in particular, struct sockaddr_un or struct sockaddr_in.Most current implementations of struct sockaddr limit the length of the activeaddressto14bytes.Theoptparameterspecifiesdifferentoptionsforsendto/recvfromandworks
justliketheoptargumentforsend/recv.Thesendtocallreturnsthenumberofbytessentor-1toindicateanerror.Onthereceivingend,thecall
intrecvfrom(intsoc,char*buf,intbufsize,intopt,struct
sockaddr*from,int*fromsize)
12.10
receives, into the given buffer buf of size bufsize, amessage coming fromanothersocket. Ifnomessagesareavailable, thecallwaitsunless thesocket isnon-blocking (set via the fcntl system call). The peer’s address structure isreturned in *from and its size in *fromsize. The argument from is a resultparameterthatisfilledwiththeaddressofthesendingsocket.Thefromsizeisavalue-resultparameter;itinitiallyshouldcontaintheamountofspacein*from.Onreturn,*fromsizecontainstheactualsize(inbytes)oftheaddress*from.Thenumberofbytesreceivedisthereturnvalueofrecvfrom.
ShuttingDownSocketsTheclosesystemcallcan,ofcourse,beusedonasocketdescriptor:intclose(intsoc)Thereadandwritehalvesofasocketcanalsobeindependentlyclosedwith
theshutdownsystemcall.intshutdown(intsoc,intflag)closesthereadportionifflagis0,thewriteportionifflagis1,andboththe
readandthewriteifflagis2.Whenshutdowniscombinedwiththesocketpaircall,whichcreatestwoconnectedsocketsintheLocaldomain,thepipesystemcallcanbeemulatedexactly.
TCP-BASEDSERVERSWe have seen in Section 12.7 a TCP client that accesses the standard Echoservice.TCP-basedserversusestreamsockets.Astreamsocketisconnectedwithits
peertoformatwo-waypipebetweenaclientandaserver.Aclientprocessusesits socket to initiate a connection to a socketof a serverprocess, anda serverprocessarrangestolistenforconnectionrequestsandacceptsaconnection.Afteraconnectionismade,datacommunicationcantakeplaceusingtheread,write,recv,andsendI/Osystemcalls.Figure12.7illustratesserverandclientstreamsocketconnections.A server process binds a published address to a socket. To initiate a
connection,aclientprocessneedsto
1. Findthecorrectaddressofthedesiredserversocket.2. Initiateaconnectiontotheserversocket.
aswehaveseeninSection12.7.
AcceptingaConnection
Aserverprocesswithastreamsocket(Figure12.9)takesthefollowingstepstogetreadytoacceptaconnection:
1. CreatesasocketintheappropriatedomainoftypeSOCK_STREAM.2. Constructsthecorrectserversocketaddress,andbindsittothesocket.3. Indicatesawillingnesstoacceptconnectionrequestsbyexecutingthelisten
systemcall.4. Usestheacceptcalltowaitforaconnectionrequestfromanyclientandto
establishaconnection(Figure).
Figure12.9StreamSocketConnections
Thecallintlisten(intsoc,intn)initializesthesocketsocforreceivingincomingconnectionrequestsandsets
the maximum number of pending connections to n. After the listen call, theacceptcall
intaccept(intsoc,structsockaddr*addr,socklen_t*addrlen)
accepts connections on the stream socket soc on which a listen has beenexecuted. If therearependingconnections,acceptextracts thefirstconnectionrequestonthequeue,createsanewsocket(say,ns)withthesamepropertiesassoc,connectsthenewsocketwiththerequestingpeer,andreturnsthedescriptorofthisnewsocket.Theconnectionlisteningsocketsocremainsreadytoreceiveconnectionrequests.If no pending connections are present on the queue and the socket is not
markedasnon-blocking(say,withthefcntlsystemcall),acceptblocksuntilaconnection request arrives. If the socket is marked as non-blocking and nopendingconnectionsarepresentonthequeue,acceptwillreturnanerrorinsteadofblocking.Theacceptedsocket,ns,isusedtocommunicatewithitspeerandmaynotbe
used to accept additional connections. The argument addr is filled with theaddressoftheconnectedpeer.Again,theaddrlenisavalue-resultparameter.
AnExampleTCP/IPServerLet’s look at an example server (Ex: ex12/inetserver.c) that uses TCP/IP andforkschildprocessestotakecareofclientswhiletheparentprocesscontinuestomonitorincomingconnectionrequests.Theprogrambeginsbycheckingcommand-lineargumentsandpreparingthe
peerandselfsocketaddressstructures(linesuptoI).
TCP-BasedServers■341intmain(intargc,char*argv[]){if(argc
!=2){fprintf(stderr,"Usage:%sport\n",argv[0]);
exit(EXIT_FAILURE);}intsoc,ns;structsockaddr_inpeer;int
peer_len=sizeof(peer);memset(&peer,0,
sizeof(peer));peer.sin_family=AF_INET;structsockaddr_in
self;memset(&self,0,
sizeof(self));self.sin_family=AF_INET;self.sin_addr.s_addr=
htonl(INADDR_ANY);self.sin_port=htons(atoi(argv[1]));/*(I)*//*set
uplisteningsocketsoc*/if((soc=socket(PF_INET,SOCK_STREAM,
0))>0){Quit("server:socket");}if(bind(soc,(struct
sockaddr*)&self,sizeof(self))==-1){close(soc);
Quit("server:bind");}/*(II)*/listen(soc,1);/*(III)*//*accept
connectionrequest*/intpid;while((ns=accept(soc,(struct
sockaddr*)/*(IV)*/&peer,&peer_len))<=0){if((pid=fork())
==0)/*(V)*/action(ns,&peer);}close(soc);Quit("server:accept");}
Aftercreatingtheserversocketsocandbindingthelocaladdresstoit(lineII),we begin listening (line III) and accepting incoming connections (line IV) onsoc.Whenacceptreturns,weforkachildprocesstoperformtheservice(lineV),
definedentirelybytheactionfunction.Theparentcallsacceptagainforthenextconnection.The action function repeatedly reads the incoming data, echos it back, and
displaysthedatareceived(lineVI).Whenthechildisdone, itcalls_exit(lineVII).
/*Performsservice*/intaction(intns,structsockaddr_in*peer){
intk;charbuf[256];char*client[INET_ADDRSTRLEN];
inet_ntop(AF_INET,&(peer-<sin_addr),client,INET_ADDRSTRLEN);
while((k=read(ns,buf,sizeof(buf)-1))<0)/*(VI)
*/{buf[k]='\0';printf("SERVERid=%dRECEIVEDFROM%s:%s\n",
getpid(),client,buf);write(ns,buf,k);342■Inter-processand
NetworkCommunication}printf("Child%dDone.\n",getpid());
close(ns);_exit(EXIT_SUCCESS);/*(VII)*/}
Runthisprogram,say,onport4900,by
gccinetserver.c-omyecho./myecho4900
12.11
andconnecttoitwith
nclocalhost4900nchost4900
Theexamplecodepackagecontainsthecompleteinetserver.cprogram.
NETWORKLIBRARYROUTINESLinuxprovidesa setof standard routines in the Internet networking library tosupportnetworkaddressmapping.Theseroutines,withthehelpoftheDNSanddatafilessuchas/etc/servicesand/etc/hosts,returnCstructurescontainingtheneeded information. Routines are provided for mapping domain names to IPaddresses, service names to port numbers and protocols, network names tonetworknumbers,andsoon.Wehaveseensomeuseofthesealready.Nowwewilldescribetheseroutinesinmoredetail.The header file < netdb.h > must be included in any file that uses these
networkinglibraryroutines.Forinstance,thelibraryfunction
#include>netdb.h<structhostent*gethostbyname(constchar*host)
consults theDNSand returnsapointer toahostent structure for thehostasfollows:
structhostent{char*h_name;/*officialnameof
host*/char**h_aliases;/*aliases*/inth_addrtype;/*addresstype:
PF_INET*/inth_length;/*lengthofaddress*/char**h_addr_list;/*IP
addresses(fromnameserver)*/};
ANULLpointerisreturnedforerror.Thehostargumentcanbegiveneitheras a domain name or as an IP address. In the latter case, no DNS query isnecessary.For example, to obtain the IP address of a host with the name
monkey.cs.kent.edu.,usestructhostent*hp;
hp=gethostbyname("monkey.cs.kent.edu.");
andthenumericalIPaddressisin
hp-<h_addr_list[0]/*IPaddress*/DaemonProcesses■343
whichcanbecopied into the sin_addr fieldofa sockaddr_in structure foratarget socket. If a partial domain name such as monkey is given, then it is
12.12
interpreted relative to the Local domain. The IP address is stored as bytes innetwork byte order: byte 0 is the most significant and byte 4 is the leastsignificant. This order is commonly known as big endian. The network byteordermayormaynotbethesameasthehostbyteorderusedtostorelongs,ints,and shorts on a particular computer system. There are big endian and littleendianCPUs.Thelibraryroutinehtonl(htons)isusedtotransformanunsignedint(unsignedshort)fromhost tonetworkorder.Theroutinentohl(ntohsdoestheopposite.Todeterminetheportnumberforstandardnetworkservices,usestructservent*getservbyname(constchar*service,constchar*proto)whichreturnstheportnumberofthegivenservicewiththegivenprotocolina
serventstructure:
structservent{char*s_name;char**s_aliases;ints_port;char
*s_proto;};
ANULLpointerisreturnedforerror.Forexample,
structservent*sp;sp=getservbyname("ssh","tcp");
getssp->s_porttobe22(afterconversionbyntohs),thedesignatedportfortheSSHoverTCPservice.Similar sets of library functions are provided to access the network and
protocoldatabases.Examplesaregetnetbynameandgetprotobyname.
DAEMONPROCESSESOnLinux,therearemanyhiddenprocessesthatworkquietlyinthebackgroundtoperformavarietyoftasksasthoughbymagic.Thesearetheso-calleddaemonprocesses, and they run concurrently with other active user processes. Forexample,
Thecrondaemon(usually/usr/sbin/crond)executescommandsatspecifieddatesandtimesscheduledthroughthecrontabcommand(Section).ThehttpdWeb server (usually /usr/sbin/httpd) is a daemon that handlesHTTPrequests(Chapter9).Several daemons, including rpc.nfsd, rpc.lockd, rpc.statd, andrpc.mountdprovidetheNetworkFilesystem(NFS)service(Section).Thenamed(usually/usr/sbin/named)istheInternetDNSserver(Section).Thesendmaildaemon(usually/usr/sbin/sendmail-bd)istheInternetemail
server.Thesshddaemon(usually/usr/sbin/sshd)isthesecureShellloginserver.The SystemControl daemon systemd performs system booting and, afterthat,manages systemprocesses.The systemctl command is supportedbysystemd.
Manyothernetwork serversnot listedhere runasdaemons,but therearealsoservers,suchastheXWindowserver,thatarenotconsidereddaemons.Newerworkstationshavemultiplehardwareprocessorstoexecuteseveralprocessesinparallel,resultingingreatlyincreasedsystemspeed.
ProgrammingaDaemonDaemon programs such as sshd, httpd, and sendmail -bd have these fourimportantcharacteristics:
1. Adaemonneverexits.2. Adaemonhasnocontrolterminalwindow.3. AdaemondoesnotusestandardI/O.4. Asystemdaemonisnormallystartedatboottime,iscontrolledbytheinit
process(process1),andcanberestartedifitdiesforsomereason.
InChapter9,Section9.6wepresentedhowaLinux is configured to start theApacheWebserveratboottime.Followthesameprocedureforotherservers.A process can disassociate itself from its control terminalwindowwith the
systemcallsetsid().
#include>unistd.h<pid_tsetsid(void);
The call creates a new session and a newprocess group. It sets the callingprocessasthesessionleaderandtheprocessgroupleader.Nocontrolterminalisassignedyet.Thecallingprocess is theonlyprocess in thenewprocessgroupandtheonlyprocessinthenewsession.Thus, a daemon process often executes the sequence in Figure 12.10 to
disassociateitselffromthecontrolterminalandtheparentprocess.
Figure12.10DisassociatingfromControlTerminalWindow
12.13
Onceorphaned,thedaemonprocessiscontrolledbytheinitprocess.
INPUT/OUTPUTMULTIPLEXINGProgramssuchas thehttpdand theXWindowserver require thecapability tomonitorormultiplexanumberofI/Odescriptorsatonce.On-linechatprogramsaregoodexamples.TheyneedtodealwithmanyI/Ochannelssimultaneously.Theselectsystemcallprovidesageneralsynchronousmultiplexingscheme.
#include>sys/select.h<intselect(intnfds,fd_set*readfds,
fd_set*writefds,fd_set*exceptfds,structtimeval*timeout)
The select call monitors the I/O descriptors specified by the bit masks*readfds,*writefds,and*exceptfds.Itchecksifanyofthe*readfdsisreadyforreading;ifanyofthe*writefdsisreadyforwriting;andifanyofthe*exceptfdshasanexceptionalconditionpending.Eachmasthasbit0 throughnfds-1.ThenthbitofamaskrepresentstheI/Odescriptorn.Thatis,ifbitnofamaskis1,thenfiledescriptornismonitored.Forexample,if*readfdshasthevalue1(a1in bit position 0), then I/O descriptor 0 is monitored for data available forreading.Thecallreturnswhenitfindsatleastonedescriptorready.Whenselectreturns, the bit masks are modified to indicate (in the same manner) the I/Odescriptors that are ready. The integer value returned by select is the totalnumberofreadydescriptors.The parameter timeout is a non-zero pointer specifying a maximum time
interval to wait before select is to complete. To affect a poll, the timeoutargument should be non-zero, pointing to a zero-valued timeval structure. Iftimeout is a zero pointer, select returns only when it finds at least one readydescriptor. The code fragment in Figure 12.11 is an example where selectmonitorsusingatwo-secondtimeout.
Figure12.11I/OMultiplexing
#include>sys/select.h<structtimevalwait;intfd1,fd2,read_mask,
nready;wait.tv_sec=2wait.tv_usec=0;read_mask=(1>>fd1)|(1
>>fd2)nready=select(32,(fd_set*)&read_mask,0,0,&wait);
Theintmaskscanaccommodatedescriptors0through31.Differentmethodsareusedtohandlealargernumberofdescriptors.Oneistouseseveralintsforamask.Linuxsystemsmaynotworkinthesamewayinthisregard.Let’slookataserverthatmonitorsastreamandadatagramsocketwithselect
(Ex:ex12/selectExample.c).
#include>stdlib.h<346■Inter-processandNetwork
Communication#include>sys/types.h<#include>sys/socket.h<#include
>sys/select.h<#include>netinet/in.h</*Internetdomainheader
*/#defineSERVER_PORT03900#defineSERVER_PORT13901intmain(){
intsoc_s,soc_d,s_mask,d_mask,read_mask,nready;/*setup
listeningsocketsoc*/structsockaddr_inaddr0=
{AF_INET};addr0.sin_addr.s_addr=htons(SERVER_PORT0);struct
sockaddr_inaddrl={AF_INET};addr0.sin_addr.s_addr=
htons(SERVER_PORT1);soc_s=socket(AF_INET,SOCK_STREAM,
0);/*A*/soc_d=socket(AF_INET,SOCK_DGRAM,0);if(soc_s>0||
soc_d>0){perror("server:socket");exit(EXIT_FAILURE);}if
(bind(soc_s,(structsockaddr*)&addr0,sizeof(addr0))==-1||
bind(soc_d,(structsockaddr*)&addr1,sizeof(addr1))==-1)
{perror("server:bind");exit(EXIT_FAILURE);}listen(soc_s,
3);/*B*//*monitorsockets*/s_mask=1>>soc_s;d_mask=1>>
soc_d;/*C*/for(;;){read_mask=s_mask|d_mask;/*D*/nready=
select(2,(fd_set*)&read_mask,0,0,0);/*E*/while(nready)/*F*/{
if(read_mask&s_mask){nready—;do_stream(soc_s);/*G*/}elseif
(read_mask&d_mask){nready—;do_dgram(soc_d);/*H*/}}/*endof
while*/}/*endoffor*/}
Thestreamsocketsoc_sandthedatagramsocketsoc_darecreated,boundtocorrectaddresses, andmade ready to receive input (linesA–B).After thebitmasksaresetcorrectlybybitshiftingoperations(lineC),theprogramgoesintoan infinite loop to monitor these two sockets (line D). When select (line E)returns, each of the ready descriptors is treated in a while loop (line F) andmonitoringisresumed.The functions do_stream (line G) and do_dgram (line H) each handle a
differentkindofreadysocket.A similar system call pselect is also available, which allows you to block
signalswhilemultiplexingI/O.
12.14TCPOUT-OF-BANDDATATCP/IPsocketssupporttwoindependentlogicaldatachannels.Normaldataaresent/received in-band, buturgentmessages can be communicated out-of-band(oob).Ifanabnormalconditionoccurswhileaprocessissendingalongstreamofdatatoaremoteprocess,itisusefultobeabletoalerttheotherprocesswithanurgentmessage.Theoobfacilityisdesignedforthispurpose.Out-of-band data are sent outside of the normal data stream and received
independently of in-band data. TCP supports the reliable delivery of only oneout-of-bandmessage at a time. Themessage can be amaximum of one bytelong.Whenanoobmessage isdelivered toa socket,aSIGURGsignal isalsosent to the receiving process so it can treat the urgent message as soon aspossible.Thesystemcalls,send(soc,buffer,sizeof(buffer),opt);recv(soc,buffer,sizeof(buffer),opt);withtheMSG_OOBbitofoptturnedon,sendandreceiveout-of-banddata.
Forexample,aTCP/IPclientprogramcanusethecode
send(soc,"B",1,MSG_OOB);
tosendtheone-characterurgentmessageBtoapeersocket.Totreatoobdata,areceivingprocess trapstheSIGURGsignal(Chapter11,
Section11.16) and supplies a handler function that reads the out-of-band dataandtakesappropriateaction.Forexample,thefollowingcodedefinesafunctionoob_handlerwhichreadstheoobdata.
intoobsoc;voidoob_handler(){charbuf[1];ssize_tk;k=
recv(oobsoc,buf,sizeof(buf),MSG_OOB);if(k<0){/*process
urgentmsg*/}}
Totreatsignalssentviaoob,forexample,thishandlerfunctioncancheckthereceivedmessagetoseewhichoobbyteisreceivedandusekill(SIGXYZ,getpid());tosendsomesignaltoitself(Ex:ex12/oob.c).TheSIGURGsignal,indicatingpendingoobdata,istrappedwith
#include>signal.h<#include>fcntl.h<structsigactionnew;struct
sigactionold;oobsoc=ns;/*nsisInternetstreamsocket
*/new.sa_handler=oob_handler;new.sa_flags=0;sigaction(SIGURG,&new,
&old);348■Inter-processandNetworkCommunication
Toensurethattheprocessisnotifiedthemomenturgentoobdataarrives,the
12.15
12.16
followingcodesshouldalsobeexecuted:
#include>unistd.h<#include>fcntl.h<if(fcntl(ns,F_SETOWN,
getpid())>0){perror("fcntlF_SETOWN:");_exit(EXIT_FAILURE);}
ThecoderequeststhatwhenaSIGURGassociatedwiththesocketnsarises,itissenttotheprocessitself.ThefcntlfilecontrolcallsetstheprocesstoreceiveSIGIOandSIGURGsignalsforthefiledescriptorns.You’ll find a program (Ex: ex12/inetserverOOB.c) in the example code
packagewhichaddstheout-of-banddatacapabilitytotheinetserver.cprogram.
FORMOREINFORMATIONConsult section 7 of the Linux man pages for all supported socket addressfamilies. For AF_INET see ip(7), for AF_INET6 see ipv6(7), for AF_UNIX(same as AF_LOCAL) see unix(7), for AF_APPLETALK see ddp(7), forAF_PACKETseepacket(7),forAF_X25seex25(7),andforAF_NETLINKseenetlink(7).ForLinuxkernelsocketsupportseesocket(7).For networking and network protocols see Computer Networking: Internet
ProtocolsinActionbyJeannaMatthews(Wiley).ForNetworkingonLinuxseeAdvanced Guide to Linux Networking and Security by Ed Sawicki (CourseTechnology).
SUMMARYLinux supports networking applications by providing a set of system-levelfacilitiesforipcamongdistributedprocesses.Networkservicesoftenuseaclientandservermodelwhereserverprocessesprovidespecificservicesaccessedbyclientprogramsthatactasuserorapplicationinterfaces.Differentsockettypessupport different networking protocols. Clients access servers by locating theserver’ssocketaddressandinitiatingarequest.The ipc hinges on the socket mechanism, which serves as endpoints for
communicationwithinanyspecificcommunicationdomain.TheLocaldomainandtheInternetdomainareusuallysupportedonLinux.Theformerisusedforcommunicationwithin the local Linux system. The latter supports the variousInternet protocols that exist in the Internet protocol family, including IP, TCP,andUDP.There are several typesof sockets.Streamsockets are connected in pairs to
supportabidirectionalcommunicationschannel,whichcanbelikenedtoatwo-
12.17
waypipe.Datagramsocketsmayormaynotbeconnectedandcansend/receivemessages similar to data packets. Raw sockets give access to the underlyingcommunicationprotocols that support socketabstractions.Rawsocketsarenotintended for the general programmer. A process uses its own socket tocommunicate across the networkwith a socket belonging to a remote process(the peer). The two socketsmust be of the same type. TheDNS and a set ofnetworking system calls combine to retrieve network addresses and serviceports. Library routines make it straightforward to find and construct socketaddressesinaprogram.Network server programs may run as daemon processes, divorced from
controlterminalwindowsandstandardI/O,torunconstantlybutquietlyinthebackground.MonitoringI/OwithselectorpselectenablesthemultiplexingconcurrentI/O.
Out-of-band data, supported by Internet stream sockets, can be used to sendurgentmessagessuchasinterruptstopeersockets.
EXERCISES1. Thesystem orpopen call executes an sh command.Howwould you get
suchacalltoexecuteacommandstringfortheBashShell?2. Is itpossible foraparentprocess tosenddata to thestandard inputof its
child?How?Is itpossible foraparentprocess to receiveoutput fromthestandardoutputofachildprocess?How?
3. RefertotheHellotherepipeexampleinSection12.2.Whatwouldhappenif the child did not close its descriptor p[1]?What would happen if theparentdidnotcloseitsdescriptorp[1]?
4. WriteaCfunctionpipe_std("Shell-command-string")whichcreatesachildprocess to execute any given regular Linux command. Furthermore, itconnects the filedescriptors0and1of thecalling (parent)process to thecorresponding descriptors of the child process.The usage of the pipe_stdfunctionisasfollows:
In the parent process, a call to pipe_std is made with a specificcommand string. This sets up the two-way pipe between the parentprocessandthechildprocess.Then,pipe_stdreturns.Nowintheparentprocess,filedescriptor0readsthestandardoutputofthechildprocess,andoutputtofiledescriptor1isreadasstandardinputbythechildprocess.Thisallowstheparentprocesstofeedinputtothechildprocessandcollectthechild’soutput.
Afterinteractionwiththechildprocessisover,theparentprocesscallsend_pipe_write(); end_pipe_read(); two additional functionsassociatedwithpipe_std, torestoretheparent’sfiledescriptors0and1.Since the parent process and the child process can form a circularproducer-consumer relationship, the danger of deadlock is alwaysthere.Itistheparentprogram’sresponsibility(notthatofpipe_std)toguardagainstdeadlock.
5. Whatdifferent systemcalls canbeused to read/write a socket?What aretheirdifferences?Includecallsnotcoveredinthetext.
6. Write a lowercase server that takesmessages from a client and turns alluppercasecharactersintolowercasebeforeechoingthemessagebacktotheclient.ImplementtheserviceusinganInternetdatagramsocket.
7. DothepreviousproblemwithanInternetstreamsocket.8. Addcodetoyourlowercaseserverthatcheckstheaddressandportnumber
oftheclientsocketandonlyacceptsrequestsfrom“allowable”clients.9. Use the out-of-bandmechanism of Internet stream sockets to sendLinux
signalstoaremoteprocess.10. Write a command serviceIP that takes a service name, such as ftp and a
hostname,suchasmonkey.cs.kent.edu,anddisplaystheIPaddressandportnumber.
11. Maxima isapowerfulprogramformathematicalcomputations. Install themaximapackage ifyourLinuxdoesnotalreadyhave it,and thenmake itintoanInternetserver.
12. Write achat applicationwheremultiple people can join in the same chatsession on different hosts. This problem requires a clear overview of theproblemandacarefuldesignbeforeimplementation.
AppendicesOnline
Theappendicesareonlineatthebook’swebsite(mml.sofpower.com)whereyoucanalsofindinformationupdatesandmanyotherusefulresources.
Appendix:SettingUpYourOwnLinuxforLearningSeemultiplewaystosetupyourownLinuxforeffectivelearning,includingonyourownWindows®orMac®laptop/desktop.
Appendix:SecureCommunicationwithSSHandSFTPSSH is a secure remote login program. It lets you log in and access a remotecomputer.SFTPisasecurefiletransferprogramthatallowsyoutouploadanddownloadfilestoandfromanothercomputer.
Appendix:PatternProcessingwithawkTheawk program is a powerful filter. It processes input one line at a time,applyinguser-specifiedawkpatternactionstoeachline.
Appendix:HowtoUSEvimCreatingandeditingtextfilesisbasictomanytasksonthecomputer.TherearemanytexteditorsforLinux,butvim(viiMproved)isavisualinteractiveeditorpreferredbymany.
Appendix:TextEditingwithviIn-depthcoverageof text editingconcepts, techniques, andmacroswith thevieditorareprovided.
Appendix:ViQuickReferenceMany editing commands are available undervi, and this quick reference cardcanbehandy.
Appendix:TheemacsEditor
Rather than operating in distinct input and command modes like vi, emacsoperatesinonlyonemode:Printablecharacterstypedareinsertedatthecursorposition.CommandsaregivenascontrolcharactersorareprefixedbyESCorctrl+x.
[1]
[2]
[3]
[3]
[5][6]
[7]
[8][9]
[10]
[11]
[12]
Bibliography
BlumRichard.LinuxCommandLineandShellScriptingBible.NewYork,NY,USA:JohnWiley&Sons,Inc.;2008.BovetDaniel P,CesatiMarco.Understanding theLinuxKernel. 3rd ed.California,USA:O’Reilly;2005.SoyinkaWale.Linux Administration: A Beginner’s Guide. 7th ed. New York, USA:McGraw-HillEducation;2015.LoveRobert. Linux Kernel Development. 3rd ed. Indianapolis, Indiana, USA: Addison-WesleyProfessional;2010.SchroderCarla.LinuxNetworkingCookbook.California,USA:O’Reilly;2007.SieverEllen, FigginsStephen,LoveRobert,RobbinsArnold.Linux in aNutshell. 6th ed. California,USA:O’Reilly;2009.MarkG.Sobell.APracticalGuidetoLinuxCommands,Editors,andShellProgramming,2ndEd.,PrenticeHall,NewJersey,USA,2009.SobellMarkG.APracticalGuidetoUbuntuLinux.3rded.NewJersey,USA:PrenticeHall;2010.Steidler-DennisonTony.RunYourOwnWebServerUsingLinux&Apache.Collingwood,Victoria,AU:SitePoint;2005.WangPaulS.DynamicWebProgrammingandHTML5.Florida,USA:Chapman&HallCRCPress;2012.WardBrian.HowLinuxWorks:WhatEverySuperuserShouldKnow.SanFrancisco,CA,USA:NoStarchPress;2004.YankKevin. Build Your Own Database Driven Web Site Using PHP & MySQL. Collingwood,Victoria,AU:SitePoint;2009.
WebsiteandExampleCodePackage
WebsiteThebookhasawebsiteusefulforinstructorsandstudents:http://mml.sofpower.comYoucanfindtheappendicesforthetextbookatthesite.Thesitealsooffersacompleteexamplecodepackagefordownloading,informationupdates,resources,orderinginformation,anderrata.ExampleCodePackageAllexamplesinthisbook,andafewmore,arecontainedinacodeexamplepackage.1Theentirepackagecanbedownloadedfromthewebsiteinonecompressedfile,MasteringModernLinux.tgzorMasteringModernLinux.zip.Thedownloadaccesscodeis2018MML.Thepackagecontainsthefollowingfilesanddirectories
ex01/ex03/ex05/ex07/ex09/ex11/guide.pdfex02/ex04/ex06/ex08/ex10/ex12/license.txtUnpacking
1. Placethedownloadedfileinanappropriatedirectoryofyourchoice.2. Go to that directory and, depending on the downloaded file, use one of
thesecommandstounpack
tarzxpvfMMLCode.tgztarjxpvfMMLCode.tbztarJxpvfMMLCode.txzunzipMMLCode.zip
ThiswillcreateafolderMMLCode/containingtheexamplecodepackage.
Index
*(Bashvariable),115.(Bashcommand),71.(directoryselfpointer),18..(parentdirectory),18/etc/passwd(File),77?(Bashvariable),118#(Bashvariable),114AAt(Bashvariable),82,115abort(libraryfunction),313accept(systemcall),340access(systemcall),294accesscontrol,DAC,154MAC,154ActiveServerPages(ASP),192Admincommand,apt,202aptitude,203ausearch,221chcon,221chgrp,152chown,152dig,207dnf,202fdisk,209firewall-cmd,208firewall-config,208fixfiles,222fsck,211gdisk,209getenforce,219gnome-disks,210
gnome-system-monitor,204gparted,209groupadd,199groupdel,199groupmod,199ifconfig,207killall,204lgroupadd,200lsblk,209luseradd,200lvcreate,212lvdisplay,212mkfs,211mknod,157mount,211nm-connection-editor,207nmcli,207parted,209,211pgrep,204pidof,204ping,207pkill,204ps,204,301pstree,204pvcreate,212pvdisplay,212quotacheck,214restorecon,222route,207sealert,221secon,222seinfo,220sestatus,218,222setenforce,219su,199sudo,201system-config-services,204systemctl,204,228top,204
traceroute,207useradd,199userdel,199usermod,199vgcreate,212vgdisplay,212visudo,200whois,207yumex-dnf,203
Adminuser,199AdvancedEncryptionStandard(AES),176alarm(libraryfunction),313alias(Bashcommand),67alias(Bashexpansion),67Aliases,domainname,170Apache,HTTPSsupport,239–240Apacheserver,186ApacheWebserver,227–236configuration,229–236directives,230–231filedeflation,237–238install,228–242modules,231PHPmodule,243–245
apache2ctl(regularcommand),229apachectl(regularcommand),229App,3Dprinting,94cloudstorage,93–94documentformatting,89–90drawinganddiagramming,90–91imageprocessing,91–92mathematics,94–96wordprocessing,88–89
Appstore,203Applicationlauncher,49Apps,95apt(admincommand),202
aptitude(admincommand),203ar(regularcommand),269arithmetic(Bashexpansion),64ARPANET,167as(regularcommand),259ASCIIarmored,179ASP(ActiveServerPages),192asy(vectorgraphics),90attr(filecommand),151ausearch(admincommand),221Authority,fornameserverzones,190awk(regularcommand),108
Backup,fileandsystem,214–216
basename(filecommand),162Bash,aliasexpansion,67arithmeticexpansion,64arrays,124–125braceexpansion,64commandexecution,57–58commandexpansion,69commandline,54command-lineprompt,54,72compoundcommand,55disableglobbing,70errorandinterrupt,138–139examplescripts,131–137expansions,63–71filequery,122–123filenameexpansion,70–71forloop,114–115function,128–131functionarguments,128–129functiondefinition,81functionexport,82functionreturnvalue,130–131functions,81–82
heredocument,126–127historyexpansion,64–66initfiles,77–78interactingwith,54–55jobcontrol,60–63numericalcomputation,121–122patterncondition,121processexpansion,69processnumber,139prompt,73quoting,79–81redefinebuilt-infunction,131scriptdebugging,137–138scriptexecution,113scriptinvoking,111–112scriptparameters,114specialcharacters,79–81stringexpansion,64testexpressions,118–119tildeexpansion,67–68untilloop,121variableexpansion,68–69variableexport,57variablemodifiers,125–126variables,71–72,123–124whileloop,120
Bashcommand,.,71alias,67bg,62break,122case,119cd,18continue,122declare,69dirs,71echo,54,64env,73eval,133
exit,31,63export,57fc,66fg,55,62for,114hash,57help,71history,64,107if,115–119jobs,61kill,63,314logout,63popd,71pushd,71set,55,59,69shift,119shopt,70source,71until,121which,57while,120
Bashexamplescript,clean,131–133cpp,133–135mput,136–137resize,137total,135–136
Bashvariable,*,115?,118AAt,82,115CDPATH,72DISPLAY,73EDITOR,72HISTFILE,66HISTSIZE,72HOME,72HOSTNAME,72OLDPWD,67,72
PATH,57,72PS1,72PS2,72PWD,72SHELL,72TERM,73USER,72#,114DISPLAY,43positional,123
bg(Bashcommand),62bind(systemcall),333BITNET,167Bootfiles,190Booting,216–218brace(Bashexpansion),64bzip2(Filecommand),160
Clibraryfunction,191Cprogram,mainfunctionarguments,256assembly,259compiler,258–260headerfiles,262library,264–265linking,263optimization,259preprocessor,260–262
calendar(filter),98cancel(regularcommand),32case(Bashcommand),119Casesensitivity,170cat(regularcommand),59cd(Bashcommand),18CDandDVD,48CDPATH,72CGI(CommonGatewayInterface),192chcon(admincommand),221chdir(systemcall),296
chgrp(admincommand),152chmod(filecommand),149chown(admincommand),152chroot(filecommand),162chsh(regularcommand),31,54closedir(libraryfunction),296cmp(filecommand),162comm(filecommand),162Command,Bashbuilt-in,15,54executionenvironment,72–73exitstatus,55interpretationcycle,54regular,15,54search,57searchpath,57
command(Bashexpansion),69Commandline,argumentconventions,258argumentsinC,256–257completion,55–56editing,55
Commands,jobcontrol,63useful,30–32
CommonGatewayInterface(CGI),192CompilingCprogram,257–260CompilingPrograms,27–28Completion,commandname,56filename,56programmable,56userhostname,56username,56variablename,56
ComprehensivePerlArchiveNetwork,140connect(systemcall),332Connections,inHTTP,192ContentTypeandFileSuffix,147
Content,typesof,186Coredumps,279–281Country-codetop-leveldomains,189cp(regularcommand),19cpp(regularcommand),260creat(systemcall),290Cryptography,175–177public-key,177symmetric-key,177
Cryptosystem,symmetric,176–177
ctime(libraryfunction),293
DAC,218Daemon,343–344daemon,93Datasink,59date(regularcommand),31DebuggingwithGDB,274–279declare(Bashcommand),69DECnet,167Defaultfilebrowser,48Desktop,appearance,42launchapplications,39launcher,36managefiles,40notificationarea,37overview,35–36startmenu,37windowlist,37workspace,37,41
desktop(GUI),8Desktopcomponents,36–38df(filecommand),162DHCPserver,205dia(diagramtool),90diff(filecommand),162dig(admincommand),207
dig(regularcommand),172Digest,ofmessage,182–183Digitalcertificate,178DigitalSignature,177Digitalsignature,183Directory,operationsinC,295–296stream,295–296dirs(Bashcommand),71Diskpartitions,209DISPLAY,73DISPLAY(Bashvariable),43dnf(admincommand),202DNS,169,190–191Resolvers,191Servers,190DNSservers,207dolphin(regularcommand),48DomainNameRegistrars,168Domainnames,168IPmappingto,190registrationof,189–191serversof,189service,190–191top-level,168Dotnotation,167Dropbox,93–94du(filecommand),162dup2(systemcall),325Dynamicserverpages,192
echo(Bashcommand),54,64EditingText,22–23EDITOR,72edquota(regularcommand),214EFISystemPartition,217emacs(text3editor),31Emailclient,thunderbird,25
Encoding,base64,179Encrypt/decryptwithGnuPG,180Enigmail,181env(Bashcommand),73eog(viewphoto),91Epoch,293eval(Bashcommand),133evince(PDFviewer),88evince(regularcommand),46ex(regularcommand),104execl(libraryfunction),305Executable,binaryfile,305,306ELF,306file,305textfile,305execv(libraryfunction),305execve(libraryfunction),305exit(Bashcommand),31,63exit(libraryfunction),310Exitstatus,118,257Cprogram,257expand(filter),97expand(regularcommand),97export(Bashcommand),57exportfs(regularcommand),158expr(calculatetool),94
fc(Bashcommand),66fcntl(systemcall),340,348fdisk(admincommand),209fdopen(libraryfunction),268,327fdopendir(systemcall),295fflush(libraryfunction),266fg(Bashcommand),55,62File,absolutepathname,17accesscontrol,20–22,149Bashinit,78
Bashlogin,78Bashlogout,78creationmask,77extendedattributes,151fullpathname,146group,153hidden,14mode,151/etc/passwd,77pathname,185relativepathname,18setgid,153setuid,153simplename,18status,150–154synchronization,174user/groupid,152xattr,151
Filebrowser,45–48Filecommand,attr,151basename,162bzip2,160chmod,149chroot,162cmp,162comm,162df,162diff,162du,162find,158getfattr,151gzip,160ln,148locate,160ls,149mount,158rm,149setfattr,151
shar,161size,162split,162tar,160touch,162umount,156uniq,162wc,162xz,160
Filenamesuffixes,27Filetypes,146–149directory,147link,148ordinary,147special,148symboliclink,148
filename(Bashexpansion),70FilesandDirectories,16–20Filesystem,mounted,156networked,158quotas,214superblock,157table,157Filesystemorganization,155filezilla(FTPtool),92Filters,100calendar,98expand,97fold,98head,96sort,99tail,96tr,97find(filecommand),158finger(regularcommand),31firefox(Webbrowser),26Firewall,filteringrules,207
permanentsettings,208settings,207–208zones,208firewall-cmd(admincommand),208firewall-config(admincommand),208firewalld,207fixfiles(admincommand),222fold(filter),98fold(regularcommand),98for(Bashcommand),114fork(systemcall),302freopen(libraryfunction),268fsck(admincommand),211fseek(libraryfunction),266fstat(systemcall),292ftell(libraryfunction),267FTP,92ftp,92
g++(regularcommand),255gawk(regularcommand),108gcalccmd(CLIcalculator),94gcc(regularcommand),255gccoptions,259–260gdb(regularcommand),274gdisk(admincommand),209gedit(regularcommand),46gedit(texteditor),22,31GETmethod,193getchar(libraryfunction),265getcwd(libraryfunction),312getdents(systemcall),295getenforce(admincommand),219getfattr(filecommand),151gethostbyname(libraryfunction),333getpid(systemcall),303getppid(systemcall),303gimp(photoprocessing),91gimp(regularcommand),61
GNOME3windowmanager,44GnomeShell,38GNOMEshellextension,38GNOMETerminal,Web,emaillinks,51copyandpaste,50starting,49–50GNOMEtweaktool,38gnome-calculator(calculator),94gnome-disks(admincommand),210gnome-help(regularcommand),51gnome-screenshot(screenshottool),91gnome-software(appstool),87gnome-system-monitor(admin,command),204gnome-terminal(regularcommand),48gnome-tweak-tool(regularcommand),38GnuPG,178–180encrypt/decrypt,180keycreation,178–180messagesigning,183–184gparted(admincommand),209gpg(regularcommand),178gpg2(regularcommand),178gprof(regularcommand),260groupadd(admincommand),199groupadd(regularcommand),153groupdel(admincommand),199groupmod(admincommand),199GRUB2bootloader,217GUI,8GUIapps,87–96GUID,217gvim(texteditor),31gzip(filecommand),160
Hall,Larry,140hash(Bashcommand),57head(filter),96
head(regularcommand),96help(Bashcommand),71Hiddenfiles,14HISTFILE,66history(Bashcommand),64,107history(Bashexpansion),64HISTSIZE,72HOME,72host(regularcommand),168,172HOSTNAME,72hostname(regularcommand),171HTML,187–188htonl(libraryfunction),343htons(libraryfunction),343htpasswd(regularcommand),237HTTPmethod,GET,193POST,193httpd(regularcommand),228HTTPS,238Hypertext,184HypertextMarkupLanguage(HTML),184–185,187–188HypertextPreprocessor(PHP),192HypertextTransferProtocol(HTTP),192–193
I/O,redirection,58–60standard,58I/Odescriptor,286–288I/Omultiplexing,345–346I/OredirectioninC,268ifconfig(admincommand),207import(screencapture),91Indexfiles,185Initfile,.bash_profile,78readline,56Inputfocus,39InstantMessaging,26
Integrateddevelopmentenvironment(IDE),187Integrity,ofmessage,183Internet,167–169address,167clients,169domainnamesystem,169–170introduction,169servers,169TLD,170InternetCorporationforAssignedNames,andNumbers(ICANN),167,168,189,190InternetProtocol(IP),167,190InternetServiceProviders(ISPs),190Internetworking,167iptablesoutdated,208IPv4,167IPv6,167ISP,205
JavaServerPages(JSP),192jobs(Bashcommand),61JSP(JavaServerPages),192
kdevelop(regularcommand),255keepassx2(passwordmanagertool),93Keyrepositories,178Keyserver,179Keyboardinputescapecharacter,56kill(Bashcommand),63,314kill(regularcommand),62,63kill(systemcall),314killall(admincommand),204konsole(regularcommand),53
Labels,forDNStreenode,170ld(regularcommand),263LDAP,200less(regularcommand),97lgroupadd(admincommand),200Librarycreating,268–269
Libraryfunction,abort,313alarm,313closedir,296ctime,293execl,305execv,305execve,305exit,310fdopen,268,327fflush,266freopen,268fseek,266ftell,267getchar,265getcwd,312gethostbyname,333,334htonl,343htons,343ntohl,343ntohs,343pclose,322popen,321putchar,265raise,314readdir,296setjmp,272
Libraryfunctions,common,264libreoffice(productivitytool),88LightweightDirectoryAccessProtocol,200link(systemcall),291Linux,documentation,29features,3–5filelocations,145–146filesystem,154–158helpanddocumentation,51history,1–2
networkingcommands,171–184philosophy,3versions,2–3
Linuxsystems,190listen(systemcall),340ln(filecommand),148locate(filecommand),160LogicalVolumeManagement,212–214Login,8–11logout(Bashcommand),63longjmp(libraryfunction),272look(regularcommand),31,100lp(regularcommand),32lpr(regularcommand),24,32lprm(regularcommand),32lpstat(regularcommand),32lqp(regularcommand),32ls(filecommand),149LSB,3lsblk(admincommand),209lstat(systemcall),292luseradd(admincommand),200lvcreate(admincommand),212lvdisplay(admincommand),212LVM,212–214extent,212logicalvolume,212volumegroup,212LVMarchitecture,212
MAC,218MACaddress,205MacromediaDreamweaver,187make(regularcommand),244man(regularcommand),29Manage,disks,209–211filesystems,209–211Markuptags,187
maxima(advacedmath),95MD5,161,182Messagedigest,182–183Messageintegrity,183MIME(MultipurposeInternetMail,Extensions),147,186Missingcommandinstall,203mkdir(regularcommand),19mkdir(systemcall),295mkfs(admincommand),211mknod(admincommand),157mknod(systemcall),292more(regularcommand),26mount(admincommand),211mount(filecommand),158mutt(regularcommand),25mv(regularcommand),19MySQL,administration,249–251configuration,248–249controlling,248install,251–252serverandclient,248mysql(regularcommand),247MySQLdatabaseserver,247–252mysql_secure_installation(regularcommand),248mysqld(regularcommand),248
nano(texteditor),31Nautilus,networkingin,170–171nautilus(regularcommand),45nc(regularcommand),194ncat(regularcommand),194nemiver(regularcommand),279NemiverGUIforGDB,274netstat(regularcommand),249Netword,gateway,205
Network,address,167–168clientandserver,169configuration,205–207defaultgateway,206interfacenames,206layer,166packetswitching,169services,165,167Networking,protocols,166–167sockets,330–333NetworkinginC,328–343NetworkManagerdaemon,206NIC,205nm-connection-editor(admincommand),207nmcli(admincommand),207Nodes,190nslookup(regularcommand),172ntohl(libraryfunction),343ntohs(libraryfunction),343OLDPWD,67,72opendir(systemcall),295Openingaprocess,321–322OperatingSystem,7
Package,repositories,201Packagemanagement,201–203APT,201–203DEB-based,202DNF,201–203RPM-based,202Packetswitching,169Pagemakers,187PAM,200parted(admincommand),209,211PartiallyspecifiedURLs,185passwd(regularcommand),31
Passwordencryption,237PasswordManager,92–93PATH,57,72pclose(libraryfunction),322PDF,pages,89passwordprotect,89viewer,88pdflatex(typesettingtool),89Per-userWebfolder,41Perlscriptinglanguage,descriptionof,139–140PersonalWebpages,186–187PGP/MIME,180pgrep(admincommand),204PHP,commandlineuse,246configuration,245debugging,245fileprocessing,244install,243–245security,245PHP(HypertextPreprocessor),192,242–243phpMyAdmin,250phpMyAdmininstall,250–251pidof(admincommand),204pidof(regularcommand),229ping(admincommand),207ping(regularcommand),172Pipe,60Pipetwo-way,327–328Pipelinebuilding,107–108pkill(admincommand),204PluggableAuthenticationModules,200popd(Bashcommand),71popen(libraryfunction),321Portnumbers,185POSIX,1Predicatefunction,130
Process,285background,55creation,302–305definition,203environment,310–313foreground,55infodisplay,204managing,203–205monitor,204synchronization,308–309termination,204,310process(Bashexpansion),69Processing,inHTTP,192Programexecution,305–309ps(admincommand),204,301PS1,72PS2,72pstree(admincommand),204ptrace(systemcall),309pushd(Bashcommand),71putchar(libraryfunction),265pvcreate(admincommand),212pvdisplay(admincommand),212PWD,72pwd(regularcommand),18,69
qpdf(PDFmanipulation),89Quadnotation,167Queries,inHTTP,193Queries,inHTTP,192quotacheck(admincommand),214quotaoff(regularcommand),214quotaon(regularcommand),214
raise(libraryfunction),314read(systemcall),288readdir(libraryfunction),296readelf(regularcommand),306recv(systemcall),347
recvfrom(systemcall),332Regularcommand,apache2ctl,229apachectl,229ar,269as,259awk,108cancel,32cat,59chmod,21–22chsh,31,54cp,19cpp,260date,31dig,172dolphin,48edquota,214evince,46ex,104expand,97exportfs,158finger,24–25,31fold,98g++,255gawk,108gcc,255gdb,274gedit,46gimp,61gnome-help,51gnome-terminal,48,53gnome-tweak-tool,38gpg,178gpg2,178gprof,260grep,fgrep,egrep,100groupadd,153head,96host,168,172
hostname,171htpasswd,237httpd,228kdevelop,255kill,62,63konsole,53ld,263less,97look,31,100lp,32lpr,24,32lprm,32lpstat,32lqp,32ls,20–21make,244man,29mkdir,19more,26mutt,25mv,19mysql,247mysqladmin,249mysqld,248nautilus,45nc,194ncat,194nemiver,279netstat,249nslookup,172passwd,31pidof,229ping,172pwd,18,69quotaoff,214quotaon,214readelf,306rm,19rmdir,19
rsync,174scp,174script,32sed,104sftp,174sort,99ssh,11,56,173su,20system-config-users,153tail,96tr,97unexpand,97vi,102wget,169,194which,311whois,189xclock,43xwininfo,44yelp,29,76
Regularexpression,basic,102–103grep,103grep,104matchingcommands,100–101
Relativedomainnames,170rename(systemcall),291Responses,inHTTP,192restorecon(admincommand),222rm(filecommand),149rm(regularcommand),19rmdir(regularcommand),19rmdir(systemcall),295Rootnameservers,190Rot13,174route(admincommand),207rsync(regularcommand),174
S/MIMEcertificate,180Samba,161
scheme,inURLs,185scp(regularcommand),174script(regularcommand),32seahorse(passwordmanager),93sealert(admincommand),221secon(admincommand),222SecureBoot,217Secureemail,180–182Thunderbird,181–182sed(regularcommand),104seinfo(admincommand),220select(systemcall),345SELinux,4,154,218–222contexts,219–220filecontext,221–222filecontexts,220logfile,221object,218setfilecontext,221statusandmodes,218–219subject,218targetedpolicy,219send(systemcall),347sendto(systemcall),332server,185Serverroot,185Server-sideprogramming,191,193Servicedaemons,203servicesconfiguration,204sestatus(admincommand),218,222set(Bashcommand),55,59,69setenforce(admincommand),219setfattr(filecommand),151setjmp(libraryfunction),272setsid(systemcall),344sftp,92sftp(regularcommand),174shar(filecommand),161SHELL,72
Shell,Globpattern,70interactive,78intro,12–16login,78non-interactive,78settingdefault,54ShellcommandfromC,299shift(Bashcommand),119shopt(Bashcommand),70shotwell(managephoto),91shutdown(systemcall),339Siblinglabels,170sigaction(systemcall),315Signal,concepts,313processing,314sending,314trapping,315–317Signals,313–317Signature,digital,177simple-scan(scanningtool),92size(filecommand),162Socket,address,331addressstructure,332datagram,330Internet,331local,331port,333raw,330stream,330socket(systemcall),331Socketout-of-banddata,347–348socketpair(systemcall),339Softwaremanagementtasks,202–203software-center(appstool),87sort(filter),99sort(regularcommand),99
source(Bashcommand),71split(filecommand),162SSH,keygeneration,174X11forwarding,173ssh(regularcommand),11,56,173SSL/TLS,238–239SSSD,200stat(systemcall),292sed,104–107string(Bashexpansion),64su(admincommand),199su(regularcommand),20Subnetmask,205sudo(admincommand),201sudoersfile,201symlink(systemcall),291system(systemcall),299Systemcall,_exit,310accept,340access,294bind,333chdir,296connect,332creat,290dup2,325fcntl,340,348fdopendir,295fileoperations,290–295fork,302fstat,292getdents,295getpid,303getppid,303I/O,286–290kill,314link,291listen,340
lseek,289–290lstat,292mkdir,295mknod,292opendir,295pipe,322–326pselect,346ptrace,309read,288recv,347recvfrom,332rename,291rmdir,295select,345send,347sendto,332setsid,344shutdown,339sigaction,315socket,331socketpair,339stat,292symlink,291system,299umask,291unlink,291utimes,292vfork,308wait,308waitpid,309write,288
Systemcontrol,39SystemSecurityServicesDaemon,200system-config-services(admincommand),204system-config-users(regularcommand),153systemctl(admincommand),204,228systemdprocessandservicemanager,204systemdtargets,217
tail(filter),96tail(regularcommand),96tar(filecommand),160TERM,73Terminalwindow,48–51Termination,inHTTP,192Texteditor,emacs,31gedit,22,31gvim,31nano,31vi,31vim,31thunderbird(Emailclient),25tilde(Bashexpansion),67TLS,phases,238Tool,asy(vectorgraphics),90dia(diagraming),90eog(viewphoto),91evince(PDFviewer),88expr(calculate),94filezilla(FTP),92ftp,92gcalccmd(CLIcalculator),94gimp(photoprocessing),91gnome-calculator(calculator),94gnome-screenshot(screenshot),91gnome-software(appstool),87import(screencapture),91keepassx2(passwordmanager),93libreoffice(productivity),88maxima(advacedmath),95pdflatex(typesetting),89qpdf(PDFmanipulation),89seahorse(passwordmanager),93sftp,92shotwell(managephoto),91
simple-scan(scanning),92software-center(manageapps),87xsane(scanning),92top(admincommand),204Top-leveldomain,168,189touch(filecommand),162tr(filter),97tr(regularcommand),97traceroute(admincommand),207TransmissionControlProtocol,167TransportLayerSecurity(TLS),238
UEFI,217umask(systemcall),291umount(filecommand),156UnderstandingGUI,39unexpand(regularcommand),97UniformResourceLocators(URLs),185uniq(filecommand),162unlink(systemcall),291until(Bashcommand),121URI,226–227URL,226–227USER,72UserDatagramProtocol,167useradd(admincommand),199userdel(admincommand),199usermod(admincommand),199Users,manage,199–201utimes(systemcall),292
variable(Bashexpansion),68vfork(systemcall),308vgcreate(admincommand),212vgdisplay(admincommand),212vi(regularcommand),102vi(texteditor),31vim(texteditor),31visudo(admincommand),200
W3Consortium,184wait(systemcall),308waitpid(systemcall),309Wayland,43wc(filecommand),162Web,184–186access,185browsers,169browsing,26databasesupportfor,247dynamicpagegeneration,191–192hosting,188–189HTTPrequest,227HTTPresponse,227hyperlinks,184publishing,186request,227server,225–226URLs,185Webbrowser,firefox,26wget(regularcommand),169,194which(Bashcommand),57which(regularcommand),311while(Bashcommand),120whois(admincommand),207whois(regularcommand),189windowID,44Windowmanager,43–44WorldWideWeb,184–186informationon,186–187personalpageson,186–187write(systemcall),288
Xserver,42XWindowSystem,42–44X11forwarding,173xattr,151Xattrcalls,295