Download - Setting up an Amazon Web Services Account
SettingupanAmazonWebServicesAccount
DianaMacleanforCS448G2011Firstofall,whatdoweneed?Forthisassignment,we’llberunningMapReduceworkflowsonAmazon’sElasticMapReduceframework.You’llneedtodoacoupleofthingstogetsetup:
1. CreateanAWSaccounta. SignupforS3(thisiswhereyou’llstoreyourdataonthecloud)&
createyourmainandloggingbuckets.i. SignupforS3Fox(optional,butrecommended)
b. SignupforEC2(themachineswe’llusetorunourworkflowson)c. SignupforEMR(thepackagedframeworkwe’llusetorunour
MapReducejobs;EMRwillrunyourworkflowsonEC2foryou.)2. Createakeypairandaccessyouraccountcredentials3. Redeemyour(free)credit!(Andcheckhowmuchyouhaveleft.)4. FamiliarizeyourselfwithS3,EMRandEC2
1:CreateanAWSaccountGotohttp://aws.amazon.comandsignupforanaccount,ifyoudonothaveonealready.Fornow,pleaseentertherequireddetails,includingpaymentdetailsifrequired.Amazonhasgenerouslyagreedtoprovideeachstudentwithcreditforthisclass;moreonhowtoredeemthislater.Onceyou’vecreatedanaccount,youwillneedtosignupfortheservicesthatwewillbeusingfortheAssignment.1a:SignupforS3AmazonSimpleStorageService(S3)isjustastorageframework–youcanuploadanydatahereandpay(verysmallamounts,unlessyouhaveverylargedata)forstoragespace.YoucanreadmoreaboutS3here:http://aws.amazon.com/s3GototheAWSManagementConsole,andclickthe“S3”tab.ThereshouldbeamessageinformingyouthatyouneedtosignupforS3.Enteryourownpaymentdetailsfornow–wewillloadcreditontotheaccountoncewe’reallsetup.
WeneedS3fortworeasons:1.)anEMRworkflowrequirestheinputdatatobeonS3;2.)EMRworkflowoutputisalwayssavedtoS3.Data(orobjects)inS3arestoredinwhatwecall“buckets”.Youcanthinkofbucketsasfolders.Forthisclass,we’veputthedatayou’llneedforPart1ofAssignment2inapublicbucketcalled:cs448g‐dataYou’llseehowtoreferencethisforEMRinputlateron.Inthemeanwhile,you’llneedsomebucketsofyourownto1.)storeyourEMRoutputin,and2.)storeyourlogfilesifyouwanttodebugyourEMRruns.Onceyou’reallsignedup,youshouldseeamessagepromptingyoutogetstartedbycreatingyourownbuckets.We’llcreatethelogbucketfirst.1.IntheS3console,clickon“CreateBucket”.
2.AllS3bucketshavetohaveuniquenames,socallyourloggingbucketcs448g‐lastname‐logging.Clickon“Create”(noton“SetUpLogging>>”).
3.Yournewbucketwillappearinthesidebar,withaninformativemessagetellingyouthatitisempty.
4.Nowwe’llcreateourmainbucket.Again,clickon“CreateaBucket”.Callthisonecs448g‐lastname.We’lllinkourloggingbuckettotheonewe’recreatingnow,soclickon“SetUpLogging>>”.
5.“Enablelogging”,andstarttypinginthenameofyourloggingbucket.Itshouldappearinthedropdownmenu.Selectit,and“Create”.
You’redonecreatingyourbuckets!Awesome
1ai)SignupforS3Fox(Optional,butrecommended)YoucanmanageyourbucketsthroughtheS3console,butthereisafantasticpluginforFirefoxthatmakesuploadinganddownloadingbetweenS3andyourlocalmachinereallyeasy.Ofcourse,you’llneedtobeusingFirefox.Fromhttp://www.s3fox.net,clickon“Download”.Youshouldgetapopupwindowaskingyouifyouwanttoinstalltheplugin(examplehere:http://www.s3fox.net/DownloadPage.aspx).Onceyou’veinstalled,restartFirefox.Youshouldbeabletoaccess“S3Organizer”byclickingonthe“Tools”tabintheFirefoxtaskbar,asshownbelow.
WhenyoufirstopenS3Fox,you’llneedtoenteryourAccessKeyandyourSecretKey.Youcanfindthesebyclickingon“Account”intheAWSManagementConsole(http://aws.amazon.com/account),andthenonthe“SecurityCredentials”link.ThiswilltakeyoutoapagewhereyoucanviewyourAccessKeys.
Onceyou’vecorrectlyenteredyourcredentialsinS3Fox,youshouldseeaconsolethatlookssomethinglikethefollowing(onlyyourswon’thavefilesinityet).S3FoxwillmakeitmucheasiertouploadanddownloadyourscriptsandoutputfilesoncewestartrunningourEMRjobs.
1b:SignupforEC2IntheAWSconsole,clickonthe“EC2”tab,andsignupforEC2.
Proceedasbefore;thistimeyouwillprobablybeaskedtoverifyyouridentityviaphone.Thisshouldnottakemorethanacoupleofminutestocomplete.
Onceyou’veverifiedyouridentity,youshouldgetanemailverificationstatingthatyou’rereadytouseEC2.Awesome!YoucanreadmoreaboutEC2here:http://aws.amazon.com/ec2.EC2isessentiallyabigmassofmachinesthatyoucan“rent”torunyourownworkloadson.EC2isgreatbecauseyoucancustomizeyourEC2instancesaccordingtoyourcomputerequirements(e.g.specifyRAM,softwardetc.).Wewon’tlaunchinstancesdirectly,butwhenwerunMapReducejobsthroughEMR,EMRwilluseEC2instances.1c:SignupforEMRRepeatstep1b,exceptunderthe“ElasticMapReduce”tabontheAWSManagementConsole.YoucanreadmoreaboutEMRhere:http://aws.amazon.com/elasticmapreduceInanutshell,EMRprovidesaframeworkforrunningMapReducejobsusingdatastoredonS3,andmachinesfromtheEC2service.We’llbeworkingmostlywithEMRfromhereon.2:Accessyouraccountcredentials,andcreateakeypairIfyousignedupforS3Fox,thenyoualreadyknowhowtoaccessyouraccountcredentials.Takealookatsection1aiofthistutorial,ifnot.WhenwerunjobsonEMR,you’llneedtohaveavalidpublic/privatekeypair.YoucanalsousethesekeystosshintoEC2instancemachinesthatyoulaunched,butwewon’tgothroughthathere.(Comeandfindmeifyouwanttodothis.)Tocreateyourfirstkeypair,clickonthe“EC2”tabintheAWSManagementConsole.
Ontheright,youshouldseealinkstating“0KeyPairs”.Clickonthis.
You’llbegivenanoptionto“CreateKeyPair”.Nameyourkeypairasyouwish(trytobemorecreativethanme).
Uponprovidinganameandclickingon“Create”,yourprivatekey(a.pemfile)willautomaticallydownload.Saveitinasafeplacewhereyou’llbeabletofinditagain.
Ifyouneedtoaccessyourpublickey,you’llbeabletofinditinthesameplacewhereyoufoundyouraccountcredentials.Amazonkeepsnorecordofyourprivatekey,however,soifyouloseit,you’llneedtogenerateanewset.(Note:Youwon’treallyneedtoaccessyourprivatekeyifyouusetheAWSManagementConsole,butyouwillbeaskedtonameyourkeypaireachtimeyourunanEMRjob.)3:AddCredittoyourAccountFinally,thepartyou’vebeenholdingyourbreathfor!Adding(free)credittoyouraccount!Inordertobecredited,you’llneedyouruniqueCreditCouponCode.Ifyouhavenotreceivedthisyet,e‐mailmeandIwillissueyouone.
Onceyouhaveyourcode,gotoyouraccountpage(http://aws.amazon.com/account)andclickon“PaymentMethod”.You’llseeyourbillingdetails;ifyouscrolldown,youwillalsoseeanoptiontoredeemaCreditCoupon.
ClickonRedeem/ViewAWSCredits,andenteryourcodetogetyourcredit.PleaseemailtheCS448Gcoursestaffimmediatelyifthisdoesnotwork.Unfortunately,wecanonlygiveyousomuchfreecredit,sodon’tgotoowild!Youcancheckonhowmuchcredityouhaveleftbyclickingonthe“AccountActivity”linkfromyouraccountpage.Sometimesthiscantakeawhiletoupdate,sodon’tbesurprisedifrecentchangesaren’timmediatelyapparent.4:FamiliarizeyourselfwithS3,EMRandEC2NowyoushouldbeallsetuptostartwithAssignment2.ThemorefamiliaryouarewithAWS,however,theeasieritwillbe.BrowsearoundtheAWSsite–thereareplentyofhelpfultutorialsandresourcesthere.Partyon
