Upload File

data centers - courses.cs.washington.edu– readfileblock (vs. posix file read) – writefileblock...

  • Home
  • Documents
  • Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each
39
Data Centers Tom Anderson

Upload: others

Post on 20-May-2020

4 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

DataCenters

TomAnderson

Page 2: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

TransportClarification

•  RPCmessagescanbearbitrarysize– Ex:oktosendatreeorahashtable– Canrequiremorethanonepacketsent/received

•  Weassumemessagescanbedropped,duplicated,reordered–  Itispacketsthataredropped,duplicated,reordered– MostRPCsaresinglepackets

•  TCPaddressessomeoftheseissues– Butwe’llassumethegeneralcase

Page 3: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

RPCSemantics

•  Atleastonce(NFS,DNS,lab1b)–  true:executedatleastonce–  false:maybeexecuted,maybemultipletimes

•  Atmostonce(lab1c)–  true:executedonce–  false:maybeexecuted,butnevermorethanonce

•  Exactlyonce–  true:executedonce–  false:neverreturnsfalse

Page 4: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Whendoesat-least-oncework?

•  Ifnosideeffects–  read-onlyoperations(oridempotentops)

•  Example:MapReduce•  Example:NFS–  readFileBlock(vs.Posixfileread)– writeFileBlock–  (butnot:deletefile)

Page 5: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

AtMostOnce

ClientincludesuniqueID(UID)witheachrequest–  usesameUIDforre-send

ServerRPCcodedetectsduplicaterequests–  returnpreviousreplyinsteadofre-runninghandlerifseen[uid]{r=old[uid]}else{r=handler()old[uid]=rseen[uid]=true}

Page 6: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

SomeAt-Most-OnceIssues

HowdoweensureUIDisunique?–  Bigrandomnumber?–  CombineuniqueclientID(IPaddress?)withseq#?–  Whatifclientcrashesandrestarts?Canitreusethe

sameUID?

•  Inlabs,nodesneverrestart–  Equivalentto:everynodegetsnewIDonstart–  Clientaddress+seq#isunique–  Seq#aloneisNOTunique

Page 7: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

WhenCanServerDiscardOldRPCs?

Option1:Never?Option2:uniqueclientIDs–  per-clientRPCsequencenumbers–  clientincludes"seenallreplies<=X"witheveryRPC

Option3:oneclientRPCatatime–  arrivalofseq+1allowsservertodiscardall<=seq

LabsuseOption3

Page 8: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

WhatifServerCrashes?

Ifat-most-oncelistofrecentRPCresultsisstoredinmemory,serverwillforgetandacceptduplicaterequestswhenitreboots–  DoesserverneedtowritetherecentRPCresultsto

disk?–  Ifreplicated,doesreplicaalsoneedtostorerecent

RPCresults?InLabs,servergetsnewaddressonrestart–  Clientmessagesaren’tdeliveredtorestartedserver–  (discardifsenttowrongversionofserver)

Page 9: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

ADataCenter

Page 10: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

InsideaDataCenter

Page 11: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Datacenter

10k-100kservers:250k–10Mcores

1-100PBofDRAM

100PB-10EBstorage

1-10Pbpsbandwidth(>>Internet)

10-100MWpower-1-2%ofglobalenergyconsumption

100sofmillionsofdollars

Page 12: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Servers

Limitsdrivenbythepowerconsumption

1-4multicoresockets

20-24cores/socket(150Weach)

100sGB–1TBofDRAM(100-500W)

40Gbpslinktonetworkswitch

Page 13: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Serversinracks

19”wide

1.75”tall(1u)

(definedin1922!)

40-120servers/rack

networkswitchattop

Page 14: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Racksinrows

Page 15: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Rowsinhot/coldpairs

Page 16: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Hot/coldpairsindatacenters

Page 17: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Whereisthecloud?

Amazon,intheUS:-NorthernVirginia

-Ohio

-Oregon

-NorthernCalifornia

Whythoselocations?

Page 18: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

MTTF/MTTR

MeanTimetoFailure/MeanTimetoRepairDiskfailures(notreboots)peryear~2-4%– Atdatacenterscale,that’sabout2/hour.–  Ittakes10hourstorestorea10TBdisk

Servercrashes– 1/month*30secondstoreboot=>5mins/year– 100K+servers

Page 19: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

TypicalYearinaDataCenter(2008)•  ~0.5overheating(powerdownmostmachinesin<5mins,~1-2daystorecover)

•  ~1PDUfailure(~500-1000machinessuddenlydisappear,~6hourstocomeback)

•  ~1rack-move(plentyofwarning,~500-1000machinespowereddown,~6hours)

•  ~1networkrewiring(rolling~5%ofmachinesdownover2-dayspan)

•  ~20rackfailures(40-80machinesinstantlydisappear,1-6hourstogetback)

Page 20: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

TypicalYearinaDataCenter(2008)•  ~5racksgowonky(40-80machinessee50%packetloss)

•  ~8networkmaintenances(4mightcause~30-minuterandomconnectivitylosses)

•  ~12routerreloads(takesoutDNSandexternalvipsforacoupleminutes)

•  ~3routerfailures(havetoimmediatelypulltrafficforanhour)

•  ~dozensofminor30-secondblipsfordns•  ~1000individualmachinefailures•  ~thousandsofharddrivefailures•  slowdisks,badmemory,misconfiguredmachines,flakymachines,etc

Page 21: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

ThisWeek:PrimaryBackup

•  Howdowebuildsystemsthatsurviveasinglenodefailure?

•  Statereplication:runtwocopiesofserver–  primary,backup–  differentracks,differentPDUs,differentDCs!–  Ifprimaryfails,backuptakesover

•  Howdoweknowprimaryfailedvs.isslow?

Page 22: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

DataCenterNetworks

EveryserverwiredtoaToR(topofrack)switch

ToR’sinneighboringaisleswiredtoanaggregationswitch

Agg.switcheswiredtocoreswitches

Page 23: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Earlydatacenternetworks

3layersofswitches-Edge(ToR)

-Aggregation

-Core

Page 24: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Earlydatacenternetworks

3layersofswitches-Edge(ToR)

-Aggregation

-CoreOptical

Electrical

Page 25: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Earlydatacenterlimitations

Cost-Core,aggregationrouters=highcapacity,lowvolume

-Expensive!

Fault-tolerance-Failureofasinglecoreoraggregationrouter=large

bandwidthloss

Bisectionbandwidthlimitedbycapacityoflargestavailablerouter

-Google’sDCtrafficdoubleseveryyear!

Page 26: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Closnetworks

HowcanIreplaceabigswitchbymanysmallswitches?

Bigswitch

Smallswitch

Page 27: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Closnetworks

HowcanIreplaceabigswitchbymanysmallswitches?

Bigswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Page 28: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

ClosNetworks

Whataboutbiggerswitches?

Page 29: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Page 30: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Smallswitch

Page 31: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Multi-rootedtree

Everypairofnodeshasmanypaths

Faulttolerant!Buthowdowepickapath?

Page 32: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Multipathrouting

Lotsofbandwidth,splitacrossmanypaths

ECMP:hashonpacketheadertodetermineroute

-(5tuple):SourceIP,port,destinationIP,port,prot.

-Packetsfromclient–serverusuallytakesameroute

Onswitchorlinkfailure,ECMPsendssubsequentpacketsalongadifferentroute

=>Outoforderpackets!

Page 33: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

DataCenterNetworkTrends

RTlatencyacrossdatacenter~10usec40Gbpslinkscommon,100Gbpsontheway– 1KBpacketevery80nsona100Gbpslink– Directdeliveryintotheon-chipcache(DDIO)

Upperlevelsoftreeare(expensive)opticallinks– Thintreetoreducecosts

Withinrack>withinaisle>withinDC>crossDC– Latencyandbandwidth:keepcommunicationlocal

Page 34: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

LocalStorage

•  Magneticdisksforlongtermstorage– Highlatency(10ms),lowbandwidth(250MB/s)– Compressedandreplicatedforcost,resilience

•  Solidstatestorageforpersistence,cachelayer– 50usblockaccess,multi-GB/sbandwidth

•  EmergingNVM– LowenergyDRAMreplacement– Sub-microsecondpersistence

Page 35: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

DayintheLifeofaWebRequest

Usertypes“google.com”,whathappensDNS=DomainNameSystem– Globaldatabaseofname->IPaddress

DNSquerytorootnameserver– Rootnameserverisreplicated(600+)– HardcodedIPmulticastaddresses– Updateshappenasyncinbackground(rare)

Rootreturns:IPaddressofgoogle’snameserver–  Cachedonclient(foraconfigurableperiod)–  Canbeoutofdate

Page 36: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

NameServertoDC

•  DNSreturnsgoogle’snameserver•  Googlenameserverreturnslocalnameserver– NameserverinDCclosetouser’sIPaddress

•  LocalnameserverreturnsIPofwebfrontend– Alsoinlocaldatacenter– Withshorttimeout,rerouteiffrontendfailure

•  BrowsersendsHTTPrequesttofrontend

Page 37: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

ResilientScalableWebFrontEnd

•  IPaddressoffrontendisanarrayofmachines•  Packetsfirstgothroughloadbalancer– Actually,anarrayofloadbalancers(allthesame)

•  Loadbalancerhashes3or5tuple->frontend– SourceIP,port#,DestinationIP,port#,protocol– Thisway,alltrafficfromusergoestosameserver

•  Whenfrontendfails,loadbalancerredirectsincomingpacketselsewhere

Page 38: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

Challenge

Howdowebuildahashfunctionthat’sstable?Thatreroutesonlythepacketsforthefailednode,leavingthepacketstoothernodesalone.

Page 39: Data Centers - courses.cs.washington.edu– readFileBlock (vs. Posix file read) – writeFileBlock – (but not: delete file) At Most Once Client includes unique ID (UID) with each

WebFrontEnd->Cache,StorageLayer

Key-valuestoreShardedacrossmanycacheandstoragenodesHash(key)->cache,storagenodeIfcachenodefails,rehashtonewcachenodeIfstoragenodefails,???


Next Generation Archive Management · Private Cloud Disk Systems Tape POSIX Object & POSIX Enable OBJECT. 2. Accelerate VSM File System VSM Policy Engine Archive Set 1 Archive Set

0x04.net0x04.net/doc/posix/Ieee Posix 1003.1B.pdf0x04.net

Posix Conf

POSIX IEEE 1003

Notes Posix

A New Network File System is Born: SMB2€¦ · Locking semantics differ between CIFS and POSIX at the application layer. CIFS locking is mandatory, POSIX advisory. CIFS locking stacks

POSIX I/O, System Calls - courses.cs.washington.edu

Virtual File System - Computer Scienceporter/courses/cse506/f12/slides/vfs.pdf · Single programming interface ! (POSIX file system calls – open, read, write, etc.) ! Single file

Approximation - courses.cs.washington.edu

seenubabueducom.files.wordpress.com file · Web view2.Write a C/C++ POSIX compliant program that prints the POSIX defined configuration options supported on any given system using

Posix Shm Slides

POSIX Threads Programming

CSE 333 Section - courses.cs.washington.edu€¦ · POSIX Posix is a family of standards specified by the IEEE. These standards maintains compatibility across variants of Unix-like

Practica POSIX

Xenomai POSIX skin API fileContents 1 Module Index 1 1.1 Modules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 2 File Index 3 2.1 File List

POSIX Commands of the C/C++ Compiler - Fujitsumanuals.ts.fujitsu.com/file/10462/cpp_pcmd.pdf · This manual is primarily intended for use as a reference manual for the POSIX commands

Hilos POSIX

“Implementation of SMB3.0 in Scale-Out NAS” · (MPA) File States. SMB3. Distributed Access Service (DAS) File States. Transport. File System. NTFS Interface. POSIX Interface

posix posix

POSIX Threads

File%System%Reliability% (part2)% - courses.cs.washington.edu

Scalable POSIX File Systems in the Cloud

POSIX - SOCKETS/XTI for POSIX - Fujitsu Technology …manuals.ts.fujitsu.com/file/4686/posix_s.pdf ·  · 2011-10-06together the common characteristics of processes which communicate

POSIX - Commands - Fujitsumanuals.ts.fujitsu.com/file/11127/posix_k.pdf · POSIX Commands BS2000/OSD User Guide Valid for BS2000/OSD V7.0/V8.0/V9.0. Comments… Suggestions… Corrections…

Reasoning About POSIX File Systems

POSIX Thread Programming

POSIX and Real-Time POSIX Portable Operating System

Cs Posix Ssoo2

IEEE Posix Standards

Extending the POSIX I/O Interface: A Parallel File System ... · Extending the POSIX I/O Interface: A Parallel File System Perspective Murali Vilayannur ... (open, close, read, write,

plantr. - courses.cs.washington.edu

File%System%Reliability% - courses.cs.washington.edu · LastTime:%File%System%Layout • Tree%structure% – Asymmetric:%FFS% – Balanced:%NTFS% • Disk%oriented%free%space%allocaon%

MarFS: A Scalable Near-POSIX File System over Cloud · PDF fileLA-UR-15-27431 . MarFS: A Scalable Near-POSIX File System over Cloud Objects . Kyle E. Lamb HPC Storage Team Lead Nov

Card3000 - courses.cs.washington.edu

POSIX BS2000 file system bs2fs - Fujitsumanuals.ts.fujitsu.com/file/11129/posix_bs2fs.pdf · POSIX BS2000 file system bs2fs BS2000/OSD ... file systems are embedded in POSIX and which