Distributed Shared MemoryOutline Introduction Shared Memory vs. Distributed Memory Distributed Shared Memory Architecture DSM vs. Message Passing Design and Implementation Consistency Models DSM Algorithms ConclusionIntroductionFrom the system interconnection perspective, parallel systems fall into to main categories! MultiprocessorsMulticomputers"ightly#Coupled Multiprocessor$oosely#Coupled MultiprocessorShared Memory architecture Distributed Memory architectureShared MemoryMemoryMemoryMemoryMemory%etor&CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'CP'Shared Memory vs. Distributed MemoryCache coherent means if one processor updates a location in shared memory, all the other processors &no about the update.Shared Memory Distributed MemoryGlobal Address Space No concept of global address spaceCache Coherent No concept of cache coherencyLack of scalability Scalable performanceExpensive to build Cost effectiveness: can use commodity off!the!shelf processors and net"orkingEasy to program reusability programmer is responsible for data communication#ata sharing fast and uniform #ata sharing by message passing non!uniform memory access timesDistributed-Shared Memory ArchitectureDefnitionA Distributed Shared Memory DSM is an abstraction that allos the physically separated memories to be addressed as one logically shared address space.General Characteristics(ybrid architectureShared Memory model implemented over physically distributed memory)irtual Space shared beteen all processesShared#Memory programming techni*ues can be used+hen reading and updating, processes see DSM as an ordinary memory ithin their address spaceMapping Manager! Maps shared#memory address to physical memory ,remote or local-P.M.P/M/PnMn0.MM MM MM 0.Shared )irtual SpaceInterconnection %etor&Distributed-Shared Memory Architecture (contd.)General CharacteristicsCovert Communication operationsCommunications are still needed to e1change data, but they are hiddenfrom the programmer. Inter#process communication transparency (eterogeneous %odes"he shared memory component can be a cache#coherent SMP machine and2or a graphics processing unit 3P'Processes on di4erent computers observe the updates made by one anotherCache#coherentSMP or 'MA ,bus#based- ! A model ith identical processors that have e*ual access times to a shared memory'MA ! 'niform Memory AccessSMP! Symmetric MultiprocessorP.M.P/M/PnMn0.MM MM MM 0.Shared )irtual SpaceInterconnection %etor&Distributed-Shared Memory Architecture (contd.)AdvantagesImplicit data sharingShields programmer from Send25eceive primitives$ess e1pensive to build and scalableInherited from the distributed#memory architecture)ery large total physical memory for all nodes$arge programs can run more e6cientlySoftare Portability and 5eusabilityPrograms ritten for shared memory multiprocessors can be run on DSM systems ith minimum changesDisadvantages$ittle programmer control over actual messages being generatedDSM implementations use asynchronous message#passingcannot be more e6cient than Message Passing implementationsDistributed-Shared Memory Architecture (contd.)7est Suitable+hen individual shared data items can be accessed directlye.g. Parallel Applications$ess appropriate+hen data is accessed by re*ueste.g. Client#Server SystemsA server may be used to assist in providing DSM functionality for data shared beteen clientsDSM vs. Messae !assin%o evidence against or in favor to any of the to communication mechanismsProperty DSM Message Passing$arshalling No% &ariables shared directly 'es% (rogrammer)s *obAddress space Single% +nterference may occur (rivate% (rocesses are protected#ata representation ,niform -eterogonousSynchroni.ation Normal construct forshared!memory programming$essage passing primitives(rocess execution Non!overlapping lifetimes At the same timecommunication)s cost +nvisible /bviousDesin and Im"lementationMain Issues3ranularity refers to the si8e of sharing unit that can be uniform chun&s of memory or data structures byte, page or comple1 data structureSmall Pages ! increased parallelismincrease in directory si8e$arge Pages ! reduce paging overhead, but increase sharing overheadStructure refers to the arrangement of shared data Most systems vie DSM as a linear array of ords5eplacement StrategiesSimilar to caching mechanisms in MPIn cache systems, $5' is often usedIn DSM, shared pages need to be given higher priority than e1clusively oned pagesthey could be replaced 9rstSynchroni8ation PrimitivesCoherence protocols must ensure the consistency of shared dataDSM must allo simultaneous access to shared data on di4erent machinessingle riter, multiple readers, etc.Consistency ModelsDe9nitionA memory consistency model for a shared address space speci9es constraints on the order in hich memory operations must appear to be performed ,i.e. to become visible to the processors- ith respect to one another.Strict Consistency ModelAny read to a certain memory location returns the value stored by most recent rite operation to that address, irrespective of the locations of the processors performing the read and the rite operation.Se*uential Consistency Modelif the result of any e1ecution is the same as if the operations of all processors ere e1ecuted in the same se*uential order, and the operations of each individual processor appear in this se*uence in the order speci9ed by its program. ,$eslie $amport-Defnition restated: Se*uential consistency re*uires that a shared memory multiprocessor appears to be amultiprogramming uniprocessor system to any program running on it.All instructions are e1ecuted in order;very rite operation becomes instantaneously visible throughout the systemConsistency Models (contd.);1ample .P. P/ Data < /===hile ,(ead ?(ead < . 0 < DataSe*uential consistency re*uires program order"he rite to Data has to complete before the rite to (ead can begin"he read of (ead has to complete before the read of Data can begin;1ample /Initially A < 7 < =P. P/ P@A < . if ,A