![Page 2: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/2.jpg)
About your speaker…Ph.D. Robotics, Carnegie Mellon, ‘91 - ’97Path Planning for Multiple Mobile Robots
Researcher, Microsoft Research, ‘98 - ’02Ubiquitous Computing
Software Eng., Microsoft Games, ‘03 - ’05AI for Forza Motorsport
Software Engineer, Google, ‘05 - nowMaps: Pathfinder
Systems: Infrastructure
![Page 3: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/3.jpg)
![Page 4: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/4.jpg)
![Page 5: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/5.jpg)
Indexing Large Datasets
All web pages Data CenterIndex Files
![Page 6: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/6.jpg)
Indexing Large Datasets
Geographic Data Data CenterIndex Files
...not so useful for user-facing applications…
![Page 7: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/7.jpg)
Pointer Following (or) Joining
Feature List1: <type=Road>, <intersections=(3)>, <geom>, …
2: <type=Road>, <intersections=(3)>, <geom>, …
3: <type=Intersection>, stop_type, POI? …
4: <type=Road>, <intersections=(6)>, <geom>,
5: <type=Road>, <intersections=(3,6)>, <geom>, …
6: <type=Intersection>, stop_type, POI?, …
7: <type=Road>, <intersections=(6)>, <geom>, …
8: <type=Town>, <name>, <geom>, …
.
.
.
Intersection List3: <type=Intersection>, stop_type, <roads=(
1: <type=Road>, <geom>, <name>, …
2: <type=Road>, <geom>, <name>, …
5: <type=Road>, <geom>, <name>, … )>, …
6: <type=Intersection>, stop_type, <roads=(
4: <type=Road>, <geom>, <name>, … ,
5: <type=Road>, <geom>, <name>, … ,
7: <type=Road>, <geom>, <name>, … )>, …
.
.
.
Input Output
1
2
7
5
4
63
![Page 8: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/8.jpg)
Inner Join PatternInput Map Shuffle Reduce Output
Apply map() to each;Key = intersection id
Value = featureSort by key
Apply reduce() to listof pairs with same key,gather into a feature
Feature list,aggregated
Feature list
1: Road
2: Road
3: Intersection
4: Road
5: Road
6: Intersection
7: Road
(3, 1: Road)
(3, 2: Road)
(3, 3: Intxn)
(6, 4: Road)
(3, 5: Road)
(6, 5: Road)
(6, 6: Intxn)
(6, 7: Road)
(3, 1: Road)
(3, 2: Road)
(3, 3: Intxn.)
(6, 4: Road)
(3, 5: Road)
(6, 5: Road)
(6, 6: Intxn.)
(6, 7: Road)
3
6
3: Intersection1: Road,2: Road,5: Road
6: Intersection4: Road,5: Road,7: Road
1
2
7
5
4
63
![Page 9: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/9.jpg)
Inner Join Pattern in SQL
e
d
c
b
a
D
67
65
64
32
31
IR
y6
x3
DI
e
d
c
b
a
r.D
x7
y5
y4
x2
x1
i.Dr.R
roads intsSELECT roads.R, roads.D, ints.DFROM roads INNER JOIN intsON roads.I = ints.I
x36e7
y63a1
y63b2
y66c4
y66d5
y66e7
6
6
3
3
r.I
3
3
3
3
i.I
d
c
b
a
r.D
x5
x4
x2
x1
i.Dr.R
1
2
7
5
4
63
“Cross Join”
![Page 10: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/10.jpg)
Inner Join Pattern in SQL
e
d
c
b
a
D
57
65
64
32
31
IR
y6
x3
DI
SELECT roads.R, roads.D, ints.DFROM roads, intsWHERE roads.I = ints.I
(aka “an Equi Join”)
e
d
c
b
a
r.D
x7
y5
y4
x2
x1
i.Dr.R
roads intsSELECT roads.R, roads.D, ints.DFROM roads INNER JOIN intsON roads.I = ints.I
1
2
7
5
4
63
![Page 11: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/11.jpg)
Tables vs. Flat File?
Roads Features
Intersections
Towns
Tables Flat File
Road Intersection Town
Road Intersection Town
Road Intersection Town
Message GeoFeature { enum Type { ROAD = 1; INTERSECTION = 2; TOWN = 3; } required Type type = 0; optional Road road = 1; optional Intersection intersection = 2; optional Town town = 3 ;}
“ Protocol Buffer ”
![Page 12: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/12.jpg)
References vs. Duplication?
• References: Common primary key; easy restructuring• Duplication: Avoids additional MR passes; denormalizes data• …an engineering space / time / complexity tradeoff
References1: <type=Road>, <intersections=(3)>, <geom>, …
2: <type=Road>, <intersections=(3)>, <geom>, …
3: <type=Intersection>, <roads=(1,2,5)>, …
4: <type=Road>, <intersections=(6)>, <geom>,
5: <type=Road>, <intersections=(3,6)>, <geom>, …
6: <type=Intersection>, <roads=(5,6,7)>, …
7: <type=Road>, <intersections=(6)>, <geom>, …
8: <type=Town>, <name>, <geom>, …
.
.
.
Duplication3: <type=Intersection>, <roads=(
1: <type=Road>, <geom>, <name>, …
2: <type=Road>, <geom>, <name>, …
5: <type=Road>, <geom>, <name>, …)>, …
6: <type=Intersection>, <roads=(
4: <type=Road>, <geom>, <name>, … >
5: <type=Road>, <geom>, <name>, … >
7: <type=Road>, <geom>, <name>, …)>, …
.
.
.
1
2
7
5
4
63
![Page 13: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/13.jpg)
Code Exampleclass IntersectionAssemblerMapper : public
Mapper { virtual void Map(MapInput* input) { GeoFeature feature; feature.FromMapInput(input); if (feature.type()==INTERSECTION) { Emit(feature.id(), input); } else if (feature.type() == ROAD) { Emit(feature.intersection_id(0), input); Emit(feature.intersection_id(1), input); } }};REGISTER_MAPPER(IntersectionAssemblerMapper);
class IntersectionAssemblerReducer : publicReducer {
virtual void Reduce(ReduceInput* input) { GeoFeature feature; GraphIntersection intersection; intersection.id = input->key(); while(!input->done()) { feature.FromMapInput(input->value()); if (feature.type()==INTERSECTION) intersection.SetIntersection(feature); else intersection.AddRoadFeature(feature); input->next(); } Emit(intersection); }};REGISTER_REDUCER(IntersectionAssemblerReducer);
(3, 1: Road)
(3, 2: Road)
(3, 3: Intxn)
(6, 4: Road)
(3, 5: Road)
(6, 5: Road)
(6, 6: Intxn)
(6, 7: Road)
(3, 1: Road)
(3, 2: Road)
(3, 3: Intxn.)
(6, 4: Road)
(3, 5: Road)
(6, 5: Road)
(6, 6: Intxn.)
(6, 7: Road)
3
6
![Page 14: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/14.jpg)
Join, but no pointers or keys?
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: Road
1
2
7
5
4
63
Input Map Shuffle Reduce OutputApply map() to each;emit (key,val) pairs
Sort by keyApply reduce() to listof pairs with same key
New list of itemsList of items
3: 1,2,5
6: 4,5,7
?
![Page 15: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/15.jpg)
Bucketing (or) Grace Hash Join
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: RoadA B
C D
Input Map Shuffle Reduce Output
1
2
7
5
4
63
(A-Road, 1)
(C-Road, 2)
(A-Town, 3)
(D-Road, 4)
(C-Road, 5)
(B-Town, 6)
(B-Road, 7)
(C-Road, 1)
(B-Town, 3)
(C-Town, 3)
(D-Road, 5)
(D-Town, 6)
(D-Road, 7)
Emit (key, item) pairKey = geometric hashSecondary key = Type
Sort by keysIntersect all townswith all roads; emitintersecting pairs
(town, road) pairFeature List
![Page 16: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/16.jpg)
Reduce on Key A
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: RoadA B
C D
Input Map Shuffle Reduce OutputEmit (key, item) pairKey = geometric hashSecondary key = Type
Sort by keysIntersect all townswith all roads; emitintersecting pairs
(town, road) pairFeature List
1
2
7
5
4
63
(A-Road, 1)
(C-Road, 2)
(A-Town, 3)
(D-Road, 4)
(C-Road, 5)
(B-Town, 6)
(B-Road, 7)
(C-Road, 1)
(B-Town, 3)
(C-Town, 3)
(D-Road, 5)
(D-Town, 6)
(D-Road, 7)
(A-Road, 1)
(A-Town, 3) (3, 1)
![Page 17: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/17.jpg)
Reduce on Key B
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: RoadA B
C D
Input Map Shuffle Reduce Output
1
2
7
5
4
63
(A-Road, 1)
(C-Road, 2)
(A-Town, 3)
(D-Road, 4)
(C-Road, 5)
(B-Town, 6)
(B-Road, 7)
(C-Road, 1)
(B-Town, 3)
(C-Town, 3)
(D-Road, 5)
(D-Town, 6)
(D-Road, 7)
(B-Town, 6)
(B-Road, 7)
(B-Town, 3) (6, 7)
Emit (key, item) pairKey = geometric hashSecondary key = Type
Sort by keysIntersect all townswith all roads; emitintersecting pairs
(town, road) pairFeature List
![Page 18: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/18.jpg)
Reduce on Key C
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: RoadA B
C D
Input Map Shuffle Reduce Output
1
2
7
5
4
63
(A-Road, 1)
(C-Road, 2)
(A-Town, 3)
(D-Road, 4)
(C-Road, 5)
(B-Town, 6)
(B-Road, 7)
(C-Road, 1)
(B-Town, 3)
(C-Town, 3)
(D-Road, 5)
(D-Town, 6)
(D-Road, 7)
(C-Road, 2)
(C-Road, 5)
(C-Road, 1)
(C-Town, 3)
(3, 2)
(3, 5)
Emit (key, item) pairKey = geometric hashSecondary key = Type
Sort by keysIntersect all townswith all roads; emitintersecting pairs
(town, road) pairFeature List
(3, 1)
![Page 19: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/19.jpg)
Reduce on Key D
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: RoadA B
C D
Input Map Shuffle Reduce Output
1
2
7
5
4
63
(A-Road, 1)
(C-Road, 2)
(A-Town, 3)
(D-Road, 4)
(C-Road, 5)
(B-Town, 6)
(B-Road, 7)
(C-Road, 1)
(B-Town, 3)
(C-Town, 3)
(D-Road, 5)
(D-Town, 6)
(D-Road, 7)
(D-Road, 4)
(D-Road, 5)
(D-Town, 6)
(D-Road, 7)
(6, 4)
(6, 5)
Emit (key, item) pairKey = geometric hashSecondary key = Type
Sort by keysIntersect all townswith all roads; emitintersecting pairs
(town, road) pairFeature List
(6, 7)
![Page 20: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/20.jpg)
Output… not quite...
1: Road
2: Road
3: Town
4: Road
5: Road
6: Town
7: RoadA B
C D
Input Map Shuffle Reduce OutputApply map() to each;emit (key,val) pairs
Sort by keyApply reduce() to listof pairs with same key
List of itemsFeature List
1
2
7
5
4
63
(6, 7)
(3, 1)
(3, 2)
(3, 5)
(6, 4)
(6, 5)
(3, 1)
(6, 7)
![Page 21: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/21.jpg)
…recall earlierJoin PatternInput Map Shuffle Reduce Output
Apply map() to each;emit (key,val) pairs
Sort by keyApply reduce() to listof pairs with same key
New list of itemsList of items
1: Road
2: Road
3: Intersection
4: Road
5: Road
6: Intersection
7: Road
(3, 1: Road)
(3, 2: Road)
(3, 3: Intxn)
(6, 4: Road)
(3, 5: Road)
(6, 5: Road)
(6, 6: Intxn)
(6, 7: Road)
(3, 1: Road)
(3, 2: Road)
(3, 3: Intxn.)
(6, 4: Road)
(3, 5: Road)
(6, 5: Road)
(6, 6: Intxn.)
(6, 7: Road)
3
6
3: Intersection1: Road,2: Road,5: Road
6: Intersection4: Road,5: Road,7: Road
1
2
7
5
4
63
![Page 22: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/22.jpg)
(6, 5)
Recursive Key Join PatternInput Map Shuffle Reduce Output
Identity Mapper,key = town
Sort by keyReducer sorts, gathers,
remove duplicates;similar to join
Index of roadsin each town
Output fromprevious phase
3
6
(3:1,2,5)
(6:4,5,7)
(3, 1)
(3, 2)
(3, 1)
(6, 7)
(6, 7)
(6, 5)
1
2
7
5
4
63
(6, 7)
(3, 1)
(3, 2)
(3, 5)
(6, 4)
(6, 5)
(3, 1)
(6, 7)
(3, 5)
(6, 4)
(6-7, 7)
Could use 2ndry keysto avoid reduce sort(),eg:
(6, 7)
(3, 1)
(3, 2)
(3, 5)
(6, 4)
(3, 1)
(6, 7)
![Page 23: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/23.jpg)
Chained MapReduce’s Pattern
Input Map Shuffle Reduce Output
Identity Mapper,key = town
Sort by keyReducer sorts, gathers,
remove duplicates;similar to join
Index of roadsin each town
(town, road) pair
Emit (key, item) pairKey = geometric hashSecondary key = Type
Sort by keysIntersect all townswith all roads; emitintersecting pairs
(town, road) pairFeature List
![Page 24: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/24.jpg)
Distributing Costly Computation:e.g. Rendering Map Tiles
Input Map Shuffle Reduce OutputEmit each to all
overlapping latitude-longitude rectangles
Sort by key(key= Rect. Id)
Render tile usingdata for all enclosed
featuresRendered tiles
Geographicfeature list
I-5
Lake Washington
WA-520
I-90
(N, I-5)
(N, Lake Wash.)
(N, WA-520)
(S, I-90)
(S, I-5)
(S, Lake Wash.)
(N, I-5)
(N, Lake Wash.)
(N, WA-520)
(S, I-90)
N
S (S, I-5)
(S, Lake Wash.)
…
…
…
…
(Bucket pattern) (Parallel rendering)
![Page 25: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/25.jpg)
Finding Nearest Points Of Interest (POIs)Feature List1, Type, Road, Intersection, …2, Type, Road, Intersection, …3, Type, Road, Intersection, …
. . .
Nearest POI within5mi of Intersection
(1, 1)(2, 1)(3, 1)(4, 1)(5, 7)(6, 7)(7, 7)(8, 7)(9, 7)
Input Output1
2
3
4
5
67
8
9
12
3
4
5
67
8
9
![Page 26: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/26.jpg)
Finding Nearest POI on a Graph
Input Map Shuffle Reduce Output
Nodes withnearest POIs
Graph
Perform Dijkstra from each POI node.
Emit POI & dist. Ateach node in search.
For each node, emitclosest POI
![Page 27: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/27.jpg)
Finding Nearest POI on a Graph
Input Map Shuffle Reduce Output
Nodes withnearest POIs
Graph
Perform Dijkstra from each POI node.
Emit POI & dist. Ateach node in search.
For each node, emitclosest POI
![Page 28: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/28.jpg)
Finding Nearest POI on a Graph
Input Map Shuffle Reduce Output
Nodes withnearest POIs
Graph
Perform Dijkstra from each POI node.
Emit POI & dist. Ateach node in search.
For each node, emitclosest POI
![Page 29: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/29.jpg)
Finding Nearest POI on a Graph
Input Map Shuffle Reduce Output
Nodes withnearest POIs
Graph
Perform Dijkstra from each POI node.
Emit POI & dist. Ateach node in search.
For each node, emitclosest POI
![Page 30: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/30.jpg)
Putting it all together: Nearest POI
Input Map Shuffle Reduce Output
Nodes with edges Feature List
Subgraphs Nodes with edges
Nodes withnearest POI & dist
Subgraphs
Perform Dijkstra from each POI node.
Emit POI & dist. Ateach node in search.
For each node, emitclosest POI
Sorted nodeswith
nearest POI
Nodes withnearest POI
& dist
Use key-join pattern to createnodes,edges out of intersections,roads
Use bucketing pattern to create“appropriate”(overlapping, large-enough) subgraphs
Use identity mapper & gather patternto sort and clean-up node, POI pairs
![Page 31: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/31.jpg)
Hard Problems for MapReduce
• Following multiple pointer hops
• Iterative algorithms• Algorithms with global state• Operations on graphs without good embeddings
• [insert your favorite challenge here]
![Page 32: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/32.jpg)
SummaryMapReduce eases:• Machine coordination• Network communication• Fault tolerance• Scaling• Productivity
MapReduce patterns:• “Flat” data structures
• Foreign / Recursive Key Joins(aka pointer following)
• Hash Joins (aka bucketing)• Distribute $$ computation• Chain MapReduce phases• Simplify Reduce() by using
secondary keys• [ insert your pattern here ]
![Page 33: MapReduce Design Patterns - courses.cs.washington.eduInner Join Pattern Input Map Shuffle Reduce Output Apply map() to each; Key = intersection id Value = feature Sort by key Apply](https://reader030.vdocuments.site/reader030/viewer/2022040721/5e2e2ac096c03f6c9d6681c7/html5/thumbnails/33.jpg)
Questions?• MapReduce: Simplified Data Processing on Large Clusters,
Jeffrey Dean and Sanjay GhemawatOSDI'04: Sixth Symposium on Operating System Design andImplementation
• Contact: [email protected]