µ ^ µ u } µ o d Æ ] u ] Ì ] } v ] v d ] À , µ Ç> x e p µ Ç...
TRANSCRIPT
![Page 1: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/1.jpg)
Distributed SubmodularMaximization in Massive Datasets
Joint work with Rafael Barbosa, Alina Ene, Justin Ward
Huy L. Nguyen
![Page 2: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/2.jpg)
Combinatorial Optimization• Given
– A set of objects V– A function f on subsets of V– A collection of feasible subsets I
• Find– A feasible subset of I that maximizes f
• Goal– Abstract/general f and I– Capture many interesting problems– Allow for efficient algorithms
![Page 3: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/3.jpg)
SubmodularityWe say that a function is submodular if:
We say that is monotone if:
Alternatively, f is submodular if:
for all and Submodularity captures diminishing returns.
![Page 4: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/4.jpg)
SubmodularityExamples of submodular functions:
– The number of elements covered by a collection of sets– Entropy of a set of random variables– The capacity of a cut in a directed or undirected graph– Rank of a set of columns of a matrix– Matroid rank functions– Log determinant of a submatrix of a psd matrix
![Page 5: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/5.jpg)
Example: Multimode Sensor Coverage• We have distinct locations where we can place sensors• Each sensor can operate in different modes, each with a distinct coverage profile• Find sensor locations, each with a single mode to maximize coverage
![Page 6: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/6.jpg)
Example: Identifying Representatives In Massive Data
![Page 7: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/7.jpg)
Example: Identifying Representative Images• We are given a huge set X of images.• Each image is stored multidimensional vector.• We have a function d giving the difference between two images.• We want to pick a set S of at most k images to minimize the loss function:
• Suppose we choose a distinguished vector e0 (e.g. 0 vector), and set:
• The function f is submodular. Our problem is then equivalent to maximizing f under a single cardinality constraint.
![Page 8: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/8.jpg)
Need for Parallelization• Datasets grow very large
– TinyImages has 80M images– Kosarak has 990K sets
• Need multiple machines to fit the dataset• Use parallel frameworks such as MapReduce
![Page 9: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/9.jpg)
Problem Definition• Given set V and submodular function f• Hereditary constraint I (cardinality at most k, matroid constraint of rank k, … )• Find a subset that satisfies I and maximizes f• Parameters
– n = |V|– k = max size of feasible solutions– m = number of machines
![Page 10: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/10.jpg)
Greedy AlgorithmInitialize S = {}While there is some element x that can be added to S:
Add to S the element x that maximizes the marginal gainReturn S
![Page 11: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/11.jpg)
Greedy Algorithm• Approximation Guarantee
• 1 - 1/e for a cardinality constraint• 1/2 for a matroid constraint
• Inherently sequential• Not suitable for large datasets
![Page 12: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/12.jpg)
Distributed GreedyMirzasoleiman, Karbasi, Sarkar, Krause '13
![Page 13: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/13.jpg)
Performance of Distributed Greedy• Only requires 2 rounds of communication• Approximation ratio is:
(where m is number of machines)• Can construct bad examples• Lower bounds for the distributed setting
(Indyk et al. ’14)
![Page 14: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/14.jpg)
Power of Randomness
![Page 15: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/15.jpg)
Power of Randomness• Randomized distributed Greedy
– Distribute the elements of V randomly in round 1– Select the best solution found in rounds 1 & 2
• Theorem: If Greedy achieves a C approximation, randomized distributed Greedy achieves a C/2 approximation in expectation.• Related results: [Mirrokni, Zadimoghaddam ’15]
![Page 16: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/16.jpg)
Intuition• If elements in OPT are selected in round 1 with high probability
– Most of OPT is present in round 2 so solution in round 2 is good• If elements in OPT are selected in round 1 with low probability
– OPT is not very different from typical solution so solution in round 1 is good
![Page 17: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/17.jpg)
Power of Randomness• Randomized distributed Greedy
– Distribute the elements of V randomly in round 1– Select the best solution found in rounds 1 & 2
• Provable guarantees– Constant factor approx for several constraints
• Generality– Same approach to parallelize a class of algorithms– Only need a natural consistency property– Extends to non-monotone functions
![Page 18: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/18.jpg)
Optimal Algorithms?• Near-optimal algorithms? • Framework to parallelize algorithms with almost no loss?
YES, using a few more rounds
![Page 19: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/19.jpg)
![Page 20: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/20.jpg)
![Page 21: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/21.jpg)
Core Set
![Page 22: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/22.jpg)
Core SetSend Core Setto every machine
![Page 23: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/23.jpg)
Core Set
![Page 24: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/24.jpg)
Core Set
![Page 25: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/25.jpg)
Core SetGrow Core Setover 1/ rounds
![Page 26: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/26.jpg)
Core SetGrow Core Setover 1/ rounds
![Page 27: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/27.jpg)
Core SetGrow Core Setover 1/ rounds
![Page 28: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/28.jpg)
Core SetGrow Core Setover 1/ rounds
Leads to only an lossin the approximation IntuitionEach round adds an fraction of OPT to the Core Set
![Page 29: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/29.jpg)
Matroid Coverage (n=900, r=5) Matroid Coverage (n=100, r=100)
It's better to distribute ellipses from each location across several machines!
Matroid Coverage Experiments
![Page 30: µ ^ µ u } µ o D Æ ] u ] Ì ] } v ] v D ] À , µ Ç> X E P µ Ç vvova/sublinear2016/slides/Nguyen.pdf · 2018-01-28 · d Z v l z } µ J Y µ ] } v M. Title: Microsoft PowerPoint](https://reader030.vdocuments.site/reader030/viewer/2022040621/5f33f2a259eae06cfe166fa6/html5/thumbnails/30.jpg)
Thank You!Questions?