The Role of Communication Complexity in Distributed Computing

Rotem Oshman

Background: Distributed Computing

Distributed Computing

• Typical model:– Local computation for free– Charge for “communication”

Distributed Lower Bounds

• “All or nothing”:– If have not communicated, they know nothing

about each other– If communicated, they know everything about

each other• Recently: interest in quantifying the amount of

communication• Natural to use communication complexity

Shared Memory

• Processes communicate by accessing objects in shared memory– Read/write registers– Read-modify-write: CAS, T&S, …

• Typically asynchronous:– Schedule = sequence of process IDs– Adversarially chosen– Sometimes processes may crash

Shared Memory Lower Bounds

• “Covering arguments”• Examples:– Deterministic consensus impossible if one process

may fail [FLP’79]– Mutual exclusion requires shared registers [BL’90]

• Can we introduce a cell-probe-like model for shared memory lower bounds?

Message Passing

• Processes communicate by sending messages– Over some network graph, often complete

• Fully synchronous, fully asynchronous, or anywhere in between

• Processes can crash, recover, cheat, lie,…

• Many successful applications of CC

Some differences…

Complexity measure

Problems

Input

Distributed Computing Comm. Complexity#rounds (limited bandwidth) total #bits

Search Decision

Number-In-Hand Number-on- Forehead (usually)

Message-Passing Models

MESSAGE-PASSING

LOCAL

CONGEST

SHARED BLACKBOARD#rounds total CC

Talk Overview

I. Lower bound techniquesa. CONGEST (#rounds): reductions from 2-party

communication complexityb. Total CC with private channels

II. Shared blackboarda. Number-in-handb. “Not-quite-number-in-hand”

The CONGEST Model

• nodes communicate over a network graph (small diameter)

• Computation proceeds in synchronous rounds• Each message = bits• Several variants:– Communication by unicast vs. local broadcast– Arbitrary graph vs. complete graph

The CONGEST Model

• Arbitrary graph topology: strong lower bounds– [Distributed Verification and Hardness of Distributed

Approximation, by Das Sarma, Holzer, Kor, Korman, Nanongkai, Pandurangan, Peleg, Wattenhofer]• Nearly-tight lower bound on MST approximation• Non-tight lower bounds on all-pairs shortest path,

minimum cut, shortest s-t path, …• Almost all lower bounds

– Very few bounds known, no super-linear bounds

The CONGEST Model

• Complete graph topology, unicast:– Known to be extremely powerful, e.g.,• Sort values in rounds• MST construction in rounds

– No explicit super-constant lower bound known– [Drucker, Kuhn, O. ‘14]: even slightly super-

constant lower bound new ACC lower bound• Complete graph topology, broadcast = shared

blackboard

CONGEST Lower Bounds for Arbitrary Graphs

… by reduction from 2-party disjointness

2-Party Reductions

• Textbook reduction [Kushilevitz Nisan]:Given algorithm for solving task …

Solution for answer for, e.g., Disjointness

bits

𝑌𝑋

Based on

Based on

Simulate

2-Party Reductions

• More generally:Given algorithm for solving task …

Solution for answer for, e.g., Disjointness

𝑌𝑋

Based on

Based on

Multi-Player NIH Communication with Private Channels

The Message-Passing Model

• players• Private channels• Private -bit inputs • Private randomness

• Goal: compute • Cost: total communication

The Coordinator Model

• players, one coordinator• The coordinator has no input

Message-Passing vs. Coordinator

≈

Secure multi-party computation!

Message-Passing Lower Bounds

For players with -bit inputs…• Woodruff, Zhang ‘12: estimation• Phillips, Verbin, Zhang ’12:– for bitwise problems (AND/OR, MAJ, …)

• Woodruff, Zhang ‘13:– for threshold and graph problems

• Braverman, Ellen, O., Pitassi, Vaikuntanathan ‘13: for disjointness

Symmetrization [Phillips, Verbin, Zhang ’12]

Lower bound for -player problem :• Choose hard 2-player problem • Fix hard distribution for , let

Alice:

Bob:

“Smear” across players

Symmetrization [Phillips, Verbin, Zhang ’12]

Lower bound for -player problem :

– -player distribution must be symmetric– Answer to answer to

cost for = cost for

Alice:

Bob:

“Smear” across players

Symmetrization Example: Bitwise-XOR

• 2-party problem:– Input: uniform independent – Goal: Alice outputs Bob’s input,

• Reduction:– Sample uniform , Alice sets – Bob chooses uniformly s.t. – From , Alice can reconstruct

Alice:

Bob:

Set Disjointness

Disj𝑛 ,𝑘    =¿ 𝑖=1¿𝑛¿ 𝑗=1¿𝑘 𝑋 𝑖𝑗¿

?

𝑋 1

𝑋 2

𝑋 3

𝑋 4𝑋 5

Symmetrization vs. Disjointness

• Consider any symmetric distribution…

• How many 0s in coord. , given ?– More than one ⇒ Bob probably sees a zero• Ignore this coordinate

– Only one Pr[ Alice got it ] – 2-party CC

Alice:

Bob:

[BEOPV’13] Lower Bound Outline

1. Direct sum: 2. One-bit lower bound:

Disj𝑛 ,𝑘    =¿ 𝑖=1¿𝑛¿ 𝑗=1¿𝑘 𝑋 𝑖𝑗¿

Reduction from DISJ tograph connectivity [Based on WZ’13]

1

2

3

4

5

6

𝑝1

𝑝2

𝑝𝑘

(Players)

𝑋 𝑖

[𝑛 ]∖⋃𝑋 𝑖

(Elements)

input graph connected

Number-In-Hand Shared Blackboard

Why Should We Care?

• Some fundamental question still open• Natural model for distributed computing– Single-hop wireless network– More generally, abstracts away network topology– Related to MapReduce, etc. [Hegeman and

Pemmaraju’14]

Example: NIH Multi-Party Disjointness

• Trivial upper bound: – Also easy to get

• Simultaneous CC: [Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13]

– In rounds?– Looks like for [current work with Ran Raz]

• Unbounded rounds: [with Mark Braverman]

“Not-Quite Number in Hand”

• In undirected networks, each edge is known to both endpoints

• Distributed graph property testing:– Players , input graph – Input to player : its neighbors in – Goal: test if satisfies property

• Example: subgraph detection [Drucker, Kuhn, O. ‘14]

Example: Lower Bound for

• Claim: -round algorithm for detection solve DISJ in bits

• Reduction outline:– Alice and Bob get inputs – Construct input graph on nodes, such that

contains – Simulate the run of -detection algorithm on

12

34

12

34

12

34

12

34

Construction of from

Construction of from 1

2

34

12

34

12

34

12

34

Top-bottom: Bottom-top:

12

34

12

34

12

34

12

34

Construction of from

(𝟐 ,𝟒 )∈𝑿∩𝒀

Simulating the Algorithm

• Alice simulates nodes, Bob simulates • To simulate one round, each player:– Locally computes message broadcast by each

node it simulates– Sends all messages to the other player– Cost: per round

• Total cost:

What About ?

• More complicated….Can contain regardless of

Solution: use extremal -free

graph Elements of DISJ

= edges of

Upper Bound on Subgraph Detection

• Turán number: max # edges in -free graph on vertices

• Upper bound: solve -subgraph detection in rounds– Example: , nearly tight– Open problem: is this tight for all subgraphs?

Detecting Triangles

• Trivial upper bound: rounds• Lower bound?– 2-party (black box) reduction cannot prove it– For each triangle, one player knows all 3 edges

Triangles to 3-Party NOF Disjointness

• [Ruzsa and Szemerédi ’76]: there is a tripartite graph where

– contains triangles– Each edge in belongs to exactly one triangle

• Reduce from 3-party NOF Disjointness on elements, each representing one triangle in

Triangles to 3-Party NOF Disjointness

• Input: sets of triangles • Let be the unique triangle edge belongs to• Construct , including:– iff – iff – iff

• Note: endpoints of each edge agree on its inclusion!

𝑍

𝑋

𝑌

𝐴

𝐶

𝐵

Triangles to 3-Party NOF Disjointness

• Input: sets of triangles • Triangle appears in • Cost of simulation:– bits per round⇒ Round complexity of triangle detection

for 3-party NOF

𝑍

𝑋

𝑌

𝐴

𝐶

𝐵

3-Party NOF Disjointness

• Randomized CC:– Sherstov’13: – We get nothing:

• Deterministic CC:– Rao and Yehudayoff’14: ⇒ for deterministic triangle detection

Conclusion

MESSAGE-PASSING

LOCAL

CONGEST

SHARED BLACKBOARD#rounds total CC

Directions for Future Research

• Exploiting asynchrony and faults to get stronger communication lower bounds

Example 1: Dynamic Networks

• Abstract model for dynamic networks:– In each round we get a different graph

• [Kuhn, Lynch, O. ‘10]:– Assume each is connected– Any function can be deterministically computed in

rounds using -bit messages– Lower bounds?

Example 1: Dynamic Networks

• For exchanging all inputs:– Determistic, “routing-based” algorithms: [Haeupler,

Kuhn’12]

– Randomized??– Non-routing based??

Example 2: Byzantine Consensus

• processes, synchronous message-passing• Each process receives a bit• Goal:– Everyone outputs the same bit– If everyone received , the output is

• Byzantine faults: up to processes may behave arbitrarily

Example 2: Byzantine Consensus

• [King, Saia ‘10]: can solve with total bits• No general lower bound better than