couchconf london syncpoint

1

Connecting Millions ofMobile Devices to the Cloud

Marty Schoch@mschoch

2

THE WATER CYCLE

3

The water cycle

Fun fact: In 20 minutes, one thunderstorm can drop 125,000,000 gallons of water.

4

11101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101

11101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101

The water data cycle

5

11101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101

11101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101111010110101101000101010111010101010111010100101010010101010100010101010111101011010110100010101011101010101011101010010101001010101010001010101011110101101011010001010101110101010101110101001010100101010101000101010101

Increasingly, it starts and ends with a mobile device

Capture Consume

Collect

6

Actually, it starts and ends with a LOT of mobile devices

Mobile-broadband subscriptions have grown 45% annually over the last four years and today there are twice as many mobile-broadband as fixed-broadband subscriptions.

The number of mobile-connected devices will exceed the world’s population in 2012; and monthly global mobile data traffic will surpass 10 exabytes in 2016.

Fun Fact: By the end of 1999, the sum of human-produced information (including all audio, video recordings and text/books) was about 12 exabytes of data, according to Berkeley study.

7

PUTTING DATA TO WORK

8

Make farming data work

9

Make intelligence data work

10

Make industrial data work

11

Make business data work

12

Connecting millions of mobile devices to the cloud

otherfarms

Type 1backup and sync

my stuff

Type 2sync relevant portion

of a database

Type 3collect machine-generated data

analysis

13

We’ll focus on Type 2

Type 2sync relevant portion

of a database

Apple iC

loud

Smar

tMet

er N

etwork

Most applications combine patterns.

14

CHALLENGES … SOLUTIONS

15

Type 2 Mobile Sync

The ultimate home of the data for a business application is not on the device, but is a central data store in the cloud.

16

Challenges

Which data – temporal, spatial, user?

Bandwidth conservation

Scalability of consolidated store

Conflict detection and resolution

Battery and memory conservation

Selection performance

Deletion propagation

Schema evolution

Referential integrity on interruption

New user provisioning

Security

Hard at “enterprise scale.” Excruciating at “web scale.”

17

A multi-tier, scalable type 2 sync architecture

C-channelF-channel

Load

bal

ance

Sync

poin

t

Couc

hbas

e

Elas

tic S

earc

h

App

Serv

er

M

CHANNEL MEMBERSHIP

DOCUMENTS (TO INDEX)

DOCUMENTS SYNC METADATA

DOCUMENTS

18

Details Subject to

Change

19

ElasticSearch Queries and Syncpoint Channel

{ "range": { "abv": { "from": 7, "to": 10 } }}

Register ElasticSearch query named strongBeer for percolation

Then define a Syncpoint channel with strongBeer as a member query

20

The architecture, in more detail

Couchbase Cluster

ElasticSearch Cluster Syncpoint Cluster

DocumentMutation

TAPStream

• Q1• Q2• Q3

PercolationResults

• C1 (Q1, Q2)• C2 (Q2)• C3 (Q1, Q2, Q3)

21

Syncpoint Client API

Mobile Client Syncpoint Cluster

Establish Session

• Pluggable Authentication• Pairs user with device• Establishes control database

Subscribe to Channel(s)

• Creates a local installation of this channel• Bi-directional sync of channel documents

22

TouchDB – Foundation for Syncpoint

Native Library

23

TouchDB Betas – Works with CouchDB Today

TouchDB for iOSTouchDB for

Android

Not a complete product but a functional open source projectswith production deployments.Open source, and available fromCouchbase labs.

24

JOIN THE COMMUNITY

http://groups.google.com/group/mobile-couchbase

25

THANKS

MARTY@COUCHBASE.COM