ricon/west 2013: adventures with riak pipe

DISTRIBUTING WORK ACROSSCLUSTERSAdventures With Riak Pipe

Last Updated: October 31, 2013

Susan Potter @SusanPotter

Background Concepts & Intuitions Applications

WHOAMI

2

..

WHOAMI..

2013-10-31

Distributing Work Across Clusters

whoami

An app + middleware dev who has felt Ops pain too

BACKGROUND


MODEL/ASSUMPTIONS

4


MOTIVATING USES

5


COMMON APPROACHES

6


WHY DISTRIBUTE? WHY DECENTRALIZE?

→ Probability of system failure→ Large datasets (data locality)→ Eliminate SPOFs→ Commodity hardware

7


8

CONCEPTS & INTUITIONS


LAYERS

10


TERMINOLOGY (CORE)

→ Partition→ Vnode→ Hinted Handoff→ Read Repair

11


TERMINOLOGY (PIPE)

→ Pipe→ Fitting ("phase")→ Queue→ Worker

12


EXAMPLE UNIX PIPE

1 find . -name "*.rb" \2 | xargs egrep "#.*?TODO:" \3 | wc -l

Character-based, through file descriptors

13


EXAMPLE FUNCTION COMPOSITION

1 (length . mapToUpper . sanitize) input

Value based, through functions

14


EXAMPLE RIAK PIPE

1 [ #fitting_spec{ name=fetch_trades2 , module=riskmgr_fetch_trades3 , ...}4 , #fitting_spec{ name=calc_var5 , module=riskmgr_calc_var6 , ...}7 ]

Message-based, across nodes

15


VNODE AND WORKERS

→ vnodemanages lifecycle and queues of workers

→ vnode workerprocesses inputs

16


VNODE WORKER BEHAVIOR

1 behaviour_info(callbacks) ->2 [{init,2},3 {process ,3},4 {done,1}];5 behaviour_info(_Other) ->6 undefined.7 %% Optionally two more too, not required.

17


DISTRIBUTION: CHASHFUN

→ Use random uniform hash fun to saturate workers→ See Bryan Fink's previous Pipe presentations

18


VALIDATION

1 % #fitting_spec { module = ..., arg = 43, ...}

2 % validate_arg/13 validate_arg(Arg) when is_integer(Arg) -> ok;4 validate_arg(Arg) when is_atom(Arg) -> ok;5 validate_arg(_) -> {error, "Argument must be

a valid integer or atom"}.

19


LEAKY PIPES

http://www.flickr.com/photos/thirteenofclubs/

20


RIAK (CORE) MECHANICS

→ HandoffMigrate vnodes from node to node

1 % Old Node2 % Called with last known State of worker3 archive/1 % returns {ok, Archive}4

5 % New Node6 % Called with Archive and State from old node7 handoff/2 % returns {ok, NewState}

21


FAILURES

→ validate_argSeen above

→ Pipe client process diesYOLO

→ nvalHow many vnodes to ask before failing, default 1

→ vnode worker logicThink using dataflow semantics

22


EXISTING FITTINGS

→ riak_pipe_w_teeUseful for intermediate results

→ riak_pipe_w_xformSimple delegater: 3-arity function

→ riak_pipe_w_reduceWhat you would expect: simple accumulating reduce

→ riak_kv_pipe_getTake advantage of data locality with cohosted KV store

23

APPLICATIONS


KNOWN USES

→ Riak's Map/Reduce→ Risk metrics→ Tenant Usage

25


TROUBLESHOOTING

→ riak_pipe:status/1Provides fittings, processed, failures, queue_length, etc stats

→ riak_pipe_w_crash fittingUsed to test Riak Pipe

→ riak_pipe:active_pipelines/1See all active pipelines

→ riak_pipe_cinfo modulecluster info interrogation module

26


FURTHER WORK

→ More applications (e.g. genetics, 3rd party APIs with ratelimits)

→ Decentralized pipe control→ Measure "completeness" of resultset

27


RELATED WORK

→ "Pipe" libraries EVERYWHERE (pipe, scalaz-stream)→ riak_pg→ Map-Reduce frameworks→ Staged Event Driven Architecture→ Event Stream Processing→ Dataflow multi-stage processing

28


ROYAL FAIL

http://www.flickr.com/photos/dadavidov/

29


QUESTIONS

Questions?

30

ricon/west 2013: adventures with riak pipe

Technology

arg arg

integer arg

atom arg

var module

trades module

crash fitting

old node handoff

riak piperiak