Thinking Like An Erlanger

Torben HoffmannCTO @ Erlang Solutions

torben.hoffmann@erlang-solutions.com@LeHoff

Do You Want To See The Erlang Code?

You Can’t Handle The Erlang Code!!

Source: http://static7.businessinsider.com/image/4d9d0c7bcadcbbaf20120000/jack-a-few-good-men.jpg

Syntax Is Irrelevant

Thinking Is Everything

General vs Domain Specific

Telecom

Erlang

C++/Java

Smaller gap =

money!

Protocols

Paxos

{ Acceptors }Proposer Main Aux Learner| | | | | | -- Phase 2 --X----------->|->|->| | | Accept!(N,I,V)| | | ! | | --- FAIL! ---|<-----------X--X--------------->| Accepted(N,I,V)| | | | | -- Failure detected (only 2 accepted) --X----------->|->|------->| | Accept!(N,I,V) (re-transmit, include Aux)|<-----------X--X--------X------>| Accepted(N,I,V)| | | | | -- Reconfigure : Quorum = 2 --X----------->|->| | | Accept!(N,I+1,W) (Aux not participating)|<-----------X--X--------------->| Accepted(N,I+1,W)| | | | |

Source: https://en.wikipedia.org/wiki/Paxos_(computer_science)#Byzantine_Paxos

The Golden Trinity Of Erlang

Processes

They are dirt cheap!

Use lots of them! Lots!

Focus on their interactions

Game of Life

Simple cellular automaton

Evolution of cells

Discrete time

2,3 neighbours: survive

empty, 3 neighbours: new cell

all others: empty

Traditional Approach 2-d array - one per time step

Issues with Traditional Approach

Does not scale well

Imperative data structures are ugly in Erlang

Basic Erlang Idea

One process per cell

Talk to neighbour cells

Collect from all neighbours

Update own content and go to T+1

Is this Erlangy enough?

Spawn a process to collect from all neighbours

collector_loop([], NeighbourCount, Cell, Content) ->

Cell ! {self(), {next_content, next_content(Content, NeighbourCount)}};

collector_loop(WaitingOn, NeighbourCount, Cell, Content) ->

receive

{cell_content, {{{_,_},_}=XYatT, NeighbourContent}} ->

case lists:member(XYatT, WaitingOn) of

true ->

collector_loop(lists:delete(XYatT, WaitingOn),

NeighbourCount + NeighbourContent,

Cell, Content);

false -> %% ignore messages we are not waiting for

collector_loop(WaitingOn, NeighbourCount, Cell, Content)

end

end.

Will this work?

It works if…

you only take one step at a time

let the cells run freely…

cells get out of sync

requests for old time

requests for future time

Fixes

Old time requests:

keep history

Future time requests:

queue the response

What About Failures?

Supervising Cells

The Erlang supervisor will restart with the original args…

all state and history is lost

Fix:

monitor all cells

step them forward to current time

Cell Mgr Codeloop(#state{cells=Cells}=State) ->

receive

{'DOWN', Ref, process, _Pid, _Info} ->

{value, {_,_, XY}, Rest} = lists:keytake(Ref, 2, Cells),

NewPid = await_restart(XY),

NewRef = erlang:monitor(process, NewPid),

NextState = State#state{

cells=[{NewPid, NewRef, XY}|Rest]},

spawn ( fun () -> pacer(NewPid) end ),

loop(NextState)

end.

Process down

Cell Mgr Codeloop(#state{cells=Cells}=State) ->

receive

{'DOWN', Ref, process, _Pid, _Info} ->

{value, {_,_, XY}, Rest} = lists:keytake(Ref, 2, Cells),

NewPid = await_restart(XY),

NewRef = erlang:monitor(process, NewPid),

NextState = State#state{

cells=[{NewPid, NewRef, XY}|Rest]},

spawn ( fun () -> pacer(NewPid) end ),

loop(NextState)

end.

Wait for supervisorto restart it

Cell Mgr Codeloop(#state{cells=Cells}=State) ->

receive

{'DOWN', Ref, process, _Pid, _Info} ->

{value, {_,_, XY}, Rest} = lists:keytake(Ref, 2, Cells),

NewPid = await_restart(XY),

NewRef = erlang:monitor(process, NewPid),

NextState = State#state{

cells=[{NewPid, NewRef, XY}|Rest]},

spawn ( fun () -> pacer(NewPid) end ),

loop(NextState)

end.

Get the new processup-to-date

Deadlock Remaining…

N1 dies with Cell(i,j) queued

Re-cap

A process per cell

Short-lived processes for small tasks

Focus on the protocols between processes

Supervisor to restart

Monitoring manager process to get restarted cells up-to-speed

Stock Exchange

The Trigger…

Erlang-Questions on using ETS for sell and buy orders:

http://erlang.org/pipermail/erlang-questions/2014-February/077969.html

Painful…

An Exchange

Connects buyers and sellers

Buyers post buy intentions

Sellers post sell intentions

Basic Erlang Idea

One process per buy/sell intention

Processes to negotiate deals by exchanging messages

Communication

Use gproc as pub-sub mechanism to announce buy and sell intentions

All buyers listen to sell intention

All sellers listen to buy intentions

Can happen when

Negotiation by 3-way handshake

Deals

priceseller pricebuyer

Buyer Arrives

Unique reference to identify the sell offerSeller’s Pid

5 pt

Seller Arrives

What About Failures?

What Can Go Wrong?

1. Buyer dies

3. Buyer dies

2. Seller dies

1 & 2 can be fixed by timing out

Danger!! Seller has closed the deal on his side

Simple re-start leaves the buyer at 3@5

Monitor each other

Removes the need for timeouts

Still not sure how far the other side got

Transaction Log Per Process

Just replay back to the last state

Issues:

Messages cannot be replayed

Must ask partner about their view on the status of the deal

Ledger

Create Ledger process that tracks all completed deals

Each buyer and seller get a unique OfferID when started

Thinking in Erlang

Focus on protocols (MSCs)

Ask “What could go wrong here?”

Tools Lots of processes!!

Spawn short-lived processes for small things

Supervisors

Link and monitor

gproc for registering and pub/sub

Timeouts

Transaction logs (ledgers)

Food for Thought

What can I only do in Erlang?

http://erlang.org/pipermail/erlang-questions/2014-November/081570.html

You can avoid writing your own service framework.

Craig Everett

Testing

Async protocols are nasty

Use EQC - Property Based Testing

Focus on one process

Mock the calls to others

WIP in repo!

Code

Game of life

https://github.com/lehoff/egol

Erlang Exchange

https://github.com/lehoff/erlang_exchange

Deadlock left to fix

Testing incomplete

No ledger

Learning Erlang

Source: http://www.despair.com/mistakes.html

Learning Erlang

ESL training courses

Learn You Some Erlang

http://learnyousomeerlang.com/

Use the erlang-questions mailing list

Do it hands-on

Give it time to sink in!!!

Elixir

Built on top of the Erlang VM

More Ruby-like syntax

Hygienic macros - easy to do DSLs

Better support for data handling

But… you still have to learn the Erlang programming model

Key building blocks

Share nothing processes

Message passing

Fail fast approach

Link/monitor concept

You can deal with failures in a sensible manner because you have a language for them.