observability, event sourcing and state machines...observability, event sourcing and state machines...

39
Peter Lawrey – Chronicle Software QCon London - 2017 Observability, Event Sourcing and State Machines

Upload: others

Post on 23-May-2020

20 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Peter Lawrey – Chronicle Software

QCon London - 2017

Observability,EventSourcingandStateMachines

Page 2: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

PeterLawrey

JavaDeveloper/Consultantforinvestmentbanksandhedgefunds

for10years.

MostanswersforJavaandJVMonstackoverflow.com

Page 3: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

KeypointsInpureJavayoucan• AccessTBsofdatainprocess• DatacanbesharedacrossJVMs• Thiscanspeedupyourapplication

Page 4: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Why?Observability• Reducestimetofix• Reducestimetodeliveraqualitysolution• Improvesperformance

Page 5: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

TypicalSolutionsMarketdataprocessinganddistributionOrdergenerationandmanagementPositionnotificationanddistributionRealtimeCompliance

30micro-secondstypical,100micro-seconds,99%ofthetime

Page 6: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

128 KB RAM

Page 7: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Howmuchdoesrecordeverythingcost

2TBSSD~£1K

Page 8: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

ScaletohighvolumeswithlessmemoryWriting1TBona128GBmachine

Page 9: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

ScaletohighvolumeswithlessmemoryWriting1TBona128GBmachine

Page 10: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

ScaletohighvolumeswithlessmemoryWriting1TBona128GBmachine

Page 11: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Scaletohighthroughputwithlowlatencies.

Page 12: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

HowtoaccessTBsofpersisteddata

MemorymappedfilesDatastructuresonthesefiles

ConcurrentaccessbetweenJVMsUsereplicationinsteadofsync

Page 13: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Eventsourcing

persiststhestateofabusinessentity…asasequenceofstate-changingevents.

Theapplicationreconstructsanentity’scurrentstatebyreplayingtheevents.

http://microservices.io/patterns/data/event-sourcing.html

Page 14: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

UsingEventSourcing

Eachoutputistheresultofoneinputmessage.Thisisusefulforgateways,bothinandoutofyour

system.Highlyconcurrent.

Page 15: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Buildinghighlyreproduciblesystems

EachoutputistheresultofALLtheinputs.InsteadofreplyingALLinputmessageeachtime,theFunction

couldsaveanaccumulatedstate.

Page 16: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Yourcriticalpathasaseriesoflowlatency,nonblockingtasks.Thiskeepsyourlatencies

endtoendconsistentlylow.

Page 17: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for
Page 18: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Recordeverythingmeans

GreaterTransparencyHighReproducibilityFastertimetofix

Fasterdeliveryofaqualitysystem

Page 19: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Togofaster,doless

Perfectionisachieved,notwhenthereisnothingmoretoadd,butwhenthereisnothinglefttotakeaway.

AntoinedeSaint-Exupery

Page 20: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

NoFlowControl?

MarketData

Compliance

Page 21: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Reproduceeachcomponentindependently

Whetheryouareenrichingdatafromadatabaseorproductioniscomplex,eachservicecanbetestedin

isolation.

Page 22: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

TestingandDebuggingMicroservices

Frameworkscanmaketestinganddebuggingharder.

Youneedtobeabletotestanddebugyourcomponentswithouttheframework,oratransport.

Page 23: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

TurningaMonolithintoMicroservices

BusinessComponent+Transport=Service.

Page 24: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Startingwithasimplecontract

Anasynchronousmessagehasatype,apayloadanddoesn’treturnaresult.

public interface SidedMarketDataListener {void onSidedPrice(SidedPrice sidedPrice);

}

public interface MarketDataListener {void onTopOfBookPrice(TopOfBookPrice price);

}

Page 25: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

ADataTransferObjectpublic class SidedPrice extends AbstractMarshallable {

String symbol;long timestamp;Side side;double price, quantity;

public SidedPrice(String symbol, long timestamp, Side side, double price, double quantity) {

this.symbol = symbol;this.timestamp = timestamp;this.side = side;this.price = price;this.quantity = quantity;return this;

}}

Page 26: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Deserializable toString()Forittodeserialize thesameobject,noinformationcanbelost,whichusefultocreatingtestobjectsfromproductionlogs.

SidedPrice sp = new SidedPrice("Symbol", 123456789000L, Side.Buy, 1.2345, 1_000_000);

assertEquals("!SidedPrice {\n" +" symbol: Symbol,\n" +" timestamp: 123456789000,\n" +" side: Buy,\n" +" price: 1.2345,\n" +" quantity: 1000000.0\n" +"}\n", sp.toString());

// from stringSidedPrice sp2 = Marshallable.fromString(sp.toString());assertEquals(sp2, sp);assertEquals(sp2.hashCode(), sp.hashCode());

Page 27: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Writingasimplecomponent

Wehaveacomponentwhichimplementsourcontractandinturncallsanotherinterfacewitharesult

public class SidedMarketDataCombinerimplements SidedMarketDataListener {

final MarketDataListener mdListener;

public SidedMarketDataCombiner(MarketDataListener mdListener) {this.mdListener = mdListener;

}

Page 28: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Writingasimplecomponent

Thecomponentcalculatesaresult,usingprivatestate.

final Map<String, TopOfBookPrice> priceMap = new TreeMap<>();

public void onSidedPrice(SidedPrice sidedPrice) {TopOfBookPrice price = priceMap.computeIfAbsent(

sidedPrice.symbol, TopOfBookPrice::new);if (price.combine(sidedPrice))

mdListener.onTopOfBookPrice(price);}

Page 29: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Testingoursimplecomponent

Wecanmocktheoutputlistenerofourcomponent.

MarketDataListener listener = createMock(MarketDataListener.class);listener.onTopOfBookPrice(new TopOfBookPrice("EURUSD", 123456789000L,

1.1167, 1_000_000, Double.NaN, 0));listener.onTopOfBookPrice(new TopOfBookPrice("EURUSD", 123456789100L,

1.1167, 1_000_000, 1.1172, 2_000_000));replay(listener);

SidedMarketDataListener combiner = new SidedMarketDataCombiner(listener);combiner.onSidedPrice(new SidedPrice("EURUSD", 123456789000L,

Side.Buy, 1.1167, 1e6));combiner.onSidedPrice(new SidedPrice("EURUSD", 123456789100L,

Side.Sell, 1.1172, 2e6));

verify(listener);

Page 30: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

TestingmultiplecomponentsWecanmocktheoutputlistenerofourcomponent.

// what we expect to happenOrderListener listener = createMock(OrderListener.class);

listener.onOrder(new Order("EURUSD", Side.Buy, 1.1167, 1_000_000));

replay(listener);

// build our scenarioOrderManager orderManager =

new OrderManager(listener);

SidedMarketDataCombiner combiner =new SidedMarketDataCombiner(orderManager);

Page 31: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Testingmultiplecomponents

// events in: not expected to triggerorderManager.onOrderIdea(

new OrderIdea("EURUSD", Side.Buy, 1.1180, 2e6));

combiner.onSidedPrice(new SidedPrice("EURUSD", 123456789000L, Side.Sell, 1.1172, 2e6));

combiner.onSidedPrice(new SidedPrice("EURUSD", 123456789100L, Side.Buy, 1.1160, 2e6));

combiner.onSidedPrice(new SidedPrice("EURUSD", 123456789100L, Side.Buy, 1.1167, 2e6));

// expected to triggerorderManager.onOrderIdea(

new OrderIdea("EURUSD", Side.Buy, 1.1165, 1e6));

verify(listener);

Page 32: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Addingatransport

Anymessagingsystemcanbeusedasatransport.Youcanuse• RESTorHTTP• JMS,Akka,MPI• AeronoraUDPbasedtransport.• RawTCPorUDP.• ChronicleQueue.

Page 33: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Makingmessagestransparent

--- !!data #binaryonOrderIdea: {

symbol: EURUSD,side: Buy,limitPrice: 1.118,quantity: 2000000.0

}

orderManager.onOrderIdea(new OrderIdea("EURUSD", Side.Buy, 1.1180, 2e6));

Page 34: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

WhyuseChronicleQueue

ChronicleQueuev4hasanumberofadvantages• Brokerless,onlytheOSneedstobeup.• Lowlatency,lessthan10microseconds99%ofthe

time.• Persisted,givingyourreplayandtransparency.• Canreplaceyourloggingimprovingperformance.• KernelBypass,SharedacrossJVMswithasystemcall

foreachmessage.

Page 35: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

--- !!meta-data #binaryheader: !SCQStore { wireType: !WireType BINARY, writePosition: 777, roll: !SCQSRoll { length: 86400000, format: yyyyMMdd, epoch: 0 }, indexing: !SCQSIndexing { indexCount: !int 8192, indexSpacing: 64, index2Index: 0, lastIndex: 0 } }

# position: 227 --- !!data #binaryonOrderIdea: { symbol: EURUSD, side: Buy, limitPrice: 1.118, quantity: 2000000.0 }

# position: 306 --- !!data #binaryonTopOfBookPrice: { symbol: EURUSD, timestamp: 123456789000, buyPrice: NaN, buyQuantity: 0, sellPrice: 1.1172, sellQuantity: 2000000.0 }

# position: 434 --- !!data #binaryonTopOfBookPrice: { symbol: EURUSD, timestamp: 123456789100, buyPrice: 1.116, buyQuantity: 2000000.0, sellPrice: 1.1172, sellQuantity: 2000000.0 }

# position: 566 --- !!data #binaryonTopOfBookPrice: { symbol: EURUSD, timestamp: 123456789100, buyPrice: 1.1167, buyQuantity: 2000000.0, sellPrice: 1.1172, sellQuantity: 2000000.0 }

# position: 698 --- !!data #binaryonOrderIdea: { symbol: EURUSD, side: Buy, limitPrice: 1.1165, quantity: 1000000.0 } ... # 83885299 bytes remaining

Page 36: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Measuringtheperformance?MeasurethewritelatencywithJMH(JavaMicrobenchmarkHarness)

Percentiles, us/op:

p(0.0000) = 2.552 us/op

p(50.0000) = 2.796 us/op

p(90.0000) = 5.600 us/op

p(95.0000) = 5.720 us/op

p(99.0000) = 8.496 us/op

p(99.9000) = 15.232 us/op

p(99.9900) = 19.977 us/op

p(99.9990) = 422.475 us/op

p(99.9999) = 438.784 us/op

p(100.0000) = 438.784 us/op

Page 37: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for
Page 38: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

WherecanItrythisout?

LowLatencyMicroservicesexampleshttps://github.com/Vanilla-Java/Microservices

TheOSSChronicleproductsareavailablehttps://github.com/OpenHFT/

Page 39: Observability, Event Sourcing and State Machines...Observability, Event Sourcing and State Machines Peter Lawrey Java Developer / Consultant for investment banks and hedge funds for

Q&A

Blog:http://vanilla-java.github.io/

http://chronicle.software

@ChronicleUG

[email protected]

https://groups.google.com/forum/#!forum/java-chronicle