ekon21 microservices - event driven design
TRANSCRIPT
Microservices – Event Driven Systems
• Arnaud Bouchez
– Delphi Expert
• Various solutions (from Vatican to gaming industry)
• IoT solution (RSI)
• Real-time Monitoring solution (LiveMon)
– Open Source
• mORMot (SOA ORM MVC framework)
• SynPDF SynMustache SynDB SynCrypto
– Training and consultancy
• Synopse one-man company
Microservices – Event Driven Systems
Event-Driven Systems
• Using Events?
• Event-Driven Design
• Event-Driven mORMot
Microservices – Event Driven Systems
Using Events?
• Imperative Architecture
– Events are high-level commands
Microservices – Event Driven Systems
Using Events?
• Imperative Architecture
– Events are high-level commands
• Easy modelization into regular code
procedure TZen.HandleIAmGoingOut(Sender: TPeople); var temp: TTemperature; begin temp := TemperatureSensor.GetTemperature; if temp > Threshold then RoboValet.FetchDownJacket(Sender);
end;
Microservices – Event Driven Systems
Using Events?
• Imperative Architecture
– State is stored in a main shared DB
– Events are high-level commands
translated into SQL statements
Microservices – Event Driven Systems
Using Events?
• Imperative Architecture
– State is stored in a main shared DB
– Events are high-level commands
translated into SQL statements
• DB-centric approach
• Atomicity is ensured, but history is lost
• DB becomes a bottleneck
Microservices – Event Driven Systems
Using Events?
• Event-Driven Architecture
– Collaboration is done using Events
Microservices – Event Driven Systems
Using Events?
• Event-Driven Architecture
– Collaboration is
done using Events
Just like
VCL/FMX /LCL !
Microservices – Event Driven Systems
Using Events?
• Event-Driven Architecture
– Collaboration is done using Events
– System consists in uncoupled nodes
which could be added or removed on-the-fly
– Nodes publish events, and subscribe to them
– Each node stores the information it needs
– Some API gateways expose state via REST
Microservices – Event Driven Systems
Event-Driven Architecture
• Nodes implemented as services procedure TZen.Init; begin TemperatureSensor.SubscribeToTempChange(OnTempChanged); end;
procedure TZen.OnTempChanged(NewTemp: TTemperature); begin CurrentTemp := NewTemp; end;
procedure TZen.HandleIAmGoingOut(Sender: TPeople); begin if CurrentTemp > Threshold then RoboValet.NotifyNeedsJacket(Sender); end;
Microservices – Event Driven Systems
Event-Driven Architecture
– Collaboration is done using Events
Not natural to imperative code (Delphi, C#, Java)
– System consists in uncoupled nodes
Architecture becomes complex
– Nodes publish events, and subscribe to them
May be subject to unexpected loops
– Each node stores the information it needs
Duplicated data, with eventual consistency
Microservices – Event Driven Systems
Event-Driven Architecture
• More complex than imperative code procedure TZen.HandleIAmGoingOut(Sender: TPeople); var temp: TTemperature; begin temp := TemperatureSensor.GetTemperature; if temp > Threshold then RoboValet.FetchDownJacket(Sender);
end;
Microservices – Event Driven Systems
Event-Driven Architecture
• Events may be received for no use e.g. the following method is always called procedure TZen.OnTempChanged(NewTemp: TTemperature); begin CurrentTemp := NewTemp;
end;
but this value may never be used in procedure TZen.HandleIAmGoingOut(Sender: TPeople); begin if CurrentTemp > Threshold then RoboValet.NotifyNeedsJacket(Sender); end;
Microservices – Event Driven Systems
Event-Driven Architecture
• Danger of unexpected infinite loop procedure TZen.HandleIAmGoingOut(Sender: TPeople); begin if CurrentTemp > Threshold then RoboValet.NotifyNeedsJacket(Sender); end;
What happens
if RoboValet.NotifyNeedsJacket
triggers (indirectly) TZen.HandleIAmGoingOut ?
Since nodes may be added at any time,
how to prevent it to happen?
Microservices – Event Driven Systems
Event-Driven Architecture
• Danger of unexpected infinite loop procedure TZen.HandleIAmGoingOut(Sender: TPeople); begin if CurrentTemp > Threshold then RoboValet.NotifyNeedsJacket(Sender); end;
What happens
if RoboValet.NotifyNeedsJacket
triggers (indirectly) TZen.HandleIAmGoingOut ?
Enter the Event-Driven debugging hell:
actual event process depends on the system it runs on
Microservices – Event Driven Systems
Event-Driven Architecture
• So, when to use it?
– Your are modeling evolving information
• e.g. scale to a lot of devices or users
– You need to track changes of state
• A monolithic DB only stores the current state
(via Update) - otherwise it may sink under Inserts
• Event-Sourcing and CQRS (see later)
Microservices – Event Driven Systems
Event-Driven Architecture
• So, when to use it?
– You expect a modular design
• Process not written in stone during compilation
(as in imperative programming): add/enable nodes
to change the system behavior at runtime
• Microservices, SaaS, horizontal scaling
Microservices – Event Driven Systems
Event-Driven Architecture
• So, when to use it?
– You defined a stateless domain
• From state comes complexity,
so DDD usually modelizes a time snapshot
• DDD events are a way to introduce temporality
– Never 100% Event-Driven
• You will eventually define REST API gateways
Microservices – Event Driven Systems
Event-Driven Architecture
• Message Bus
– Most common way to implement Event-Driven
• Cloud, Local or in-house Queues
• Like VCL TApplication.ProcessMessages
– Publish/Subscribe mechanism
• Nodes get only needed information
• Nodes could be added/removed
– Used in conjunction with regular DB and REST
Microservices – Event Driven Systems
Event-Driven Architecture
• Peer to peer communication
– Nodes communicate directly between them
• Each node manages its own queue(s)
• Good integration with REST
• Need a centralized catalog service, P2P ZeroMQ,
or convention-over-configuration setup
• As offered by mORMot over WebSockets
Microservices – Event Driven Systems
Event-Driven Architecture
• Event Sourcing
– Business logic is implemented through Events
– Events are persisted, not state
• Play the events to compute a state
• Intermediate states may be cached/persisted
– Events are transmitted, not state
• Publish/Subscribe instead of REST
• Define API gateways for the outer world
Microservices – Event Driven Systems
Event-Driven Architecture
• Event Sourcing
– Business logic is implemented through Events
– Convenient way to define Microservices
• Subscribe to events, to get the information
• Publish events, as result of the process
– Microservices persistence
• Most Microservices will use a regular state storage
• Some Microservices will use Event Sourcing
Microservices – Event Driven Systems
Event-Driven Architecture
• Events and CQRS
– Command Query Responsibility Segregation
• Commands triggers events
• Events are gathered and stored
• State/Events store is uncoupled from Commands
– CQRS follows a non CRUD pattern
• Create Read Update Delete
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services type
ICalculator = interface(IInvokable)
['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
/// add two signed 32 bit integers
function Add(n1,n2: integer): integer;
end;
– Defines the contract of the service
– Microservice: should follow SOLID principles
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services procedure ResolveCalculator(out: ICalculator); ... function TMyClient.MyAdd(a,b: integer): integer;
var Calculator: ICalculator;
begin
Client.Services.Resolve(ICalculator, Calculator);
result := Calculator.Add(a,b);
end;
– Runtime injection of the actual implementation
• May be in-process, local or remote
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services var Calculator: ICalculator;
begin
Client.Services.Resolve(ICalculator, Calculator);
result := Calculator.Add(a,b);
end;
– Client is a mORMot TSQLRestClientURI
– Will communicate using JSON over HTTP(S)
Microservices – Event Driven Systems
Event-Driven mORMot
• RESTful Local and Client-Server – In-process
Stand-alone client, fast server-side access
– Named pipes or Windows messages Stand-alone client, fast server-side access
– HTTP/1.1 via kernel-mode http.sys API • Kernel-mode execution, IOCP driven, https – Windows specific
– HTTP/1.1 via OS socket API • Windows or Linux, using a Thread Pool, https via Reverse Proxy
• Upgradable to WebSockets
Microservices – Event Driven Systems
Event-Driven mORMot
• RESTful Client
TSQLRestServer
TSQLRest
TSQLRestClientURI
TSQLRestClient
Microservices – Event Driven Systems
Event-Driven mORMot
• RESTful Server
TSQLRestServerDB
TSQLRestServer
TSQLRest
TSQLRestServerRemoteDB TSQLRestServerFullMemory
Microservices – Event Driven Systems
Event-Driven mORMot
• RESTful Local and Client-Server
– Needed methods are available at TSQLRest
• REST auto-routing
• JSON high-performance serialization
• Following Liskov Substitution Principle
– Focus on the logic
– Abstract from the plumbing
Microservices – Event Driven Systems
Event-Driven mORMot
• REST/JSON Persistence via ORM/ODM
– Agnostic Storage for Events or State
• ORM over local SQlite3 store
• ORM over remote SQL databases
• ODM over TObjectList • ODM over MongoDB
Switching from one to another
in a few lines of code, or through settings
Microservices – Event Driven Systems
Event-Driven mORMot
• REST/JSON Persistence via ORM/ODM
– Advanced ORM features
• REST Convention-Over-Configuration routing
• Direct DB-to-JSON serialization (no TDataSet)
• Built-in statement and JSON cache
• Batch (Unit-Of-Work) writes: bulk SQL/NoSQL insert
• SQL logging/timing, real-time replication,
data sharding, remote proxying
Microservices – Event Driven Systems
Event-Driven mORMot
• Cross-platform
– Main mORMot units
• Delphi: Windows x86/x64
Best IDE experience for coding and debugging
• FPC: Windows, MacOS, Android, Linux, BSD
x86/x64, ARM32/AARCH64 (PPC32/PPC64)
• (Cross)Kylix: Linux x86
Microservices – Event Driven Systems
Event-Driven mORMot
• Cross-platform
– Generated client code using Mustache templates
• Delphi: Windows, MacOS, Android, iOS
• FPC: Windows, MacOS, Android, iOS, Linux, …
• (Cross)Kylix: Linux
• SmartMobileStudio: JavaScript Ajax / HTML5
– Featuring almost all framework abilities
• JSON marshalling, security, instance lifetime
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Specify the callback as an interface parameter
• Native way of coding in object pascal
– Real-time push notifications over WebSockets
• Upgraded from a standard HTTP connection
– Peer To Peer communication
• No need of a centralized message bus / server
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Real-time push notifications over WebSockets
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Specify the callback as an interface parameter ILongWorkCallback = interface(IInvokable)
...
end;
ILongWorkService = interface(IInvokable)
['{09FDFCEF-86E5-4077-80D8-661801A9224A}']
procedure StartWork(const workName: string;
const onFinish: ILongWorkCallback);
function TotalWorkCount: Integer;
end;
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Real-time push notifications over WebSockets
• Once upgraded, communicates using frames
over a bi-directional socket connection
using application-level protocols
• Security, frames gathering, REST emulation
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Real-time push notifications over WebSockets
• Once upgraded, communicates using frames
over a bi-directional socket connection
using application-level protocols
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Security, Frames gathering, REST emulation
• TWebSocketProtocolBinary = SynLZ + AES-256
• Regular blocking methods expecting results
will be emulated like regular REST requests,
with REST security and parameters marshalling
• Non-blocking methods with no result
will be asynchronously sent,
optionally gathered as “jumbo frames”
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Publish/Subscribe Pattern
also known as “Observer”
Publisher
Subscriber 1
Event
Subscriber 2
Event
Subscriber 3
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Beware of “Race Conditions”
• Use critical sections (e.g. TSynLocker)
to protect your shared data on multi-threaded server
• You can gather all non-blocking callback process in a
background thread via TSQLRest.AsynchRedirect
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Beware of “Dead Locks”
• If your callback triggers another method
which shares the same critical section
in another thread, you may block both threads
• You can gather all non-blocking callback process in a
background thread via TSQLRest.AsynchRedirect
Microservices – Event Driven Systems
Event-Driven mORMot
• Interface-based services callbacks
– Sample 31
• Long-Work Push Notification
also known as “Sagas”
• Publish/Subscribe Pattern
also known as “Observer”
Microservices – Event Driven Systems
• Credits
Martin Fowler
https://martinfowler.com
code and samples at https://synopse.info