building occasionally connected applications using event sourcing

Download Building occasionally connected applications using event sourcing

Post on 27-Jan-2015

128 views

Category:

Technology

2 download

Embed Size (px)

DESCRIPTION

I've recently got the opportunity to work on a large enterprise-class system that needs to be deployed on multiple occasionally connected oil platforms and boats. Already the system's architecture was based on the Command Query Separation principles, this gave us a completely new challenge. After several months of looking at alternatives, we decided to go the Event Sourcing direction. In this in-depth session, I'd like you to learn the many alternatives we observed, the pros and cons, and the technical details of our final solution in which we use EventStore 3.0 and elements of NCQRS and Lokad.CQRS to synchronize systems over unreliable connections in a very efficient way.

TRANSCRIPT

  • 1. Occassionally Connected ApplicationsWith Event SourcingDennis Doomen 2014 Aviva Solutions 21 november 2014dennis.doomen@avivasolutions.nl

2. About Me Principal Consultant 17 years in IT C++ origins but C# since 2001 Specialties .NET Architecture Scrum/Kanban/XP ALM Speaker Public initiatives C# Coding Guidelines Fluent Assertions Internet www.dennisdoomen.net DZone MVB @ddoomen 2014 Aviva Solutions Dennis Doomen 21 november 2014 3. Contents What we tried to solve Our options What we choose for Design Challenges Scalability Challenges Architectural Challenges Concurrency Challenges Future? 2014 Aviva Solutions 21 november 2014 4. What we tried tosolve 2014 Aviva Solutions 21 november 2014 5. Central FarmSeabornServersHigh-LatencyLANDecentralizedServersattachments)MobileClientsIntermittentSatelliteConnection3G (w/oWLAN(attachments)Only synchronizewhat is relevant!>100XCopy RDBMS 2014 Aviva Solutions 21 november 2014 6. What we hadCommandsWeb ApplicationCommand ServiceProjectionsQueriesQuery Processor Command HandlersDomain ModelRepositoriesDomain ServicesSQL /OracleService AgentsBackofficeSystemsQuery HandlersRepositoriesPatterns & Principles Domain Driven Design CQRS Repository Abstraction TDD/BDD SOLIDFramework & Libraries NHibernate Autofac ASP.NET WebForms / MVC SQLite 2014 Aviva Solutions 21 november 2014 7. Our options1. SQL Server Replication2. Command Synchronization3. Aggregate Serialization4. RavenDB5. Event Sourcing 2014 Aviva Solutions 21 november 2014 8. 1. SQL Server ReplicationAutomatic reconnectsProven technologyAutomatic schema updatesDevelopment can be outsourcedSynchronization over HTTPOnly SQL Server (Express)Filtering is limitedAffects client infrastructureNo real-world examplesUnsure about limitationsRequires SQL expertize.Difficult to diagnose issues duringdevelopment 2014 Aviva Solutions 21 november 2014 9. 2. Command SynchronizationSupports any databaseNo special deployments requirementsPotentially unresolvable out-of-syncissueRequires bypassing business rulesCommand != actual changeLimited filtering possibilitiesCustom serializationSeparate queue of commands 2014 Aviva Solutions 21 november 2014 10. 3. Aggregate SerializationSupports any databaseNo special deployments requirementsDoesnt interfere with business rulesNo out-of-sync issuesRequires specialized code peraggregateNeeds an enforced order ofsynchronization.Payload is not optimizedRequires a lot of querying and changetracking 2014 Aviva Solutions 21 november 2014 11. 4. RavenDBReduces complexitySmart reconnectsSynchronization built-inArchiving built-inBuilt-in cachingVery scalableImproves productivityImproves query performanceLucene basedAcceptance by enterprise clientsNew querying (map/reduce) conceptsLimited track record 2014 Aviva Solutions 21 november 2014 12. Event SourcingWebApplicationCommandServiceExecute query ChangeUserEmailCommandQueryProcessorChangeUserEmailHandlerInvoke methodUserAggregateSubmit changesGet(key, version)DomainUOWUserProjectorQueryHandlerLINQ, HQL, SQLProjectionsUOWRead DBLoad(events)DAOCommitsWrite DBSubmitApplyGet changesDispatcher CommitHandle(UserEmailChangedEvent) 2014 Aviva Solutions 21 november 2014 13. 5. Event SourcingTransparent to clientsWorks with any SQL/No-SQL DBCan improve performanceAdvance conflict resolutionNatural unit of synchronizationCan be very scalableLots of new conceptsBig impact on domain modelSynchronization must be custom-builtLots of unforeseen challenges 2014 Aviva Solutions 21 november 2014 14. What we chose for 2014 Aviva Solutions 21 november 2014 15. The Revised ArchitectureCommandsCommand ServiceDomain ModelEventsDispatcherDomain RepositoryCommand HandlersDomain ServicesEventStoreService AgentsWeb ApplicationBackofficeSystemsQueriesProjectionsQuery ProcessorQuery HandlerProjectorsProjectionRepositoriesQueryStoreEvents 2014 Aviva Solutions 21 november 2014 16. Advantages Optimized projections Projections = cache -> rebuildanytime Reporting node Intrinsic auditing Supports temporal projections 2014 Aviva Solutions 21 november 2014 17. Synchronization OverviewJust changesownership of anaggregate andPrimary Node Laptop / Secondary NodeCheck-out / Check-inforcessynchronizationEach aggregate is owned by anode to prevent functionalconflictsSOAP/WS-*E.g. UserPasswordChanged,DocumentSigned,Contains asubset of themasterChanges ChangesImages, etc629862986297321User InterfaceChanges QueriesAttachments,BlobsQuery DataUsersGroupsDocumentsObjectsFolders630163006299321User InterfaceQueries ChangesAttachments,Images, etcBlobsCan be rebuild from thechanges table at any pointin timeQuery DataUsersGroupsDocumentsObjectsFoldersTitleChanged, RoleRevokedWIFI/LAN/3GWIFI/LANOptimized for queryingSOAP/WS-*HTTPS-REST 2014 Aviva Solutions 21 november 2014 18. Design Challenges 2014 Aviva Solutions 21 november 2014 19. NEventStore vs Lokad.CQRS vs NCQRSApply(EmailChangedEvent)User Aggregatenew User()EventStoreDataMapperCallerChangeEmail()Open Stream(guid)Write Into Stream Per AggregateNEventStore 3CommitsUser StateGetAggregateRoot(key)Domain Unit Of WorkLoad(key)Load(events)OnEventApplied()SubmitChanges 2014 Aviva Solutions 21 november 2014 20. Immediate vs Eventual ConsistencySubmit ChangesEventStoreDataMapperNEventStore 3QueriesProjectorNHibernateDispatched EventProjections CommitsTransactional Boundary Transactional Boundary 2014 Aviva Solutions 21 november 2014 21. Event VersioningEvent V1 Event V2Some EventCombined EventOther EventCombined EventSeparate EventSeparate Event 2014 Aviva Solutions 21 november 2014 22. Other Challenges No direct AR dependencies Granularity of events Domain Events as facts, not triggers 2014 Aviva Solutions 21 november 2014 23. Scalability Issues 2014 Aviva Solutions 21 november 2014 24. Scalability Limitations Commits table grows and gets slower No partitioning of global data Single point of failure Upgrades take too much time 2014 Aviva Solutions 21 november 2014 25. Out-of-place migrationVersion 1.0 Version 3.0Changes629962986297321BlobsQuery DataUsersGroupsDocumentsObjectsFoldersChanges630163006299321BlobsQuery DataUsersGroupsDocumentsObjectsFoldersSchema 1.0Pass 2= down timePass 1= no down timeSchema 3.0SetupUpconvertscommits,blobs, etc 2014 Aviva Solutions 21 november 2014 26. Multi-Tenant Scalingsystems Specific reportingIntegration NodeReportingServerConnects to back-officeNode(owns tenants 1 & 2)Node(owns tenants 3 & 4)Node(owns tenants 5 & 6)Node (failover fortenants 5 & 6)representations ofthe dataNode(owns tenants 9 & 10)Node(owns ref. data &node config)Node (uses tenants 1& 2, owns tenant 11)Decentralized nodeNode(uses tenants 1 & 11)Node(uses tenants 3 & 5)Node(uses tenants 5 & 10)Owns global dataand ownershipconfiguration 2014 Aviva Solutions 21 november 2014 27. Architectural Issues 2014 Aviva Solutions 21 november 2014 28. Limiting Constraints Recovery in web farm Selective rebuilding of projections No asynchronous dispatching Relational database is bottleneck during loadbalancing Network overhead web->database 2014 Aviva Solutions 21 november 2014 29. Load BalancerFirst AttemptWeb Site Web SiteWorkerThreadRDBMSProjectionsWorkerThreadRDBMSCommitsApplication TierDatabase Tier 2014 Aviva Solutions 21 november 2014 30. Load BalancerSecond AttemptWeb Site Web SiteWorkerThreadRDBMSProjectionsRDBMSCommitsWorkerThreadRDBMSProjectionsApplication TierDatabase Tier 2014 Aviva Solutions 21 november 2014 31. Final AttemptLoad BalancerWeb Site Web SiteApplication TierWorkerThreadRavenDBProjectionsRDBMSCommitsWorkerThreadRavenDBProjectionsDatabase Tier 2014 Aviva Solutions 21 november 2014 32. RavenDBAdvantages No migrations needed No more column length problems Faster Less SQL Server load More scalable Dynamic Indexing Keyboard/faceted search w/Lucene Concious decision onasynchronicityDisadvantages New technology, new concepts No LINQ Async indexes Requires rewrite of allprojectors and queries 2014 Aviva Solutions 21 november 2014 33. Front-End ServerWeb Site WCF Services?WorkerThread ? ?RavenDBProjectionsCommitsApplication TierDatabase TierJobScheduler? 2014 Aviva Solutions 21 november 2014 34. Front-End Server/queries/{query}Worker Job SchedulerThreadRavenDBProjectionsWeb SiteCommitsApplication TierDatabase TierWCF Services 2014 Aviva Solutions 21 november 2014 35. OWINStartupMain Repository Component RepositoryWeb SiteModule ModuleQueries QueriesQueryHandlersQueryHost RavenDBQueryRavenSession Checkpoint StoreHTTP-basedQueryProcessor InterfacesQueryHandlers HTTP (network or memory)Projectors ProjectorsRavenDBCommitEventStoreQueryHost RavenQueryHost ClientQueryHost/queries/{query}QueryingControllerThread PoolOwinWebAPI 2NEventStoreKatanaAutofacQuery HostSettingsDurable CommitDispatcher 2014 Aviva Solutions 21 november 2014 36. ConcurrencyIssues 2014 Aviva Solutions 21 november 2014 37. Event MergingPrimary Node Laptop / Secondary NodeStreamId: User-Dedo, Rev: 6RoleRevokedEventStreamId: User-DedoPasswordChangedEventStreamId: User-Dedo, Rev: 5PasswordChangedEventStreamId: User-Dedo, Rev: 4GrantedRoleEventStreamId: User-Dedo, Rev: 3PhoneNumberAddedEventGrantedRoleEventUserCreatedEventStreamId: User-Dedo, Rev: 7PasswordChangedEventStreamId: User-Dedo, Rev: 6GrantedRoleEventStreamId: User-Dedo, Rev: 5RoleRevokedEventStreamId: User-Dedo, Rev: 4PasswordChangedEventStreamId: User-Dedo, Rev: 3PhoneNumberAddedEventGrantedRoleEventUserCreatedEventLast Sync PointTime 2014 Aviva Solutions 21 november 2014 38. Future? 2014 Aviva Solutions 21 november 2014 39. Opportunities In-memory projections Gossip-based detection ATOM feeds 2014 Aviva Solutions 21 november 2014 40. Reading Material NEventStore on GitHub Domain Events by Udi Dahan Event Versioning by Rinat Abdullin Effective Aggregate Design by Vaughn Vernon NHibernate vs Entity Framework in DDD by me DomainDrivenDesign.org RAFT, a distributed consensus protoc

Recommended

View more >