replication mechanism of zemis refscg.unibe.ch/download/softwarecomposition/2018-01... · xml les...
TRANSCRIPT
![Page 2: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/2.jpg)
ZEMIS Ref
”ZEMIS Referenzdatenverwaltung”
Administration application for so called ’reference data’
• About 380 tables with relational dependencies
• Several applications use the data (clients)
• New applications outside scope of project ZEMIS
• Focus on replication mechanism
![Page 3: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/3.jpg)
Replication: Initial State
Push architecture with direct database accessZEMIS Ref replicates changes directly into the client’s db
• Direct database access
• Client - data mapping
• Database connection and schema
• ZEMIS Ref release required to add new clients
• Tight coupling - makes changing the schema complex
• Robust, no synchronisation issues, failures are detected, noheavy workload
![Page 4: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/4.jpg)
Replication: Variant 1
Push architecture with web serviceZEMIS Ref pushes changes via SOAP web service
• Client provides web service
• Client - data mapping
• Dynamic subscriber list
• No direct database access
• Coupling loosend, schema can be changed
![Page 5: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/5.jpg)
Variant 1: Pro and Contra
Push architecture with web serviceWill be used for new ZEMIS application (temporary solution)
• Closest to initial state
• Robust
• No synchronisation issues
• Failure detection
• No heavy workload
• Contradicts pull over pushpolicy
• Clients provide web service
• Coupling still tight
![Page 6: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/6.jpg)
Replication: Variant 2
Direct pull architecture (on the go)
Clients pull each time they need information
• Always up do date
• Permission and authentication instead of mapping
• No direct access to database
• No need to store the ref data
![Page 7: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/7.jpg)
Variant 2: Pro and Contra
Direct pull architecture (on the go)
30’000 users, 2500 calls/minute
• Always up do date
• No client - data mapping
• No direct access to database
• No need to store the refdata ⇒ browser caching
• Depends on ZEMIS Refavailability
• High performancerequirements ⇒ severalservers ⇒ high cost
• Robustness low, heavyworkload
![Page 8: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/8.jpg)
Replication: Variant 3
Pull architecture with caching
ZEMIS ref offers SOAP web service, clients pull and cache the data
• No direct access to database
• Deltas can be pulled
• Loose coupling, clients can be added easily, schema can bechanged
• Clients can have differing data ⇒ conflicts
• Robustness medium high
![Page 9: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/9.jpg)
Variant 3: Pro and Contra
Pull architecture with caching
ZEMIS ref offers SOAP web service, clients pull and cache the data
• Follows pull over push policy
• Web service provided byZEMIS Ref
• Loose coupling
• Robustness medium high
• When shall the clients pull?
• Clients can have differingdata ⇒ conflicts
![Page 10: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/10.jpg)
Variant 3: First Draft
![Page 11: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/11.jpg)
Variant 3: Validation of First XML Draft
Pull architecture with caching
Implemented web service for pull architecture with caching
• Functionality testing with soapUi
• Load testing with Apache JMeter
• XML files too big
• Problem: XML-Firewall
![Page 12: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/12.jpg)
Variant 3: Final XML Version
![Page 13: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/13.jpg)
Variant 3: Conclusion and Further Work
• Define requirements
• Talk to stakeholders
• WSDL and XSD are thebase
• XSD can be used for variant1 (push)
• Human readable ⇒ moreabstraction
• Reduce size of XML
• Do not send informationtwice
![Page 14: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/14.jpg)
Replication: Variant 4
Push notify to pull architecture with web services
ZEMIS ref pushes notifications about updates, clients pull if needed
• Web service provided by clients to push notification
• Web service provided by ZEMIS Ref to pull
• No heavy workload, responses can be scheduled
• Clients need to determine whether to pull or not
• Robustness good, no differing data
• What if one client doesn’t get a notification?
![Page 15: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/15.jpg)
Variant 4: Pro and Contra
Push notify to pull architecture with web services
ZEMIS ref pushes notifications about updates, clients pull if needed
• No direct access to database
• Loose coupling: Easy to addnew clients, schema can bechanged
• No heavy workload
• Robustness good, nodiffering data
• Clients need to knowwhether to pull or not
• Clients and ZEMIS Refprovide web service
![Page 16: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/16.jpg)
Replication: RabbitMQ Proof of concept
Using an open source message broker
ZEMIS Ref sends messages to exchange which forwards it toqueues, clients fetch them, Advanced Message Queuing Protocol
• Each client has its ownqueue
• Queues have binding keys
• Messages have routing keys
• If keys match, message issent to queue
![Page 17: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/17.jpg)
RabbitMQ: Pro and Contra
• Easy to set up and use
• Loose coupling
• No heavy workload
• No web service needed
• Not bound to XML - Bytearrays
• Robust
• Against architecturalconcept ⇒ optimal solution?
![Page 18: Replication Mechanism of ZEMIS Refscg.unibe.ch/download/softwarecomposition/2018-01... · XML les too big Problem: XML-Firewall. Variant 3: Final XML Version. Variant 3: Conclusion](https://reader033.vdocuments.site/reader033/viewer/2022051903/5ff3df7635e00750bf1ede95/html5/thumbnails/18.jpg)
Summary
• ZEMIS Ref administrates reference data shared amongapplications
• Decapsulation of ZEMIS Ref and its clients
• Analysed several approaches, prototypes
• One not optimal approach is already in development
• PoC for messaging service was success