Download - No sql mongodb_publicsector2014_pp
![Page 1: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/1.jpg)
NoSQL: Delivering scale without complexity
Chris Smith Chief Architect Companies House
![Page 2: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/2.jpg)
O!
nce Upon A Time....
![Page 3: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/3.jpg)
1985 - 2008 ICL Mainframe
![Page 4: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/4.jpg)
1997 Companies House First Website
![Page 5: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/5.jpg)
MainframeWebsite
![Page 6: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/6.jpg)
2004Web enabled filing of information
![Page 7: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/7.jpg)
Data ModelWebsite required data storage
![Page 8: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/8.jpg)
Fetch data from Mainframe
Store in website DB
Populate HTML form
Submit data back to DB
When complete, send to Mainframe
User iterate
![Page 9: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/9.jpg)
Directors
Secretaries
Shareholders
Addresses
Shares
![Page 10: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/10.jpg)
First stepsDesign a data model
![Page 11: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/11.jpg)
NORMALISATION
![Page 12: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/12.jpg)
Data ModellingCompanies House data is not normalised
![Page 13: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/13.jpg)
Observation:A normalised model
increased software complexity
![Page 14: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/14.jpg)
What about performance?
![Page 15: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/15.jpg)
Normalised model:Multiple inserts and joins per entity
![Page 16: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/16.jpg)
Core System
HTML form
XML
DB Tables Director Person Address Address
Iterate
XML
DB Tables Director Person Address Address
Website
![Page 17: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/17.jpg)
Observation:Companies with large datasets experienced poor performance
![Page 18: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/18.jpg)
Our first LessonUnderstand your data
![Page 19: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/19.jpg)
ComplexityMultiple websites and API’s
![Page 20: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/20.jpg)
Companies House Direct WebFiling WebCheck XML Gateway
Core Registry Additional Resources
3rd Party Systems
![Page 21: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/21.jpg)
Where did this lead us?
![Page 22: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/22.jpg)
It’s all about dataInformation is everything
![Page 23: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/23.jpg)
It’s about customersWhat do customers need from the data?
How do they use it?
![Page 24: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/24.jpg)
Second LessonUnderstand your use cases
![Page 25: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/25.jpg)
What did we do?
![Page 26: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/26.jpg)
Expose our dataFirst design a use case API
JSON, REST
![Page 27: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/27.jpg)
Data ModellingModel to satisfy the API
![Page 28: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/28.jpg)
Challenge conventionDuplicate data where required
![Page 29: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/29.jpg)
Single website uses APISame use cases across channels
![Page 30: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/30.jpg)
Choose a “database”Relational didn’t work Need an object store
![Page 31: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/31.jpg)
Other criteriaHigh performance and dynamic
queries on rapidly changing data
![Page 32: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/32.jpg)
Why MongoDB?Met all our needs
Agile, highly scalable and resilient Objects map neatly to language types
![Page 33: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/33.jpg)
Design for performancePre-load MongoDB with API data
![Page 34: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/34.jpg)
Reduce complexityData stored exactly as exposed by API
![Page 35: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/35.jpg)
Data mirrorMongoDB mirrors core system data
![Page 36: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/36.jpg)
Core system Oracle triggers
JSON API Delta update system
MongoDB Collections
HTTP
Queue
SynchronisationOracle triggers drive delta-update process
Information change
![Page 37: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/37.jpg)
Handling deviationFunction to re-sync any data on demand
![Page 38: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/38.jpg)
SimplicitySingle website and API
![Page 39: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/39.jpg)
Companies House Direct WebFiling WebCheck XML Gateway
Core Registry Additional Resources
3rd Party Systems
![Page 40: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/40.jpg)
Website
JSON API
Core Registry Additional Resources
3rd Party Systems
![Page 41: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/41.jpg)
MaintainabilitySchemas restrict flexibility
![Page 42: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/42.jpg)
Customer benefitsReliable, open access to data suiting need
Open RESTfull JSON API
![Page 43: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/43.jpg)
Developer benefitsReduced complexity and learning curve,
flexibility
![Page 44: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/44.jpg)
Service benefitsHighly available and horizontally scalable
Highly maintainable
![Page 45: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/45.jpg)
MongoDB infrastructureThree replica set, SSD and spindle shards
![Page 46: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/46.jpg)
System infrastructureMesos/Marathon nodes on
virtualised blade infrastructure NetApp storage
![Page 47: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/47.jpg)
Software technologies
MongoDB ElasticSearch Mesos Marathon AWS Kibana Fluentd
Perl Scala Java nodejs Mojolicious compass swagger
statd git Ansible Jenkins HAProxy nginx Linux
![Page 48: No sql mongodb_publicsector2014_pp](https://reader033.vdocuments.site/reader033/viewer/2022052910/559a9b271a28ab603d8b46c0/html5/thumbnails/48.jpg)
Questions?