oak , the architecture of the new repository
DESCRIPTION
Oak , the Architecture of the new Repository. Michael Dürig, Adobe Research. Design goals. Scalable Big repositories Clustering Customisable, flexible OSGi friendly. Outline. CRUD Changes Search. Tree model. a. d. b. c. Updating. ?. a. d. b. c. x. MVCC. HEAD. r1: /. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/1.jpg)
APACHE SLING & FRIENDS TECH MEETUPBERLIN, 22-24 SEPTEMBER 2014
Oak, the Architecture of the new RepositoryMichael Dürig, Adobe Research
![Page 2: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/2.jpg)
Design goals
adaptTo() 2014 2
Scalable Big repositories Clustering Customisable, flexible OSGi friendly
![Page 3: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/3.jpg)
Outline
adaptTo() 2014 3
CRUD Changes Search
![Page 4: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/4.jpg)
Tree model
adaptTo() 2014 4
a d
b c
![Page 5: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/5.jpg)
Updating
adaptTo() 2014 5
?
x
a d
b c
![Page 6: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/6.jpg)
MVCC
adaptTo() 2014 6
a d
b c
r2: /dr1: /d
r1: /a/br2: /a/b
r1: / r2: /
r2: /d/x
HEAD
![Page 7: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/7.jpg)
adaptTo() 2014 7
Refresh and Garbage Collection
![Page 8: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/8.jpg)
garbage
Refresh
adaptTo() 2014 8
![Page 9: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/9.jpg)
garbage
Garbage collection
adaptTo() 2014 9
![Page 10: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/10.jpg)
adaptTo() 2014 10
Concurrency and Conflicts
![Page 11: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/11.jpg)
r2a
Concurrent updates
adaptTo() 2014 11
r2br1
![Page 12: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/12.jpg)
Merging
adaptTo() 2014 12
r1 r3merge
r2b
r2a
upates
![Page 13: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/13.jpg)
Conflict handling: serialisation
adaptTo() 2014 13
Fully serialised Fail, no concurrent update
Partially serialised Concurrent conflict free updates
![Page 14: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/14.jpg)
Conflict handling strategies: merging
adaptTo() 2014 14
Partial merge Conflict markers, deferred resolution
Full merge Need to choose victim
![Page 15: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/15.jpg)
adaptTo() 2014 15
Replicas and Sharding
![Page 16: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/16.jpg)
Replica and caches
adaptTo() 2014 16
master copy full replica cache
![Page 17: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/17.jpg)
Sharding strategies
adaptTo() 2014 17
by path by level by hash with caching
![Page 18: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/18.jpg)
adaptTo() 2014 18
Implementations
![Page 19: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/19.jpg)
MicroKernel / NodeStore
adaptTo() 2014 19
Tree / Revision model implementation
Responsible for
Clustering
Sharding
Caching
Conflict handling
Not responsible for
Validation
Access control
Search
Versioning
![Page 20: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/20.jpg)
Current implementations
adaptTo() 2014 20
DocumentMK TarMK (SegmentMK)
Persistence MongoDB, JDBC Local FS
Conflict handling Partial serialisation Full serialisation
Clustering MongoDB clustering Simple failover
Sharding MongoDB sharding N/A
Node Performance Moderate High
Key use cases Large deployments (>1TB), concurrent writes
Small/medium deployments, mostly read
![Page 21: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/21.jpg)
adaptTo() 2014 21
Access Control
![Page 22: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/22.jpg)
Accessible paths
adaptTo() 2014 22
a d
cb
![Page 23: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/23.jpg)
xistentialism
adaptTo() 2014 23
All paths traversable Node may not exist Decorator on NodeStore
E
root.getChildNode("a").exists();
root.getChildNode("a") .getChildNode("b").exists();
⟹ false
⟹ true
![Page 24: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/24.jpg)
adaptTo() 2014 24
Comparing Revisions
![Page 25: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/25.jpg)
Content diff
adaptTo() 2014 25
What changed between trees Cornerstone for
Validation Indexing Observation …
![Page 26: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/26.jpg)
What changed?
adaptTo() 2014 26
∆
![Page 27: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/27.jpg)
Example: merging
adaptTo() 2014 27
r1
r2a
r2b
r3
r1 ➞ r2a“a” modified
“b” removed
∆
r1 ➞ r2b “d” modified
“x” added
∆
![Page 28: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/28.jpg)
adaptTo() 2014 28
Commit Hooks
![Page 29: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/29.jpg)
Commit hooks
adaptTo() 2014 29
Key plugin mechanism Higher level functionality
Validation (node type, access control, …) Trigger (auto create, defaults, …) Updates (index, …)
![Page 30: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/30.jpg)
Editing a commit
adaptTo() 2014 30
∆ ∆ + x
![Page 31: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/31.jpg)
Commit hooks
adaptTo() 2014 31
Based on content diff pass a commit fail a commit edit a commit
Applied in sequence
![Page 32: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/32.jpg)
Type of hooks
adaptTo() 2014 32
CommitHook Editor Validator
Content diff Optional Always Always
Can modify Yes Yes No
Programming model
Simple Callbacks Callbacks
Performance impact
High Medium Low
![Page 33: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/33.jpg)
adaptTo() 2014 33
Observers
![Page 34: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/34.jpg)
Observers
adaptTo() 2014 34
Observe changes After commit Often does a content diff Asynchronous Optionally synchronous
Local cluster node only
![Page 35: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/35.jpg)
Examples
adaptTo() 2014 35
JCR observation External index update Cache invalidation Logging
![Page 36: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/36.jpg)
adaptTo() 2014 36
Search
![Page 37: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/37.jpg)
Query Engine
adaptTo() 2014 37
SELECTWHERE x=y
/a//*
Parser
Parser
Parser
Index
Index
Traverse
Parser
Index
parse execute post process
![Page 38: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/38.jpg)
Index Implementations
adaptTo() 2014 38
Property (ordered) Reference Lucene
In-content or file system Solr
Embedded or external
![Page 39: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/39.jpg)
adaptTo() 2014 39
Big Picture
![Page 40: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/40.jpg)
Big picture
adaptTo() 2014 40
MicroKernel
Oak Core
Oak JCR
Oak API
NodeStore API
JCR API
Plugins
![Page 41: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/41.jpg)
Resources
adaptTo() 2014 41
http://jackrabbit.apache.org/oak/
![Page 42: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/42.jpg)
adaptTo() 2014 42
Appendix
![Page 43: Oak , the Architecture of the new Repository](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813f29550346895da9c7ac/html5/thumbnails/43.jpg)
Resources
adaptTo() 2014 43
http://jackrabbit.apache.org/oak/
http://jackrabbit.apache.org/oak/docs/
https://svn.apache.org/repos/asf/jackrabbit/oak/trunk
/