Download - perfSONAR Multi-LS
![Page 1: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/1.jpg)
perfSONARMulti-LS
Jason Zurawski
Martin Swany
![Page 2: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/2.jpg)
04/22/23Swany
OverviewIntroduction
Functionality
Dev Status
Future Plans
Demo
![Page 3: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/3.jpg)
04/22/23Swany
IntroductionA single LS instance would have problems scaling to the increased volume of global deployment
Need multiple LS instancesThey need a way to discover one another
The key idea is to relate local LS instances through the exchange of information 'summaries'.
This transformed data is smaller, yet attempts to retain necessary lookup properties.
![Page 4: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/4.jpg)
04/22/23Swany
IntroductionThrough federation it is possible to link many LS instances on the inter-domain level, as well as extending this notion between domains through the use of a dedicated representative (leader).
Queries will attempt to be resolved locally first, before either iterating through multiple layers of LS, or simply returning a pointer to the proper location to query.
![Page 5: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/5.jpg)
04/22/23Swany
IntroductionLocal LS Schematic
![Page 6: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/6.jpg)
04/22/23Swany
IntroductionGlobal LS Schematic
![Page 7: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/7.jpg)
04/22/23Swany
Functionality
XSLT Transformation
New XMLDB organization, procedures
Join Algorithm
Token Passing Algorithm
Leader Election / Leader Mirroring
Query Interface
XML Formats
![Page 8: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/8.jpg)
04/22/23Swany
Functionality - XSLT
XSLT is a part of the The Extensible Stylesheet Language Family (XSL)
XSLT is a language for transforming XML documents into other XML documents.
Idea:Define a set of definitions that match a specific pattern.
Feed source documents that fit this pattern, get back a new document defined by the stylesheet
![Page 9: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/9.jpg)
04/22/23Swany
Functionality - XSLTExample (initial Document):
![Page 10: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/10.jpg)
04/22/23Swany
Functionality - XSLTExample (stylesheet):
![Page 11: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/11.jpg)
04/22/23Swany
Functionality - XSLTExample (final document):
![Page 12: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/12.jpg)
04/22/23Swany
Functionality - XSLT
Use in the LS:Define 'Local' and 'Global' patterns summarize the contents of 'LSStore'.
LocalCompact both the service definition (MP, MA, etc.), and the actual metadata description (interface, endPoint, etc).
GlobalSame idea, but compact multiple services/metadata instances further (reduce domain scope, etc.)
![Page 13: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/13.jpg)
04/22/23Swany
Functionality - XSLTUse in the LS:
All summarized data is stored in the 'LSCache' file.
Periodic event performs the summarization of LSStore information, stores results in LSCache.
Summaries sent by peer LS instances are stored in LSCache
LSCache is subject to the same cleanup rules as LSStore (should summary life be longer?)
![Page 14: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/14.jpg)
04/22/23Swany
Functionality - XSLTOpen Issues
Format/Style of summaries depends on how much information an LS is willing to share
Formats must be similar (to the same generic pattern of the LSStore) or queries are useless.
If an LS drops out of the ring, how long before it's summary should be removed from other's caches?
Should be a function of the size and speed of the ring.
![Page 15: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/15.jpg)
04/22/23Swany
Functionality - Joining
When a new LS comes online, it will try to join up with other LS instances.
For now depends on list of 'known' LS deployments (don't need to be active, but could be)
List will be updated over time
Once other running LS instances are made aware of a new member, the logical ring can grow.
![Page 16: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/16.jpg)
04/22/23Swany
Functionality - Joining
Joining Example:LS1 is started, looks through its peer list:
![Page 17: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/17.jpg)
04/22/23Swany
Functionality - Joining
Joining Example:LS2 Inserts LS1 into it's peer list, and will inform other members (LS3) as well.
![Page 18: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/18.jpg)
04/22/23Swany
Functionality - Joining
Joining Example:LS1 is added to the ring.
![Page 19: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/19.jpg)
04/22/23Swany
Functionality - Joining
Joining Example:The Peer List is kept 'updated' by an artificial mechanism meant to measure how well each LS knows each other:
![Page 20: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/20.jpg)
04/22/23Swany
Functionality - JoiningPeer List:
The 'familar' parameter is incremented/decremented (max=3 min=0) to indicate when the last communication between a source and destination LS.
When starting, all values are set to '0'.
Each update from an LS increments the value
If an LS can't contact a particular LS, the value is decremented.
![Page 21: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/21.jpg)
04/22/23Swany
Functionality - JoiningPeer List:
A Peer List with all 0 values means needing to rejoin the ring again.
Any time a single LS reaches 0 they are taken out of the ring.
![Page 22: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/22.jpg)
04/22/23Swany
Functionality – Token Passing
The Join procedure touches on the nature of token passing.
Basic Idea:Pass around 'your' list of active peers.
Each LS will add 'new' peers to their own list
After waiting some time (token delay) a summary message (from the 'LSCache') will be sent to all active peers.
Token is passed again.
![Page 23: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/23.jpg)
04/22/23Swany
Functionality – Token Passing
LS1 is currently holding the token. It is now time to pass it:
![Page 24: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/24.jpg)
04/22/23Swany
Functionality – Token Passing
Summary information is retrieved from LSCache through an XMLDB query.
A summary Message is created, each Peer gets the message; update the peer list for success/failure
Create a Token Message from the Peer list, pass it to the 'next' LS.
![Page 25: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/25.jpg)
04/22/23Swany
Functionality – Token Passing
Same Idea, new LS:
![Page 26: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/26.jpg)
04/22/23Swany
Functionality – Token Passing
What about new additions to the Ring?When a new LS joins, only one LS instance in the ring knows at first
When the 'new' LS gets the token for the first time it will send its own summary to the group
New summaries are treated just like a Join.
![Page 27: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/27.jpg)
04/22/23Swany
Functionality – Token PassingAdding a new member:
![Page 28: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/28.jpg)
04/22/23Swany
Functionality – Token PassingAdding a new member:
![Page 29: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/29.jpg)
04/22/23Swany
Functionality – Leader ElectionEach Local group must be aware of the 'Leader' that communicates to the outside world:
Sends/Receives Group Summaries from related Domains
Acts as a 'non-authoritative' answer to queries that may need to go outside of the domain
How to 'elect' this leader?
![Page 30: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/30.jpg)
04/22/23Swany
Functionality – Leader ElectionElection:
Take your hostname as an IP, convert to a 'long' value:
127.0.0.1 = 127*256^2 + 0*256^2 + 0*256^1 +1*256^0 = 2130706432
Assign some 'rank' between 0 and 5(?) (this can be used to ensure a certain LS is always the leader)
CAT 1 2130706432 = 12130706432
![Page 31: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/31.jpg)
04/22/23Swany
Functionality – Leader ElectionThese ID values are used when communicating the summaries and tokens around the ring
The 'smallest' ID is the leader of a group.
The next 1 or 2 smallest entries can be selected to be 'Secondary' and 'Tertiary' LS instances in the logical ring
Are given the Global summaries in case the leader fails.
![Page 32: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/32.jpg)
04/22/23Swany
Functionality – Query Interface
A query, be it from a client or a service, needs to respond in a uniform manner:
Iterative:A query that results in information being NOT on the inital LS returns only the 'location' (i.e. LS information) of where the true data lives.
Recursive:A query will iterate through LS instances automatically returning the information, and caching this data as it progresses.
![Page 33: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/33.jpg)
04/22/23Swany
Functionality – Query InterfaceIterative Queries:
Simple Xpath/XQuery statements can be created to search both the LSCache/LSStore and return either the requested data, or the LS that has the requested data.
This requires either expert knowledge of Xpath (a pretty tall assumption) or publishing 'cookbook' queries.
![Page 34: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/34.jpg)
04/22/23Swany
Functionality – Query Interface
Iterative Queries:declare namespace nmwg="http://ggf.org/ns/nmwg/base/2.0/";
declare namespace nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";
for $m in /nmwg:store[@type='LSStore' or @type='LSCache']/nmwg:metadata
for $d in /nmwg:store[@type='LSStore' or @type='LSCache']/nmwg:data
where $m/@id=$d/@metadataIdRef and $d//nmwgt:hostName/text()[fn:matches(.,'.*.abilene.ucaid.edu')]
return $m
This particular query is looking for anything on the Abilene network, it will return either an LS metadata, or the actual MP metadata (if it is local to this LS).
![Page 35: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/35.jpg)
04/22/23Swany
Functionality – Query InterfaceRecursive Queries:
May be able to get away with using the same Xpath expression, but requires logic to send the query to the LS in question when we get a Hit.
Logic to Cache results is needed as well.Do we treat the returned results as 'registration' data, or store it in our LSCache?
![Page 36: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/36.jpg)
04/22/23Swany
Functionality – XML FormatsIntroduced a new message:
Token
Created a new function for an old message:
LSRegistration, used for summaries
Created 2 new storage mediums:
Peer List
Cache
![Page 37: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/37.jpg)
04/22/23Swany
Functionality – Token MSGType = LSToken
Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token.
Body consists of a peer list.
![Page 38: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/38.jpg)
04/22/23Swany
Functionality – Token MSGFormat:
![Page 39: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/39.jpg)
04/22/23Swany
Functionality – Reg MSGType=LSRegisterRequest
Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token, and that it is a summary message.
Body consists of a metadata tag describing the sending LS, and related data that is a summary of that LS's LSStore.
![Page 40: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/40.jpg)
04/22/23Swany
Functionality – Reg MSGFormat:
![Page 41: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/41.jpg)
04/22/23Swany
Functionality – Peers StoreType=LS(Local|Global)Peers
Same Idea as the LSStore/LSStore-control, meant to house peer entries for each level.
![Page 42: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/42.jpg)
04/22/23Swany
Functionality – Cache StoreType=LSCache
Same Idea as the LSStore/LSStore-control, meant to house cache information that we created, or others give us.
![Page 43: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/43.jpg)
04/22/23Swany
Dev Status
Completed:XSLT, XML DB organization, Peer Tracking, Join Alg., Token Passing Alg., Leader Election, Leader Mirroring, eXist changes (xquery+update instead of xupdate, v1.1)
In Progress:Testing, recursive LS queries, finalize XML formats, experiment on timing and efficiency, code cleanup, error handling
ToDo:Junit, using jar repository?, merge recent trunk changes into my branch, What becomes of my branch?
![Page 44: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/44.jpg)
04/22/23Swany
Future PlansComplete testing/experimentation
Start Deployment to various locations
Merge into trunk in preparation for next release
![Page 45: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/45.jpg)
04/22/23Swany
DemoDemo:
Two installations of MLS, we will be simulating 'local' interaction only
Different data will be registered to each LS, it will be summarized, and exchanged via the token passing protocol
This will be verified by examining the eXist DB contents.
![Page 46: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/46.jpg)
Lookup Service
![Page 47: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/47.jpg)
Lookup Service
Service DiscoveryFrom the client
System component inter-communication
Broad Design ConsiderationsUse existing tooling as much as possible
Flexible levels of information exchange
Scalability via hierarchical organization
Tolerant of component failures
![Page 48: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/48.jpg)
Service Registration with an LS
Express a perfSONAR service as a SubjectReuse of existing tools and protocol elements inside the perfSONAR tool namespace
Each service creates a self-identification record consisting of contact point and administrative detailsEncode information to be advertised as Data
For an MA, encode Metadata of stored measurements inside Data
![Page 49: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/49.jpg)
LS Registration with other LSes
Create registration infoset that summarizes local registration data
Including other known LS instances
Periodically register self-information and registration infoset with other LS services in the same scope
A service (including an LS) may register in multiple scopes
![Page 50: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/50.jpg)
Multi-LS
Simple case in whichLS instances register
with one another
![Page 51: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/51.jpg)
Multi-LS
Group nodes by someadministrative scope
(DNS, X.500)
Full mesh registrationhas scalability issues
Circulate control token
![Page 52: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/52.jpg)
Multi-LS
Token rotation protocolalso allows for leader
election of “designatedLS”
![Page 53: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/53.jpg)
Multi-LS
Hierarchyof domains
formed
![Page 54: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/54.jpg)
LS Registration Protocol OverviewRegistration token includes current LS set
This allows the ring to heal in the case of failures
Initial leader can potentially be chosen based on a variety of metrics
Currently, static priority and IP address are used (allows an administrator to designate defaults)
Leader initiates token at tunable target interval, but all nodes notice absence of token to regenerate/re-elect
![Page 55: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/55.jpg)
Propagation of Registration Information
Essentially, summarize registrations and propagateMust be easy to changeWe don’t know all the answers here and leaving as much room as possible for future customization is very desirable
Approach: use XML Stylesheet Language Transformations (XSLT)
This transformation language allows us to change one XML document into another formGeneric, standalone, portable, language-independent, extensible
“Fast” may not be among these
![Page 56: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/56.jpg)
Status of Prototype
Basic LS exists in the upcoming distribution
Implementation by Maciej Glowiak at PSNC
Multi-LS features implemented in early form
Implementation by Jason Zurawski at U. Delaware
![Page 57: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/57.jpg)
Status of Prototype
Summarization stylesheets
Token protocolHandles nodes joining and leaving
“Internal” and “External” summaries maintained in the XML store
![Page 58: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/58.jpg)
Design Issues
UDDIDesigned with certain things in mind, thus fixed fields don’t map well to our purposes
Version 3 has extensibility via arbitrary XMLIf the majority information of the goes into this XML document, what do we gain?
Generic XML databaseFlexibility, but standards are evolving
![Page 59: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/59.jpg)
Design Issues
Query and update mechanismsXPath certainlyXQuery very likelyXUpdate necessary?XSLT
Iterative vs RecursiveIterative queries are more simpleRecursive queries allow opportunities to speed responses by caching data
![Page 60: perfSONAR Multi-LS](https://reader035.vdocuments.site/reader035/viewer/2022062218/56815b17550346895dc8c793/html5/thumbnails/60.jpg)
Design Issues
DiscoveryCurrently, use a “cached-zone” like DNS in that knowing one server will allow us to get a relatively current list
Better solutions are possiblemDNS
Multicast and Anycast