perfsonar multi-ls

perfSONARMulti-LS

Jason Zurawski

Martin Swany

04/22/23Swany

OverviewIntroduction

Functionality

Dev Status

Future Plans

Demo

04/22/23Swany

IntroductionA single LS instance would have problems scaling to the increased volume of global deployment

Need multiple LS instancesThey need a way to discover one another

The key idea is to relate local LS instances through the exchange of information 'summaries'.

This transformed data is smaller, yet attempts to retain necessary lookup properties.

04/22/23Swany

IntroductionThrough federation it is possible to link many LS instances on the inter-domain level, as well as extending this notion between domains through the use of a dedicated representative (leader).

Queries will attempt to be resolved locally first, before either iterating through multiple layers of LS, or simply returning a pointer to the proper location to query.

04/22/23Swany

IntroductionLocal LS Schematic

04/22/23Swany

IntroductionGlobal LS Schematic

04/22/23Swany

Functionality

XSLT Transformation

New XMLDB organization, procedures

Join Algorithm

Token Passing Algorithm

Leader Election / Leader Mirroring

Query Interface

XML Formats

04/22/23Swany

Functionality - XSLT

XSLT is a part of the The Extensible Stylesheet Language Family (XSL)

XSLT is a language for transforming XML documents into other XML documents.

Idea:Define a set of definitions that match a specific pattern.

Feed source documents that fit this pattern, get back a new document defined by the stylesheet

04/22/23Swany

Functionality - XSLTExample (initial Document):

04/22/23Swany

Functionality - XSLTExample (stylesheet):

04/22/23Swany

Functionality - XSLTExample (final document):

04/22/23Swany

Functionality - XSLT

Use in the LS:Define 'Local' and 'Global' patterns summarize the contents of 'LSStore'.

LocalCompact both the service definition (MP, MA, etc.), and the actual metadata description (interface, endPoint, etc).

GlobalSame idea, but compact multiple services/metadata instances further (reduce domain scope, etc.)

04/22/23Swany

Functionality - XSLTUse in the LS:

All summarized data is stored in the 'LSCache' file.

Periodic event performs the summarization of LSStore information, stores results in LSCache.

Summaries sent by peer LS instances are stored in LSCache

LSCache is subject to the same cleanup rules as LSStore (should summary life be longer?)

04/22/23Swany

Functionality - XSLTOpen Issues

Format/Style of summaries depends on how much information an LS is willing to share

Formats must be similar (to the same generic pattern of the LSStore) or queries are useless.

If an LS drops out of the ring, how long before it's summary should be removed from other's caches?

Should be a function of the size and speed of the ring.

04/22/23Swany

Functionality - Joining

When a new LS comes online, it will try to join up with other LS instances.

For now depends on list of 'known' LS deployments (don't need to be active, but could be)

List will be updated over time

Once other running LS instances are made aware of a new member, the logical ring can grow.

04/22/23Swany

Functionality - Joining

Joining Example:LS1 is started, looks through its peer list:

04/22/23Swany

Functionality - Joining

Joining Example:LS2 Inserts LS1 into it's peer list, and will inform other members (LS3) as well.

04/22/23Swany

Functionality - Joining

Joining Example:LS1 is added to the ring.

04/22/23Swany

Functionality - Joining

Joining Example:The Peer List is kept 'updated' by an artificial mechanism meant to measure how well each LS knows each other:

04/22/23Swany

Functionality - JoiningPeer List:

The 'familar' parameter is incremented/decremented (max=3 min=0) to indicate when the last communication between a source and destination LS.

When starting, all values are set to '0'.

Each update from an LS increments the value

If an LS can't contact a particular LS, the value is decremented.

04/22/23Swany

Functionality - JoiningPeer List:

A Peer List with all 0 values means needing to rejoin the ring again.

Any time a single LS reaches 0 they are taken out of the ring.

04/22/23Swany

Functionality – Token Passing

The Join procedure touches on the nature of token passing.

Basic Idea:Pass around 'your' list of active peers.

Each LS will add 'new' peers to their own list

After waiting some time (token delay) a summary message (from the 'LSCache') will be sent to all active peers.

Token is passed again.

04/22/23Swany

Functionality – Token Passing

LS1 is currently holding the token. It is now time to pass it:

04/22/23Swany

Functionality – Token Passing

Summary information is retrieved from LSCache through an XMLDB query.

A summary Message is created, each Peer gets the message; update the peer list for success/failure

Create a Token Message from the Peer list, pass it to the 'next' LS.

04/22/23Swany

Functionality – Token Passing

Same Idea, new LS:

04/22/23Swany

Functionality – Token Passing

What about new additions to the Ring?When a new LS joins, only one LS instance in the ring knows at first

When the 'new' LS gets the token for the first time it will send its own summary to the group

New summaries are treated just like a Join.

04/22/23Swany

Functionality – Token PassingAdding a new member:

04/22/23Swany

Functionality – Token PassingAdding a new member:

04/22/23Swany

Functionality – Leader ElectionEach Local group must be aware of the 'Leader' that communicates to the outside world:

Sends/Receives Group Summaries from related Domains

Acts as a 'non-authoritative' answer to queries that may need to go outside of the domain

How to 'elect' this leader?

04/22/23Swany

Functionality – Leader ElectionElection:

Take your hostname as an IP, convert to a 'long' value:

127.0.0.1 = 127*256^2 + 0*256^2 + 0*256^1 +1*256^0 = 2130706432

Assign some 'rank' between 0 and 5(?) (this can be used to ensure a certain LS is always the leader)

CAT 1 2130706432 = 12130706432

04/22/23Swany

Functionality – Leader ElectionThese ID values are used when communicating the summaries and tokens around the ring

The 'smallest' ID is the leader of a group.

The next 1 or 2 smallest entries can be selected to be 'Secondary' and 'Tertiary' LS instances in the logical ring

Are given the Global summaries in case the leader fails.

04/22/23Swany

Functionality – Query Interface

A query, be it from a client or a service, needs to respond in a uniform manner:

Iterative:A query that results in information being NOT on the inital LS returns only the 'location' (i.e. LS information) of where the true data lives.

Recursive:A query will iterate through LS instances automatically returning the information, and caching this data as it progresses.

04/22/23Swany

Functionality – Query InterfaceIterative Queries:

Simple Xpath/XQuery statements can be created to search both the LSCache/LSStore and return either the requested data, or the LS that has the requested data.

This requires either expert knowledge of Xpath (a pretty tall assumption) or publishing 'cookbook' queries.

04/22/23Swany

Functionality – Query Interface

Iterative Queries:declare namespace nmwg="http://ggf.org/ns/nmwg/base/2.0/";

declare namespace nmwgt="http://ggf.org/ns/nmwg/topology/2.0/";

for $m in /nmwg:store[@type='LSStore' or @type='LSCache']/nmwg:metadata

for $d in /nmwg:store[@type='LSStore' or @type='LSCache']/nmwg:data

where $m/@id=$d/@metadataIdRef and $d//nmwgt:hostName/text()[fn:matches(.,'.*.abilene.ucaid.edu')]

return $m

This particular query is looking for anything on the Abilene network, it will return either an LS metadata, or the actual MP metadata (if it is local to this LS).

04/22/23Swany

Functionality – Query InterfaceRecursive Queries:

May be able to get away with using the same Xpath expression, but requires logic to send the query to the LS in question when we get a Hit.

Logic to Cache results is needed as well.Do we treat the returned results as 'registration' data, or store it in our LSCache?

04/22/23Swany

Functionality – XML FormatsIntroduced a new message:

Token

Created a new function for an old message:

LSRegistration, used for summaries

Created 2 new storage mediums:

Peer List

Cache

04/22/23Swany

Functionality – Token MSGType = LSToken

Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token.

Body consists of a peer list.

04/22/23Swany

Functionality – Token MSGFormat:

04/22/23Swany

Functionality – Reg MSGType=LSRegisterRequest

Use Parameters (at the msg level) to indicate if this is a 'Local' or 'Global' token, and that it is a summary message.

Body consists of a metadata tag describing the sending LS, and related data that is a summary of that LS's LSStore.

04/22/23Swany

Functionality – Reg MSGFormat:

04/22/23Swany

Functionality – Peers StoreType=LS(Local|Global)Peers

Same Idea as the LSStore/LSStore-control, meant to house peer entries for each level.

04/22/23Swany

Functionality – Cache StoreType=LSCache

Same Idea as the LSStore/LSStore-control, meant to house cache information that we created, or others give us.

04/22/23Swany

Dev Status

Completed:XSLT, XML DB organization, Peer Tracking, Join Alg., Token Passing Alg., Leader Election, Leader Mirroring, eXist changes (xquery+update instead of xupdate, v1.1)

In Progress:Testing, recursive LS queries, finalize XML formats, experiment on timing and efficiency, code cleanup, error handling

ToDo:Junit, using jar repository?, merge recent trunk changes into my branch, What becomes of my branch?

04/22/23Swany

Future PlansComplete testing/experimentation

Start Deployment to various locations

Merge into trunk in preparation for next release

04/22/23Swany

DemoDemo:

Two installations of MLS, we will be simulating 'local' interaction only

Different data will be registered to each LS, it will be summarized, and exchanged via the token passing protocol

This will be verified by examining the eXist DB contents.

Lookup Service

Lookup Service

Service DiscoveryFrom the client

System component inter-communication

Broad Design ConsiderationsUse existing tooling as much as possible

Flexible levels of information exchange

Scalability via hierarchical organization

Tolerant of component failures

Service Registration with an LS

Express a perfSONAR service as a SubjectReuse of existing tools and protocol elements inside the perfSONAR tool namespace

Each service creates a self-identification record consisting of contact point and administrative detailsEncode information to be advertised as Data

For an MA, encode Metadata of stored measurements inside Data

LS Registration with other LSes

Create registration infoset that summarizes local registration data

Including other known LS instances

Periodically register self-information and registration infoset with other LS services in the same scope

A service (including an LS) may register in multiple scopes

Multi-LS

Simple case in whichLS instances register

with one another

Multi-LS

Group nodes by someadministrative scope

(DNS, X.500)

Full mesh registrationhas scalability issues

Circulate control token

Multi-LS

Token rotation protocolalso allows for leader

election of “designatedLS”

Multi-LS

Hierarchyof domains

formed

LS Registration Protocol OverviewRegistration token includes current LS set

This allows the ring to heal in the case of failures

Initial leader can potentially be chosen based on a variety of metrics

Currently, static priority and IP address are used (allows an administrator to designate defaults)

Leader initiates token at tunable target interval, but all nodes notice absence of token to regenerate/re-elect

Propagation of Registration Information

Essentially, summarize registrations and propagateMust be easy to changeWe don’t know all the answers here and leaving as much room as possible for future customization is very desirable

Approach: use XML Stylesheet Language Transformations (XSLT)

This transformation language allows us to change one XML document into another formGeneric, standalone, portable, language-independent, extensible

“Fast” may not be among these

Status of Prototype

Basic LS exists in the upcoming distribution

Implementation by Maciej Glowiak at PSNC

Multi-LS features implemented in early form

Implementation by Jason Zurawski at U. Delaware

Status of Prototype

Summarization stylesheets

Token protocolHandles nodes joining and leaving

“Internal” and “External” summaries maintained in the XML store

Design Issues

UDDIDesigned with certain things in mind, thus fixed fields don’t map well to our purposes

Version 3 has extensibility via arbitrary XMLIf the majority information of the goes into this XML document, what do we gain?

Generic XML databaseFlexibility, but standards are evolving

Design Issues

Query and update mechanismsXPath certainlyXQuery very likelyXUpdate necessary?XSLT

Iterative vs RecursiveIterative queries are more simpleRecursive queries allow opportunities to speed responses by caching data

Design Issues

DiscoveryCurrently, use a “cached-zone” like DNS in that knowing one server will allow us to get a relatively current list

Better solutions are possiblemDNS

Multicast and Anycast