IBM Confidential

Link Index Service – Where we are

© 2013 IBM Corporation

Vishy RamaswamyPaul Tasillo

IBM Confidential

Agenda

1. Requirements.

2. Design Ideas

3. Preferred Approach

4. Things to validate

5. Adoptions

6. Plan items to track

IBM Confidential

Detailed Design RequirementsDescription Priority

No performance degradation in any of the applications High

Ability to support applications deployed across multiple JTS High

Ability to store configuration aware link information for link across application

High

Approach should be consistent for all applications – 0.6 SDK and Triple store based

High

Low adoption cost for all applications High

Should work for external OSLC applications High

Needs to support data integrity and disaster recovery scenarios High

Applications need to migrate their old back link data High

Information on linked data should not be visible to users that do not have access to the data.

High

IBM Confidential

Centralized Index using TRS and LQE

IBM Confidential

Centralized Index using TRS and LQE - sequence

IBM Confidential

Details on LQE based approach – Preferred Approach1. TRS polling is not a huge bottleneck anymore

1. Application can push notification via JAF API to Link Index, so it can pull instead of poll on time interval

2. Have a separate index for cross repository links to address global configuration scope and scalability

3. This index will only contain link information from the TRS feeds (leverage User-Agent headers to request application to subset link triples only)

4. Fits well with existing or new OSLC partners as they are familiar with TRS protocols

5. We will provide a POJO API on top of this index to avoid apps writing SPARQL and allowing the underlying technology of LQE to evolve and move away from JENA

1. It will be index rebuild using the TRS feed (not migration), could be done online

6. All queries from all applications go to one single end point which is discoverable for getting cross repository links. This discovery and query is done by the POJO API

7. No changes to existing link schemas

8. Adoptions related to providing link data are natural extensions to existing TRS support in applications

9. Adoption for querying links is minimal (calling one query end point – POJO API)

10. Data integrity is built in the TRS model as apps can send new change logs to keep the data in sync

11. Security – show or not showing labels for backlinks

12. Degrade gracefully if index is corrupted or if index server is down

IBM Confidential

Things to validate in LQE based approachM7

1.Have a LQE for links only

1. How to subset link data using User-Agent or other request headers

2. What aspects need to change or be configured in an existing LQE technology

1. Application send notification to LQE, so LQE can pull instead of poll

2. LQE GET resource with special User-Agent value for link data only

3. How to register LQE as the central link index provider, pattern is similar to GC application

2.Foundation to build JAF POJO API for query backlinks

3.Foundation to build JAF API for notification to link index

4.Application should start thinking about how to expose configure aware info via TRS

M8 – CLM Application adoption

1.Need to validate how the GC to LC TRS feed looks like

1. Need to validate how to get all the LCs from GC, the list should be flattened

2.Need to validate how the LC TRS feed looks like for DNG/DM, based on CM OSLC Spec

1. Need to validate how to get all versioned resources from LC

2. Need to validate how to get concept resource from versioned resource

3.Need to validate how RTC exposes GC info to link index

1. RTC workitem/planning does not have LC information, will it have GC association?

2. RTC SCM link?

4.Need to validate end user web UI experience, backlink will be “discovered”

1. What use cases can or can not be supported.

5.Public API (OSLC, Reportable REST) would not return backlinks – will break things

6.Modified date of the resource is not updated (sort by date)

IBM Confidential

Application adoption

• Application provide subset of data - (link data only)

• By checking User-Agent or other request headers

• Conditionally remove the backlink creation code

• By calling foundation api that could determine application compatibility

• For older (n-1) application, we will continue create backlinks

• NOTE - Application will still get its forward link from its local repository same way they do today, so user can see the link immediately (even though that info is also available in LQE)

• CLM applications call local foundation POJO API to query the backlinks

• 3rd Party OSLC application will discover the query end point to query

• Application need to display backlink separate

IBM Confidential

Application adoption/**

* Find all back links for an artifact under the current global configuration that are the particular link type

*

* @param globalConfigUri - the current global configuration uri

* @param linkType - the publicly known OSLC link type predicate

* @param targetUri - the target artifact concept resource uri

* @return – one page of concept resource uris that are pointing to target artifact that are the particular link type under the GC

*/

public List<URI> getAllLinkToMeByLinkType(URI globalConfigUri, String linkType, URI targetUri, int pageSize);

public PagedResult<URI> getAllLinkToMeByLinkType(URI globalConfigUri, String[] linkType, URI[] targetUri, int pageSize);

public PagedResult<URI> getAllLinkToMeByLinkType(URI globalConfigUri, URI[] targetUri, int pageSize); // all link types

public PagedResult<URI> getAllLinkToMeByLinkType(URI globalConfigUri, URI[] targetUri, int pageSize); // all link types

public PagedResult<URI> getAllLinkToMeByLinkType(URI globalConfigUri, URI[] targetUri, int pageSize, serviceProviderUri); // scope to project area

/**

* Find all back links for an artifact without configuration aware that are the particular link type

*

** @param linkType - the publicly known OSLC link type predicate

* @param targetUri - the target artifact concept resource uri

* @return – one page of concept resource uris that are pointing to target artifact that are the particular link type */

public PagedResult<URI> getAllLinkToMeByLinkType(String linkType, URI targetUri, int pageSize);

•To Do: Expose as RESTful/OSLC? Service for 3rd party OSLC application to use

IBM Confidential

Plan items to track

CLM Plan Item:

314431: [CLM] Implement removal of explicit backlinks across CLM Solution

• Install/Setup LQE

• Include LQE as part of CLM installation

• Customize LQE to poll only link data (use User-Agent=linkInfoOnly)

• Register this LQE instance as the centralized link index provider

• LQE notification service

• Foundation:

• Create POJO API to query backlinks

• Create REST service to query backlinks against central index

• All CLM application:

• TRS feed for configuration (local and global)

• Return subset (link only) info when User-Agent = linkInfoOnly

• Conditionally remove backlink creation

• Display backlink as “discovered” links on web UI

IBM Confidential

Back up

IBM Confidential

Central link index across multiple JTS deployments

IBM Confidential

Internal implementation of the POJO API for query backlinks

• Call foundation POJO API to query the backlinks, internal implementation will

1. Discover LQE link index provider and query end point, establish remote connection

2. Construct and send sparql to registered link index (LQE), sparql need to UNION the following results

1. Query backlink for config aware app

1. GC “includesContributor” LC

2. LC “includesVersionedResource” VersionedResource

3. VersionedResource“isVersionOf” ConceptResource

4. ConceptResource “hasLinkType” TargetArtifactResource

2. Query backlink for config participate app who registers a GC provider and a LQE link index provider(e.g. RTC or 3rd party)

1. ConceptResource “hasLinkType” TargetArtifactResource

2. ConceptResource “hasGC” GC

3. ConceptResource DOES NOT have any versionedResource (how to do this in sparql??)

3. Parse the XML/Json result to PagedResult

IBM Confidential

Linking approach

Today most * of the Rational and partner integrations are modeled as a pair between for each link type: validates/ validated by and links are stored on each side of the relationship.

Proposal:

For customer who do NOT opt-in (default for new installs and migration):

• Continue to create back links as we do today for all links

• This will provide n-1 compatibility and integration for 3rd parties- DOORS 9, CQ, etc.

• ETLs will continue to work unchanged (Java, DCC, DM).

• Existing reporting will continue to work. BIRT, RRDI, Insight, RPE.

• Requirements:

• Apps must support a ‘default’ or ‘primordial’ stream that the project areas use. This must NOT be changed. If the stream changed, the DW would need to be reloaded and backlinks would start to break.

IBM Confidential

Opt-in = True

Links here have a direction. For each type, there is a defined side that stores the link and points to the other side.

Question: Are we ok fixing this at development time or does this need to be configurable?

Opting into CM support is a big deal. This is crossing the chasm.

• Customers can’t opt out again. This needs to be consistent across the apps.

• Granularity of opt-in is the project area

• DW based reporting is disabled. We need to move customers down the path of LQE based reporting and not setting false expectation of DW.

• Only support linking to “CM aware” OSLC providers. Specify this with new property in service provider discovery doc. Requirements:

• Create only one way links

• Publish TRS with link data and configuration data (optional- only if you want your links to be seen on other side)

• Some level of GC awareness. This enables rich hover and link navigation. (possibly optional)

• Existing non-CM OSLC links are shown but are read only. No new links to these providers can be made (can be removed).

IBM Confidential

Link migration

This is a per-project area migration (requires new invention from JAF). Migration is triggered by opt-in.

Migration scans all OSLC links in a project area (also some cross project area links for DNG, CM? )

if link’s service provider doesn’t support CM (get on URL to get service provider link, get on service provider URL to get properties):

update local link storage to indicate read only

Else // service provider does support CM

if link type of link owned by me do nothing //assume other side will remove their backlink

else archive back link // we don’t own storage for the link

To ensure we don’t miss any links that can happen when links aren’t consistent, all links will be already loaded into LQE. A query will be run to identify for this project area being migrated which links have missing forward links*. These are re-created.

Link inconsistent cases: B side owns link storage, B is migrating

A B Not a problem

A ---> B Problem case. We own the storage but don’t have a forward link

A - B Not a problem for B migrating

IBM Confidential

Distributed Cache

IBM Confidential

Issues with distributed cache

1. Changes to link schemas for configuration awareness are at the repository level and are mandatory

2. Support for existing OSLC applications requires new OSLC protocols around eventing – push + pull model

3. Too many background jobs and synchronization issues – Overengineered for data integrity and synchronization

4. Adoption costs are high – Triple store based apps need big adoptions. 0.6 Apps have to adopt the changes to IAuditable link schema changes

5. Each application DB will grow and requires cloning for streams and baselines

6. Relies on GC to LC cache being there (requires all LCs) for quick resolution of relevant LCs. Not sure how this will work with RTC

7. If we are not cloning then each app has to make n REST calls to query for links. Cloning requires data to be duplicated on target servers when source side creates baselines and streams

8. Expects bi-directional friend relationships

9. Overall this approach is solving many of the things that a central index like LQE is already addressing