1

StaffPlan Conduit

Interfacing StaffPlan Rostering Solution

Table of Contents

About Conduit ............................................................................................................................... 2

When applicable ..................................................................................................................................................................... 2

StaffPlan Rostering Solutions ............................................................................................................................................ 3

Conduit component deployment .................................................................................................................................... 4

Conduit XML Schema Library .................................................................................................... 6

Schemas for rostering .......................................................................................................................................................... 6

Communication parties ........................................................................................................................................................ 7

Social Care System – Rostering Solution data flow ............................................................................................... 7

How the data is handled ................................................................................................................................................. 11

Transport-level structures ................................................................................................................................................ 12

Message envelopes ................................................................................................................................................................................. 13

Acknowledgements ................................................................................................................................................................................. 14

Message sets .............................................................................................................................................................................................. 14

Appendix: Developer documentation .................................................................................... 15

General concept ................................................................................................................................................................... 15

Transport .................................................................................................................................................................................. 17

Service contract and management software .......................................................................................................... 17

ConduitMessageQueue service contract (interface) ............................................................................................................... 18

Conduit Management Pages .............................................................................................................................................................. 19

Security ..................................................................................................................................................................................... 20

Collaboration ......................................................................................................................................................................... 20

General idea ................................................................................................................................................................................................ 21

Details of message passing ................................................................................................................................................................. 22

Messages ................................................................................................................................................................................. 24

Basic message schema (entity or acknowledgement) ........................................................................................................... 24

Message sets (unit of work) ............................................................................................................................................................... 25

Transactions ............................................................................................................................................................................ 25

2

Versioning and compatibility ......................................................................................................................................... 26

About Conduit

StaffPlan Conduit is a communication solution designed and implemented by StaffPlan to enable

automated (unattended) communication between StaffPlan products and other systems (referred

as external systems). The core components of Conduit are:

XML Schema Library (expressed in XSD) – a formal definition of messages that may be

passed between StaffPlan and External systems. The schemas are independent of a

particular transport method.

Message Relay Service (MRS) – a web service working between systems as a queue of

messages. Both StaffPlan and External systems connect to the MRS directly (SOAP/WSDL)

or via StaffPlan client software (XML). The MRS performs XML validation against Conduit

XML schemas (XSD).

StaffPlan Conduit Client – a service running in the local network where StaffPlan Roster is

installed, responsible for reading from and writing to StaffPlan Roster database. It consists

of:

o Key mapping engine – the XML messages can refer to entities using keys defined

in and managed by external systems. The engine is built into StaffPlan side and

provides primary key translation where required

o Business rule validation module – the incoming XML messages are checked

against a subset of business rules defined for StaffPlan products.

Clients of external systems – these may connect directly to MRS or use a client provided

by StaffPlan (like XML File client) to interact with remote StaffPlan products.

When applicable

Conduit is applicable when continuous, unattended communication between two systems is

required. If data is to be exported from Roster in one-off or periodical, manually triggered

manner (e.g. monthly), then other built-in solutions like Data Exports are simpler to implement.

Similarly, small-scale, manual imports may be easier to implement as bespoke software.

Conduit defines a strict language of communication (expressed as XML schemas). Due to the lack

of standardisation in the Homecare marketplace, the XML must be formatted or interpreted by

dedicated software attached/running within the second system. In some cases a message bus can

be located between Conduit and other systems to translate messages between Conduit XML and

the other system language. Setting up the message bus requires effort and high IT skills. A

message bus is a dedicated server application (like MS BizTalk or nServiceBus) which may require

additional licensing.

3

Implementation of the Conduit solution requires IT consultancy. StaffPlan or third-party

developers need to be involved in order to write software which accepts Conduit XML messages

and produces them from the other system data. The decision on who is responsible for data

translation must be made as part of the contract agreement. The standard approach is to involve

Customer’s IT or a third-party company (specializing in integration) as the provider/implementer

of the translation modules. If the other system already exposes well-documented interfaces, the

work can be done by StaffPlan as a bespoke task. In either case StaffPlan consultancy is required

during the testing and start-up phase to identify and resolve problems stemming from data

format discrepancy or misuse. If the other system already has Conduit interfaces in place, then

very little implementation effort is needed.

Conduit is especially well-suited for long-distance, unattended solutions (e.g. data transfer

between Providers and their Commissioners over secure SSL internet connections). The

components of Conduit assure reliability and asynchrony, which means that the data will be

always delivered even if temporarily one of the sides does not respond (e.g. because of a

maintenance break). It works also in local solutions (between two local systems e.g. Roster and HR

system) – especially if the required communication is asynchronous and unattended.

By default, Conduit does not support synchronous operations (i.e. there are no standard methods

to query about e.g. a Service User code 123123). It relies on subscriptions: whenever data changes

in Roster, Conduit sends a data update out to the other system. The same behaviour is expected

from the other systems, which send data to Conduit. Conduit waits for updates, does not query

for them. If the other system does not support data change detection it may require a lot of

development effort to implement it.

Conduit is recommended for any small unattended solutions where there is a perspective of

further development (additional data sent/received is foreseen).

Customers may find Conduit useful if they already have eGIF-based external interfaces in place.

This is due to the fact that Conduit XML Schemas are based on Government eGIF standard

schemas which allow developers to re-use whole units of interfacing code when

interpreting/producing Conduit messages.

StaffPlan Rostering Solutions

A rostering solution is software that models service provision. It accepts commissioned referrals

and returns actuals of provided services. Optionally, Care worker / staff data may be sent along

with the referrals. The actuals can optionally contain invoicing information (charges) and pay data

(gross payroll).

4

Each rostering solution can be seen as a provider of services, although it may merely mediate

between commissioner and the actual providers. StaffPlan provides 2 Rostering Solutions:

1. StaffPlan Roster – application used by the actual service providers

2. StaffPlan Exchange – portal that acts as a point of contact between providers and their

commissioners (and as a further option the StaffPlan Exchange Online Roster).

Care worker data is transferred directly between the commissioning system and the rostering

solution only in case of the commissioner’s in-house provision.

The commissioner’s systems that communicate with the StaffPlan Rostering Solutions are referred

to as the Social Care System.

Conduit component deployment

Depending on the actual StaffPlan rostering product interfaced, there are two Conduit

deployment models as follows:

1. Interface between Social Care System and in-house StaffPlan Roster

Both StaffPlan Roster and the Social Care System belong to the same organisation and

run in the same local or virtual network.

In this case, all of the Conduit components reside in the same network. The components

required are the:

1.1 Social Care System Client (either a dedicated one or provided by StaffPlan): connects

on port 443 to Message Relay Service.

1.2 Message Relay Service (IIS web server) which listens for incoming HTTPS connections.

When a client submits a message, the message is persisted in the MRS message

queue for later delivery. When a recipient connects to the MRS, the service returns

contents of recipient’s “inbox”.

1.3 StaffPlan Conduit Client which connects on port 443 to the Message Relay Service.

The components may be deployed on separate machines:

5

StaffPlan Roster Conduit Client

Social Care System user StaffPlan Roster userAdministrator

Ro

ster

D

atab

aseMessage Relay Service (IIS)

External System Conduit Client

Soci

al C

are

Syst

em

Internal Conduit connection Internal Conduit connection

The components can also be deployed on one machine:

Social Care System userStaffPlan Roster userAdministrator

Ro

ster

D

atab

ase

Message Relay Service (IIS),External System Conduit Client,StaffPlan Roster Conduit Client

Soci

al C

are

Syst

em

2. Interface between Social Care System and StaffPlan Exchange

StaffPlan Exchange is hosted on StaffPlan’s servers. Multiple providers interact with the

StaffPlan Exchange servers by using StaffPlan Monitoring or Point of Care services or

when provider representatives log into the StaffPlan Exchange web portal.

The Social Care System resides on a separate network in the commissioner’s organisation.

The physical distance between Social Care System and StaffPlan Exchange can be large.

In this case, the deployment of Conduit components is as follows:

2.1. Social Care System client (either a dedicated one or provided by StaffPlan) resides in

the commissioner’s organisation network and connects on port 443 to the MRS.

2.2. The MRS resides on StaffPlan servers along with StaffPlan Exchange and listens for

incoming HTTPS connections. When a client submits a message, the message is

persisted in the MRS message queue for later delivery. When a recipient connects to

the MRS, the service returns the contents of the recipient’s “inbox”.

2.3. StaffPlan Exchange Conduit Client resides on StaffPlan servers along with the MRS. It

connects on port 443 to the MRS.

6

Provider Scheduling System(Roster or third-party)Social Care

userStaffPlan Exchange

user

Administrator

Staf

fPla

n

Exch

ange

Message Relay Service (IIS),StaffPlan Exchange Conduit Client

Social Care System Conduit Client

Soci

al C

are

Syst

em

Commissioner Organisation

Provider Organisation

StaffPlan

Co

nd

uit co

nn

ection

: HTTP

S over In

ternet

Co

nd

uit ad

min

istration

: H

TTPS o

ver Intern

et

StaffPlan Exchange (IIS)

StaffPlan Exchangeuser

Oth

er c

om

min

icat

ion

inte

rfac

es

Exchan

ge Web

bro

wser co

nn

ection

: HTTP

S over In

ternet

Conduit XML Schema Library

The systems interface via XML messages, the format of which is formally defined by XSD schemas.

The actual transport mechanism may not use XML (can be binary). However Conduit clients

always pass data as XML messages. The format of the messages is the same regardless of the

actual transport method (e.g. SOAP, XML files).

Although the schemas may evolve, StaffPlan guarantees support for all older versions. It’s even

possible to talk to the same StaffPlan product using different schema versions. When starting a

new project, we always base the project on the newest schema version. For full set of Conduit

XML schemas please refer to http://conduit.staffplan.co.uk/SchemasByVersion.aspx. The schemas

are documented in XSD annotations. You may refer to the HTML documentation automatically

generated from the annotations (http://conduit.staffplan.co.uk/SchemaHtmlDocs.aspx).

Schemas for rostering

Conduit Schemas cover a wide range of scenarios (e.g. providing monitoring services). Only a

subset of them is applicable in case of transferring data between Social Care Systems

(commissioner) and StaffPlan Roster or StaffPlan Exchange (provider). The messages used with

rostering solutions are listed in further chapters. For a full description of the schemas, please refer

to http://conduit.staffplan.co.uk/SchemasByVersion.aspx and

http://conduit.staffplan.co.uk/SchemaHtmlDocs.aspx).

7

Communication parties

The XML Schemas refer to two parties (i.e. systems):

1. Rostering Provider – the system which accepts referrals and produces actuals (it does not

have to be a scheduling system itself like StaffPlan Exchange).

2. Rostering Purchaser – (commissioner) – the system which sends referrals and requests

the actuals including financial data.

Although it is possible to identify jobs, workers and service users with the Rostering Provider’s

primary keys, the preferable way of managing entity identity is via utilisation of the Rostering

Purchaser’s primary keys. Appropriate mapping is assured by the StaffPlan Conduit Client.

Social Care System – Rostering Solution data flow

The following diagram presents the typical data flow between Provider and Purchaser of the care

services. The dashed lines indicate entities and messages that are used in case of in-house

StaffPlan Roster installations. The solid lines denote entities and messages that are used with

both StaffPlan Roster and StaffPlan Exchange as the Provider.

8

Lookups: booking types, task types, service user types, when required

In-House HR database

Purchaser(Social Care System)

Lookups: employment finish reasons etc...

In-House provider

Provider(StaffPlan Exchange or in-house)

Tim

e

Tim

e

Care workers

Referrals (service user data + careplans)

Actuals (including charges)

Carer allocation

Scheduling

Care provision

Care provision validation, monitoring

StaffPlan Exchange only:Setting up mapping between target provider and purchaser lookups.

StaffPlan Exchange only:Providers committing to deliver the services

StaffPlan Exchange only:Comparison of actuals and commissioned services. Charge computation and confirmation..

The corresponding Conduit messages are:

1. Provider (lookups) – a handshake message containing general information about Service

provider or about how the Purchaser wants the provider system to behave. As shown

above, Provider message can be used to impose Purchaser’s lookups on StaffPlan Roster.

It means that e.g. the Social Care System can use its own visit types when interacting with

9

StaffPlan Roster. The visit types are then mapped to appropriate booking types in

StaffPlan Roster. Descriptions are passed along lookup entry keys. StaffPlan Roster uses

the descriptions to display lookup elements, while the keys are used in communication

between systems. For in-house solution that allows the lookup entries to be created and

deleted in external system only, while Roster automatically reflects the changes. <t:ProviderInformation ProviderID="demoStaffPlan1">

<p:ProviderName>Me</p:ProviderName>

<p:BookingTypes>

<p:BookingType>

<p:BookingTypeKey>[YourBT1]</p:BookingTypeKey>

<p:BookingTypeName>Personal Care</p:BookingTypeName>

<p:Category>ServiceProvision</p:Category>

[...]

</p:BookingType>

[...]

<p:TaskTypes>

<p:TaskType>

<hct:Key>[YourTaskTypePK1]</hct:Key>

<hct:Description>Get up</hct:Description>

</p:TaskType>

[..]

2. Employment (workers) – data of the workers and their employment contract – applicable

for in-house solution, where data of care workers can be transferred to StaffPlan Roster

from the Social Care System. <t:Employment ProviderID="demoStaffPlan1">

<e:ContractKey>[YourCCntrctPK]</e:ContractKey>

<e:Worker>

<hct:PersonKey>[YourWrkrPK]</hct:PersonKey>

<hct:PersonDetails>

<hct:Name>

<pd:PersonGivenName>Adam</pd:PersonGivenName>

<pd:PersonGivenName>Stephen</pd:PersonGivenName>

<pd:PersonFamilyName>Milton</pd:PersonFamilyName>

</hct:Name>

<hct:BirthDate>

<pd:PersonBirthDate>1979-11-30</pd:PersonBirthDate>

<pd:VerificationLevel>Level 0</pd:VerificationLevel>

</hct:BirthDate>

<hct:Gender Code="1">Male</hct:Gender>

<hct:Addresses>

<hct:Address AddressType="Home">

<hct:Street>28D Bigbear Estate</hct:Street>

<hct:Street>South Rd</hct:Street>

[...]

</e:Worker>

<e:TeamKey>North</e:TeamKey>

<e:ContractDate>2010-06-01</e:ContractDate>

[...]

3. Referral (service users and careplans) – data of the service users and their contracts. <t:Referral ProviderID="demoStaffPlan1">

<r:Purpose>CreateUpdate</r:Purpose>

<r:ReferralKey>[YourSvcContractPK]</r:ReferralKey>

<r:TeamKey>North</r:TeamKey>

<r:ServiceUser>

<hct:PersonKey>[YourSUPrimaryKey]</hct:PersonKey>

<hct:PersonDetails>

<hct:Name>

<pd:PersonGivenName>Adam</pd:PersonGivenName>

<pd:PersonGivenName>Steven</pd:PersonGivenName>

[...]

<hct:ServiceUserType>Elderly</hct:ServiceUserType>

<hct:WarningText>Sample Warning Text</hct:WarningText>

<hct:Comments>Comments on Mr Clark</hct:Comments>

</hct:ServiceUserDetails>

[...]

<r:ReferralDate>2010-06-01</r:ReferralDate>

10

[...]

The Referral message consists of many sections that can be updated independently (any

sections that are not to be updated can simply be omitted). The main sections of the

Referral are shown in the full list, available at:

http://conduit.staffplan.co.uk/SchemaHtmlDocs.aspx

3.1. Service User demographics including service user type, banding, contacts (other

person demographics). Also comments and warnings if required.

3.2. Service User team allocation – for in-house providers – service users can be grouped

into teams.

3.3. Contributions – describes funding scheme, i.e. lists care funders along with funding

type:

periodic with hourly cap,

hourly with periodic cap,

percentage with hourly or periodic cap,

commission with hourly or periodic cap,

balance (pays the rest).

3.4. Referral suspension or termination (including reasons).

3.5. Careplan:

3.5.1. General constraints (Min/Max Weekly Hours/Visits)

3.5.2. Optionally: set of regular jobs. A job is described by:

Key (to which Actuals try to refer),

Start time and duration,

Number of carers or – for in-house teams – keys of particular care

workers.

Tasks

Job comments

Recurrence – when the recurrence starts and finishes plus type of

recurrence:

o OneOff (single occurrence on a specific date)

o DailyRepeat (every day)

o WeeklyRepeat (every 1,2,3… weeks on particular days of week)

The careplan jobs can be expressed with fuzzy start times (used with StaffPlan

Exchange).

4. Actuals – actuals referring to occurrences of particular care plan jobs or to jobs created in

the provider system in response to the commissioner’s specification that can have a form

of fuzzy jobs (with fixed duration but fuzzy start time). The actuals may (in case of

StaffPlan Exchange they always do) contain information about charges. The charge is

computed using StaffPlan Roster tariffs (in-house solution) or is a result of financial rules

implemented in StaffPlan Exchange (comparison and calculation based on the

commissioned and actual values). Actuals may refer to a particular invoice generated by

StaffPlan Exchange or StaffPlan Roster.

11

<t:Actuals ProviderID="demoStaffPlan1">

<a:ActualJob>

<a:CareplanJobKey>[YourJobPK_c1]</a: CareplanJobKey>

<a:ActualJobKey>17500</a:ActualJobKey>

<a:BookingTypeKey>Personal Care</a:BookingTypeKey>

<a:StartDateTime>2010-06-02T13:00:00</a:StartDateTime>

<a:FinishDateTime>2010-06-02T15:00:00</a:FinishDateTime>

<a:Tasks>

<a:TaskTypeStatuses />

</a:Tasks>

<a:Staff System="Provider">

<hct:StaffKey>40331</hct:StaffKey>

</a:Staff>

<a:Validated>true</a:Validated>

<a:ReferralPurchaserKey>[YourSvcContractPK]</a:ReferralPurchaserKey>

<a:ReferralProviderKey>189</a:ReferralProviderKey>

<a:JobComments>No comments...</a:JobComments>

<a:Charge>42.80</a:Charge>

</a:ActualJob>

</t:Actuals>

[...]

The final invoice actuals can be also transferred by StaffPlan Exchange back to the Service

Provider’s scheduling system, which generated the original care plan. This requires a

Conduit Client running at the Service Provider end (and using either StaffPlan Roster or a

third-party scheduling system).

How the data is handled

The messages sent between systems enclose data not operations. A message does not say

explicitly “Create a service user” or “Add service user address”. The schema of all messages is

designed to represent either full data of an entity or a part of its description. A message can refer

to another message using a primary key (e.g. “<StaffKey>40331</StaffKey>”).

Conduit clients stick to the following set of rules when processing an incoming message or

producing one to be sent out:

1. Any data passed is saved in the destination system. That may mean creation of an entity

(if it does not exist yet) or the update of an existing one.

2. If the structure passed is incomplete (i.e. optional elements are omitted), then the valid

action is specified in annotation of the omitted XML element (in XSD schema). That may

be:

a. Resetting an existing value or inserting a blank value – for “atomic structures”.

E.g. if you omit an address line, the address line will be cleared in the target

database.

b. Or leaving the existing value unmodified – for “composite structures”. E.g. if you

omit the person details section of a service user, the person details will remain

unchanged in the target database.

3. The above means, that XSD validation allows for incomplete XML structures of entities

even if the entities do not yet exist in the destination system. However e.g. creation of a

service user without personal data is not valid. In such cases, the destination system

12

rejects the message (using a negative acknowledgement). The XSD documentation

explicitly states “Required when creating a new…” constraint where applicable.

4. No entities are explicitly deleted except Jobs (which are deleted via JobCancellation

element). Use the following guidance for delete operations:

a. Lookups (booking types, task types, etc. = elements of Provider message):

Specify ValidityEndDate to logically delete a lookup entry.

b. Workers – specify EmploymentFinishDate to logically delete a worker.

c. Service Users – specify the ReferralFinishDate to logically delete a service user.

d. Jobs – pass job’s primary key with the JobCancellation element.

Note that all optional elements can be omitted in a message triggering delete operation.

5. Some messages consist of two elements: Key and Details (can also have multiple details

sections). They are specially designed to simplify 2 scenarios:

a. When the parent element needs to specify full data of the child element. Example:

Usually you want to pass full personal data of a service user when you create a

new service user contract.

b. When the parent element merely needs a reference to an existing child element.

Example: if a service user contract bases on self-funding, you do not need to

repeat full service user personal data in his/her funder list – you specify only the

service user person key.

6. If a message element refers to another entity via entity primary key (e.g. a Funder

element refers to a funder person), but the referred entity does not exist in the

destination system (it has not been imported yet), then two scenarios are possible:

a. The destination system reports an error (via negative acknowledgement) and

rejects the message.

b. Or a stub of the referred entity is created using default values for all of its

properties and a link is established between the referring entity and the entity

stub.

The entity stubs are created as temporary objects, which will be updated by further

transmissions. Note, that a stub can be created only if it is possible to set defaults to all

its properties. StaffPlan Conduit Client uses stubs to handle lookup entries. E.g. if you

transfer a job with reference to booking type

<BookingTypeKey>XYZ123</BookingTypeKey> and booking type XYZ123 does not yet

exists, a stub will be created as a default booking type having both key and description

set to XYZ123. A user-friendly description may be set via Provider message later. This

won’t break the link between the job and XYZ123 booking type. A stub is simply

promoted to a fully described booking type entity.

Transport-level structures

13

Message envelopes

Conduit messages are built in a standardized way. The core business XML structure (like

Employment, Referral or Actuals) is embedded within an envelope and decorated with a header.

This construction follows SOAP pattern. It is not based on SOAP protocol though, instead being

much simpler and explicitly described within the Conduit Transport Schemas (part of Conduit

Schema Library). The envelope is required by MRS and its clients. No business information is

carried within the header and no transport-specific information is referred from inside of the inner

business message.

<t:Message xmlns:t="http://www.staffplan.co.uk/Conduit/Transport">

<t:Header>

[... the transport metadata goes here ...]

</t:Header>

<t:Body>

[... the business message goes here ...]

</t:Body>

</t:Message>

Note that the Conduit envelope and header can be seamlessly converted to the corresponding

SOAP elements, if needed for other transport environments.

The minimum information that needs to be included in a message header is shown below:

<t:Header>

<t:MessageContentID>Jobs/2010/06/12</t:MessageContentID>

<t:ReceiverAddress>monitoring</t:ReceiverAddress>

<t:CreationTime>2010-06-03T09:31:48.921875+01:00</t:CreationTime>

<t:MessageCredentials>

<ct:Username>demo</ct:Username>

<ct:Password>demo</ct:Password>

</t:MessageCredentials>

<t:IsRetransmission>false</t:IsRetransmission>

</t:Header>

MessageContentID uniquely identifies the body. Two messages having the same

MessageContentID are deemed to be duplicates – only the first one is accepted by MRS. The

other ones are silently ignored.

ReceiverAddress uniquely identifies the destination system. It’s assigned by MRS administrators.

MessageCredentials specify how the destination Conduit client should log into the destination

system / database. It’s possible to produce messages that are executed (and audited against)

various accounts. Confidentiality is assured by transport-level encryption (SSL).

There is a strict association between messages and transactions in destination system. Each

message must be processed within an ACID transaction. Note that some messages can carry more

14

than one entity (e.g. OneOffJobs). All entities are created/updated within the same ACID

transaction, unless split into separate Conduit messages. An error reported for one of the jobs,

causes rollback of all updates.

Acknowledgements

Every successful operation may be acknowledged by an Acknowledgement message (defined in

Transport set of Conduit Schema Library).

Every failure (message rejection) must be acknowledged by a negative acknowledgement.

<t:Acknowledgement>

<t:AcknowledgedMessageContentID>Jobs/2010/06/12</t:AcknowledgedMessageContentID>

<t:Status>Success</t:Status>

<t:IssueClass>Other</t:IssueClass>

<t:SuccessErrorWarningMessage>Message processed successfully.</t:SuccessErrorWarningMessage>

<t:UserHint />

<t:AdminHint />

</t:Acknowledgement>

The header of an acknowledgement message must contain the ReferredMessageInstanceID

element set to the value of MessageInstanceUUID read from the header of the acknowledged

message. Value of MessageInstanceUUID is automatically set by MRS for all incoming

messages and is guaranteed to be globally unique.

<t:Header>

[...]

<t:ReferredMessageInstanceID>a5eda8fb-0896-[…]-cb5922a0ea1f</t:ReferredMessageInstanceID>

[...]

</t:Header>

Note: negative acknowledgements can be seamlessly transformed to SOAP faults, if needed.

Message sets

To ensure efficient and transactional transport, the messages in their envelopes are packed into

message sets. A message set is a group of messages delivered together in an atomic and durable

manner. Receipts are used to ensure that no message set is lost. Message sets contain also

information about the sender including authentication details used with MRS. You don’t need to

understand the message set concept, unless you plan to develop your custom MRS SOAP Web

Service client.

<t:MessageSet

xmlns:t="http://www.staffplan.co.uk/Conduit/Transport"

SchemaVersion="2.0">

<t:Sender>

<t:Address>demoStaffPlan1</t:Address>

<t:Password>xyz</t:Password>

</t:Sender>

<t:SendTime>2010-06-03T17:00:36.9375+01:00</t:SendTime>

15

<t:Messages>

<t:Message>[...]</t:Message>

<t:Message>[...]</t:Message>

[...]

</t:Messages>

</t:MessageSet>

Appendix: Developer documentation

The original Conduit development project description is attached below. You may find the

information helpful, if you plan to implement your own Conduit client.

General concept

The Conduit system consists of 3 main components:

1. Roster Conduit Client (delivered as a module to Roster application server)

2. Other Client (implemented by Other System provider as a standard, plain W3C Web

Service client).

3. Conduit Message Relay Service – a plain W3C Web Service residing on a separate web

server, responsible for queuing and passing messages between the two other

components.

The Conduit Message Relay Service will be further referred shortly as a Conduit Service or The

Service.

The simplest configuration is shown below:

Conduit Web Service is a central message relay service, exposing a single interface which contains

the following operations:

Other System

Conduit Message Relay

Service

Roster Business

Logic Layer

SOAP Service

16

1. Sending messages to the receiver (similar to SMTP service)

2. Requesting incoming messages (similar to POP3 service).

The system which calls either operation can be: Home Care Roster or Other System. Both Roster

and Other System act as clients and are only able to call service operations. They do not expose

any services themselves.

The Conduit service is passive i.e. it never initiates any communication itself. Both Roster and

Other System are active i.e. they are responsible for calling Conduit service in order to deliver

messages or receive messages waiting for them.

Conduit Web Service provides persistent queues for messages passed between Roster and Other

System parties. Messages delivered to the Conduit Web Service remain there as long as the

second party requests contents of its incoming message queue.

Given instances of both Roster and Other systems can act as multiple Conduit Web Service

clients:

The Conduit Web Service is able to serve many locations and many instances of cooperating

systems:

Other System

Conduit Message Relay

Service

Roster Business

Logic Layer

SOAP Service

17

In either case the Conduit Service must know the identity of each Conduit Client (A..F in the

picture above). From Conduit point of view there is no difference between Roster (blue) and

Other System (red) clients. Both are simply parties interested in message passing and are further

referred simply as clients.

Transport

Parties communicate with the Conduit Service using standard W3C Web Service calls. They pass

SOAP messages over HTTPS. HTTPS (HTTP over SSL) provides security of the conversation.

Conduit Server may reside in the Open Internet.

A single call to a service operation is performed within one TCP session. This implies that if there

is a lot of information queued at either side of the communication channel, then the transmission

may last a relatively long time. In the event of network reliability problems a longer connection

implies higher probability of unexpected TCP disconnection. In such case no information is

transmitted and the whole operation must restart. A sequence of disconnections may lead to

starvation of the system as no data can be effectively transferred.

To avoid system starvation, service contracts must include limitation of message size. In case of

large amounts of data queued for transmission the data must be divided into chunks.

Service contract and management software

The main contract ConduitMessageQueue (http://conduit.staffplan.co.uk/ServicesByVersion.aspx)

is used by clients to post and receive messages.

Local

Authority 1

Social Care

System

Conduit Message Relay

Service

Provider 1

Roster

SOAP Service

A E

B

Provider 2

Roster C

D

Local

Authority 2

Social Care

System

F

18

The additional management software (http://conduit.staffplan.co.uk/) is used by administrators

for purposes such as checking current communication queue state, reading logs, acquiring copies

of sent messages or initiating retransmission.

ConduitMessageQueue service contract (interface)

The contract consists of three messaging operations:

1. Enqueue(MessageSetStructure) no return value.

2. BeginDequeue(ConduitCredentialsStructure) returns MessageSetStructure.

3. CommitDequeue(ReceiptStructure) no return value

ReceiptStructure = ConduitCredentialsStructure+MessageSetStructure.UUID.

To minimise the number of remote calls the two latter operations may be also performed in a

single SOAP request (operation CommitBeginDequeue).

The Enqueue operation takes a list of messages as parameters and queues the message for

delivery (FIFO). Recipient of each message is specified in the message body. It is the caller’s

responsibility to limit (split into chunks) posted messages (see 0 - starvation problem).

The BeginDequeue takes the caller credentials as parameter and returns a sequence of messages

waiting for the caller. The returned sequence object should also contain information indicating

whether there is more data waiting for the caller (MoreMessageSetsAwaiting, see 0 - starvation

problem).

The CommitDequeue operation takes the following parameters: caller credentials and UUID (aka

GUID) of message set which has been recently read from BeginDequeue and successfully

persisted for processing. A Client calls this method to confirm successful persisting of the data. If

the receipt is missing, then BeginDequeue will return again the same set of messages, assuming

the previous transmission was successful but the data sent could not be persisted for some

reason. This method need not be called if WS-Atomic-Transaction-enabled client calls the

BeginDequeue operation. However our tests of the WS-AT solution reveal many difficulties in

management and also problems with robustness of that approach in Windows environments.

Example (simplified - refer to 0 (Transactions) for detailed description of a reliable algorithm):

1. Receiver request data – calls the service BeginDequeue.

2. The service call ends successfully – the data is in Receiver’s RAM.

3. Before data persisting, an unexpected receiver’s system shutdown occurs. The receiver

loses the data. If there was no Receipt mechanism the service would be in doubt if the

data persisting operation was successful or not.

4. After restarting the receiver again calls BeginDequeue. Because the last transmission UUID

was not sent within any CommitDequeue, the service assumes the data was lost and

sends the same messages again.

19

5. This time data is persisted successfully. The receiver sends the Receipt calling

CommitDequeue.

6. Basing on the receipt received the server marks the data as persisted successfully – the

data will not be resent.

WS-Atomic-Transaction is a W3C standard which extends default Web Services protocol adding

the functionality described above. However our tests of the WS-AT solution reveal many

difficulties in management and also problems with robustness of that approach in Windows

environments.

ConduitMessageQueue service exposes also an infrastructure operation:

AddListener(MessageQueueListenerStructure). The purpose of this operation is to adjust the

service contract. The client may pass one of 3 communication Models:

Pull

The default, POP3-like approach described in this document. CMRS and the client are

most of time disconnected.

Push

CMRS calls the client directly to push MessageSets rather than keep them waiting in

queue until they are requested by BeginDequeue. The client must provide their URL of

service conforming to ConduitMessageConsumer contract. The approach is not covered in

this document.

BlockedPoll

The general scheme is identical to the described in this document. The only difference

from Pull model is that the BeginDequeue (or preferably CommitBeginDequeue) HTTPS

request does not return immediately when there are no messages in client’s input queue.

It waits instead until a long (e.g. 1min) timeout occurs or a new message arrives. This

approach gives closer synchronisation (less message delivery delays) retaining possibility

of using CMRS located beyond private network (beyond Network Address Translation and

firewall).

If a Conduit client does not call AddListener at all, it is assumed that it works according to Pull

communication scheme.

Conduit Management Pages

The purpose of this web application is to provide an administration tool.

Two sample use cases of the service could be:

20

1. A client system receives negative acknowledge “Invalid message format” after sending a

message. The administrator inspects message which still exists in the Conduit queue. They

may modify and insert it into the queue (retransmit).

2. Target system refused processing of 1000 messages reporting infringement of some

business rule. Administrator inspects some of the rejected messages which are still in the

queue. Finally she discovers that the problem is caused by improper change in

configuration of recipient system. She fixes configuration of the recipient system and

retransmits all 1000 messages without intervention in sender system.

3.

Security

In order to assure reach of the service (every potential Other System provider should be able to

easily integrate with it) the protocol used should be as basic as possible. Currently that is fulfilled

by plain W3C Web Service which does not support authentication or authorization.

The Conduit Service will authenticate users to avoid attacks. Being unable to actively delegate

authentication process to any of its clients, the Conduit Service must store trusted user credentials

in its local database. Conduit Management Pages may be used for management of such data.

The credentials (client id/username and shared secret (password)) are included in each Conduit

message as part of that message schema (ConduitCredentialsStructure).

The credentials described above are used only to authenticate the client within Conduit Web

Service. The password is not passed to the receiver. To authenticate and authorize sender in

receiver system the second credentials (MessageCredentials) are used.

Those credentials are stored in a message queue together with message body and delivered to

recipient client on incoming message request. It is the responsibility of recipient client to

authenticate and authorize sender based on credentials passed through Conduit Service. Lack of

authorization must result in standard negative acknowledge (administrator may retransmit

messages in case of unexpected account expiration at target system).

Transport (HTTPS) is responsible for credential security.

Collaboration

Every communication is initiated by a client. Whether the client is going to send Referral data or

Actuals data or receive either, the procedure of posting and receiving is always the same and

21

message-independent. That enables the system to process messages whose schema and meaning

is currently unknown.

General idea

The general procedure is presented below. This is a description for a single message and its

acknowledgement transfer between sender and receiver.

1. The sender periodically sends messages built from its internal data changes since last

posting (e.g. changes in Referrals or new Actuals). The sender posts a message using

Enqueue operation. A successful service call means that the message conforms to the

XSD schema and that the message has been queued for delivery. Failure is signalized

using standard SOAP Fault response (Exception).

2. Service sends the message as a return value from call to BeginDequeue. The message

posted remains in the queue until the recipient additionally calls CommitDequeue

operation. That marks the message in its queue as delivered (i.e. persisted, not

acknowledged).

22

3. The recipient is responsible for producing an acknowledgement for each message

received (except acknowledgements). He posts acknowledgements in the same way as

other messages (Enqueue).

Acknowledgements may be positive (successful processing confirmation) or negative

(error, message rejected).

4. Having received an acknowledgement for the message, the Service marks the

corresponding message as acknowledged and queues the acknowledgement for delivery

to the sender of the acknowledged message.

5. The sender of the original message periodically checks for new incoming messages calling

a service operation BeginDequeue. The sender may use received acknowledgement to

update status of corresponding data in its internal database (e.g. Referal or Actuals).

6. Successful persisting of received data or acknowledgement is confirmed by receiver

through call to CommitDequeue operation.

Details of message passing

The following procedures describe the required behaviour of clients and the relay service.

Message posting

Client:

1. The Client is responsible for posting data for changed local entities and

acknowledgements for entities modified according to previously received messages.

Its responsibilities concerning local entity change detection include:

a. Versioning (timestamps).

b. Managing entity identities (identities refer usually to database keys of updated

entities)

Responsibilities concerning acknowledgements include persisting acknowledgements of

successful updates or errors. No acknowledgement can be lost.

2. The Client builds a message set, attaches the credentials, sets up other header fields and

passes the MessageSetStructure to the Enqueue method. The size of structure should be

limited (see 0. Transport). The service rejects messages which size exceeds a server

parameter value (maximum size).

Conduit Service:

3. The Conduit Service validates credentials received within ConduitCredentialsStructure

(authentication) and validates other message header fields.

23

4. Returns immediately with a SOAP fault if credentials are invalid or if the message does

not adhere to the XSD Schema (it detects also version compatibility at this step, see 0 -

Versioning and compatibility).

5. If any other of the fields is invalid, then it accepts the message but produces a negative

acknowledge (inserts it into the queue – does not return it immediately from the current

service call).

6. Ignores duplicates (based on MessageContentID) – without an acknowledgement – only

logs appropriate information.

7. Inserts proper messages into receiver’s queue.

8. Acknowledgement messages are linked with appropriate messages in the Service database

for administrating purposes.

Client:

9. On success: records timestamp in local data store so that the future transmissions do not

include sent items.

10. On failure: administrator intervenes (authentication or network error).

Message receiving

Client:

11. The client is responsible for requesting for data waiting in her input queues. The data can

be of two kinds:

a. Entity update messages (e.g. update of Referral or Actuals)

b. Acknowledgement of a message previously sent out.

12. The client passes the ConduitCredentialsStructure to the BeginDequeue method.

Conduit Service:

13. The Conduit Service validates credentials (authentication).

14. Returns immediately with a SOAP fault if credentials are invalid or if the message does

not adhere to the XSD Schema (it detects also version compatibility at this step, see 0 -

Versioning and compatibility).

15. Reads client’s receiver queue. In order to avoid large messages, the Service divides it into

parts and constructs a single Message Set from the oldest part of the queue.

16. Sends the Message Set as a return value from BeginDequeue operation call.

It is possible that more than one acknowledgement comes for a given message

(administrator-initiated retransmission or Status=Waiting acknowledgements).

Client:

24

17. On success (= data available or an empty message set received): reads the Message Set

and persists it for further processing.

18. On failure (= no data available, SOAP Fault): administrator intervenes (authentication or

network error).

19. After successful persisting of the Message Set the client invokes the CommitDequeue

operation passing UUID of persisted message set within a ReceiptStructure (the receiver’s

ConduitCredentialsStructure is also passed as part of this message: Issuer element).

Duplicate calls with the same UUID are allowed (duplicates have no effect).

Conduit Service:

20. The Service marks the appropriate message set as received and persisted by client. If the

service does not receive at least one CommitDequeue call before the next call to

BeginDequeue it will transfer the same data again assuming the client was unable to

persist them (avoidance of data loss).

Messages

Messages are developed according to eGIF standards. If eGIF defines appropriate structures, then

the eGIF schema types are used (preserving eGIF XML namespace).

Basic message schema (entity or acknowledgement)

Conduit messages consist of:

1. Header:

a. MessageInstanceUUID. There are no two messages with the same id even if they

represent a command duplicated intentionally.

b. MessageContentID – pair of entity id and entity version – a string which allows

detecting message duplicates.

c. Who sends (Sender as ConduitCredentialsStructure)

d. To whom (ReceiverAddress)

e. When

f. MessageCredentials: username and password at the receiver's system.

g. Referred message UUID (in case of acknowledges or for future use).

h. ... (Extensibility)...

2. Body:

a. XML data according to XML Schema published at

http://conduit.staffplan.co.uk/SchemasByVersion.aspx

25

Or

b. Positive or Negative Acknowledge according to ConduitTransport-v?-?.xsd schema.

It contains:

i. Status: Success / Failure / Waiting

ii. IssueClass– typical issues:

1. “Warning”

2. “Message type not supported”

3. “Invalid Format”

4. “Authorization Failed”

5. ...

6. “Other”

iii. Success, Error or Warning message (if accepted conditionally)

iv. Optional: User hint – how to deal with from the user point of view.

v. Optional: Administrator hint – how to deal with from the user point of

view.

vi. Referred message UUID (in case of acknowledgements or for future use).

Message sets (unit of work)

Usually more than one message will be sent during a given service operation call. In order to

support transactional atomicity (see 0. Transactions) the messages can be gathered into sets. A

Set is a higher-level message which represents a single batch to be transmitted and persisted by

the client or server. The schema of MessageSet contains:

1. Set Header

a. Unique set UUID

b. Sending time

c. Flag MoreMessageSetsAwaiting – set if data remains in incoming queue after

transmission of this set

d. (...extensibility...)

2. Body – a sequence of messages according to a schema presented in 24 - Basic message

schema (entity or acknowledgement).

Transactions

This section summarises techniques used to assure message delivery reliability and points out

possible issues which must be handled by client implementers:

1. If any call to the service operation fails, then the client must assume that none of data

passed as parameters to the operation has been processed by the service. The client must

retry with the whole message set (single unit of work).

26

2. The service assumes that data acquired from it by calling BeginDequeue may be lost (e.g.

client reads the whole set of data, operation call finishes successfully but then transaction

at client site rolls back and the data received is lost). The Service will retransmit data if it

does not receive an explicit confirmation in the form of CommitDequeue operation call.

3. The client must handle message duplicates if it is not able to assure CommitDequeue calls

each time the received messages are persisted.

Below is an example of correct client implementation, which would avoid incoming

message duplicates. This assumes a single, cyclic thread, a. is an entry point:

c. Client reads data from BeginDequeue

d. Client begins a database transaction

e. Stores the data

f. Stores a CommitDequeue call command as a row in the same database.

g. Commits the transaction.

If the transaction rolls back then also the CommitDequeue call command rolls

back and the data will come from the service again as expected.

...

a. Before each call to BeginDequeue, the client reads CommitDequeue command

rows.

b. For each CommitDequeue command row the Client:

i. Calls CommitDequeue operation

ii. On success: deletes the command row

iii. On failure: retries.

If the client was unable to delete a command row after successful call to

CommitDequeue (e.g. client machine shuts down) then the client would

produce a duplicate call to CommitDequeue operation which is allowed by

service and has no effect.

Versioning and compatibility

According to the eGIF standard an instance of XML document must explicitly indicate the version

of its schema. Following the eGIF recommendations a SchemaVersion attribute is required on

MessageSet root element. The attribute value must be equal to the version of ConduitTransport-

v?-?.xsd. The version of the schema is also indicated in its file name (e.g. –v1-0.xsd).

The eGIF standard requires also that any change in any XML schema referenced by

ConduitTransport-v?-?.xsd causes a change of ConduitTransport schema version.

The version number (e.g. 3.23b) consists of three parts:

1. Major version number (e.g. 3) – obligatory.

27

2. Minor version number (e.g. 23) – obligatory.

3. Version letter (e.g. b) – optional – used only for draft versions.

The major version number is changed when the change from the previous version of schema will

cause existing documents to fail to validate. This could occur, for example, if a new mandatory

element is added.

The minor version number is changed when the change to the schema will result in all existing

documents continuing to validate. However, some new documents which validate against the new

version will fail against the old version. For example, this could occur with the addition of a new

optional element.

The version letter should change every time a new draft is issued. In the example above, the

version 3.23b is the second draft based on an existing release 3.23, and will lead to a new release

3.24 or 4.0.

The Conduit Service adheres to these rules when validating incoming message sets. Each service

endpoint represents a single service version (e.g.

http://conduit.staffplan.co.uk/v0.0d/ConduitMessageQueue.svc endpoint provides access to a

service which contract is based on ConduitTransport-v0-0d.xsd schema).

The service reads the SchemaVersion attribute of the incoming message set, compares it to its

own version and rejects the message set if:

1. The major version numbers differ

2. The message minor version number is greater than the minor version number of the

service.

When a client requests incoming messages by calling BeginDequeue operation, the service returns

only these messages which version fulfils the above conditions comparing to the version of a

service endpoint used to invoke BeginDequeue operation.

It is possible to use two versions of the service at the same time. Newer messages can then go

through newer service endpoints. Older messages can go through both (if minor version number

differs) or only through older endpoint (if major version number differs).