swift message retrieval and replay capabilities for cls members cls member gateway elimination...

SWIFT message retrieval and replay capabilities for CLS membersCLS Member Gateway Elimination Project

Pieter Herrebout/Damien Vanderveken/Pamela Amick

22-23 January 2014

Session objectives

• Objective of today’s session is to – Provide an overview of the various

retrieval capabilities available on SWIFTNet

– Illustrate the recovery process for the MI Channel using the new replay service

2SWIFT message retrieval and replay capabilities – 22-23 January 2014

SWIFT message retrieval and replay capabilities – 22-23 January 2014 3

Agenda

• Overview of SWIFTNet retrieval capabilities

• Recovery for MI Channel

Retrieval on SWIFTNet For store-and-forward traffic

• Message retrieval• Message retrieval

Retrieval options available via a store-and-forward interface

• generic message(s) retrieval - similar to FIN

• to support investigations or message recovery

• generic message(s) retrieval - similar to FIN

• to support investigations or message recovery


Retrieval options available via MI Channel

• Short-term bulk retrieval

• Short-term bulk retrieval

• Message replay• Message replay

• retrieves all traffic in bulk (max last 24 hrs)

• allows to "fill the gap" in case of unplanned site takeover with missing traffic

• retrieves all traffic in bulk (max last 24 hrs)


• retrieves ("replays") messages in an easy way


• retrieves ("replays") messages in an easy way


Message retrieval

SWIFT message retrieval and replay capabilities – 22-23 January 2014

• SWIFT stores message traffic in an archive when customers send traffic via a store-and-forward service

• Customers can retrieve a copy of their traffic sent and/or received from the archive

• Find a message e.g. dispute between sender/receiver,

question from regulator, ...

• Recover lost traffic e.g. after an interface crash

Typical use

5

SWIFTNet message retrieval process

Trigger Selection Processing Delivery Use

1

2


• Two ways to trigger the retrieval request• Using a GUI screen (in the SWIFTNet Online Operations Manager)• Using a messaging interface that sends system message

• Received messages arrive in a store-and-forward queue, the messaging interface picks them up (as usual)

• for manual treatment, or • possibly automated treatment by interface or backoffice

6

• Features:• Each customer can retrieve their traffic (sent or received)

• covers messages sent/received with InterAct store-and-forward• includes traffic to/from central institution in case of Y-Copy

• 124 days retrieval capability• (pilot and ITB traffic 4 days)

• Online retrieval • trigger through system message or GUI screen (SWIFTNet Online Operations Manager)

• retrieved traffic arrives in a store-and-forward queue

• Input or output retrieval, with various selection criteria• Access control using an RBAC role

• Interfaces:• Requires updated interface release to submit retrieval requests

and process retrieval responses (e.g. Alliance Access 7.0.75)

• Overall functionality is similar to FIN• there are some technical differences mainly relevant to developers

Message retrievalHighlights

7

Message retrievalExample screen - Online Operations Manager GUI


Message retrievalExample system message xsys.015.001.01 Retrieval Request


See SWIFTNet system messages (User Handbook):https://www2.swift.com/uhbonline/books/protected/en_uk/sn_7_0_ufsm/index.htm

SWIFTNet short-term bulk retrievalOverview


What• Retrieve all InterAct/FileAct of max

24 hours old

• Allows to "fill the gap" if customer lost data in transit

• Subscription based

How• Request via system message

• Delivery of traffic is bulked inside files delivered via FileAct

• Requires specific development to process file contents

Ongoing sync

Site disaster – possible data loss

011 101 01001

SWIFT

Bulk retrieval request to "fill the gap"

10

SWIFTNet short-term bulk retrievalHighlights

• Covers (max) last 24hrs of traffic• Single retrieval criterion: timerange• Retrieves all store-and-forward traffic sent and received• Requires some development• The customer sends a Bulk Retrieval Request system message to SWIFT,

mentioning the time range to cover.

• SWIFT replies with a Bulk Retrieval Report system message that lists the files that have been generated.

• SWIFT delivers the files through FileAct store-and-forward. They arrive in the notification queue that is specified when sending the Bulk Retrieval system message to SWIFT.

• Customer needs to process the bulk files (e.g. filter on service name).


Message replay (with MI Channel)Overview


SWIFT

site1 site2

copy

Normal traffic delivery

SWIFT

site1 site2

Switch to site2

SWIFT

site1 site2

Automatically resumenormal traffic deliveryActivate replay

Message replay (with MI Channel)Highlights

• MI Channel has an integrated "replay" recovery feature:• SWIFT will re-deliver messages received in the same way as the

original delivery (+ "potential duplicate" indication)

• Allows the secondary site to "fill the gap" by identifying and processing the missing messages

• After the replay, normal traffic delivery resumes automatically

• Preserves message delivery order

• User initiates replay using the following parameters:• timestamp: re-delivery of all messages received since this time

• + optional OSN(s) per SnF queue: re-delivery of only messages with higher OSN


Comparison


Message retrieval Short term bulk retrieval

Messagereplay

Typical use case Message investigation or interface recovery

Point-in time recovery to other site

Point-in time recovery to other site

Scope Any message All traffic One market infrastructure service

Selection criteria Flexible (timeframe, reference, sequence numbers, ...)

Timeframe Timestamp ortimestamp+OSN

Supported period Last 124 days, requests of 10000 messages.

Last 24 hours, requests of 1 hour max

Last 8 hours

Eligibility All customers Subscribed customers Market Infrastructure customers using MI Channel

Trigger mechanism System message or O2M screen

System message Alliance Gateway GUI (or command line)

Delivery mechanism Message-per-message re-delivery

Traffic bulked in files Message-per-message re-delivery

Retrieved message handling

Application required to process retrieved traffic

Application required to process retrieved files, and extract and process the needed information

Easy integration, messages are re-delivered as they were delivered initially

Messaging services supported

InterAct Store and Forward

InterAct and FileAct Store and Forward

InterAct Store and Forward


Agenda

• Overview of SWIFTNet retrieval capabilities

• Recovery for MI Channel

MI Channel recovery in case of data loss


SWIFTNetStore and Forward

Alliance GatewayWebSphere MQ queue manager

HSMsBack-Office application

MI Channel component

Alliance GatewayWebSphere MQ queue manager

HSMs

Primary site

Secondarysite


Active

Inactive

MI Channel component status

MI Channel – Emission flow – From member to CLS

17

Alliance Gateway

MQ clientBack-Office

application



Emission Queue

Notification Queue

Error Queue


1 2

367

5

4

MI Channel

Recovery in case of data loss – Emission flow

18

Batch1

S&F

Msg1

Msg2

SWIFT

Primary site

Secondary site

Emission Queue

Msg3

Notification Queue

Notif1

Notif2

Back Office

Msg4

Msg1

Msg2

Msg3

Alliance Gateway

MQ client


MI Channel

Alliance Gateway

MQ client


MI Channel

ACK received

Waiting ACK

Back-Office emission status

Ready to send

Emission Queue

Notification Queue

Example – Identifying status of sent messages


Element Type Description

SentMessage [1] SentMessage Contains resulting transfer details of the message which was attempted to be sent

SentHeader [0..1] SentHeader Contains the routing information found for the message in the queue

MessageRef [1] String Unique reference of the message

BatchIndex [0..1] Integer Optional element present when messages have been batched together and share the same SnFRef. The value represents the index position of the message in the batch starting from one.

BatchCount [0..1] Integer Optional element present when messages have been batched together and share the same SnFRef. The value represents the total number of messages in the batch.

TransferAnswer [1] String

“Accepted”, “Failed Storage”, “Failed Delivery”

Indicates that SWIFT has accepted the storage of a message, failed to store the message, or a message that was stored successfully failed to be delivered

SnFRef [0..1] String SWIFT reference of the request. Includes a trusted time stamp.

TransferError [0..1] TransferError Present if the Transfer Answer is not “Accepted”

ErrorCode [1] String Error code of the error

ErrorText [1] String Error description

extract from MI Channel Interfaces Specifications

01 <SwMsg:SentMessage xmlns:SwMsg="urn:swift:snl:ns.SwMsg">

02 <SwMsg:SentHeader>

03 <SwMsg:MessageRef>MyUniqueRef</SwMsg:MessageRef>

04 <BatchIndex>1</BatchIndex>

05 <BatchCount>2</BatchCount>

05 </SwMsg:SentHeader>

06 <SwMsg:TransferAnswer>Accepted</SwMsg:TransferAnswer>

07 <SnFRef>... </SnFRef>

08 </SwMsg:SentMessage>

Example – Identifying status of sent messages


Batch3

Batch2

Recovery in case of data loss – Emission Flow

21

Batch1

S&F

Msg1

Msg2

SWIFTPrimary site

Secondary site

Msg3 Notif1

Notif2

Back Office

Msg4

Msg1

Msg2

Msg3

Alliance Gateway

MQ client


MI Channel

Alliance Gateway

MQ client


MI Channel

Msg1PDE

Msg2PDE

Msg3PDE

Notif1

Notif2

Notif3

Msg1PDE

Msg2PDE

Msg3PDE

Msg4

Msg4 Notif4

ACK received

Waiting ACK

Back-Office emission status

Ready to send

Emission Queue

Emission Queue

Notification Queue

Notification Queue

MI Channel - Reception flow

22

Alliance Gateway


application



Reception Queue

Non-repudiation

Queue

Notification Queue


5 2

1

4

Error Queue

3 MI Channel

Recovery in case of data loss – Reception flow

23

S&F

Batch1

Batch2

SWIFTPrimary site

Secondary site

NR Queue

Batch1

Batch2

ReceptionQueue

B1.Msg2

Back Office

B1.Msg1

Batch3

Batch4

B2.Msg1

B2.Msg2

Alliance Gateway

MQ client


MI Channel

Alliance Gateway

MQ client


MI Channel

Delivered

Not yet delivered

SWIFT delivery status

Batch1

B1.Msg1

B1.Msg2

NR Queue

ReceptionQueue

Options to determine the replay restart point


Option

Behaviour

Pro’s

Con’s

Based on knowledge of the local routing, replication, queue processing, determine the point in time where all messages have been processed.

Based on knowledge of the local routing, replication, queue processing, determine the point in time where all messages have been processed.

Input criteria

Starting timeOSN & associated

delivery time

All traffic as of the starting time will be replayed in the same way as the original delivery with "potential duplicate" indication

All traffic as of the starting time will be replayed in the same way as the original delivery with "potential duplicate" indication

All traffic as of sequence number +1 will be replayed in the same way as the original delivery with "potential duplicate" indication

All traffic as of sequence number +1 will be replayed in the same way as the original delivery with "potential duplicate" indication

• Easy to implement• Easy to implement

• More precise selection criterion hence limiting the number of duplicates and the time required to receive replayed messages.

• More precise selection criterion hence limiting the number of duplicates and the time required to receive replayed messages.

• Less precise selection criterion hence increasing the number of duplicates and the time required to receive replayed messages.

• Less precise selection criterion hence increasing the number of duplicates and the time required to receive replayed messages.

• Requires back office application to implement complex reconciliation logic

• Requires back office application to implement complex reconciliation logic

CLS member application must determine, for each SnF queue, the last OSN received (without gaps) where all messages within the batch have been processed

CLS member application must determine, for each SnF queue, the last OSN received (without gaps) where all messages within the batch have been processed

Example – Identifying the replay trigger parameters – extract from MI Channel specs


Element Type

ReceivedMessage [1] ReceivedMessage Contains the detailed format of the messages sent to SWIFT andreceived by the member or CLS

ReceivedHeader [1] ReceivedHeader Contains the routing information for the message in the queue

Requestor [1] String Requestor DN used to send the original message

Responder [1] String Responder DN of the receiving institution

Service [1] String SWIFTNet service name used when sending the original message

RequestType [1] String ISO standard business request type in the format xxxx.nnn.nnn.nn

MessageRef [1] String Unique Sender reference of the message

PDIndication [0..1] DateTime PDIndication [0..1] DateTime Indicator of possible duplicate emission in UTC

BatchIndex [0..1] Integer Optional element present when messages have been batched together and share the same SnFRef. The value represents the index position of the message in the batch.

BatchCount [0..1] Integer Optional element present when messages have been batched together and share the same SnFRef. The value represents the total number of messages in the batch.

SnFRef [1] String SWIFT reference of the request. Includes a trusted time stamp.

RetrievedMessageIndication [0..1] Boolean Indicates that the message has previously been delivered and has been re-sent as a result of an operational recovery.

SnFInfo [1] SnFInfo Store-and-forward related transfer information

SnFDeliveryInfo [1-10] SnFDeliveryInfo Provides the delivery attempt history

SnFSessionId [1] String Identifies the Output Session used to deliver the message from SWIFT. Can be used in union with the OutputSeq number to verify that all messages sent can be accounted for once Delivered

SnFOutputSeq [1] Integer Store-and-forward output sequence number for the message retrievedwithin the session

SnFDeliveryTime [1] DateTime Time of delivery of the message in UTC

RequestPayload [1] XML[Any] An XML string that contains the ISO message to be sent

Identifying the replay trigger parameters


B1.Msg2

B1.Msg1

B2.Msg1

01 …02 <SwMsg:BatchIndex>1</SwMsg:BatchIndex>03 <SwMsg:BatchCount>2</SwMsg:BatchCount>04 …05 <SwMsg:SnFSessionId>bankbebb_clsinstrnot:d:12345</SwMsg:SnFSessionId>06 <SwMsg:SnFOutputSeq>1</SwMsg:SnFOutputSeq>07 <SwMsg:SnFDeliveryTime>2014-01-22T08:00:00Z</SwMsg:SnFDeliveryTime>08 …



Batch 1 has been completely received

Batch 2 has not been completely received

Triggering replay from Alliance Gateway:Generic approach for operating profile

• Operating profile functions available for MI Channel support interface

• Define operating profile with relevant functions for starting / stopping message flow:– Enable a Message Flow Instance– Start Replay for a Message Flow Instance– Disable a Message Flow Instance

• Define operator and assign operating profile


Tentative

Triggering replay from Alliance Gateway


Tentative

Copyright <text>

Configuration

Alliance Gateway: <instanceName>-

Licensing Configuration

System

User Management+Event Log+Application Interface+SWIFTNet Interface+MI Channel Support Interface-

File Transfer+Routing+

User: <name> Alliance Server Instance: <name> Status

Help LogoutChange Password

AboutAlliance Gateway Administration <n>Configuration MonitoringHome HSM Management

Message Flow Instances

Name Port

Message Flow Instances Rows in list: <n>, in selection: <n>

Window Size Min. Delay

<nameOfMI>+

Emission EndpointsReception Endpoints

Reliable Messaging-

Change View Add Delete Enable Disable

Max. Delay Base Port Condition

< Previous Next >

Report

State

Refresh


Batch Classes

SitesMQ Managers

MQ QueuesMQ Channels

SnF QueuesRouting Rules

Routing Rule Sets

Replay

MFP<nn> <value> <value> <value> <value> <value> Disabled <value> stoppedStatus

Cancel OK

Replay for Message Flow Instance

Name

Help

Status

State

SnF Queue Parameters

Remove

Add

Edit

stopped

Disabled

MFP<nn>

File Path

Use Existing File No

UTC Time yyyy-mm-ddThh24:mi:ssZ

Include OSN per SnF Queue No

Force Start No

Condition <value>

Triggering replay from Alliance Gateway – option 1: file input


29

Tentative

Copyright <text>

Configuration



System







Name Port



<nameOfMI>+


Reliable Messaging-



< Previous Next >

Report

State

Refresh


Batch Classes

SitesMQ Managers



Routing Rule Sets

Replay


Cancel OK


Name

Help

Status

State


Remove

Add

Edit

stopped

Disabled

MFP<nn>

File Path

Use Existing File Yes



Force Start No

Condition <value>

File built by CLS back office application

Triggering replay from Alliance Gateway – option 2: manual input


Tentative

Copyright <text>

Configuration



System







Name Port



<nameOfMI>+


Reliable Messaging-



< Previous Next >

Report

State

Refresh


Batch Classes

SitesMQ Managers



Routing Rule Sets

Replay


Cancel OK


Name

Help


Remove

Add

Edit

MFP<nn>

File Path




Status

State stopped

Disabled

Force Start No

Condition <value>

Parameters to be manually filled in

Triggering replay from Alliance Gateway – option 2 manual input – continued


Tentative

Copyright <text>

Configuration



System







Name Port



<nameOfMI>+


Reliable Messaging-



< Previous Next >

Report

State

Refresh


Batch Classes

SitesMQ Managers



Routing Rule Sets

Replay


Cancel OK


Name

Help

Status


Disabled

MFP<nn>

File Path



Include OSN per SnF Queue Yes

2014-01-22T08:00:00Z

Remove

Add

Edit

bankbebb_clsopdata

bankbebb_clsrepbankbebb_clsinstrnot

State stopped

Force Start No

Condition <value>


Triggering replay from Alliance Gateway – option 2 manual input – continued

32

.


Tentative

Copyright <text>

Configuration



System







Name Port



<nameOfMI>+


Reliable Messaging-



< Previous Next >

Report

State

Refresh


Batch Classes

SitesMQ Managers



Routing Rule Sets

Replay


Cancel OK


Name

Help

Status


Disabled

MFP<nn>

File Path



Include OSN per SnF Queue Yes

<value>

Remove

Add

Edit

bankbebb_clsopdata

bankbebb_clsrepbankbebb_clsinstrnot

State stopped

Force Start No

Condition <value>

Cancel OK

Edit SnF Queue Parameters Help

Session Details bankbebb_clsinstrnot:d:12345

OSN

queue:d:nnnnn

1


Recovery in case of data loss – Reception flow


S&F

Batch1

Batch2

SWIFTPrimary site

Secondary site

Batch1

Batch2B1.Msg2

Back Office

B1.Msg1

Batch3

Batch4

B2.Msg1

B2.Msg2

Alliance Gateway

MQ client


MI Channel

Alliance Gateway

MQ client


MI Channel

Batch2 B2.Msg1

B2.Msg2PDM PDM

PDM

B2.Msg1

B2.Msg2PDM

PDM

Batch3

Batch4B3.Msg2

B4.Msg1

B4.Msg2

B3.Msg1

B3.Msg2

B4.Msg1

B4.Msg2

B3.Msg1 Resumes normal delivery

Delivered

Not yet delivered


Delivered

Not yet delivered


Batch1

B1.Msg1

B1.Msg2

NR Queue

NR Queue

ReceptionQueue

ReceptionQueue


Recovery using replay Summary

Recover from

data loss

resulting from

site failover

Requires

traffic

reconciliation

from Back

Office

Easy to

integrate

Triggered

through Alliance

Gateway

Administration

GUI*

while preserving message sequencing

* Or command line

Q&A



Thank you


Appendix

MI Channel Emission side – error flow

38

Alliance Gateway


application


CLS Member Gateway Elimination Project workshop – 12 November 2013

Emission Queue 1

Notification Queue 1

Error queue


1 2

35

4

4

6MI

Channel

MI Channel Reception side – error flow

39

Alliance Gateway


application


CLS Member Gateway Elimination Project workshop – 12 November 2013

reception Queue 1

Non-repudiation

queue

Notification queue


1

4

Error queue

3

2

2

2

MI Channel

5

5

5

Replay InvocationStandalone SNLswiftnet start [-g <group> | -s <subsys>] [ -F ] [ -R <argument> ]

Where :

-g <group>: optional parameter. sub-system group.

Example - mfp

-s <subsys>: optional parameter. sub-system name.

Example - Support

-F : Force Start.

-R <argument> : Replay Mode

argument - <timestamp> or< file-name>

<timestamp> format yyyy-mm-ddThh24:mi:ssZ

<file-name> - file with replay information as per interface spec.

<file-name> is the name of the replay parameters file. MI Channel subsystem will fail to start if there are no snf queues configured or if snf queues in file-name do not match the configured snf queues for this MI Channel.

<timestamp> is in UTC. MI Channel subsystem will start replay from all the configured snf queues of the MI Channel. It fails if there are no snf queues configured for this MI Channel.

Some examples: to start/resume replay:

swiftnet start -R <file-name>

swiftnet start –R <timestamp>

swiftnet start -F -R <file-name>

swiftnet start –F –R <timestamp>


Replay File

Element Type Description

Replay Replay Required information to restart in replay mode

SnFInfo [0..n] SnFInfo Store-and-forward related transfer information for each queue

SnFDeliveryInfo [1-10] SnFDeliveryInfo Provides the delivery attempt history

SnFOutputSeq [1] Integer Store-and-forward output sequence number for the message retrieved within the session

SnFSessionId [1] String Identifies the Output Session used to deliver the message from SWIFT. Can be used in union with the OutputSeq number to verify that all messages sent can be accounted for once delivered.

SnFDeliveryTime [1] DateTime Time of delivery of the message in UTC

RecoveryTime [1] DateTime Mandatory.

The time when the customer is sure that all traffic has been replicated up to the last OSN gap when all the messages in the batch have been processed.


swift message retrieval and replay capabilities for cls members cls member gateway elimination...

Documents

bulk retrieval system

retrieval requests

output retrieval

swift bulk retrieval

swift stores message

process retrieval responses

missing traffic slide

days online retrieval