websphere mq managed file transfer - mqug.org.uk - wmq03 - introduction t… · agenda common...

WebSphere MQ Managed File TransferGeoff Judd

WebSphere MQ Development

IBM

© 2009 IBM Corporation

1

Agenda

� Common problems transferring file data

� Introduction to MQ Managed File Transfer

� IBM’s Managed File Transfer Portfolio

– Introducing IBM Sterling Commerce products


2

� Key MQ Managed File Transfer concepts

� Usage scenarios for MQ Managed File Transfer

How do most organizations move files today?

� FTP

– Typically File Transfer Protocol (FTP) is combined with writing and maintaining homegrown code to address its limitations

� Why is FTP use so widespread?

– FTP is widely available – Lowest common denominator

– Promises a quick fix – repent at leisure

– Simple concepts – low technical skills needed to get started

Most organizations rely on a mix of homegrown code, several legacy products and

different technologies … and even people!


3

– Simple concepts – low technical skills needed to get started

– FTP products seem “free”, simple, intuitive and ubiquitous

� Legacy File Transfer products

– A combination of products often used to provide silo solutions

– Often based on proprietary versions of FTP protocol

– Can’t transport other forms of data besides files

– Usually well integrated with B2B but rarely able to work with the rest of the IT infrastructure – especially with SOA

� People

– From IT Staff to Business staff and even Security Personnel

– Using a combination of email, fax, phone, mail, memory keys…

Shortcomings of Basic FTP

Limited Security

Limited Reliability

�Unreliable delivery – Lacking checkpoint restart – Files can be lost

�Transfers can terminate without notification or any record – corrupt or partial files can be accidentally used

�File data can be unusable after transfer – lack of

�Often usernames and passwords are sent with file –

as plain text!�Privacy, authentication and

encryption often not be available

�Non-repudiation often lacking


4

Limited Flexibility

Limited visibility and traceability

after transfer – lack of

Character Set conversion

�Transfers cannot be monitored and managed centrally or remotely

�Logging capabilities may be limited and may only record transfers between directly connected systems

�Cannot track the entire journey of files – not just from one machine to the next but from the start of its journey to its final destination

�Changes to file transfers often require updates to many ftp scripts that are typically scattered across machines and require platform-specific skills to alter

�All resources usually have to be available concurrently�Often only one ftp transfer can run at a time�Typically transfers cannot be prioritized

What is WebSphere MQ Managed File Transfer?

� Auditable Full logging and auditing of file transfers + archive audit data to a database

A B C X Y Z

……

WebSphere MQ Managed File Transfer

Adds managed file transfer capabilities to WebSphere MQ


5

� Auditable Full logging and auditing of file transfers + archive audit data to a database

� Reliable Checkpoint restart. Exploits solid reliability of WebSphere MQ

� SecureProtects file data in transit using SSL. Provides end-to-end encryption using

AMS

� Automated Providing scheduling and file watching capabilities for event-driven transfers

� Centralized Provides centralized monitoring and deployment of file transfer activities

� Any file size Efficiently handles anything from bytes to terabytes

� Integrated Integrates with MB, WSRR, ITCAMs for Apps, DataPower + Connect:Direct

� Cost Effective Reuses investment in WebSphere MQ.

A consolidated transport for both files and messages

� Traditional approaches to file transfer result in parallel infrastructures

– One for files – typically built on FTP

– One for application messaging – based on WebSphere MQ, or similar

� High degree of duplication in creating and maintaining the two infrastructures


6

� Managed File Transfer reuses the MQ network for managed file transfer and yields:

– Operational savings and simplification

– Reduced administration effort

– Reduced skills requirements and maintenance

File TransfersApplication Messaging

Consolidated Transportfor Messages & Files

Agenda





You are here


7



With Sterling Commerce, IBM offers comprehensive MFT Capabilities

� WebSphere MQ Managed File Transfer provides file

transfer optimized for data delivery across WebSphere

MQ networks

� Sterling Connect Direct provides peer-to-peer file

transfer optimized for data delivery within and between

enterprises across Connect:Direct protocol

Addressing multiple use cases and scenarios for both internal and multi-

enterprise file transfer


8

enterprises across Connect:Direct protocol

� Sterling File Gateway provides trading partner on-

boarding, broad protocol support, management and

visibility

� For comprehensive file transfer needs IBM provides

integration between WebSphere MQ Managed File

Transfer, Sterling Connect:Direct, and Sterling File

Gateway

IBM MFT Vision

Accelerate and simplify governance of the growing volume of business-

critical data movement within and beyond the enterprise with Smarter MFT

Maximize the agility and performance of dynamic business networks by reducing

the complexity, risk, and cost of file transfer

Visibility•Single view of transfer activity•Transaction and business monitoring•Dashboards, analytics, and scorecards

Security & Performance•Assured delivery and high availability•Protection of file data in transit and at rest•Accelerated transport and low latency


9

Usability & Management•Persona-based, easy-to-use interfaces•Unified control and configuration of infrastructure

•Community on-boarding and coordination

Universal

standards

Universal•Any transport, any protocol, and any partner

•Global and cloud-enabled deployment•Broad platform coverage and industrystandards

Connectivity• Integration with BPM to drive business processes•Leveraging ESBs to enable service orientation•SOA Registry/Repository for lifecycle governance

Agenda






10



You are here

Applications exchanging file data

Components of a typical WMQ MFT network

� Agents

– The endpoints for managed

file transfer operations

� Commands

– Send instructions to agents

� Log database or file

WebSphere MQ

Agent Agent Agent


11

– A historical record of file

transfers

� Coordination queue manager

– Gathers together file transfer

events

WebSphere MQ“Coordination”Queue Manager

Commands Log database

or file

Agents

� Act as the end points for file transfers

� Long running MQ applications that transfer files by splitting them into MQ messages

– Efficient transfer protocol avoids excessive use of MQ log space or messages building up on queues

� Multi-threaded file transfers


WebSphere MQ

Agent Agent Agent


12

Log database

or file

� Multi-threaded file transfers

– Can both send and receive multiple files at the same time

� Generate a log of file transfer activities which is sent to the “coordination queue manager”

– This can be used for audit purposes

� Associated with one particular queue manager (either v6 or v7)

– Agent state on queues


Commands

There are notes that accompany this slide

Notes on: Agents

� MFT agent processes define the end-points for file transfer. That is to say that if you want to

move files off a machine, or onto a machine – that machine would typically need to be running

an agent process

� Agent processes are long running MQ applications that oversee the process of moving file data

in a managed way. Each agent monitors a ‘command’ queue waiting for messages which

instruct it to carry out work, for example file transfers

� The MFT agent process needs connectivity to an MQ queue manager to do useful work. It can NO

TE

S


13

� The MFT agent process needs connectivity to an MQ queue manager to do useful work. It can

connect either directly to a queue manager running on the same system, or as an MQ client

using an embedded version of the MQ client library (which is kept completely separate to any

other MQ client libraries that may or may not already have been installed onto the system)*

– Each agent requires its own set of MQ queues – which means that an agent is tied to the queue manager where these queues are defined

– However – one queue manager can support multiple agents

* Note: availability of direct (bindings) connectivity or MQ client based connectivity is dependent on the version of MQ MFT in use

• WebSphere MQ Managed File Transfer on z/OS does not support the MQ client style of connectivity

• Managed File Transfer on distributed platforms has a ‘server’ and ‘client’ offering. The agent component of the ‘client’ offering is restricted to only supporting MQ client style connectivity. The agent component of the ‘server’ offering may be used either connectivity options

NO

TE

S

Commands

� Send instructions to agents and

display information about agent

configuration

– Via MQ messages

� Many implementations of

commands:

– MQ Explorer plug-in


WebSphere MQ

Agent Agent Agent


14

Log database

or file

– MQ Explorer plug-in

– Command line programs

– Open scripting language

– JCL

– Documented interface to

program to


Commands


Notes on: Commands

� “Commands” is the name we have given to anything which instructs the MFT agent. As

described on the previous slide, there are a wide range of command implementations including

graphical and non-graphical command-line based commands

� Commands instruct the MFT agent by sending it messages. The messages themselves use a

documented format which can easily be incorporated into your own applications

� The commands that are supplied with MFT can connect either as an MQ client (again based on

embedded client libraries) or directly to a queue manager located on the same systemNO

TE

S


15

embedded client libraries) or directly to a queue manager located on the same systemNO

TE

S

Log Database & File

� Keeps a historical account of transfers that

have taken place

– Who, where, when… etc.

� Implemented by the ‘logger’ component

which connects to the coordination queue

manager

– Stand alone application


WebSphere MQ

Agent Agent Agent


16

– Stand alone application

- Can log to database or file

– Or JEE application

- Can log to database only

� Queryable via Web Gateway

– Also a documented interface


Commands


Log database

or file

Notes on: Log Database

� Managed File Transfer can record a historical account of file transfers to a database using the

‘database logger’ component.

– This component is available to run as a stand alone process or as a JEE application

� The information used to populate the log database is generated, as MQ messages, by the MFT

agents participating in file transfers. This is routed to a collection point in the MQ network,

referred to as the ‘coordination queue manager’ (see next slides). The database logger

component subscribes to the messages produced by agents and reliably enters them into a NO

TE

S


17

component subscribes to the messages produced by agents and reliably enters them into a

database.

NO

TE

S

Coordination Queue Manager

� Gathers together information about

events in the file transfer network

� Not a single point of failure

– Can be made highly available

– Messages stored + forwarded


WebSphere MQ

Agent Agent Agent


18

Log database

or file

� MQ v7 publish / subscribe

– Allows multiple log databases,

command installs

– Documented interface


Commands


Notes on: Coordination queue manager

� The coordination queue manager is used as the gathering point for all the information about file

transfers taking place between a collection of MFT agents

� The queue manager uses publish/subscribe (so it must be MQ version 7) to distribute this

information to “interested parties” which typically include:

– The WebSphere MQ Explorer plug-in, which provides a graphical overview of MFT activity

– The database logger component, which archives the information to a database

– Some of the command line utilities which are part of the MFT product

�

NO

TE

S


19

� The format used to publish information is documented and can be used to develop 3rd party

applications which process this data

� Although there is only a single ‘coordination’ queue manager for a given collection of agents it

does not represent a point of failure:

– MQ stores and forwards messages to the coordination queue manager when it is available – so if the coordination queue manager is temporarily unavailable no log data is lost

– The ‘coordination’ queue manager can be made highly available using standard HA techniques such as MQ multi-instance queue manager or via a HA product such as PowerHA

NO

TE

S

Agenda






20


� Usage scenarios for MQ Managed File Transfer You are here

Example usage of monitoring + program execution

ExistingApplication

WMQMFT

Agent

WMQMFT

Agent

ExistingApplication

3. MFT transports file

to destination

5. MFT can also start another

application to process the file


21

1. Application writes

file to file system

2. Agent monitors file

system, spots arrival

of file and based on

rules, transfers the file

4. At destination MQ MFT

writes file to file system


Notes on: Monitoring & Program Execution

� Resource monitors work in two stages:

1. Poll a resource (in this case the file system – but as we’ll see later the ‘resource’ can also be an MQ queue) and identify that a condition has been met (perhaps the appearance of a file matching a particular pattern)

2. Perform an action (which can include starting a managed file transfer, or running a script), optionally propagating information about the resource (for example the name of the file triggered on) into the action

� As shown on the previous slide, resource monitors are typically used to provide integration

with an existing system without needing to make changes to the system

� Another function of MFT, used for integration with existing systems is the ability to execute NO

TE

S


22

� Another function of MFT, used for integration with existing systems is the ability to execute

programs or scripts both on the source or destination systems for a file transfer. This can be

used to:

– Start a program, on the source system, which generates the file data to be transferred prior to performing the managed file transfer

– Start, or notify, a program on the destination system when the file data has been transferred – allowing it to process the data without having to poll

NO

TE

S

XML Scripting using Apache Ant

Step 1

Invoke a File Transfer

Step 2

If Step 1 completes Ok then

invoke program to process file

1


invoke program to process file

2

3

Step 3

If Step 1 fails then

send an email to the

Administrator

Protocol Bridging Agents

� Support for transferring files located on FTP and SFTP servers

– The source or destination for a transfer can be an FTP or an SFTP server

� Enables incremental modernization of FTP-based home-grown solutions

– Provides auditability of transfers across FTP/SFTP to central audit log

– Ensures reliability of transfers across FTP/SFTP with checkpoint restart

� Fully integrated into graphical, command line and XML scripting interfaces

– Just looks like another MFT agent…


24

Audit information

Agent

WebSphere MQ

Agent Agent

ProtocolBridgeAgent

FTP/SFTP

FTP/SFTPServer

FTP/SFTPClient

FTP/SFTPClient

FTP/SFTPClient

Files exchanged between MFT and FTP/SFTP

WebSphere Message Broker Nodes

Message Flow

Execution Group

Message Broker

FTEInput FTEOutput

WMQMFT

AgentWMQMFT

Agent

WMQMFT

Agent

WMQMFT

Agent

Part ofWMB 7.0.0.1

Part ofWMB 7.0.0.1


25

� FTEInput node

– Build flows that accepts file transfers from the WMQ MFT network

� FTEOutput node

– Build flows that are designed to send a file across a WMQ MFT network

� When WMQ MFT nodes are used in a flow an MFT agent is automatically

started in the Message Broker Execution Group

Agent

Trading Partner

Integration with IBM Sterling Connect:Direct

Agent

WebSphere MQ

Agent Agent

C:DBridgeAgent

C:DNode

C:DNode

C:D

C:DNode

Reference

Available2Q2011

Available2Q2011


26

Trading Partner

MFT

Audit

C:DNode

C:D

Audit

Reference

Inside the MFT audit trail…

The audit information for each MFT transferreferences related C:D audit information

� The Connect:Direct Bridge capability supports

managed file transfers that span MFT and C:D

with a joined up audit trail

Interoperation with DataPower B2B Appliance XB60

� Documented and tested configurations for integrating with DataPower Appliances

– WebSphere DataPower XB60 B2B Appliance – for B2B connectivity

– WebSphere DataPower IX50 Integration Appliance – for ESB connectivity

� Enables sending files to trading partners over a range of protocol transports

– via DataPower Appliances acting as B2B gateways


27

DMZ

Company A

XB60

Agent

WebSphere MQ

Agent Agent HTTPS

SFTP

AS2

…etc

Company B

Company C

… etc

Internal Network Trading PartnerInternet

Agent Agent

Multi-protocol transfers to B2B trading partners

Securing file data with SSL and WMQ AMS

� WMQ MFT supports transport

level encryption using SSL

� Data is encrypted before it is

sent over a channel and

decrypted when it is received

WebSphereMQ

QueueManager

WebSphereMQ

QueueManager

svrconn channel

sndr/rcvrchannels

Agent Agent


28

WebSphereMQ

QueueManager

WebSphereMQ

QueueManager

svrconn channel

sndr/rcvrchannels

� When combined with WMQ

Advanced Message Security

– Allows file data to be encrypted

at the source system and only

decrypted when it reaches the

destination system

– Data is secure even when at

rest on a queue

AgentAgent

Staged migration to messaging

� Pain-point:

– Hard to migrate to an event driven architecture as lots of applications communicate

by transferring files

� Managed File Transfer Helps:

– Deliver files as message payloads and vice versa

– Monitor queues and transfer message payloads to files


29

WebSphere MQ ManagedFile Transfer

Options for converting data between files and messages

One file to one message

WMQMFT

One file to a group of messages

WMQMFT

� The file can be split based on:

– Size

– Binary delimiter

� One file becomes one

message


30

One message to one file

A group of messages (or all messages on the queue) to one file

MFT

WMQMFT

WMQMFT

– Binary delimiter

– Regular expression

� One message becomes one

file

� Optionally, a delimiter can be

inserted between each

message used to compose the

file

Monitoring queues for the arrival of messages

� The WMQ MFT agent can monitor queues

for the arrival of messages, then perform an

action, such as transferring the payload fro

the messages as a file (as per the previous

slide)

ExistingApplication

WMQMFT

Agent

Remember we said MFTcan monitor for files arriving…


31

� Conditions that can be monitored for:

– Queue not empty

– Complete group of messagesExisting

Application

WMQMFT

Agent

Well, it can also monitor for messages arriving on a queue…

Historical Roadmap

FTE V7.0.2� Ability to bridge to FTP networks

� iSeries® support

� Initial DataPower XB60 integration

� Consumeability enhancements

� Security enhancementsFTE V7 4Q 2010

WMB 7.0.0.1

� FTE input

and output

nodes

FTE V7.0.4

2Q 20112Q 2012

WebSphere MQ V7.5

� WebSphere MQ File Transfer Edition

became

� WebSphere MQ Managed File Transfer


32

Q4 2008

Q2 2009Q4 2009

FTE V7.0.1� Archive transfer audit log to external DB

� Enhanced directory monitoring

� ANT XML Scripting of multi-step transfer jobs

� Enhanced transfer request error reporting

� Unattended silent install

� Support for zLinux

� Enhanced z/OS Performance, Tape, GDG

FTE V7� Reliable, managed file transfer

� Remote management and audit

� Core platforms including z/OS

� Zero programming needed

� Command line and GUI interfaces

� File auditing across backbone

� Globalization

4Q 2010

FTE V7.0.3� Web browser (ad hoc) file transfers

� Convert payloads between files and messages

� Automatically start agents and DB logger on Windows

� View deployed agents in MQ Explorer (GUI)

� End-to-end encryption using WebSphere MQ AMS

� Extensions to platform support

FTE V7.0.4� Integration with existing IBM

Sterling Connect:Direct networks

� Bridge to transfer files into out of

existing C:D networks

� Enhanced Explorer tooling with

visibility of file transfers sent

into/out of C:D networks

Resources

� Information Center:

– http://publib.boulder.ibm.com/infocenter/wmqfte/v7r0/index.jsp (pre-V7.5)

– http://publib.boulder.ibm.com/infocenter/wmqv7/v7r5/index.jsp (7.5 onwards)

� Redbooks / Redguides / Redpapers:

– Getting Started with WebSphere MQ Managed File Transfer V7

• http://www.redbooks.ibm.com/abstracts/sg247760.html

– IBM WebSphere MQ Managed File Transfer Solution Overview

• http://www.redbooks.ibm.com/abstracts/redp4532.html

– Managed File Transfer for SOA using IBM WebSphere MQ Managed File Transfer


33

– Managed File Transfer for SOA using IBM WebSphere MQ Managed File Transfer


– B2B Enabled Managed File Transfer using WebSphere DataPower B2B Appliance XB60 and WebSphere MQ Managed File Transfer


– IBM Sterling Managed File Transfer Integration and WebSphere Connectivity for a Multi-Enterprise Solution

• http://www.redbooks.ibm.com/redpieces/abstracts/sg247927.html

� Trial Download:

– http://www.ibm.com/software/integration/wmq/filetransfer/

� Early Design Program

– Interested in participating in the development of future versions of MFT?

• Ask your local IBM representative to nominate you for the MFT EDP program

Any Questions?

■


34

websphere mq managed file transfer - mqug.org.uk - wmq03 - introduction t… · agenda common...

Documents