managed file transfer solutions using datapower and ... - mq02... · managed file transfer...

61
© 2011 IBM Corporation Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition

Upload: buihanh

Post on 11-Apr-2018

264 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Managed File Transfer Solutions using DataPower and WebSphere

MQ File Transfer Edition

Page 2: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation2

Agenda

Introduction

MQ FTE Overview

DataPower File handling Overview

File transfer scenarios with DataPower / MQ FTE

DataPower MQ FTE Configuration

Page 3: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

File Transfer within the Enterprise

Head OfficeHead Office Regional LocationsRegional Locations

SubsidiariesSubsidiaries

StoresStores

BranchesBranches

InventoryInventoryOrdersOrders

AccountingAccounting

Regional OfficesRegional Offices

WarehousesWarehouses

Packaged AppsPackaged AppsCRMCRM

SalesSales

Moving Files and Documents around internally within Head-office

Moving Files and Documents between Head-office and

remote locations

Page 4: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation4

How do most organizations move files today?

FTP– Typically File Transfer Protocol (FTP) is combined with

writing and maintaining homegrown code to address its limitations

Why is FTP use so widespread?– FTP is widely available – Lowest common denominator– Promises a quick fix – repent at leisure– Simple concepts – low technical skills needed to get started– FTP products seem “free”, simple, intuitive and ubiquitous

Legacy File Transfer products– A combination of products often used to provide silo

solutions– Often based on proprietary versions of FTP protocol– Can’t transport other forms of data besides files– Usually well integrated with B2B but rarely able to work with

the rest of the IT infrastructure – especially with SOA

People– From IT Staff to Business staff and even Security Personnel– Using a combination of email, fax, phone, mail, memory

keys…

Most organizations rely on a mix of homegrown code, several legacy products and different

technologies … and even people!

Page 5: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation5

Shortcomings of Basic FTP

Limited Flexibility

Limited Security

Limited visibility and traceability

Limited Reliability

Unreliable delivery – Lacking checkpoint restart – Files can be lost

Transfers can terminate without notification or any record – corrupt or partial files can be accidentally used

File data can be unusable after transfer – lack of Character Set conversion

Often usernames and passwords are sent with file – as plain text!

Privacy, authentication and encryption often not be available

Non-repudiation often lacking

Transfers cannot be monitored and managed centrally or remotely

Logging capabilities may be limited and may only record transfers between directly connected systems

Cannot track the entire journey of files – not just from one machine to the next but from the start of its journey to its final destination

Changes to file transfers often require updates to many ftp scripts that are typically scattered across machines and require platform-specific skills to alter

All resources usually have to be available concurrentlyOften only one ftp transfer can run at a timeTypically transfers cannot be prioritized

Page 6: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation6

Auditable Who transferred a file? Where? When? Was it this file?

Reliable Automatic resumption of interrupted transfers. No partial file data left lying around

Secure Limits access to authorized users. Protects file data in transit

Automated Designed for “lights out” operation

Centralized Can be monitored and managed from one central location

Any file size Imposes no practical limits on file sizes. Efficient regardless of file size

Integrated Integrates well with applications that typically perform file processing

Cost Effective Reuses existing skills and infrastructure

What is Managed File Transfer?

No agreed specification for managed file transfer products to certify against

General consensus that managed file transfer involves the following:

Reliable, controlled, auditable movement of files

Page 7: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation7

Agenda

Introduction

MQ FTE Overview

DataPower File handling Overview

File transfer scenarios with DataPower / MQ FTE

DataPower MQ FTE Configuration

Page 8: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation8

What is WebSphere MQ File Transfer Edition?

Auditable Full logging and auditing of file transfers + archive audit data to a database

Reliable Checkpoint restart. Exploits solid reliability of WebSphere MQ

Secure Protects file data in transit using SSL. Provides end-to-end encryption using AMS

Automated Providing scheduling and file watching capabilities for event-driven transfers

Centralized Provides centralized monitoring and deployment of file transfer activities

Any file size Efficiently handles anything from bytes to terabytes

Integrated Integrates with MB, WSRR, ITCAMs for Apps, DataPower + Connect:Direct

Cost Effective Reuses investment in WebSphere MQ. Wide range of support (inc. z/OS and IBM i)

A B C X Y Z

……

WebSphere MQ File Transfer Edition

Adds managed file transfer capabilities to WebSphere MQ

Page 9: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation9

A consolidated transport for both files and messages

Traditional approaches to file transfer result in parallel infrastructures– One for files – typically built on FTP– One for application messaging – based

on WebSphere MQ, or similar

High degree of duplication in creating and maintaining the two infrastructures

File Transfer Edition reuses the MQ network for managed file transfer and yields:– Operational savings and simplification– Reduced administration effort– Reduced skills requirements and

maintenance

File Transfers Application Messaging

Consolidated Transportfor Messages & Files

Page 10: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation10

Applications exchanging file data

Components of a typical WMQ FTE network

Agents– The endpoints for managed file

transfer operations

Commands– Send instructions to agents

Log database– A historical record of file transfers

Coordination queue manager– Gathers together file transfer

events

WebSphere MQ

Agent Agent Agent

“Coordination”Queue Manager

LogDatabase

Commands

Page 11: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation11

Components of a typical WMQ FTE networkAgents

Act as the end points for file transfers Long running MQ applications that

transfer files by splitting them into MQ messages– Efficient transfer protocol avoids

excessive use of MQ log space or messages building up on queues

Multi-threaded file transfers– Can both send and receive

multiple files at the same time Generate a log of file transfer

activities which is sent to the “coordination queue manager”– This can be used for audit

purposes Associated with one particular queue

manager (either v6 or v7)– Agent state on queues

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination”Queue Manager

LogDatabase

Commands

Page 12: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation12

Notes on:Agents

FTE agent processes define the end-points for file transfer. That is to say that if you want to move files off a machine, or onto a machine – that machine would typically need to be running an agent process

Agent processes are long running MQ applications that oversee the process of moving file data in a managed way. Each agent monitors a ‘command’ queue waiting for messages which instruct it to carry out work, for example file transfers

The FTE agent process needs connectivity to an MQ queue manager to do useful work. It can connect either directly to a queue manager running on the same system, or as an MQ client using an embedded version of the MQ client library (which is kept completely separate to any other MQ client libraries that may or may not already have been installed onto the system)*

– Each agent requires its own set of MQ queues – which means that an agent is tied to the queue manager where these queues are defined

– However – one queue manager can support multiple agents

* Note: availability of direct (bindings) connectivity or MQ client based connectivity is dependent on the version of MQ FTE in use

• WebSphere MQ File Transfer Edition on z/OS does not support the MQ client style of connectivity• File Transfer Edition on distributed platforms has a ‘server’ and ‘client’ offering. The agent component of the ‘client’ offering is

restricted to only supporting MQ client style connectivity. The agent component of the ‘server’ offering may be used either connectivity options

NO

TES

Page 13: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation13

Example usage of agent monitoring + program execution

1. Application writesfile to file system

ExistingApplication

WMQFTE

Agent

WMQFTE

AgentExisting

Application*ta

p*

2. Agent monitors filesystem, spots arrival of file and based on rules, transfers the file

3. FTE transports fileto destination

4. At destination MQ FTE writes file to file system

5. FTE can also start another application to process the file

Page 14: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation14

Notes on:Monitoring & Program Execution

Resource monitors work in two stages:1. Poll a resource (in this case the file system – but as we’ll see later the ‘resource’ can also

be an MQ queue) and identify that a condition has been met (perhaps the appearance of a file matching a particular pattern)

2. Perform an action (which can include starting a managed file transfer, or running a script), optionally propagating information about the resource (for example the name of the file triggered on) into the action

As shown on the previous slide, resource monitors are typically used to provide integration with an existing system without needing to make changes to the system

Another function of FTE, used for integration with existing systems is the ability to execute programs or scripts both on the source or destination systems for a file transfer. This can be used to:– Start a program, on the source system, which generates the file data to be transferred prior

to performing the managed file transfer– Start, or notify, a program on the destination system when the file data has been

transferred – allowing it to process the data without having to poll

NO

TES

Page 15: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation15

Components of a typical WMQ FTE networkCommands

Send instructions to agents and display information about agent configuration– Via MQ messages

Many implementations of commands:– MQ Explorer plug-in– Command line programs– Open scripting language– JCL– Documented interface to

program to

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination”Queue Manager

LogDatabase

Commands

Page 16: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation16

Notes on:Commands

“Commands” is the name we have given to anything which instructs the FTE agent. As described on the previous slide, there are a wide range of command implementations including graphical and non-graphical command-line based commands

Commands instruct the FTE agent by sending it messages. The messages themselves use a documented format which can easily be incorporated into your own applications

The commands that are supplied with FTE can connect either as an MQ client (again based on embedded client libraries) or directly to a queue manager located on the same system

NO

TES

Page 17: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation17

Components of a typical WMQ FTE networkLog Database

Keeps a historical account of transfers that have taken place– Who, where, when… etc.

Implemented by the ‘database logger’ component which connects to the coordination queue manager– Stand alone application– Or JEE application

Queryable via Web Gateway– Also a documented interface

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination”Queue Manager

LogDatabase

Commands

Page 18: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation18

Notes on:Log Database

File Transfer Edition can record a historical account of file transfers to a database using the ‘database logger’ component.

– This component is available to run as a stand alone process or as a JEE application

The information used to populate the log database is generated, as MQ messages, by the FTE agents participating in file transfers. This is routed to a collection point in the MQ network, referred to as the ‘coordination queue manager’ (see next slides). The database logger component subscribes to the messages produced by agents and reliably enters them into a database.

NO

TES

Page 19: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation19

Components of a typical WMQ FTE networkCoordination Queue Manager

Gathers together information about events in the file transfer network

Not a single point of failure– Can be made highly available– Messages stored + forwarded

MQ v7 publish / subscribe– Allows multiple log databases,

command installs– Documented interface

Applications exchanging file data

WebSphere MQ

Agent Agent Agent

“Coordination”Queue Manager

LogDatabase

Commands

Page 20: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation20

Notes on:Coordination queue manager

The coordination queue manager is used as the gathering point for all the information about file transfers taking place between a collection of FTE agents

The queue manager uses publish/subscribe (so it must be MQ version 7) to distribute this information to “interested parties” which typically include:

– The WebSphere MQ Explorer plug-in, which provides a graphical overview of FTE activity

– The database logger component, which archives the information to a database– Some of the command line utilities which are part of the FTE product

The format used to publish information is documented and can be used to develop 3rd party applications which process this data

Although there is only a single ‘coordination’ queue manager for a given collection of agents it does not represent a point of failure:

– MQ stores and forwards messages to the coordination queue manager when it is available – so if the coordination queue manager is temporarily unavailable no log data is lost

– The ‘coordination’ queue manager can be made highly available using standard HA techniques such as MQ multi-instance queue manager or via a HA product such as PowerHA

NO

TES

Page 21: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

WebSphere MQ File Transfer Edition Tools

Command Line tools – For operations (for example transferring a file) – Administration (for example starting an agent)

GUI – eclipse based plug-in to MQ Explorer– Used for operations only (for example transferring files, browsing transfer log history)

Page 22: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM CorporationPage 22

How to start a File Transfer Command Line Interface is consistent across all

supported platforms Transfer commands can be invoked from the

supported Operating Systems shell environment Commands can be invoked from anywhere across

the file transfer network – i.e. Command could be invoked from a

Windows machine for transfers taking place between z/OS and Unix machines

Developers can use any native command line language on the OS that can invoke these commands (shell, bat, cmd, etc.)

Application Programs can place a request using a messaging interface in XML

Examples: fteCreateTransfer Starts a new file transfer from the command line fteStartAgent Starts a File Transfer agent from the command line fteStopAgent Stops a File Transfer agent in a controlled way fteShowAgentDetails Displays the details of a particular File Transfer agent fteShowAgents Displays the status of all known Transfer agents

Page 23: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Eclipse GUI integrated into WMQ Explorer

Page 24: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM CorporationPage 24

Scheduling & Triggering File Transfers

Choose when to start the scheduled transfer

Choose when to repeat the scheduled transfer and how often

Choose the trigger for the transfer

Choose advanced options

Deploy file transfer to network

Page 25: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM CorporationPage 25

Auditing & Monitoring File Transfers

Page 26: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation26

Agenda

Introduction

MQ FTE Overview

DataPower File handling Overview

File transfer scenarios with DataPower / MQ FTE

DataPower MQ FTE Configuration

Page 27: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

InternalEmployees

Wholesalers

Retailer

The Modern Enterprise is a Network of Complex Interactions

Affiliates

B2B

B2C

Page 28: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation28

Rich Portfolio of Workload Optimized Systems and Appliances

SecurityInfoSphere Guardium Real-time Database Monitoring and SecurityProventia Management SiteProtector SystemProventia Network Enterprise ScannerProventia Network Multi-Function SecuritySecurity Network Active BypassSecurity Network Intrusion Prevention SystemWebSphere DataPower XML Security Gateway

ConnectivityWebSphere Cast Iron Cloud IntegrationWebSphere DataPower B2B ApplianceWebSphere DataPower Integration

PerformanceWebSphere DataPower Caching ApplianceWebSphere DataPower Low Latency ApplianceWebSphere DataPower XML AcceleratorPower7 with WebSphere and DB2Smart Analytics Optimizer

ManagementApplication Manager for Smart BusinessIBM CloudBurstRational Power Appliance ExpressRational Power Appliance WorkgroupRational Power Appliance EnterpriseService Manager for Smart BusinessSmart Business ApplianceVirtual Desktop for Smart BusinessWebSphere CloudBurst Appliance

StorageInformation ArchiveSAN Volume ControllerScale Out Network Attached StorageSystem Storage Productivity CenterReal Time Compression Appliance

InformationNetezza Retail Analytic ApplianceNetezza SkimmerNetezza TwinFinSmart Analytics System

Page 29: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

SIMPLE

PURPOSE BUILT

WORKHORSE

An Appliance Analogy

Page 30: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

1) Take some characteristics from integration software stacks

2) Take some characteristics of networking & firewall devices

3) Combine them into a hardware device (aka hardware form factor) that has characteristics of both application stacks and networking gear.

• Programmability by developers • Message level routing and security (including by partner for B2B)• Integration with other SOA infrastructure & governance software such as Registry, Repository, etc…• Conditional Processing (if, else, split, join)• Transformation of any-2-any formats

• High throughput• Many concurrent connections• IP level security and routing (can’t see inside message but route by source and target)• Simplified Management and reduced maintenance costs through firmware updates• Reliability

The Connectivity Appliance Recipe

=

Page 31: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

IBM WebSphere DataPower appliances

Simple architecture: – microcode firmware + purpose-built hardware

Delivered from the factory with everything you need to connect to the network and start working

– No need to provision anything but the Ethernet network and CAT cables to get started

All computationally-significant components sealed within a temper-proof casing

– Chips– Memory– Boards and cards– Flash-based file system (signed and encrypted)– Parsing and xform accelerators (patented)– Cryptographic accelerators (patented)

Page 32: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation32

Why use WebSphere DataPower appliances? Business Requirements

• Configuration driven, not programmable• Relief from maintaining diverse development and system

administrator skills • Simplified IT deployments

Appliances are Fit for Purpose, Easily Consumable• A drop-in, “plug-n-work” device• Hardened, with numerous functions integrated into a single device • Easy configuration-driven set up and management • Purpose-built hardware / software that delivers on specific business

needs

Page 33: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

WebSphere DataPower XI52 and XB62 Appliances (9005)

• 2U high density rackmount design

• New customer-replaceable Ethernet modules with far more ports than the current generation (4)

– Two 10 Gbps ports– Eight 1 Gbps ports

Higher performance chips, memory, flash, and hard drives

Far more memory than ever before– Memory aggregation at the domain level

(announced for firmware 4.01)

Enhanced hardware diagnostics

Customizable intrusion detection

33

8 1-Gigabit Ethernet NICs8 1-Gigabit

Ethernet NICs

RAID mirroring and striping across four drives

RAID mirroring and striping across four drives

2 10-Gigabit Ethernet NICs

2 10-Gigabit Ethernet NICs

Page 34: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

DataPower appliances have been the best selling WebSphere product since the DataPower company acquisition in 2005

Guiding philosophy is to take rote, repeatable integration tasks and lock them down in the appliance form factor, including

– Parsing (XML, Flat)– Validation (XML, Flat)– Transformation (XML, Flat)

– Security (Transport, XML, other) – Integration (Multi-transport, SOA, ESB)

– B2B (Partner Mgt, Persistent Tx Store)

The “Russian Doll”

DataPower Devices

Page 35: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation35

Common usage patterns

Internet Trusted Domain

Partner Apps

Devices

3 B2B Gateway Integration Gateway functions + Partner Management Enhanced qualities of service

1 Web Gateway Security policy enforcement SLA policy enforcement Web Application Firewall

Applications

System z

DMZ

Web Apps 2 Integration Gateway Web Gateway functions + Any-to-any transformation Protocol Bridging

2.1 ESB Integration Gateway functions + Routing Database connectivity more...

2.2 Service Enablement ESB functions + Rapid web service façade creation

Page 36: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation36

Web Gateway

Proxying and Enforcement• Terminate incoming connection• Terminate transport-level security• Enforce Service Level Agreement policies• Inspect message content, filter, pattern-match• Enforce security policies on message content• Call out to Access Control List(s) • Detach binaries and call out to virus checker• Transform content (XSLT, XML-to-XML)• Establish a new connection to pass results

Connection from client

New connection to target

ACLVirus

Scanner

Partner App

Internal App

Page 37: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation37

Integration Gateway

All the same capabilities as the XS40 to proxy and protect traffic, PLUS…

Support for additional transport protocols:– FTP(s), SFTP(SSH), WebSphere MQ, WebSphere

JMS, TIBCO EMS, MQ FTE Database connectivity (ODBC) File system access (NFS) Web 2.0 standards: ReST, JSON Onboard engine for parsing and transforming

virtually any message format– WebSphere Transformation Extender (WTX)

WTX Design Studio

Page 38: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

B2B Gateway

All of the capabilities of the XS40 to proxy and enforce policies

All of the additional capabilities of the XI50 for protocol bridging and universal transformation, PLUS…

Partner Management functions:– Define partners with the web management console– Associate partners with network endpoints– Attach metadata about the partners to their definitions

Enhanced Qualities of Service– Onboard persistent transaction store– Search messages by partner, time, etc– Replay messages if necessary– ebXML/ebMS, AS1, AS2, and AS3 protocol bindings

for greater reliability across traditionally unreliable protocols

ebMS /

Page 39: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Business to Business (B2B) Appliance XB60 / XB62 Purpose-built B2B hardware for simplified deployment, exceptional performance and hardened security

• Extend integration beyond the enterprise with B2B

• Hardened Security for DMZ deployments• Easily manage and connect to trading partners

using industry standards• Simplified deployment and ongoing management

• Trading Partner Management for B2B Governance; B2B protocol policy enforcement, access control, message filtering, and data security

• Application Integration with B2B Gateway Service capabilities supporting B2B patterns for AS1, AS2, AS3, ebMS v2.0, FTP(S), SFTP, HTTP(S), SFTP/POP3, MQ, MQ FTE, JMS and more

• Full featured User Interface for B2B configuration and transaction viewing; correlate documents and acknowledgments displaying all associated events

• Simplified deployment, configuration and management providing a quicker time to value by establishing rapid connectivity to trading partners

Page 40: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Applicatio ns

DMZ Trusted Domain

XB60 complements IBM‘s family of WebSphere software solutions

Standalone Consolidated Patterns– Deploy standalone for purpose built B2B gateway functionality in the DMZ utilizing exceptional security and B2B transaction volumes for

quickly connecting to trading partners

Enterprise B2B Integration Pattern– Deploy with MQFTE for B2B enabled Managed File Transfer– Deploy with WTX-TM for end-to-end EDI Processing – Deploy as B2B entry point for BPM and ESB solutions– Supplement WPG or Sterling Integrator by offloading security functions and advanced web services

Internet

WebSphere DataPower XB60

WebSphere Partner Gateway / Sterling Integrator

WebSphere TransformationExtender / Trading Manager

WPS, WESB, WMB, DP

Partner

WebSphere MQ FTE

Page 41: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation41

Agenda

Introduction

MQ FTE Overview

DataPower File handling Overview

File transfer scenarios with DataPower / MQ FTE

DataPower MQ FTE Configuration

Page 42: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

XB60 MFT Integration Pattern – Integrate to MQ FTE for end-to-end file transfer

Browser (LOB User)

XB60

Tradin g Pa rtner

XB60

B2B Gateway Service

TransactionViewer

Multi-Protocol Gateway Service

ProfileMgmt

Data Store

Internet

Browser (Admin)

Browser (Partner view)

Server

MQ ExplorerFTE Viewer

WS MQ

Agent01

ServerWS MQ

Agent02

Data Store

Application

WAN

1

23

4b4a

65

Enterprise

Page 43: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation43

Message Segmentation for Large Files

...

...

...... ... ... ...

If the payload of the received message is too big, the message payload will be segmented into smaller pieces before forwarding to the queue

Can also be configured for streaming

Page 44: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation44

Outbound

PartnerFile TransferSender Source

Agent

MQFTE Network Destination

Agent Appliance

MQ queue

DataPower portion of transferMQ File Transfer Edition portion of transfer

Logical flow of data

1. MQFTE source agent receives the transfer and sends the file to the destination agent which shares queues with DataPower

2. Destination agent places file data as one or more messages in a group.

3. MQFTE front-side handler consumes message(s) from the queue, reassemble the message if there is more than one segments, and sends the data to the external partner

Message Message

Page 45: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation45

Agenda

Introduction

MQ FTE Overview

DataPower File handling Overview

File transfer scenarios with DataPower / MQ FTE

DataPower MQ FTE Configuration

Page 46: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Outbound from MQ FTE to DataPower

Appliance

Agent

...

MQ queue : FTE2DP

FTE Agent : APCQMGR : QMPC

Network

Page 47: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Using a File to Queue transfer to the FTE2DP queue

Appliance

Agent

...

MQ queue : FTE2DP

FTE Agent : APCQMGR : QMPC

Network

Page 48: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation48

MQFTE & DataPower XB60 Integration Headers

MQFTE Source Agent can optionally configure the following metadata when integrating with XB60.

When set, B2B Gateway will use these information to match the right partner profile settings.

• DPMQFTESenderID : the business ID of internal partner

• DPMQFTEReceiverID : the business ID of external partner

• DPMQFTEContentType : the content type of message payload

Page 49: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Appliance

Agent

...

MQ queue : FTE2DP

FTE Agent : APCQMGR : QMPC

Set up of the queue manager connections within DataPower

Network

Page 50: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation50

MQFTE Front Side Protocol Handler Configuration

The Queue manager that host the shared queue

The queue where we retrieve the message segments

Page 51: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation51

Transactionality Support

PartnerFile TransferSender Source

Agent

MQFTE Network Destination

Agent ApplianceGet Queue

DataPower portion of transferMQ File Transfer Edition portion of transfer

Logical flow of data

1. MQFTE source agent receives the transfer and sends the file to the destination agent which shares queues with DataPower

2. Destination agent places file data as one or more messages in a group.

3. MQFTE front-side handler consumes message(s) from the queue

4. Error happens in transaction processing and MQFTE front-side handler rolls the transfer back and. Go back to step 1 to retry the transfer.

5. If the number of retries reaches the “backout threshold”, then MQFTE front-side handler will send the transfer to the “backout queue”.

12 3

4 (error)

Backout Queue

.

Page 52: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Agent Network Agent

Appliance

...

MQ queue

Inbound to DataPower and MQ FTE

Network

Page 53: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Agent Network Agent

Appliance

...

MQ queue

The DataPower Multi-Protocol Gateway with FTE backend URL

dpmqfte://FTEWA/?RequestQueue=DP2FTE&DestAgent=APC&DestQM=QMPC&DestFile=testcomplete.xml

Network

Page 54: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation54

MQFTE URLOpener Parameters

Mandatory– DestAgent : the destination agent in MQFTE network that will ultimately receive the file– DestQM : the destination queue manager the source agent should send the messages to– DestFile : the filename to use when the destination agent need to store the received

messages into file. – Unique filename is supported.

Optional– SenderID : the business ID of sender (e.g. external partner )– ReceiverID : the business ID of receiver (e.g. internal partner)– ContentType : the content type of the message payload

Example : – dpmqfte://QM_Demo/?RequestQueue=Q_in&DestAgent=Kai&DestQM=dummyQM&DestFile=dummyPath

Page 55: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Nothing MQ FTE specific in the Policy

Page 56: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

Agent Network Agent

Appliance

...

MQ queue

The FTE Agent Monitor configured to accept routing from DataPower

agent="${DPMQFTEDestinationAgent}"

QMgr="${DPMQFTEDestinationQM}"

<file>c:\temp\${DPMQFTEDestinationFile}</file>

FTE Agent : AWAQMGR : FTEWA

Network

Page 57: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation57

View MQ FTE Transactions in the B2B Viewer(XB60 only)

1. Outbound gateway sends a B2B message to inbound gateway

2. The DataPower appliance writes data to an MQ queue as one or more messages in a group. The message ID assigned to the first message in the group becomes the Integration ID. Other information can be passed in the RFH 2 header of the first message

3. FTE consumes the group of messages and uses the information present in the first message in the group to route the data to a back end system, where it is written as a file

4. B2B Viewer of the outbound gateway retrieve the transaction metadata from the MQFTE Logger Database

Back-endSystemAgent Network Agent

Appliance

MQ queue

DataPower portion of transfer MQ File Transfer Edition portion of transfer

Outbound GW Inbound GW

MQFTE Logger4

Page 58: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation58

New Integration ID Column in B2B Viewer ( XB60 only)

A new column to store the integration ID which could be used to correlate a B2B Transaction with a backend transaction

For MQFTE integration, the integration ID maps to the “transfer ID” of MQFTE network

For inbound, the integration-ID is the Message ID of first message we sent to the shared queue

For outbound, the integration-ID is the Message ID of the first message we received from the shared queue

Page 59: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation59

MQ FTE Data View in XB60 - The metadata display popup windows

A pop-up to display the MQFTE metadata when the integration-id is clicked ( for MQFTE-related transaction only)

Depending on weather the transaction is inbound or outbound, the popup window will show different metadata

Page 60: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation

DataPower and MQ FTE solution benefits

Integration between the B2B appliance and backend application is over WebSphere MQ instead of a shared file system.

Files transferred between the B2B appliance and MQ FTE can be correlated using the integration ID from MQ; this ID can also be seen in the B2B Viewer.

The combined B2B messaging flow through the B2B appliance and file transfer flow through MQ FTE can be viewed through the B2B Viewer on the appliance. This provides the user with an end-to-end view of the file transfer.

File transfers can be set up to occur at specified times or dates, or repeated at specified intervals. File transfers can also be triggered by a range of system events, such as new files or updated files.

Page 61: Managed File Transfer Solutions using DataPower and ... - MQ02... · Managed File Transfer Solutions using DataPower and WebSphere MQ File Transfer Edition . ... Logging capabilities

© 2011 IBM Corporation