Managed  File  Transfer:  backbone  of  your  operationsChinmaya  Tripathy,  Integration  Consultant,  YenloRob  Blaauboer,  Integration    Consultant  YenloSenduran Balasubramaniyam,  Software  Engineer  WSO2

April  5th 2016

About  the  presenters  

2

Chinmaya TripathySenior  Consultant,  YenloChinmaya Tripathy is  an  Integration  Consultant  with  WSO2  knowledge  stack  along  with  9  years  of  experience  in  cross  domain  projects  and  technologies   like  Java,  J2EE,  WebServices,  Spring,  SOA  Architecture,  JS,  AJAX,  DOJO,  GWT.  He  has  a  Master  degree  in  Computer  Application,  followed  by  graduation  of  science   as  Mathematics   as  special  course.  

Rob  BlaauboerSenior  Consultant,  YenloRob  is  a  Senior  Business  Consultant  and  Solution  Architect  with  more  than  twenty  years  experience.   In  addition  to  his  work  he  is  an  active   blogger  working  on  a  number  of  articles  on  the  'Internet  of  Things'  and  a  WSO2  'Getting  Started  with  ...'  series   in  which  he  talks  about  WSO2  components  and  their  purpose  especially   aimed  at  non  technical  readers.

Senduran BalasubramaniyamSoftware  Engineer,  WSO2Senduran is  a  software  Engineer  at  WSO2  and part  of  WSO2  Enterprise  Service  Bus  engineering  team.

3

• Premier  Partner  of  WSO2

• Global  Organization

• Offices  in  the  Netherlands,  Germany,  Belgium,  United  Kingdom  and  United  States

• Experts  is  Integration  Solutions

• Experts  in  a  ‘Connected  Business’

• WSO2  project  &  consultancy  services

• WSO2  support   services:

• Product  Support• Development  Support• Operational  Support

• WSO2  Training  services

• Enterprise  &  Solution  Architecture

Who  we  are What  we  deliver

More  info  about  us  and  our  pre-­‐build   (WSO2)  solutions:  www.yenlo.com

Topics  Webinar

4

Managed  File  Transfer  =  Still  one  of  the  backbone’s  of  many  operations

• Introduction  to  ‘File  Transfer’  in  general

• File  transfer  in  WSO2  ESB  (Transports)

• What  about  Managed File  Transfer?

• Benefits of  our MFT  solution  based  on  WSO2

• Demonstration  &  Technical  details

• You  have  questions?  We  have  answers!

The  longevity  of  File  Transfer  

For  many  organizations  file  transfer  is  still  part  of  their  daily  operations,  for  example:

o Transfer  files  between  systemso Interact  with  business  partnerso Managing  Laws  (publishing)o Tax  returns  to  consumers

File  transfer  is  here  to  stay….

6

o Files  can  be  really  big,  up  to  several  GB,  and  really  small,  some  kilobytes,  for  example

Big  files  and  small  files

7

o Files  can  be  PDF,  XML,  Plain  text,  and-­‐so-­‐ono Files  can  be  encrypted  or  even  need  to  be  encrypted  

All  kind  of  files

8

o After  uploading  you  might  need  to  send  the  uploader  a  receipto Calculating  a  hash  might  be  necessary  for  audit  purposes

ab34e562acd

ReceiptWe have received…

Files  may  need  receipt  or  hash

9

o Check  for  malicious  content,  is  there  any  unwanted  payload?

ReceiptWarning! We have received…

Malicious  content

10

o Active/Passive  or  Active/Active  setupo Make  it  scalableo Making  sure  one  ESB  picks  up  the  file,  not  two  or  moreo Needs  to  be  a  dead  letter  channel  (EIP)

Setup  can  be  challenging

11

File  transfer  in  WSO2  ESB(Transports)

Transports  in  WSO2  ESB  

For  the  complete  list:  https://docs.wso2.com/display/ESB490/ESB+Transports

o HTTP/S  NIO  Transport  o HTTP  PassThrough Transporto HTTP/S  Servlet  Transporto JMS  Transport  o VFS  Transporto FIX  Transport  o and  more...

13

VFS  Transport

Supported  Protocols

o Local  File  Systemo FTPo FTPSo SFTPo Windows  Share  (SMB)

14

File  Transports  in  ESB

15

o VFS  Transporto File  Inbound  protocolo File  Connector

File  Handling  Components

VFS  Transporto File  processing  is  done  through  Virtual  File  System  (VFS)  

transport.o VFS  Transport  has  two  parts

o VFS  Transport  Listenero VFS  Transport  Sender

o Enabling  by  Uncommenting  the    following  entries  in  axis2.xml    

<transportreceiver name="vfs"  class="org.apache.synapse.transport.vfs.VFSTransportListener"/>

<transportSender name="vfs"  class="org.apache.synapse.transport.vfs.VFSTransportSender"/>

16

o This  is  a  much  improved  alternative  to  VFS  Transport  o Supports  multi-­‐tenancy  o Supports  coordination  in  clustered  environment  

File  Inbound

File  Connector

o Speed  file  copy

17

VFS  use  cases

18

HTTP  to  VFS

VFS  use  cases

19

VFS  to  HTTP

The  YenloManaged  File  Transfersolution  based  on  WSO2

o A  robust  MFT  solution  based  on  WSO2  ESB

o Many  out-­‐of-­‐the-­‐box  features  to  address  mentioned  challenges,  and  much  more…

o Added  high  availability  by  adding  2  or  more  ESB  nodes

So  Yenlo  created...

21

In  almost  all  cases  guaranteed  delivery  is  a  necessity.  The  Yenlo  Managed  File  Transfer  offers:

o SLA’so Possible  guaranteed  deliveryoDelivery  only  onceo Receipt  with  or  without  hasho Possibility  to  work  with  encrypted  files  

Guaranteed  Delivery

22

For  continuous  operation  a  HA  setup  is  necessary:oHA  in  Active/Passive  single  setupoHA  in  Active/Passive  group  setup  (2+  groups)

AP

AP

High  availability

23

Content  and  metadata  control

Making  content  based  routing  possible  using  simple  rules  based  on,  for  example:

o RegEXo Metadata  (timestamp)o Inclusion  and  exclusion  criteria  (size,  type,  extension)o Encryption  or  the  absence  of  encryptiono Virus  scanning

24

Clustered  setup

Managing  multiple  ESB’s  and  multiple  drop  directories

25

RegEx  Sample

Example  of  Metadata  based  routing  

26

Transport  variables  (current  version)

27

Future  developments

o Encryption  – Decryptiono Hash  creation  for  audit  purposeso Malicous Payload  scanningo Receipt  to  sender  /  uploader  o Potential  Validation  of  payload  (internal)  

28

Managed  File  TransferWhat  are  the  benefits?

Benefits  are  clear  

o High  availabilityo Multi  tenanto Monitoring  &  Tracingo Guaranteed  deliveryo MFT  is  a  service;  So  no  headaches  for  the  end-­‐user

30

Managed  File  TransferDemonstration  &  Technical  stuff

Demo  

o Running  on  local  machine:  two  WSO2  ESB’s  (4.9.0)o Clustered  (Hazelcast)  with  two  workers

o 10000  (dummy)  files  o 7  too  bigo 13  corrupto 8000  XML  fileso 2000  PDF  files  

o Estimated  time  to  finish:  less  then  120  seconds

32

Managed  File  TransferYou  have  questions?We  have  answers!

Contact  &  Download

34

Interested  in  our  Managed  File  Transfer  solution?

Download  this  presentation:https://www.yenlo.com/en/free-­‐advice/webinars

Or

Contact  us:https://www.yenlo.com/en/contact

35

THANK  YOU  FOR  YOUR  

ATTENTION