handling large file with managed file transfer and oracle's cloud services

Robert van Mölken - AMIS Services

Handling Large File with Managed File Transfer and Oracle's Cloud Services

Who Am I

• Robert van Mölken• Oracle Integration Specialist since ‘07• Beta tester, blogger, SOA SIG Lead

for the Dutch User Group and author of “Implementing Integration Cloud Service”

• Linkedin: linkedin.com/in/rvmolken • Blog: technology.amis.nl • Twitter: @robertvanmolken

Middleware Partner of the Year 2011, 2013, 2014 & 2015

EMEA Partner of the Year 2014

3

What will we discuss today?

Agenda

File Transfer Challenges

File Transfer Patterns

Real World Use Case with Cloud Integrations

Demo of On-Premises 2 Cloud Integration

4

The growing challenge with Managing File Transfers

Cloud introduces a new dimension of complexity…

On-premises File Transfers

Cloud File Transfers

5

File Transfer Challenges

• Multiple ways and entry points for receiving files• Uncontrolled proliferation (spread) of FTP servers or File shares• Not integrated with enterprise security standards• Single points of failure, not HA or reliable delivery• Complex and costly to maintain

• Handling large files – How do we not clog our Middleware environment?– How do we know a file is processed successfully?

• Different styles of transfers with On-Premises and Cloud applications• Varied protocol – (s)FTP, file, SOAP WS, REST API, Cloud Adapter

6

Managed File Transfer Overview

• Simple and Secure End-to-End Managed File Gateway– Support large files, PGP encryption, auditing, monitoring – pass-by-reference, extensible with custom call-outs

• Lightweight Web based Design Time Interface– Easily build, edit and deploy end-to-end transfers

• Standards Based Middleware & Cloud Integrations– (s)FTP, File, SOA, B2B, Webcenter Content, Storage Cloud

• Additional Characteristics– Scheduling, Embedded sFTP server, FTP proxy, no

transformations– Advanced Management: Pause, Resume, Resubmit

7

Oracle SOA Cloud Overview

Oracle Applications, SAP, customer, legacy…

ON-PREMISES

3rd PartyORACLE CLOUD

Salesforce

OracleSOA Cloud Service

Oracle SaaS Applications

OracleSOA Suite

• SOA Suite in the Cloud– Platform for Orchestration, application integration, API

management, virtualization, orchestration, streaming analytics, B2B, MFT, connectivity, business monitoring dashboards

– Fully portable, lift and shift from/to on-premise– Identical components to Oracle SOA Suite for faster

deployment portability

• Complete access to product surface area – Access to Weblogic console, Enterprise Manager & Filesystem– Rapid and fully automated provisioning– Secure, Highly Available with Clustering– Fully Managed i.e. easy patching– Build anywhere, deploy anywhere

8

Oracle Storage Cloud Overview

Oracle SOA Suite, Oracle Apps, customer, legacy…

ON-PREMISES

ORACLE CLOUD

Oracle PaaS & SaaS Applications

• Secure file storage in the Cloud– Secure, elastic, reliable, and cost-effective cloud storage– Access anywhere, 24/7, from any device– Replicated on multiple nodes

• Easy-to-use solution– Store, manage & consume large amounts of unstructered data– OpenStack Swift-compatible REST API or Java API– Web- based console for monitoring & management

• Role-based access control– Very granular level, folders of on individual files– Make data accessible publicly

OracleStorage Cloud Service

3rd Party

Cloudberry

OracleManaged File Transfer

9

Oracle Document Cloud Overview

Oracle SOA Suite, Oracle Apps, customer, legacy…

ON-PREMISES

ORACLE CLOUD

Oracle PaaS & SaaS Applications

• Content Collaboration in the Cloud– Intuitive & user-friendly web-based interface– Collaborate within or outside the company

• Anytime, Anywhere, Any Device– 24 X 7 access to files, folders, people and conversations– Access content from mobile devices, sync with Mac or PC or work off

web browser (and offline access).

• Engage and communicate– Presenter App to display presentations, PDFs, etc.– Social conversations about Cloud assets

• Application extensions– Sync data between cloud and existing content server

OracleDocument Cloud Service

3rd Party

Office 365

OracleManaged File Transfer

10

Overview Summary

• Managed File Transfer

• SOA Suite Cloud Service

• Storage Cloud Service

• Document Cloud Service

File Transfer Patterns

11

12

Stand-alone MFT Patterns

Source TargetMFT

• Direct – source to target

• Fan-out – source to multiple targets

• Chained – multiple MFT transfers

Source TargetMFT

Target

Target

Source MFT MFT Target

13

SOA Cloud Patterns

• SOA web services

• SOA based appplication

Source TargetMFT SOA

Source TargetSOA MFT

Source

Target

MFT Flow 1

SOAApplication

MFTFlow 2Source

Initiate instance

Mid-process receive

14

Storage Cloud Patterns

• Cloud File Back-up

• Automatic File Encryption

Source StorageCloudMFT

StorageCloud

StorageCloud

MFTPGP Encryption

Source

StorageCloud

MFT

Target

15

Document Cloud Patterns

• Cloud Collaboration

• Sync changes to back-end

Source DocumentCloudMFT

DocumentCloud TargetMFT

Web

Mobile

Social

Mobile

16

Hybrid Integration Patterns

• Hybrid Integration

Source MFT Target

MFT

Target

DocumentCloud

SOA

StorageCloud

Web

Mobile

Social

Real World Use Case

17

18

Use Case Overview

Combining on-premises File Handling with Cloud Collaboration & Back-up

• Files are supplied by on-premises CRM on File share

• MFT transfers files to SOA Cloud Service

• SOA Cloud Service composite App combines files to single ZIP

• MFT transfers ZIP file to Document CS and back-ups file to Storage CS

• Files can be accessed by anytime, anywhere.

19

Use Case Overview

Web

Mobile

Social

Source MFT

Source MFT

SOA MFT

DocumentCloud

StorageCloud

Web

App

On-Premises 2 Cloud IntegrationDefining MFT transfers for Batch

20

21

Check availability network share

Source MFT

Source MFT

TRG

IN

22

Check availability network share

Source MFT

Source MFT

TRG

IN

23

Check availability network share

Source MFT

Source MFT

TRG

IN

24

Define trigger source

Source MFT

Source MFT

TRG

IN

25

Define trigger source

Source MFT

Source MFT

TRG

IN

26

Define trigger source

Source MFT

Source MFT

TRG

IN

27

Define file source

Source MFT

Source MFT

TRG

IN

28

Define file source

Source MFT

Source MFT

TRG

IN

29

Define file source

Source MFT

Source MFT

TRG

IN

30

Define trigger target

MFT

MFT

SOA

31

Define trigger target

MFT

MFT

SOA

IN

TRG

32

Define trigger target

MFT

MFT

SOA

IN

http://cloudhost:8001/soa-infra/services/default/ProcessBatchService/receiveTrigger

TRG

33

Define file target

MFT

MFT

SOA

TRG

IN

34

Define file target

MFT

MFT

SOA

TRG

IN

35

Define file target

MFT

MFT

SOA

TRG

IN

http://cloudhost:8001/soa-infra/services/default/ProcessBatchService/receiveFile

36

Define & deploy trigger transfer

MFT

MFT

SOA

37

Define & deploy trigger transfer

MFT

MFT

SOA

TRG

IN

38

Define & deploy trigger transfer

MFT

MFT

SOA

TRG

IN

39

Define & deploy trigger transfer

MFT

MFT

SOA

TRG

IN

40

Define & deploy trigger transfer

MFT

MFT

SOA

TRG

IN

41

Define & deploy file transfer

MFT

MFT

SOA

TRG

IN

42

Define & deploy file transfer

MFT

MFT

SOA

TRG

IN

43

Define & deploy file transfer

MFT

MFT

SOA

TRG

IN

44

Define & deploy file transfer

MFT

MFT

SOA

TRG

IN

On-Premises 2 Cloud IntegrationSOA application for MFT batching

45

46

Inspecting SOA application

MFT

MFT

SOA

47

Inspecting SOA application

MFT

MFT

SOA

48

Inspecting SOA application

MFT

MFT

SOA

49

Inspecting SOA application

MFT

MFT

SOA

50

Inspecting SOA application

MFT

MFT

SOA

On-Premises 2 Cloud IntegrationDefining MFT transfers for Cloud integration

51

52

Define FTP source

SOA MFT

DocumentCloud

StorageCloud

53

Define FTP source

SOA MFT

DocumentCloud

StorageCloud

FTP

54

Define FTP source

SOA MFT

DocumentCloud

StorageCloud

FTP

55

Define FTP source

SOA MFT

DocumentCloud

StorageCloud

FTP

56

Define Storage Cloud target

SOA MFT

DocumentCloud

StorageCloud

57

Define Storage Cloud target

SOA MFT

DocumentCloud

StorageCloud

58

Define Storage Cloud target

SOA MFT

DocumentCloud

StorageCloud

One thing to keep in mind!

Import certificate of StorageCloud domain in keystore

59

Define Document Cloud target

SOA MFT

DocumentCloud

StorageCloud

60

Define Document Cloud target

SOA MFT

DocumentCloud

StorageCloud

• NO adapter available in MFT 12.2 yet!• Document Cloud provides REST API

Solution: • Build custom Java Callout that calls REST API

• REST API upload example:– POST https://

documents-amiscloud04.documents.us2.oraclecloud.com/documents/api/1.1/files/data

– Multipart HTTP request, one part JSON object with the parentID and the other part is the content of the file itself (Mime Attachment).

– { “parentID”: “GUID-of-folder”, “primaryFile”: “name-of-file.extenstion” }  

61

Define Document Cloud target

SOA MFT

DocumentCloud

StorageCloud

• GUID of folder can be found when sharing the folder

62

Define Cloud Transfer

SOA MFT

DocumentCloud

StorageCloud

63

Define Cloud Transfer

SOA MFT

DocumentCloud

StorageCloud

64

Define Cloud Transfer

SOA MFT

DocumentCloud

StorageCloud

65

Add Pre-processing Actions

SOA MFT

DocumentCloud

StorageCloud

66

Add Pre-processing Actions

SOA MFT

DocumentCloud

StorageCloud

67

Deploy Cloud Transfer

SOA MFT

DocumentCloud

StorageCloud

On-Premises 2 Cloud IntegrationTesting the Integration

68

69

Determine state of deployments

70

Transferring ZIP file to CloudSOA puts file on FTP folder

SOA MFT

DocumentCloud

StorageCloud

FTP

71

Transferring ZIP file to CloudSOA puts file on FTP folder

SOA MFT

DocumentCloud

StorageCloud

FTP

72

Transferring ZIP file to CloudMFT picks up file and processes

SOA MFT

DocumentCloud

StorageCloud

73

Transferring ZIP file to CloudMFT picks up file and processes

SOA MFT

DocumentCloud

StorageCloud

74

Transferring ZIP file to CloudInspect MFT instance

SOA MFT

DocumentCloud

StorageCloud

75

Transferring ZIP file to CloudInspect MFT instance

SOA MFT

DocumentCloud

StorageCloud

76

Transferring ZIP file to CloudInspect MFT instance

SOA MFT

DocumentCloud

StorageCloud

77

Transferring ZIP file to CloudInspect MFT instance

SOA MFT

DocumentCloud

StorageCloud

78

Transferring ZIP file to CloudInspect Storage Cloud Container

SOA MFT

DocumentCloud

StorageCloud

79

Transferring ZIP file to CloudInspect Storage Cloud Container

SOA MFT

DocumentCloud

StorageCloud

Summary

80

81

Summary

File Transfer Challenges

File Transfer Patterns

Real World Use Cases and Cloud Integrations

Demo of On-Premises 2 Cloud Integration

Pre-order: October 2016

82

Questions?

83