high performance cloud integration platform by dipanjan sengupta
TRANSCRIPT
High Performance Cloud Integration Platform
By Dipanjan Sengupta
High Performance Cloud Integration Platform
Cloud-Era Integration Challenges and Impact
Flexibility, Change Responsiveness
Adoption, On-Boarding and Time-to-Market
Latency and Performance Over Public Network
Managing Multiple Access Credentials
Near Real-Time Data Synch-Up and Replication
Challenges Impact
Operational Efficiency
Unified Governance
Performance
Flexibility
Security
Integration Strategy
Time-to-Market
Governance
Cost
DistributedApplications
Efficiency of Dynamic Provisioning
High Performance Cloud Integration Platform
Cloud-Era Integration Considerations
Performance— Wire Formats— Deployment Footprint
Communication Protocol— HTTP the de facto Communication Protocol
Low-Latency Execution— Speed of Execution On or By the Payload within the Integration Layer— Optimizing Serialization and Deserialization Requirements
On-Boarding, Adoption and Flexibility— Offering Services for Entity Data Synch-Up between On-Premise and Cloud-Based Systems, Pre-Built Flows,
Transformation Routines, Adapters for Client Side On-Premise Systems and Cloud-Based Systems, etc.— Meta-Data Driven Configuration for Flow Selection and Flow Filters
High Performance Cloud Integration Platform
What is Cognizant Cloud Connector™?
Cognizant Cloud Connector™ is a cloud-based, high-performance, service-oriented integration platform that helps connect on-premise and cloud-based applications
It is meant for enterprises that have:— multiple cloud accounts to synch-up and manage— large volume of bulk data to move to cloud applications— data replication requirements between on-premise data centers and cloud-based
applications
It can also be used to provide integration capabilities for indigenous applications offered in Software-as-a-Service mode on the cloud
High Performance Cloud Integration Platform
What is Cognizant Cloud Connector™?
High Performance Cloud Integration Platform
Novelties
Very Fast Transfer of Data— Supports wire formats which reduce the payload size by as much as 70% of traditional formats
Integration Ready: Entity-Oriented Integration Approach — Offers Prebuilt Entity Services (like Customer, Order, etc.) for Integration
Very Low Latency Flow Execution
Industry Ready: Industry Standard-Based Message Format Support — Supports Any Domain-Specific Message Formats and Domain-Specific Services
Service Provider Ready: Built-In Transformation from Standard Formats to Software-as-a-Service Provider Format
Meta-Data Driven Flow Filtration and Routing
Rapid Provisioning: Very Low Deployment Footprint
High Performance Cloud Integration Platform
Built on Best-of-Breed Open Source Technology StackProducts from WSO2
• WSO2 Governance Registry:We have plugged-in custom modules into the WSO2 Governance Registry to include support for (A) specific artifact types such as Integration Services, Integration Flows, Data Transformations, etc. (B) service subscription management in a multi-tenant environment, and (C) advanced artifact search capability using semantics
• WSO2 Business Activity Monitor: We have extended the WSO2 Business Activity Monitor to build customized monitoring dashboards and reports with the data coming from our existing Apache Cassandra-based metrics storage repository. Cloud Connector uses Apache Cassandra for keeping it’s monitoring related information such as call counts, error counts , error information etc. and provides a portal for displaying them through graphs. We have extended the monitoring functionality by hooking in to Apache Cassandra and adding new graphs through gadgets
• WSO2 Identity Server:There are two ways how identity management, authentication and authorization are handled by Cloud Connector. We can either use our built-in services for Authentication, Authorization, Policy, Group Membership, etc. and related administrative portal to configure the same or we can plug-in calls to an external identity server – like the WSO2 Identity Server – in the request-response ‘pipeline’ to handle the same. Cloud Connector architecture is flexible enough to adhere both through configuration changes and writing handlers
High Performance Cloud Integration Platform
Architecture
High Performance Cloud Integration Platform
Use Case: MDM-on-Cloud™
Scenario DescriptionA. MDM-on-Cloud™ is an enterprise class Software-as-a-
Service-based MDM solution from Customer Solutions Practice of Cognizant Technology Solutions
B. Offers tenants a single view of enterprise entities across their various on-premise and cloud-based applications (salesforce.com)
Challenges Faced
A. Batch file transfer mechanism supported evoked security concerns
B. Inefficient file transfer support with no recovery mechanism
C. Poor query response cycle-timeD. Non-standard integration with salesforce.comE. Limited access control support
A. Cloud Connector provides the integration gateway for MDM-on-Cloud™
B. Batch file transfer supported over firewall-friendly portsC. Parallelism introduced in handling batch file transfer by
intelligently splitting the batch files at the source and appropriately sequencing and aggregating them at the target
D. Use fast wire formats for real-time service invocationsE. Standard access to salesforce.com through prebuilt
Adapter Services
Solution Offered
A. Allays customer concerns on securityB. Increase operational efficiencyC. Improve performance over public networkD. Flexible and elegant architecture offering superior
governanceE. Support multiple access control mechanisms
Business Benefits
High Performance Cloud Integration Platform
Use Case: MDM-on-Cloud™
Demonstration
High Performance Cloud Integration Platform
ArchitectureIntegration Platform for MDM-on-Cloud™
Backup Slides
Area Challenge Solution
High Performance Cloud Integration Platform Use Case: Challenges and SolutionsIntegration Platform for Cognizant MDM-on-Cloud™
Batch Integration No HTTP/S support was available for transferring batch files from on-premise systems to MDM-on-Cloud™. Using FTP protocol required opening FTP ports over internet increasing security vulnerability
Cloud Connector supports file transfer over HTTP/S. As a result, opening any non-HTTP ports (like FTP) is not needed
No recovery mechanism was present in case of transferring large files from on-premise systems to MDM-on-Cloud™, that allowed re-transmission from the failure point. The entire file needed to be resent, which was certainly not an efficient way of transmitting files over public network
Cloud Connector provides a file adapter that is capable of splitting a large files into multiple smaller chunks and transferring them reliably over HTTP/S protocol. In case of failure, only the chunk that failed needs to be resent
Real-Time Integration Support for multiple data formats and communication patterns was not present Cloud Connector supports a variety of data formats, besides SOAP and XML (REST). It also supports different fast formats that can be leveraged by on-premise applications to communicate in a much more faster manner over public networks. Cloud Connector also supports various communication patterns like point-to-point, publish-subscribe etc.
Point to point invocation of the MDM-on-Cloud™ services caused tight-coupling between the tenant applications and MDM-on-Cloud™ services
Cloud Connector provides a service abstraction layer in front of MDM-on-Cloud™ services, thereby making the provider and consumer loosely coupled
Security Management It was difficult for MDM-on-Cloud™ to store and manage different sets of security credentials for each tenant, specifically when the workflow required to invoke one or more third party application provider services (like salesforce.com, netsuite.com, sugarcrm.com, etc.)
Cloud Connector externalizes the security credential management for MDM-on-Cloud™. It also supports multiple authentication standards like SAML, OAuth, OpenID, etc, simplifying integration to different kinds of cloud-based applications and tenants
High Performance Cloud Integration Platform
Fast Format Benchmark Run Results
Serialization Time(in Micro Seconds)
1K 10K 100K 1M
Protobuf 79 86 435 7332
Protostuff 63 72 238 3288
JacksonJSON 944 862 1184 5249
Avro 396 340 485 7388
Thrift 77 137 1026 19875
XML 3340 3304 4545 13866
FastInfoset 432 487 1789 17222
MsgPack 60 105 898 18677
GsonJSON 647 802 3833 54948
JeMsonJSON 4100 4808 21126 179256
Deserialization Time
(in Micro Seconds)
1K 10K 100K 1M
Protobuf 51 52 200 2554
Protostuff 59 54 371 5398
JacksonJSON 1125 1051 1325 5872
Avro 251 217 285 3437
Thrift 96 92 640 12086
XML 4012 3927 5553 18631
FastInfoset 3126 3130 3983 16494
MsgPack 112 108 706 12764
GsonJSON 704 836 3332 48476
JeMsonJSON 3358 3591 7302 47625
TOTAL TIME
(in Micro Seconds)
1K 10K 100K 1MProtobuf 130 138 635 9886Protostuff 122 126 609 8686JacksonJSON 2069 1913 2509 11121Avro 647 557 770 10825Thrift 173 229 1666 31961XML 7352 7231 10098 32497FastInfoset 3558 3617 5772 33716MsgPack 172 213 1604 31441GsonJSON 1351 1638 7165 103424
JeMsonJSON 7458 8399 28428 226881
SERIALIZED SIZE
(in Bytes)
1K 10K 100K 1MProtobuf 3106 5326 58655 578692Protostuff 3105 5325 58656 578553JacksonJSON 3739 6410 70536 695831Avro 3003 5149 56658 558931Thrift 1505 13956 130564 1296773XML 5814 9929 108684 1071478FastInfoset 3403 5780 62827 619291MsgPack 1708 11962 111939 1112197GsonJSON 3708 6365 70120 691815
JeMsonJSON 3377 5768 63186 622847
CONCLUSIONS:
A. JacksonJSON and Protostuff are respectively the most optimum text and binary formats
B. Thrift performs sub-optimally for heavy payloads