abl data integration -...
TRANSCRIPT
© 2014 Progress Software Corporation. All rights reserved. 2
Agenda
Challenges of Data Integration
ABL Data Integration Methodologies
Loose Versus Tight Coupling at the Point of Integration
ODBC Bridge
Future Data Abstractions, Embedded Objects and Tight Integration
© 2014 Progress Software Corporation. All rights reserved. 4
The Challenges of Data Integration
Form: Consumed, hosted, (un)structured, Integrity, Volume, Velocity, Variability
Approach: Technology/cost, scalability, efficiency, throughput, availability, data analysis
Business Climate: Customer/partner demands, competition, mergers, divestments
© 2014 Progress Software Corporation. All rights reserved. 5
The Challenges of Data Integration
Integration
Transaction-Oriented OpenEdge Applications Are About Data
Discovering, cleansing, monitoring, transforming, aggregating and delivering data from disparate sources to where it is
needed, when it is needed
© 2014 Progress Software Corporation. All rights reserved. 8
Traditional ABL Language Binding: Relational Data Access
FIND FOR EACH UPDATE
OpenEdge Database
ABL Client
OpenEdge Architecture
DELETE CREATE
Homogeneous Managed Relational Data Source
CAN-FIND AVAILABLE
IMPORT EXPORT
COPY-BUFFER DEFINE QUERY/DATASET
© 2014 Progress Software Corporation. All rights reserved. 9
Traditional ABL Language Binding: File Distribution & Streaming
IMPORT / EXPORT INPUT FROM / OUTPUT TO PUT
OpenEdge Architecture
INPUT THROUGH OUTPUT THROUGH
HeterogeneousUnmanaged Data Stores
ABL Client
© 2014 Progress Software Corporation. All rights reserved. 10
Traditional ABL Language Binding: Relational Data Access via “Gateways”
OpenEdge Schema Holder
Database
OpenEdge DataServers 3rd
Party Database
OpenEdge Architecture
FIND FOR EACH UPDATE
DELETE CREATE
CAN-FIND AVAILABLE
IMPORT EXPORT
COPY-BUFFER DEFINE QUERY/DATASET
Drivers
ABL Client
Heterogeneous Managed Data Sources
© 2014 Progress Software Corporation. All rights reserved. 11
Traditional ABL Language Binding: Era of Web 1.0
FIND FOR EACH UPDATE
WebSpeed Agents
WebSpeed Broker
Web Browser
Web Server
WebSpeed Messenger
OpenEdge Architecture
3rd Party Products
DELETE CREATE
CAN-FIND AVAILABLE
IMPORT EXPORT
COPY-BUFFER DEFINE QUERY/DATASET
Heterogeneous Managed Data Sources
OpenEdge Database
OpenEdge Schema Holder
Database
DataServer and/or driver
3rd Party Database
Heterogeneous Unmanaged Data Stores
IMPORT / EXPORT INPUT FROM / OUTPUT TO PUT INPUT THROUGH OUTPUT THROUGH
OpenEdge SQL Client
JDBC ODBC
ABL Client
Custom GUI Objects
ActiveX components
COM Objects
COM-HANDLE – Automation Obj. COM-HANDLE – ActiveX Control CONTROL-FRAME (legacy DDE)
.dll’s and .so’s
© 2014 Progress Software Corporation. All rights reserved. 12
Non-Traditional, Non-Language Binding: Era of Web 2.0
WebSpeed Agents
AppServer Broker
WebSpeed Broker
WebClient
Web Server
WebSpeed Messenger
AppServer Agents
AIA
WS/REST Adapters
Java Servlet Engine
Sonic ESB SonicMQ SonicMQ
Adapter
Sonic ESB Adapter
Nam
eSer
ver
Web Services
Web Browser
ABL Client OpenEdge Multi-tenant Database
OpenEdge Schema Holder
Database
DataServer and/or driver
3rd Party Database
COM Objects
.NET Framework
(e.g.,System.Data.SQLClient) .NET/Java Open Client
Custom GUI Objects
components
.NET objects
Progress.Data.BindingSource Object Progress.Windows.Form USING System.Windows.Forms
DEFINE DATASET … NESTED USING System.Data.SQLClient USING System.Data.BindSource
components
.NET objects
.dll’s and .so’s
OpenEdge Architecture
3rd Party Products
3Rd Party Web Service
OpenEdge SQL Client
JDBC ODBC
© 2014 Progress Software Corporation. All rights reserved. 13
Language-Based Network Interface: Message-Based Data Access
WebSpeed Agents
AppServer Broker
WebSpeed Broker
WebClient
Web Server
WebSpeed Messenger
AIA
WS/REST Adapters
Java Servlet Engine
Custom Protocol
Entity
Nam
eSer
ver
OpenEdge Architecture
3rd Party Products
Web Services
3Rd Party Web Service
CREATE SERVER-SOCKET CREATE SOCKET GET/SET-SOCKET-OPTION
CONNECT/DISCONNECT READ/WRITE READ-RESPONSE
Network Data Exchange
Sockets
Custom Protocol
Entity
Sockets
Web Browser
.NET/Java Open Client
Custom GUI Objects
ABL Client components
.NET objects
AppServer Agents
OpenEdge SQL Client
JDBC ODBC
components
.NET objects
Data
Data
© 2014 Progress Software Corporation. All rights reserved. 14
Web Services Data Transport (Heavyweight)
WebSpeed Agents
AppServer Broker
WebSpeed Broker
WebClient
Web Server
WebSpeed Messenger
pServer nts
AIA
WS/REST Adapt
Java rvlet gine
OpenEdge DataServers
3rd Party
Database
Sonic ESB
SonicMQ SonicMQ Adapter
Sonic ESB Adapter
Nam
eSer
ver
OpenEdge Architecture
3rd Party Products
Web Services
Custom Protocol
Entity
Sockets
OpenEdge Schema Holder
Database
Custom Protocol
Entity
Sockets
.NET Framework
Web Browser
.NET/Java Open Client
Custom GUI Objects
ABL Client
.NET objects
OpenEdge SQL Client
JDBC ODBC
.NET, COM,
.dll’s & .so’s
OpenEdge Multi-tenant Database
3Rd Party Web Service
XML WSDL
HTTP/ SOAP
HTTP/S AJAX HTML
JavaScript JSON/XML
WS
© 2014 Progress Software Corporation. All rights reserved. 15
REST Services Data Transport (Lightweight)
WebSpeed Agents
AppServer Broker
WebSpeed Broker
WebClient
Web Server
WebSpeed Messenger
pServer nts
AIA
WS/REST Adapt
Java rvlet gine
OpenEdge DataServers
3rd Party
Database
Sonic ESB
SonicMQ SonicMQ Adapter
Sonic ESB Adapter
Nam
eSer
ver
OpenEdge Architecture
3rd Party Products
Web Services
Custom Protocol
Entity
Sockets
OpenEdge Schema Holder
Database
Custom Protocol
Entity
Sockets
.NET Framework
Web Browser
.NET/Java Open Client
Custom GUI Objects
ABL Client
.NET objects
OpenEdge SQL Client
JDBC ODBC
.NET, COM,
.dll’s & .so’s
OpenEdge Multi-tenant Database
3Rd Party Web Service
JSON WADL
HTTP/ REST
HTTP/S AJAX HTML
JavaScript JSON/XML
WS
© 2014 Progress Software Corporation. All rights reserved. 16
Reference Architecture
Common Infrastructure
WebSpeed Agents
Brok
WebSpeed Broker
Web
WebSpeed Messenger
AppServer Agents
AIA
WS/REST Adapters
Java ervlet
ine
Sonic ESB SonicMQ SonicMQ
Adapter
Sonic ESB Adapter
meS
er
OpenEd Architec e
3rd Party Products
Web Services
3Rd Party Web
Custom Protocol
Entity
Sockets
.NET Framework
OpenEdge DataServers 3rd
Party Database
OE Schema Holder
Database
Web Browser
.NET/Java Open Client
Custom GUI Objects
ABL
jects
en
OpenEdge SQL Client
JDBC ODBC
OpenEdge enant
Database
Business Components Data Sources Data
Access
Presentation
Enterprise Services
© 2014 Progress Software Corporation. All rights reserved. 17
Web Server
Java Servlet Engine
Sonic ESB SonicMQ
3rd Party Products
Web Services
Web Browser
.NET/Java Open Client
Custom GUI Objects
OpenEdge Architecture
Reference Architecture
Common Infrastructure
Presentation
Enterprise Services
Business Components Data Sources Data
Access
3Rd Party Web Service
© 2014 Progress Software Corporation. All rights reserved. 18
Web Server
Java Servlet Engine
3rd Party Products
Web Services
Web Browser
.NET/Java Open Client
Custom GUI Objects
OpenEdge Architecture
Reference Architecture
Common Infrastructure
Business Components
Data Sources
Data Access
Presentation
Enterprise Services
Business Components Data Sources Data
Access
3Rd Party Web Service
Node.js
HTTP, REST, SOAP XML, JSON, Odata
© 2014 Progress Software Corporation. All rights reserved. 20
The SaaS Argument to Application Integration
SOA-2: Event-Driven Service Oriented Architecture
External data flow
Internal data flow
Pending data flow
System appears twice NAME
Planned systems
Key:
JOB MOBE
TMM
NI
DFG
PVI CID/SAM
LOC
INPLANS
ISIS
ASOS
Mech Eng
TESS
CCP
TOPS
NRSS
Billstar 3
Billstar 1 PCDB
EC
RIMS PRECISE Pay by
Phone
DOMS Customizer 800
Custom 800
DB
RCRMS
BOSS COR
OSMOP LIDB
E911
NAA
LMOS
Exch Plus
APTOS
PREMIS
MI S arwriter
CMS (CCRS)
LFACS
FWS
PICS /DPCR
TNDS/TK
LEIS
WM
COSMOS
SOAC
PAWS
LOMS
FIRST
ORGIS
GIR BAIF
IRSS SDID
SDDL POF
SOCS PDR
OR
AOG
SORD
CRMS
EARS MAPS
CESAR
CABS
Data Svc
PR
OUR
GL
RAP
3rd Pty
B ll Print MP USAGE
RM REVE Bill Format
Bill Day
Billing
Directory Delivery
Corp Books
MI
CL CONF EM EXCH
List ng Svc
C/CA
SOFE
LSD&C
Da a
Warehse FIMS
PARIS
CARTS
COR
SUMMI 4 0
SBIR
MRDB
Sales Agency
ORBI S Athena
AA PBCC
SMS
DRS DCN
CPNI Sales Comp PB Awards
APTOS
PaSS
COIN
AP
PBVS
CSFT
IP
SPACE
MARCH
CSTAR
Separation
ConnectVu
CLONES
TNM NMA F
DCOS 2000 Net t
SEAS
EADAS NDS DE
AMOS NSDB
TIRKS
IPMS
MOPICS PMM
CM S F DM
SARTS
PVS | PMI
REACT 2001
N AS M AS
LA IS
CRAS CIAS
ANS
LMOS
MLT
SORD
SOAC
PBOD
Service
Manager
E ectron c
Bond ng
ALRU
PDS ERA
PDS
Customer Profile
AIM
POS R
POS
BRIS
REMS
ESS
EmFiSys
TAPS
AUTS
TRAINS
W S
IFS TWIST
A R
NSDM
MTR
TCMS
ech PDP
PMIS
Paging CNR
PBITS
MP/F
FLEXCOM
COSMOS
WFA/C
OPAS /Loopview
OPS/INE
Predictor
INA ransport
SSI PBRIMS IPMS CCSN TAN CCPL MP/F
ComnLang askmate
SCS FEPS
CUR/CAR
AGS
FDOC
Network
AT&T
Network
DSC NOR AT&T
Network
NOR
STP SCP ISCP
EDIIS
PB1
Network
AT&T
NOR
IS
ERMIS
Common Interface Layer
ATC
CIDB
Advantage
TIRKS
Tight coupling of applications acts as a barrier to agility
© 2014 Progress Software Corporation. All rights reserved. 21
Coordination Service
Transactions in an SOA World
Service 1 Service 2 Service 3
Coordinator Service
(1) Call (2) Call
(3) Change DB
(4) Return (5) Call
(6) Fails
(7) Return Fail
(8) Call Compensation (9) Apply
Compensation
(10) Return (11) Return
Failure
© 2014 Progress Software Corporation. All rights reserved. 22
My Contention…
Key Question:
1. How will you organize your SOA environment to reach your goals?
• How will you distribute consumers and providers of your services
2. What “services” and non-services operate against your key business components?
• Data Integration
• Data Transformation
• Data Persistence
• Data aggregation
• Data Storage/Access
Essential business resources and drivers of operational decisions need tight binding
© 2014 Progress Software Corporation. All rights reserved. 24
Data Sources
Web Server
Java Servlet Engine
Sonic ESB SonicMQ
3rd Party Products
Web Services
Operational Data Store
Big Data Engine
Web Browser
.NET/Java Open Client
Custom GUI Objects
OpenEdge Architecture
Reference Architecture
Common Infrastructure
Presentation
Enterprise Services
Business Components Data Sources Data
Access
3Rd Party Web Service
ODBC BRIDGE
© 2014 Progress Software Corporation. All rights reserved. 25
Web Server
Java Servlet Engine Web
Services
3Rd Party Web Service
Web Browser
.NET/Java Open Client
Custom GUI Objects
OpenEdge Architecture
Reference Architecture
Common Infrastructure
Presentation
Enterprise Services
Business Components Data Sources Data
Access
Sonic ESB SonicMQ
3rd Party Products
ODBC BRIDGE
Business Components
Data Sources
Data Access
© 2014 Progress Software Corporation. All rights reserved. 26
ODBC Bridge to Progress DataDirect Cloud and Progress Easyl
Code Samples • Available August 2013 in OE 11.3.0 for D2C
• Available March 2014 in OE 11.3.2 for EASYL
ODBC API linked/mapped; API functions in include file: “SqlStatementProto.i” ODBC processes are object-ized into OOABL
• SELECT queries return schema that is evaluated
• Full CRUD possible but not provided in sample
Results use default data type mappings Can assign results to temp tables or JSON objects No specific internationalization, LOB processing, or native ODBC extensions, etc. All API’s “available”; Only API’s needed by same are “implemented”
© 2014 Progress Software Corporation. All rights reserved. 27
ODBC Bridge Code in PDSOE
oD2CServer:ExecuteStatement
© 2014 Progress Software Corporation. All rights reserved. 28
ODBC Bridge Code in PDSOE
oD2CServer = New ODBCConnection(<connect data>). /* conn obj */
oD2CServer:ExecuteStatement(<stmt>,<result-obj>)./* run SQL */
If <result-obj> then
<result-obj>:WriteFile(<json-file>). /* or temp-table */
© 2014 Progress Software Corporation. All rights reserved. 29
Web Server
Java Servlet Engine Web
Services
3Rd Party Web Serv
Web Browser
.NET/Java Open Client
Custom GUI Objects
OpenEdge Architecture
Reference Architecture
Common Infrastructure
Presentation
Enterprise Services
Business Components Data Sources Data
Access
Sonic ESB SonicMQ
3rd Party Products
Business Components
Data Sources
Data Access
Bus
ines
s
Com
pone
nts
Dat
a S
ourc
es
Dat
a A
cces
s
Operational Integration
Analytic Integration
JSDO
XML/JSON ProDataSet TempTable TableObject
JSDO
© 2014 Progress Software Corporation. All rights reserved. 30
Web Services Front End and Business Entities Behind the Firewall
DEFINE DATASET-HANDLE pDataSet pDataSet:ReadXML pDataSet:WriteXML
pDataSet:NameSpace-Uri pDataSet:ReadJSON pDataSet:WrteJSON
Web Server
Java Servlet Engine Web
Services
3Rd Party W b Serv
Web Browser
.NET/Java Open Client
Custom GUI Objects
OpenEdge Architecture
Business Components Data Sources D
Access
Sonic ESB SonicMQ
3rd Party Products
Operational Integration
Analytic Integration
JSDO
XML/JSON ProDataSet TempTable
Object SDO
© 2014 Progress Software Corporation. All rights reserved. 32
Disclaimer
This presentation is for informational purposes only. You are cautioned that any information contained in this presentation may change in the course of product development.
This presentation may not be interpreted as any commitment on behalf of Progress, and future development, timing and release of any products, features or functionality described in this presentation remains at the sole discretion of Progress.
© 2014 Progress Software Corporation. All rights reserved. 33
Next Steps in “OpenEdge-Centric” Data Integration
1. Do Nothing
2. Improve Samples
3. OOABL built-in object supplements & embedded API
• Expose Abstractions as built-ins (using internal OTM)
4. OOABL built-in objects with language integration & embedded API
• Modernized, flexible, DataServer-equivalency + more
© 2014 Progress Software Corporation. All rights reserved. 34
Object Model Representation of ODBC Base Classes
DataDirect.DO.ODBC.API • Open Standards, Data Access Methods
DataDirect.DO.ODBC.Defs • Open Standards, Data Access Properties
Progress.Lang.SysError Progress.DO.ODBC.Error
includes getSQLState() method
CLASS myAPIObj INHERITS (or IMPLEMENTS) API: API:SQLSetStmtAttr(…). ….. END CLASS.
© 2014 Progress Software Corporation. All rights reserved. 35
Abstraction Layer on Top of Base Classes (Similar to Sample Program’s)
Embedded API’s as built-in objects • DataDirect.DO.RecordSet – Any result set definition
• DataDirect.DO.Connection – Any connected data source
• DataDirect.DO.Command – Any SQL call or statement(s)
ConnObj:Open(User, Pwd, DSN). RecordSetObj = CommandObj:ExecuteStatement(“Select * from Customer”). Do WHILE RecordSetObj:HasRecs: CREATE tt.
ASSIGN tt.name = RecordSetObj:column(“name”):VALUE. RecordSetObj:MoveNext NO-ERROR END.
© 2014 Progress Software Corporation. All rights reserved. 36
Disadvantages: Opt.1: Samples Thru Opt. 3: OOABL Built-In Object Supplements
Code practices on embedded object no correlation to existing code or practices Foreign schema references not tied to the language
• No schema/reference checks; mismatches result in corruption?
Integration capabilities data bound copies; • All CRUD by copy
Purposed Objects provide varying degrees of abstraction & function • Roll your own ODBC would be available but would require foundation work from you
No Integration Transactions, record scope, cursor management, etc. Only default mappings for derived temp-table, JSON, other targets
• No data conversion on results
Internationalization, LOB types, native extensions, language hooks loosely possible over time; must be “hacked” into ABL language.
© 2014 Progress Software Corporation. All rights reserved. 37
Foundational Abstractions for Built-In Objects With Language Binding
DataDirect.DO.DB – logical database equivalent
DataDirect.DO.Table – file equivalent
DataDirect.DO.Column – field equivalent
DataDirect.DO.Index – index equivalent
DataDirect.DO.DS – DATA-SOURCE equivalent
© 2014 Progress Software Corporation. All rights reserved. 38
Query Example Using Dynamic Schema Object Definitions
DEFINE PRIVATE VARIABLE myCouldDbObj AS CLASS DataDirect.DO.DB NO-UNDO. DEFINE PRIVATE VARIABLE custObj AS CLASS DataDirect.DO.Table NO-UNDO.
myCloudDbObj:GET-EMPTY(“CloudDSN”); myCloudDbObj:SET-CONNECT((“CloudDSN”, “sports”, UserID, Pwd). myCloudDbObj:LOAD-SCHEMA() custObj = NEW DO.TABLE(myCloudDbObj:GET-TABLE-SCHEMA(“ora-cust”)). /* foreign name */
CREATE BUFFER bh FOR TABLE (custObj.GET-LOCAL-NAME()).
CREATE QUERY qh.
qh.SET-BUFFERS(bh). qh:QUERY-PREPARE(“FOR EACH “ + custObj.GET-LOCAL-NAME()). qh:QUERY-OPEN(). qh:GET-FIRST(). qh:QUERY-CLOSE().
Bh:BUFFER-RELEASE(). DELETE OBJECT bh. DELETE OBJECT qh.
© 2014 Progress Software Corporation. All rights reserved. 39
Query Example Using Static Object Definitions & Database Persistence
DEFINE PRIVATE VARIABLE mydbObj1 AS CLASS DataDirect.DO.DB NO-UNDO. DEFINE PRIVATE VARIABLE mydbObj2 AS CLASS DataDirect.DO.DB NO-UNDO. myClouddbObj1 = new DB(“CloudDSN1”, “sports1”, UserID, Pwd). myClouddbObj2 = new DB(“CloudDSN2”, “sports2”, UserID, Pwd). myClouddbObj1:LOAD-SCHEMA(“Cust*”). myClouddbObj2:LOAD-SCHEMA(“Ord*”). myClouddbObj1:CREATE-DB(). myClouddbObj2:CREATE-DB().
prodb –db CloudDSN1 –ld “sports1” –db CloudDSN2 –ld “sports2”
DEFINE QUERY CustOrd FOR sports1.customer, sports2.order OPEN QUERY q FOR EACH sports1.customer, EACH sports2.order OF sports1.customer. GET FIRST q DO WHILE NOT QUERY-OFF-END(‘q’). GET NEXT q END CLOSE QUERY q
© 2014 Progress Software Corporation. All rights reserved. 40
Transaction Example Using Static Schema Object Definitions
DEFINE PRIVATE VARIABLE myClouddbObj AS CLASS DataDirect.DO.DB NO-UNDO. myClouddbObj = new DB(“CloudDSN”, “sports”, UserID, Pwd). myClouddbObj:LOAD-SCHEMA(“Customer”)
SAVE CACHE COMPLETE myClouddbObj:ALIAS-NAME TO VALUE(myClouddbObj:DB-NAME) + “.csh”) NO-ERROR. IF ERROR-STATUS:ERROR THEN MESSAGE “Not Saved”.
prodb –cache CloudDSN.csh /* proutil <database> -C load schema <filename> - db */ TransBlock:
DO TRANSACTION ON ERROR UNDO, LEAVE DO FIND customer. ASSIGN customer.cust-num = 12. UPDATE customer. END. END.
© 2014 Progress Software Corporation. All rights reserved. 41
DEF VAR cmdObj AS CLASS DataDirect.DO.Command.
cmdObj:SET(“UPDATE customer SET cust-num = 12”). cmdObj:EXECUTE().
cmdObj:SET(“SELECT * from customer”). cmdObj:EXECUTE().
DEF VAR custObj AS CLASS DataDirect.DO.Table. custObj:Update(SET cust-num = 12”).
custObj:FILL.
DEFINE QUERY Cust FOR sports.customer.
OPEN QUERY q FOR EACH sports.customer
DEFINE QUERY Cust FOR sports.customer.
OPEN QUERY q Select * from sports.customer
DEFINE QUERY qCust FOR sports.customer.
DEFINE DATA-SOURCE srcCust FOR QUERY qCust.
DEFINE QUERY qCust FOR sports.customer.
DEFINE DATA-SOURCE srcCust Select * from sports.customer.
DEFINE VARIABLE mydbObj AS CLASS DataDirect.DO.DB NO-UNDO.
myClouddbObj = new DB(“CloudDSN”, “sports”, UserID, Pwd).
myClouddbObj:DIALECT = “Oracle”.
Additional Integration Possibilities
© 2014 Progress Software Corporation. All rights reserved. 42
DEFINE VARIABLE mydbObj AS CLASS DataDirect.DO.DB NO-UNDO.
myClouddbObj = new DB(“CloudDSN”, “sports”, UserID, Pwd).
DEFINE QUERY qCust FOR sports.customer.
DEFINE DATA-SOURCE srcCust Select * from sports.customer.
DEFINE DATASET ds DATA-RELATION FOR ttCust, ttOrd RELEATION-FIELDS(ttCust.Cust-Num, ttOrd.Cust-Num). /* Could be derived from foreign constraints */
Additional Integration Possibilities (ProDataSet CloudServer)
© 2014 Progress Software Corporation. All rights reserved. 43
Potential Advantages of OOABL Built-In Objects With Language Integration
Flexible schema usage (no migrations or porting) Minimal data source management
• Management, configuration and other supplementary objects possible
Access standardized on D2C relational model and SQL compliance Underlying objects are extensible
• Feature mapping possible (e.g., word indexes, sequences, etc.)
• Feature integration possible (e.g., data links, bulk operations, etc.)
Tight Language Integration • Transactions (commit/rollback), record blocks, error handling, etc.
• Same code, different data source target is possible
• Schema exposure (back and front)
• Data federations and multi-source combinations
• Language extensions for SQL possible
Get session details & presentation downloads Complete a survey Access the latest Progress product literature
www.progress.com/exchange2014
Visit the Resource Portal