[wso2con eu 2017] ballerina: exploring data integration
TRANSCRIPT
Architect, WSO2
Ballerina: Exploring Data Integration
Anjana Fernando
• Source: Heterogeneous Data Sources• Processing:
– Data Quality/Cleansing– ETL
• Store/Mining:– Data Warehouse
• Expose:– Data Services
Data Integration Explained
• Built-in support for popular data formats, XML/JSON/Tables– Natural manipulation of data
• Extensive Data Connector Support (SQL, NoSQL)• Transactions (Local, Distributed)• Graphical data modelling with Composer• Scheduled data processing jobs with tasks• Rich HTTP REST service development support for
rapid data services exposure
How Ballerina Fits In
• Value Types– int/float/boolean/string/blob
• Arrays– type[]
• Maps– map
• XML– “xml” type with fully functional XML sequence API
Ballerina Data Types
• JSON– “json” type with constrained JSON support for
schema enforcement• Datatable
– “datatable” type in representing a resultset from data connectors
• User defined types– “struct” type for data binding – Automatic data validation– Record representation in datatable type
Ballerina Data Types
Data Connectors• SQL Connector (built-in)
– MySQL, MSSQL, Oracle, DB2... – Supports DML / DDL operations– Stored procedures / batch
operations– UDT support– XA transaction support
• Multiple databases / Message queues / topics
Data Connectors...• SQL Connector
Data Connectors...• Cassandra Connector (https://github.com/ballerinalang/connector-cassandra)
– CQL based data querying
– Streaming results with datatable integration
Data Connectors...• Cassandra Connector
Data Connectors...• MongoDB Connector (https://github.com/ballerinalang/connector-mongodb)
– MongoDB CRUD API support
• insert/find/update/delete
• Native JSON data handling
– Ballerina ‘json’ data type
• JSON streaming support
Data Connectors...• MongoDB Connector
Transactions• Local / XA Transactions
– Declarative transactions by defining a transaction boundary in the program
– The mode is inferred from the participants in the transaction scope
– Currently support SQL databases and message brokers for distributed transactions
Transactions...
Graphical Data Modelling
Data Mapping with Ballerina Transformer
• ETL jobs can be scheduled with the Ballerina tasks features:– Timers: delay + interval scheduling– Appointments: cron based scheduling
Scheduled Data Jobs
https://www.informatec.com/sites/default/files/inline-images/etl_2.jpg, https://www.cosculture.com/static/images/[email protected]
Scheduled Data Jobs Timers:-
Scheduled Data Jobs Appointments:-
• HTTP/WS server connectors• Ability to generate REST APIs rapidly• Native data streaming capability
– XML / JSON• Batch data request support for high performance data
loading• Power of a general purpose programming language
with the ease of a visual programming approach– Data preprocessing, filtering, access control etc...
Data Services
Data Services...
• Ballerina provides an efficient set of connectors, tools and technologies for easy data access/manipulation and serving.
More information, visit https://ballerinalang.org/.
Summary
wso2.com