[wso2con eu 2017] ballerina: exploring data integration

Architect, WSO2

Ballerina: Exploring Data Integration

Anjana Fernando

• Source: Heterogeneous Data Sources• Processing:

– Data Quality/Cleansing– ETL

• Store/Mining:– Data Warehouse

• Expose:– Data Services

Data Integration Explained

• Built-in support for popular data formats, XML/JSON/Tables– Natural manipulation of data

• Extensive Data Connector Support (SQL, NoSQL)• Transactions (Local, Distributed)• Graphical data modelling with Composer• Scheduled data processing jobs with tasks• Rich HTTP REST service development support for

rapid data services exposure

How Ballerina Fits In

• Value Types– int/float/boolean/string/blob

• Arrays– type[]

• Maps– map

• XML– “xml” type with fully functional XML sequence API

Ballerina Data Types

• JSON– “json” type with constrained JSON support for

schema enforcement• Datatable

– “datatable” type in representing a resultset from data connectors

• User defined types– “struct” type for data binding – Automatic data validation– Record representation in datatable type

Ballerina Data Types

Data Connectors• SQL Connector (built-in)

– MySQL, MSSQL, Oracle, DB2... – Supports DML / DDL operations– Stored procedures / batch

operations– UDT support– XA transaction support

• Multiple databases / Message queues / topics

Data Connectors...• SQL Connector

Data Connectors...• Cassandra Connector (https://github.com/ballerinalang/connector-cassandra)

– CQL based data querying

– Streaming results with datatable integration

Data Connectors...• Cassandra Connector

Data Connectors...• MongoDB Connector (https://github.com/ballerinalang/connector-mongodb)

– MongoDB CRUD API support

• insert/find/update/delete

• Native JSON data handling

– Ballerina ‘json’ data type

• JSON streaming support

Data Connectors...• MongoDB Connector

Transactions• Local / XA Transactions

– Declarative transactions by defining a transaction boundary in the program

– The mode is inferred from the participants in the transaction scope

– Currently support SQL databases and message brokers for distributed transactions

Transactions...

Graphical Data Modelling

Data Mapping with Ballerina Transformer

• ETL jobs can be scheduled with the Ballerina tasks features:– Timers: delay + interval scheduling– Appointments: cron based scheduling

Scheduled Data Jobs

https://www.informatec.com/sites/default/files/inline-images/etl_2.jpg, https://www.cosculture.com/static/images/Icon---Scheduled-Publisher@270.png

Scheduled Data Jobs Timers:-

Scheduled Data Jobs Appointments:-

• HTTP/WS server connectors• Ability to generate REST APIs rapidly• Native data streaming capability

– XML / JSON• Batch data request support for high performance data

loading• Power of a general purpose programming language

with the ease of a visual programming approach– Data preprocessing, filtering, access control etc...

Data Services

Data Services...

• Ballerina provides an efficient set of connectors, tools and technologies for easy data access/manipulation and serving.

More information, visit https://ballerinalang.org/.

Summary

wso2.com