[wso2con eu 2017] ballerina: exploring data integration

21
Architect, WSO2 Ballerina: Exploring Data Integration Anjana Fernando

Upload: wso2-inc

Post on 21-Jan-2018

115 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Architect, WSO2

Ballerina: Exploring Data Integration

Anjana Fernando

Page 2: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• Source: Heterogeneous Data Sources• Processing:

– Data Quality/Cleansing– ETL

• Store/Mining:– Data Warehouse

• Expose:– Data Services

Data Integration Explained

Page 3: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• Built-in support for popular data formats, XML/JSON/Tables– Natural manipulation of data

• Extensive Data Connector Support (SQL, NoSQL)• Transactions (Local, Distributed)• Graphical data modelling with Composer• Scheduled data processing jobs with tasks• Rich HTTP REST service development support for

rapid data services exposure

How Ballerina Fits In

Page 4: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• Value Types– int/float/boolean/string/blob

• Arrays– type[]

• Maps– map

• XML– “xml” type with fully functional XML sequence API

Ballerina Data Types

Page 5: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• JSON– “json” type with constrained JSON support for

schema enforcement• Datatable

– “datatable” type in representing a resultset from data connectors

• User defined types– “struct” type for data binding – Automatic data validation– Record representation in datatable type

Ballerina Data Types

Page 6: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Connectors• SQL Connector (built-in)

– MySQL, MSSQL, Oracle, DB2... – Supports DML / DDL operations– Stored procedures / batch

operations– UDT support– XA transaction support

• Multiple databases / Message queues / topics

Page 7: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Connectors...• SQL Connector

Page 8: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Connectors...• Cassandra Connector (https://github.com/ballerinalang/connector-cassandra)

– CQL based data querying

– Streaming results with datatable integration

Page 9: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Connectors...• Cassandra Connector

Page 10: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Connectors...• MongoDB Connector (https://github.com/ballerinalang/connector-mongodb)

– MongoDB CRUD API support

• insert/find/update/delete

• Native JSON data handling

– Ballerina ‘json’ data type

• JSON streaming support

Page 11: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Connectors...• MongoDB Connector

Page 12: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Transactions• Local / XA Transactions

– Declarative transactions by defining a transaction boundary in the program

– The mode is inferred from the participants in the transaction scope

– Currently support SQL databases and message brokers for distributed transactions

Page 13: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Transactions...

Page 14: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Graphical Data Modelling

Data Mapping with Ballerina Transformer

Page 15: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• ETL jobs can be scheduled with the Ballerina tasks features:– Timers: delay + interval scheduling– Appointments: cron based scheduling

Scheduled Data Jobs

https://www.informatec.com/sites/default/files/inline-images/etl_2.jpg, https://www.cosculture.com/static/images/[email protected]

Page 16: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Scheduled Data Jobs Timers:-

Page 17: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Scheduled Data Jobs Appointments:-

Page 18: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• HTTP/WS server connectors• Ability to generate REST APIs rapidly• Native data streaming capability

– XML / JSON• Batch data request support for high performance data

loading• Power of a general purpose programming language

with the ease of a visual programming approach– Data preprocessing, filtering, access control etc...

Data Services

Page 19: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

Data Services...

Page 20: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

• Ballerina provides an efficient set of connectors, tools and technologies for easy data access/manipulation and serving.

More information, visit https://ballerinalang.org/.

Summary

Page 21: [WSO2Con EU 2017] Ballerina: Exploring Data Integration

wso2.com