query real-time kafka streams with oracle sql · •query kafka messages –integrate and analyze...
TRANSCRIPT
![Page 1: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/1.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Query Real-Time Kafka Streams with Oracle SQL
Melli AnnamalaiSenior Principal Product ManagerOracleOctober 23, 2018
![Page 2: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/2.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation.
![Page 3: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/3.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Agenda
• Demo: Monitor temperature readings from food distribution centers
• Apache Kafka Concepts
• SQL Connector for Kafka
– Database objects
– SQL syntax examples
– Performance and scale
• Roadmap
3
![Page 4: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/4.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 4
Streaming Data
In real-time: Process data streams as they occur
![Page 5: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/5.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Streaming Data Examples
• Sensor data
• Clickstream data
• Machine/equipment
• Network monitoring
• Pipeline monitoring
• Fleet monitoring
• ……
5
High Velocity
High Volume
Example: 200k/second, 24x7
![Page 6: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/6.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Demo
6
![Page 7: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/7.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Monitoring Temperature at Food Distribution Centers
7
![Page 8: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/8.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Apache Kafka Concepts
8
![Page 9: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/9.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 9
Apache Kafka
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
![Page 10: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/10.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 11
Kafka Topics
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Topic 1
Topic 2
Topic 3
Topic 4
Topic 5
Messages with a common format
![Page 11: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/11.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Kafka Topic Example
MSGNUMBER,
MSGTIMESTAMP,
SENSORUNITID,
SENSORTYPEID,
TEMPERATURESETTING,
TEMPERATUREREADING
12
![Page 12: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/12.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 13
Kafka Topic Partitions
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Subset of messagesin a Topic, managed by a broker
![Page 13: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/13.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 14
Kafka Topic Partitions
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
![Page 14: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/14.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
A Kafka Consumer Application
15
Should read all partitions
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
![Page 15: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/15.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 16
Kafka Topic Partitions
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
![Page 16: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/16.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 17
Kafka Topic Partition Offsets
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Logical Position of a Message within a Partition
34.3 36.2 34.2 33.4 34.5 31.9 32.1
![Page 17: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/17.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 18
Logical Position (Offset) Maps to Sequence Numbers
…….. 34.3 36.2 34.2 33.4 34.5 31.9 32.1 32.6 33.2 33.1 ………
• Offset uniquely identifies a message
• Consumers can read from an Offset
• Starting offset is 0(at the beginning of a topic)
…….. 103 104 105 106 107 108 109 110 111 112 ………
![Page 18: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/18.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Using Data from Kafka Topics
• Kafka Consumer application
• Kafka Streams API
• Stream Kafka topic data into HDFS/Object store/databases using Kafka connectors
• KSQL: Streaming SQL engine for real-time data processing of Kafka topics
19
![Page 19: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/19.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Kafka Consumer Example
20
With Automatic Offset Commits
From: https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
![Page 20: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/20.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Oracle Database as a Kafka Consumer
21
Enable Oracle SQL access to Kafka Topics
ProducersEntities producing streaming data
Oracle DatabaseExternal tables and views
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
![Page 21: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/21.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Oracle SQL and Kafka
1. Integrate Kafka messages with Oracle Database applications
2. Enrich Kafka data with Oracle Database table data
– Feed enriched topics back to Kafka
22
![Page 22: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/22.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
1. Integrate Kafka with Database Applications• Query Kafka messages
– Integrate and analyze with Oracle Database data
– Use the full richness of Oracle SQL
• Join data in a Kafka time interval with an Oracle Database table
• Load into Oracle Database table using Oracle SQL
23
![Page 23: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/23.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 24
2. Enrich Kafka Data with Oracle Database Table Data
…….. 34.3 36.2 34.2 33.4 34.5 31.9 33.1 ………
2. Enrich Kafka Data with Oracle Database Table Data
…….. 34.3 36.2 34.2 33.4 34.5 31.9 33.1 ………
…….. 34.3, Seattle, LastOutageDate; 36.2, Austin, LastOutageDate; 34.2, Philadelphia, LastOutageDate ………
![Page 24: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/24.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
SQL Connector for Kafka
25
![Page 25: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/25.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 26
Apache Kafka + Oracle Database
Views on the external tables
Kafka Cluster
Partition 1
Partition 2
Partition 3
Partition 4
External tables
Database applicationsquery views to access Kafka messages
![Page 26: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/26.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Database Objects
• Views over Kafka Topics
• Views are created per Topic
• There can be multiple sets of views per Topic: One set per application (Consumer Group)
27
Name Null? Type------------------- -------- -------------KAFKA$PARTITION NUMBER(38)KAFKA$OFFSET NUMBER(38)KAFKA$TIMESTAMP NUMBER(38)MSGNUMBER NUMBERMSGTIMESTAMP TIMESTAMP(6)SENSORTYPEID NUMBERSENSORUNITID NUMBERTEMPERATURESETTING NUMBER(6,3)TEMPERATUREREADING NUMBER(6,3)
![Page 27: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/27.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Database Objects
• Metadata tables – DBMS_KAFKA_CLUSTER
– DBMS_KAFKA_PARTITION– DBMS_KAFKA_LOAD_METRICS– DBMS_KAFKA_APPLICATION
• Message schema table
28
Name Null? Type------------------- -------- -------------MSGNUMBER NOT NULL NUMBER(38)MSGTIMESTAMP TIMESTAMP(6)SENSORTYPEID NUMBER(38)SENSORUNITID NUMBER(38)TEMPERATURESETTING NUMBER(6,3)TEMPERATUREREADING NUMBER(6,3)
![Page 28: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/28.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Query Streaming Data with a View
29
No Need to Persist or Stage Data in a Table
…….. 34.3 36.2 34.2 33.4 34.5 31.9 33.1 ………
Name Null? Type------------------- -------- -------------KAFKA$PARTITION NUMBER(38)KAFKA$OFFSET NUMBER(38)KAFKA$TIMESTAMP NUMBER(38)MSGNUMBER NUMBERMSGTIMESTAMP TIMESTAMP(6)SENSORTYPEID NUMBERSENSORUNITID NUMBERTEMPERATURESETTING NUMBER(6,3)TEMPERATUREREADING NUMBER(6,3)
![Page 29: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/29.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 30
Join Kafka Topic View with Oracle Database Table
Name Null? Type------------------- -------- -------------KAFKA$PARTITION NUMBER(38)KAFKA$OFFSET NUMBER(38)KAFKA$TIMESTAMP NUMBER(38)MSGNUMBER NUMBERMSGTIMESTAMP TIMESTAMP(6)SENSORTYPEID NUMBERSENSORUNITID NUMBERTEMPERATURESETTING NUMBER(6,3)TEMPERATUREREADING NUMBER(6,3)
Name Null? Type------------------- -------- -------------SENSORUNITID NOT NULL NUMBER(38)LATITUDE NUMBERLONGITUDE NUMBERCITY VARCHAR2(30)
View on Kafka Topics Database table
Find cities where the temperature fluctuation is more than normal
![Page 30: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/30.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Offset Management
31
34.2 34.5 34.1 34.6 34.3 ………………………………………………….………………….
0 1 2 3 4 …………………………………………….……………..………
![Page 31: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/31.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Offset Management
32
34.2 34.5 34.1 34.6 34.3 ………………………………………………….………………….
0 1 2 3 4 …………………………………………….……………..………
Commit offset RECORD_OFFSET()
![Page 32: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/32.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Offset Management
33
34.2 34.5 34.1 34.6 34.3 36.2 34.2 33.4 34.5 31.9 33.1 34.5 31.6 34.5 …………………..…
0 1 2 3 4 5 6 7 8 9 10 11 12 13 ……………..………
Start location for next operation
SET_OFFSET()
Offsets of each Consumer Group application are managed independently
![Page 33: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/33.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
SQL Syntax Examples
34
![Page 34: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/34.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
1. REGISTER_CLUSTER
35
SQL> BEGINdbms_kafka.register_cluster(‘SENS2',‘<Zookeeper URL>:2181',‘<Kafka broker URL>:9092','DBMSKAFKA_DEFAULT_DIR','DBMSKAFKA_LOCATION_DIR‘
'Testing DBMS_KAFKA');END;
![Page 35: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/35.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
2. Create Views
36
DECLAREviews_created INTEGER;view_prefix VARCHAR2(128);
BEGINDBMS_KAFKA.CREATE_KAFKA_VIEWS(‘SENS2',
‘MONITORAPP', ‘sensor2', ‘sensor_messages_shape‘, views_created, view_prefix);END;/
One view per partition
![Page 36: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/36.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 37
3. Query with Oracle SQL
SQL> select count(*) from KV_SENS2_MONITORAPP_SENSOR2_0;
SQL> select timestamp, sensorunitid, temperaturereadingfrom KV_SENS2_MONITORAPP_SENSOR2_0;
![Page 37: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/37.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Record Offset and Set Offset
38
SQL> execute dbms_kafka.record_offset('KV_SENS2_MONITORAPP_SENSOR2_0');
SQL> execute dbms_kafka.set_offset('KV_SENS2_MONITORAPP_SENSOR2_0');
34.2 34.5 34.1 34.6 32.1
0 1 2 3 4
Record offset
34.2 34.5 34.1 34.6 32.1 32.2
0 1 2 3 4 5
Set offset
![Page 38: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/38.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 39
Loading into Oracle Database
DECLARErows_loaded number;
BEGIN
dbms_kafka.load_table('SENS2','LOADAPP','sensor2','sensor_messages_shape',rows_loaded);
dbms_output.put_line('rows loaded: ' || rows_loaded);END;/
![Page 39: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/39.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Performance and Scale
40
![Page 40: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/40.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Scaling in Kafka
• Increase number of Partitions
• Increase Consumer instances
41
![Page 41: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/41.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 42
Kafka Consumer Groups
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
Topic with 4 Partitions
One Consumer reads from all Partitions
![Page 42: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/42.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 43
Kafka Consumer Groups
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
Topic with 4 Partitions
One Consumer reads from two Partitions
![Page 43: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/43.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 44
Kafka Consumer Groups
ProducersEntities producing streaming data
Consumers Applications that read and process messages
Kafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
Topic with 4 Partitions
One Consumer per Partition
![Page 44: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/44.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 45
Kafka Consumer Groups
ProducersEntities producing streaming data
Database viewsKafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
Topic with 4 Partitions
One view reads from all Partitions
![Page 45: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/45.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 46
Kafka Consumer Groups
ProducersEntities producing streaming data
Database viewsKafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
Topic with 4 Partitions
Each view reads from two Partitions
![Page 46: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/46.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 47
Kafka Consumer Groups
ProducersEntities producing streaming data
Database viewsKafka ClusterStores and manages streaming datain a distributed, replicated,fault-tolerant cluster
Partition 1
Partition 2
Partition 3
Partition 4
Topic with 4 Partitions
Four views, one per Partition
![Page 47: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/47.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Kafka Partitions and Database Views
• Number of views can be 1……<total number of partitions>
• Scale by increasing the number of views
• Can have multiple sets of views for the same Topic
– A set of views is a ‘Consumer Group’
–Offsets for each Consumer Group managed independently
48
![Page 48: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/48.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Factors that Impact Performance
• Degree of parallelism in the database
• Network bandwidth
– Between Kafka brokers and database system
49
![Page 49: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/49.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Kafka and Oracle Products and Cloud Services
50
![Page 50: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/50.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 51
Oracle Big Data SQL
Big Data SQL external tableKafka Cluster Hadoop Cluster with Apache Hive
Shipping since 2017
![Page 51: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/51.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 52
Query Hive table from Oracle Database
Hive table reads from Kafka
CREATE TABLE TRAFFIC_JSON_TAB(
ID VARCHAR2(256),CLIENTID VARCHAR2(256),…),EVENTTIMEASSTRING VARCHAR2(256),SENDER VARCHAR2(256),TYPE VARCHAR2(256),PROPERTIES VARCHAR2(256),DIRECTION VARCHAR2(256),RECEIVEDTIME NUMBER(38,0),RECEIVEDTIMEASSTRING VARCHAR2(256),PAYLOAD VARCHAR2(4000)
)ORGANIZATION EXTERNAL(
TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIRACCESS PARAMETERS(
com.oracle.bigdata.cluster=bigdatalitecom.oracle.bigdata.tablename=oracletest.traffic_json_tab
))
![Page 52: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/52.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 53
Oracle Event Hub Cloud Service
Enterprise Kafka features in Event Hub• Scalable and highly available• Messaging/Event broker• Security• Schema registry• Connectors:
object storage, databases, other external systems
Access from Database Cloud Servicevia Database views
![Page 53: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/53.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 55
Single Infra for Data Movement, Application Messaging, Events Broker, Stream AnalyticsOracle Event Hub Cloud Service Use Cases
Kafka Use cases Oracle PaaS Service Add-Ons
Data Movement into data lakes (Firehose) and DB Change propagation (Kafka Connect) Big Data Cloud
Application messaging for Microservices; replacement for JMS; Events Broker (Kafka Connect, Kafka Streams)
Integration Cloud
Event Processing & Analytics (Kafka Streams and KSQL) Streaming Analytics
IoT Events and Logs transport (Kafka Produce/Consume, Kafka Connect) Mobile Cloud
Data Lake Firehose Application Messaging IoT Use Case Database Change Use Case
![Page 54: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/54.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Roadmap
55
![Page 55: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/55.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Planned Release Roadmap
• Addition to Big Data Connectors product suite
• BYOL for Cloud deployments
– Note: Big Data Connectors are included in Big Data Cloud Service and Big Data Cloud@Customer
• Cloud Managed Services
– Autonomous Data Warehouse Cloud Service (ADWCS)
56
![Page 56: Query Real-Time Kafka Streams with Oracle SQL · •Query Kafka messages –Integrate and analyze with Oracle Database data –Use the full richness of Oracle SQL •Join data in](https://reader031.vdocuments.site/reader031/viewer/2022040206/5e04b943aa6f9e21c713f714/html5/thumbnails/56.jpg)
Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Peek into Planned Functionality Roadmap
• Support Kafka Streams
– Enrich Kafka Topics using Oracle SQL and database tables
• Retrieval based on timestamp information
57