calcite meetup-2016-04-20

Introduction toApache CalciteJosh ElserMTS2016-04-20

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

About me

Apache Calcite is a project at the Apache Software Foundation.This name is a trademark of the Foundation.

Apache Calcite Committer and PMC

(Slowly) Re-learning SQL

Distributed systems nerd


Users

Apache Kylin

Apache SamzaQuark

SQL-Gremlin/Apache TinkerPop

See the respective project pages at the ASF


Brief History

Originally known as “Optiq” (https://github.com/julianhyde/optiq): 2012-05-07 Entered Apache Software Fundation’s Incubator: 2014-05-25 Renamed to Apache Calcite (incubating): 2014-09-30 Graduates to top-level project (TLP): 2015-10-21 2 major releases since graduation: 2016-03-XX Currently comprised of 16 committers and 14 PMC members

“The foundation for your next high-performance database.”

https://github.com/julianhyde/optiq

https://github.com/julianhyde/optiq


AgendaSQL Parser


SQL Parser

SELECT d.name, COUNT(*) as cFROM Emps as e JOIN Depts as d ON e.deptno = d.deptnoWHERE e.age < 30GROUP BY d.deptnoHAVING COUNT(*) > 5ORDER BY c DESC

Scan

Join

Filter

Aggregate

Filter

Project

Sort

Scanhttps://calcite.apache.org/docs/reference.html


AgendaSQL Parser

Cost-Based Optimizer



Extensible

Java API– Parser output– Inline Java code

AKA Relational AlgebraRelBuilder builder = RelBuilder.create(config);RelNode node = builder .scan("EMP") .project(builder.field(“DEPTNO”), builder.field(“ENAME”)) .build();

SELECT ename, deptno FROM emp;

LogicalProject(DEPTNO, ENAME) LogicalTableScan(EMP)



SELECT d.name, COUNT(*) as cFROM depts AS dJOIN emp AS e on d.deptno = e.deptnoGROUP BY d.name;

Scan Emp[deptno]

Join

Aggregate

Scan Depts[deptno,

name]

Join

Aggregate

Project[name, c]

Scan Emp[*] Scan Depts[*]


AgendaSQL Parser


Pluggable Data Sources



User-implemented– Yes, you.

Custom optimizations– Predicate pushdown– Projections

Sources of Sources– Federation

Everything but the data

Join

Aggregate

Project

Scan Emp[*] Scan Depts[*]


AgendaSQL Parser



Avatica


Avatica

Calcite sub-project Wire protocol

– Protocol Buffers– JSON

Metrics Authentication Clients

– JDBC client– Python and Go (in-progress)

JDBC over HTTP – SQL for Everyone


Thank You!Email: [email protected]: @josh_elserMailing lists: [email protected] info: https://calcite.apache.org/