meetup - exabyte big data - hpcc systems - sql to ecl
DESCRIPTION
How to do SQL like queries is ECL (Enterprise Control Language) Install HPCC and get started in a few minutes. "How to install HPCC in 5 minutes" Youtube on last slide. https://www.youtube.com/watch?v=8SV43DCUqJgTRANSCRIPT
SQL TO ECL (ENTERPRISE CONTROL LANGUAGE)
By Fujio Turner
@FujioTurner
WHAT IS ECLECL (Enterprise Control Language) is a C++ based query language for use with HPCC Systems Big Data platform. ECLs syntax and format is very simple and easy to learn.!!
Note - ECL is very similar to Hadoop’s pig ,but!more expressive and feature rich.
JDBC CONNECTOR
SELECT * FROM `allPeople`
HPCC Systems Can Do SQL
Learn ECL to Do Complex Queries
SQL
OR
Graph
http://www.slideshare.net/hpccsystems/jdbc-hpcc
1.
2.
3.
4.
Lets Practice ECL
SCHEMA
Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END;
CREATE TABLE layout_person ( PersonID INT(15) NOT NULL, FirstName VARCHAR(15) NOT NULL, LastName VARCHAR(25) NOT NULL, PRIMARY KEY (PersonID) );
SQL ECL
Apply Schema on Read
DATAallPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person);
USE ‘layout_person’;SQL ECL
allPeople := DATASET(‘~file’,Layout_Person,THOR);
or
Inline Data
Data from HPCC Systems Cluster
WHERELayout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; !allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); !somePeople := allPeople(LastName = 'Smith'); !OUTPUT(somePeople);
SELECT * FROM `allPeople` WHERE `LastName` = ‘Smith’
SQL ECL
SELECTLayout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; !allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); !//somePeople := allPeople(LastName = 'Smith'); //OUTPUT(somePeople); !OUTPUT(allPeople);
SELECT * FROM `allPeople`
SQL ECL
SELECT WITHLayout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; !allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); !OUTPUT(allPeople,{LastName,FirstName});
SELECT ‘LastName’,’FirstName’ FROM `allPeople`
SQL ECL
ORDER BYLayout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; !allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); !!OUTPUT(SORT(allPeople,LastName));
SELECT * FROM `allPeople` ORDER BY ‘LastName’
SQL ECL
GROUP BYLayout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; !allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); !!OUTPUT(GROUP(SORT(allPeople,LastName)));
SELECT * FROM `allPeople` GROUP BY ‘LastName’
SQL ECL
GROUP BY & COUNTLayout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; !allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); !!OUTPUT(TABLE(allPeople, {LastName, LastNameC := COUNT(GROUP)}, LastName) );
SELECT LastName,COUNT(*) FROM `allPeople` GROUP BY ‘LastName’
SQL ECL
LIKEIMPORT Std; !Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; STRING33 Email; END; !allPeople := DATASET([ {1,’Fred’,’Smith’,’[email protected]’}, {2,'Joe','Blow',’[email protected]’}, {3,’Jane’,’Smith',’[email protected]’} ],Layout_Person); !emails := allPeople( Std.Str.EndsWith(Email, ‘@gmail.com’) ); OUTPUT(emails);
SELECT * FROM `allPeople` WHERE `Email` LIKE ‘%@gmail.com’
SQL ECL
Import C++ libraries and/or make
your own functions & libraries
MORE SAMPLESInstall HPCC Systems and go to the admin playground
and see more samples of ECL.
ECL LANGUAGE GUIDEhttp://hpccsystems.com/download/docs/ecl-language-referenceJOIN!
MERGE!LENGTH!REGEX!
ROUND!SUM!
COUNT!TRIM!WHEN!
AVE!ABS!
CASE!DEDUP!
NORMALIZE!DENORMALIZE!
IF!SORT!
GROUP!more ….
SCHEMA MADE EZ
http://hpccsystems.com/demos/data-profiling-demo
CSV IN
Schema OUT
Click
Uploading data from a new source and want a quick schema?
Click below
http://www.youtube.com/watch?v=8SV43DCUqJg
Watch how to install HPCC Systems
in 5 Minutes
Download HPCC Systems Open Source
Community Edition
or
Source Codehttps://github.com/hpcc-systems
http://hpccsystems.com/download/