big data: big sql application development with biginsights
DESCRIPTION
IBM, Hadoop, Big Data, BigInsights, application development, SQL, Big SQL, Web console, lifecycleTRANSCRIPT
![Page 1: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/1.jpg)
Developing, publishing and deploying your first Big SQL application with InfoSphere BigInsights
Cynthia Saracco
Senior Solution Architect
IBM Silicon Valley Lab
![Page 2: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/2.jpg)
2 © 2013 IBM Corporation
Big Data Application Ecosystem
Eclipse
App library
MapReduce, "
Text Analytics
Query
App Development
• Code application program, and generate
associated App
• Deploy Apps to Enterprise ManagerApp
Development
Publish
Data integration scenario:
Pre-defined work flows simplify
loading data from various
sources
•Work flows can be configured,
deployed, executed and
scheduled
Development tooling:
•Text analytics
•MapReduce
•Query languages
• . . .
Application scenarios (web log,
email, social media, �):
• Samples provide starting
point, speed time to value
Big Data Web Console
![Page 3: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/3.jpg)
3 © 2013 IBM Corporation
BigInsights Applications Catalog (Web Console)
� Browse available applications � Manage and deploy applications (administrators only) � Execute (or schedule execution of ) a deployed application � Monitor job (application) status� Link or chain applications for sequential execution
![Page 4: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/4.jpg)
4 © 2013 IBM Corporation
Overview of Application Development Lifecycle
� Configure your Eclipse environment (one-time set up) � Develop your application using BigInsights tools � Test your application � Package and publish your application � Deploy your application on the cluster
![Page 5: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/5.jpg)
5 © 2013 IBM Corporation
Configure your Eclipse environment
� One-time set up � Download and install BigInsights tools (Eclipse plug-ins)
– Welcome tab of BigInsights Web console includes pre-req info, download &
installation instructions
![Page 6: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/6.jpg)
6 © 2013 IBM Corporation
Develop your application – Big SQL example
� Open the BigInsights perspective in Eclipse� Create a BigInsights project
![Page 7: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/7.jpg)
7 © 2013 IBM Corporation
Develop your application – Big SQL example (cont’d)
� Create a BigInsights program with a SQL script (file)
![Page 8: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/8.jpg)
8 © 2013 IBM Corporation
Develop your application – Big SQL example (cont’d)
� Populate your SQL file with the desired code
create table if not exists media_csv
(id integer not null,
name varchar(50),
url varchar(50),
contactdate string)
row format delimited
fields terminated by ','
stored as textfile;
load hive data local inpath
'/home/biadmin/sampleData/DBMS/RDBMS_data.csv'
-- overwrite
into table media_csv;
![Page 9: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/9.jpg)
9 © 2013 IBM Corporation
Test your application
� Run your application from Eclipse
![Page 10: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/10.jpg)
10 © 2013 IBM Corporation
Publish your application to the BigInsights catalog
� Package and publish your application from Eclipse � Specify application name, workflow requirements, etc.
![Page 11: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/11.jpg)
11 © 2013 IBM Corporation
Deploy your application on the cluster
� Access the Applications tab of the Web console� “Manage” the published applications � Locate your new application and deploy it � Create credentials store file in DFS (if needed) -- see next chart � Optionally, execute the application after it’s been deployed
![Page 12: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/12.jpg)
12 © 2013 IBM Corporation
Run your Big SQL application on the cluster
![Page 13: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/13.jpg)
13 © 2013 IBM Corporation
Upgrade your application (optional)
� Satisfy evolving business requirements, improve flexibility – Example: add input parm(s)
� Modify SQL code and re-package, re-publish, re-deploy
create table if not exists $TABLE(id integer not null,
name varchar(50),
url varchar(50),
contactdate string)
row format delimited
fields terminated by ','
stored as textfile;
load hive data local inpath '$FILE' -- overwrite
into table $TABLE;
![Page 14: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/14.jpg)
14 © 2013 IBM Corporation
Upgrade your application (cont’d)
� Re-publish your application from Eclipse � Adjust workflow specs for input parm(s)
![Page 15: Big Data: Big SQL Application Development with BigInsights](https://reader033.vdocuments.site/reader033/viewer/2022051816/546cd33caf795985298b5169/html5/thumbnails/15.jpg)
15 © 2013 IBM Corporation
Summary
� Eclipse tools simplify big data application development for BigInsights– Wizards
– Context-sensitive help
– Oozie workflow generation
– Built-in test environment
– Etc.
� Application catalog provides easy way to locate and launch apps of interest– Developers use Eclipse tools to package/publish their applications to this
catalog
– Application upgrades easily managed