intro to big data and hadoop ubc cs lecture series - g fawkes

© 2013 Geoff Fawkes. All Rights Reserved. 1

Introduction to Analytics and Big Data - Hadoop

The University of British ColumbiaComputer Science Alumni/Industry Lecture Series

Geoff FawkesNovember, 2013

/ 450


Who am I?

Director Engineering, Teradata HSBC, Pivotal/Aptean, Newbridge/Alcatel, etc. various

engineering roles Technology executive, mentor, software engineer

B.Sc. Comp Sci (UBC), MBA Executive (SFU)

Interruptive (disruptive?) personality Please ask questions to me / each other as we go along I don’t have all the answers – you do!

Credits: Rob Pegler, SNIA Education Storage Networking Industry Association, 2012

Who’s paying attention - 450 slides page count? Not that “big” - - about 50


Big Data and Hadoop

History Data Challenges Why Hadoop?


Customer Challenges: The Data Deluge


Big Data is Different than Business Intelligence


Questions From Business Will Vary


Web 2.0 is “Data Driven”


The World of Data-Driven Applications


Attributes of Big Data


Top Ten Common Big Data Problems


Industries Are Embracing Big Data


Why Hadoop?


Storage and Memory B/W Lagging CPU


Commodity Hardware Economics


What is Hadoop?

Hadoop Adoption HDFS MapReduce Examples Ecosystem Projects


Hadoop Adoption in the Industry


What is Hadoop?


HDFS 101 – The Data Set System


HDFS Organization and Replication


Hadoop Server Roles - Multiple


Hadoop Cluster


HDFS File Write Operation - Instance


HDFS File Read Operation - Instance


HDFS File Operation R/W Replication


MapReduce 101 – Functional Programming Meets Distributed Processing


What is MapReduce?


Key MapReduce Terminology


MapReduce Basic Concepts


Example 1: MapReduce Operation


Example 2: Sample Dataset


MapReduce Paradigm – UNIX Cmd


Example 3: Count Words


Map function

Reduce function

Run this program as aMapReduce job

Ex. 3: Lifecycle of a MapReduce Job


Map function

Reduce function

Run this program as aMapReduce job



Map Wave 1

ReduceWave 1

Map Wave 2

ReduceWave 2

Input Splits

Time

How are the number of splits, number of map and reducetasks, memory allocation to tasks, etc., determined?



190+ parameters in Hadoop

Set manually or defaults are used

MapReduce Job Configuration Parms


Putting it all Together: MapReduce + HDFS


Hadoop Ecosystem Projects

- Interactive SQL Query & Modeling

- Data flow for tedious MapReduce Jobs

- Columnar NoSQL Store


Compare: Hadoop, SQL, Massively Parallel Processing (MPP)


Compare: RDBMS and MapReduce


Hadoop Use Cases

Set Top Cable TV Boxes Pay Per View Advertising Bank Risk Modelling Product Sentiment Analysis


Example 1: Set Top Cable TV Boxes


Example 2: Pay Per View Advertising


Example 3: Bank Risk Modelling


Example 4: Product Sentiment Analysis


More Reading?

World Economic Forum: “Personal Data: The Emergence of a New Asset Class” 2011

McKinsey Global Institute: Big Data: The next frontier for innovation, competition, and productivity

Big Data: Harnessing a game-changing asset

IDC: 2011 Digital Universe Study: Extracting Value from Chaos

The Economist: Data, Data Everywhere

Data Science Revealed: A Data-Driven Glimpse into the Burgeoning New Field

O’Reilly – What is Data Science?

O’Reilly – Building Data Science Teams?

O’Reilly – Data for the public good

Obama Administration “Big Data Research and Development Initiative.”


Introduction to Analytics and Big Data – Hadoop

Q&A

Geoff Fawkes http://www.linkedin.com/pub/geoff-fawkes/1/269/202 @gfawkes

November, 2013