understanding hadoop framework

7/27/2019 Understanding Hadoop framework

http://slidepdf.com/reader/full/understanding-hadoop-framework 1/31



Week 1 – Understanding Big Data

– Introduction to HDFS

Week 2 – Playing around with Cluster

– Data loading Techniques

Week 3 – Map-Reduce Basics, types and formats

– Use-cases for Map-Reduce

Week 4 – Analytics using Pig

– Understanding Pig Latin

Week 5 – Analytics using Hive

–

Understanding HIVE QL

Week 6 – NoSQL Databases

– Understanding HBASE

Week 7 – Real world Datasets and

– Hadoop Project Environm

Week 8 – Project Reviews

– Planning a career in Big D

Course Topics



Live classes

Class recordings Module wise Quizzes, Coding Assignments

24x7 on-demand technical support

Project work on large Datasets

Online certification exam

Lifetime access to the Learning Management System

How it works

Complementary Java Classes



What is Big Data?



Facebook Example

Facebook users spend 10.5 b

(almost 20,000 years) online

network

Facebook has an average of

comments are posted every



Twitter has over 500 million re

users.

The USA, whose 141.8 million represents 27.4 percent of all T

good enough to finish well ahe

Japan, the UK and Indonesia.

79% of US Twitter users are mo

recommend brands they follow 67% of US Twitter users are mo

buy from brands they follow

57% of all companies that use

for business use Twitter

Twitter Example



Other Industrial Usecases

• Insurance

• Healthcare

• Retail

– Recommendations

–Groupings

• Genome Sequencing

• Utilities



Hadoop Users

http://wiki.apache.org/hadoop/PoweredBy



Data volume is growing exponentially

• Estimated Global Data Volum

– 2011: 1.8 ZB

– 2015: 7.9 ZB

• The world's information doubl

• Over the next 10 years:

– The number of servers world

– Amount of information mana

data centers will grow by 50x

– Number of “files” enterprise

will grow by 75x

Source: http://www.emc.com/leaders

universe.htm, which was based on the

Universe Study

http://www.emc.com/leadership/programs/digital-universe.htm






Un-Structured Data is exploding



Read 1 TB Data

10 Machine 4 I/O Channels

Each Channel – 1 4 I/O Channels

Each Channel – 100 MB/s

1 Machine

Why DFS?






1 Machine

Read 1 TB Data

45 Minutes

Why DFS?



4.5 Minut45 Minutes




1 Machine

Read 1 TB Data

Why DFS?



What Is Distributed File System? (DFS)



Apache Hadoop is a framework that allows for the distributed processing of large data sets ac

of commodity computers using a simple programming model.

Companies using Hadoop:

- Yahoo

- Google

- Facebook

- Amazon

- AOL

- IBM

- And many more at

http://wiki.apache.org/hadoop/PoweredBy

What is Hadoop?



Hadoop Eco-System

d



HDFS – Hadoop Distributed File System (storage)

MapReduce (processing)

Hadoop Core Components:

h i S?



HDFS - Hadoop Distributed File System

Highly fault-tolerant

High throughput

Suitable for applications with large data sets

Streaming access to file system data

Can be built out of commodity hardware

What is HDFS?

M i C Of HDFS



NameNode:

master of the system

maintains and manages the blocks which are present on the

DataNodes

Main Components Of HDFS:

DataNodes: slaves which are deployed on each machine and provide the actual

storage

responsible for serving read and write requests for the clients

S d N N d



Secondary NameNode:

Not a hot standby for the NameNode

Connects to NameNode every hour*

Housekeeping, backup of NemeNode metadata

Saved metadata can build a failed NameNode

Secondary NameNode:

You gi

metada

hour, I sec

Sin

F

Secondary

NameNode

NameNode

metadata

metadata

J bT k d T kT k



JobTracker and TaskTracker:

HDFS A hit t



HDFS Architecture

Job Tracker



Job Tracker

Job Tracker Contd



Job Tracker Contd.

Job Tracker Contd



Job Tracker Contd.

HDFS Client Creates a New File



HDFS Client Creates a New File

Rack Awareness



Rack Awareness

Anatomy of a File Write:



Anatomy of a File Write:

Anatomy of a File Read:



Anatomy of a File Read:



Thank YouSee You in Class Next Week

understanding hadoop framework

Documents