hadoop framework demo
TRANSCRIPT
-
8/10/2019 Hadoop Framework Demo
1/30
b y
p r a v e e n . K
Hadoop Framework
1
-
8/10/2019 Hadoop Framework Demo
2/30
Agenda
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
#
-
8/10/2019 Hadoop Framework Demo
3/30
$iscussion %oint
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
&
-
8/10/2019 Hadoop Framework Demo
4/30
What is Hadoop?
Hadoop is a software framework fordistributed storing and processingoflargedatasetsacrosslarge clustersof computers
Large datasets'era(ytes or peta(ytes of data
Large clustershundreds or thousands of nodes
Hadoopis (ased on a simple programming model calledMapReduce
Hadoop is (ased on a simple data model)any data will fit
*
-
8/10/2019 Hadoop Framework Demo
5/30
+
,
%
, -
%-
H$F,
/ap0educe
ame ode
$ata ode
2o( 'racker
'ask 'racker
,
%
H$F,
H$F, H$F,H$F, H$F,
/ap0educe
/ap0educe /ap0educe /ap0educe
ame ode
$ata ode $ata ode $ata ode
2o( 'racker
'ask 'racker 'ask 'racker 'ask 'racker
-
8/10/2019 Hadoop Framework Demo
6/30
ont.3
Hadoop framework consists on two main layers
$istri(uted file system 4H$F,5
67ecution engine 4/ap0educe5
8
-
8/10/2019 Hadoop Framework Demo
7/30
$iscussion %oint
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
9
-
8/10/2019 Hadoop Framework Demo
8/30
Origin of Hadoop
Google Hadoop
:F, H$F,
/ap0educe /ap0educe
Big'a(le HBA,6
;
Hadoop was created (yDoug CuttingandMikeCafarellain #
-
8/10/2019 Hadoop Framework Demo
9/30
$iscussion %oint
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
C
-
8/10/2019 Hadoop Framework Demo
10/30
Why Hadoop?
0eading 1 'B of $ata
raditional Hadoop
1 machine
*Hard disks
1
-
8/10/2019 Hadoop Framework Demo
11/30
erabyte#ort $enchmark%&uly '(()*
One of =ahoo>s Hadoop clusters sorted 1 tera(yte of data in'(+ seconds) which (eatthe pre@ious record of #C9 seconds in the annual general purpose 4daytona5tera(ytesort(enchmark. 'he sort (enchmark specifies the input data 41< (illion 1
-
8/10/2019 Hadoop Framework Demo
12/30
,inserabyte#ort $enchmark%'(-*
One of =ahoo>s Hadoop clusters sorted1.*#tera(yte of data in/(seconds)there is nopre@ious record for this.
-
8/10/2019 Hadoop Framework Demo
13/30
Ad@antages
1&
1. Volume, Variety, Velocity
2. Hardware Commoditization
3. CloudComputing
-
8/10/2019 Hadoop Framework Demo
14/30
ont3
1*
-
8/10/2019 Hadoop Framework Demo
15/30
$iscussion %oint
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
1+
-
8/10/2019 Hadoop Framework Demo
16/30
Users
18
-
8/10/2019 Hadoop Framework Demo
17/30
$iscussion %oint
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
19
-
8/10/2019 Hadoop Framework Demo
18/30
Features of Hadoop
Here>s what makes it especially usefulN
#calable0t can relia(ly store and process peta(ytes.
1conomical0t distri(utes the data and processing across clusters of commonly a@aila(lecomputers 4in thousands5.
1fficient0By distri(uting the data) it can process it in parallel on the nodes where the datais located.
2eliable0t automatically maintains multiple copies of data and automatically redeployscomputing tasks (ased on failures
1;
-
8/10/2019 Hadoop Framework Demo
19/30
$iscussion %oint
What is Hadoop?
Origin of Hadoop
Why Hadoop?
Users of Hadoop
Hadoop Features
Hadoop Architecture
Basic Hadoop Work Flow.
Hadoop omponents.
! " A
1C
-
8/10/2019 Hadoop Framework Demo
20/30
Hadoop Architecture
Master #lave
Name Node Data Node
&ob racker ask racker
#econdaryName Node
#