ISSN: XXXX-XXXX Volume X, Issue X, Month Year

Significance of HADOOP Distributed File System

Vivekanand. S. Reshmi

Dept of Computer Science and Engineering

BTL Institute of Technology

Bangalore, India

Ravikumar.reshmi@gmail.com

Abstract:

A Hadoop Distributed File System (HDFS) is designed to

store very large data sets reliably and to stream those data

sets at high bandwidth to user applications. By distributing

storage and computation across many servers, the resource

can grow with demand while remaining economical at every

size. An important characteristic of Hadoop is the partition-

ing of data and computation across many (thousands) of

hosts, and executing application computations in parallel

close to their data. A Hadoop cluster scales computation

capacity, storage capacity and IO bandwidth by simply add-

ing commodity servers. Hadoop Distributed File System

(HDFS) are the most common file system deployed in large

scale distributed systems such as Face book, Google and

Yahoo today.

Introduction

The Hadoop platform [1][5]provides both hadoop distributed

file system (HDFS) and computational capabilities (Map

Reduce).[2] Hadoop is an Apache project all components are

available via the Apache open source license. The newest

Hadoop versions are capable of storing petabytes of da-

ta.HDFS stores file system metadata and application data

separately As in GFS. It is designed to run on clusters of

commodity hardware. HDFS relaxes a few requirements to

enable streaming access to the file system data. Hadoop is a

Distributed parallel fault tolerant file system. It is designed

to reliably store very large files across machines in a large

cluster. It is inspired by the Google

File System. Hadoop DFS stores each file as a sequence of

blocks; all blocks in a file except the last block are the same

size. Blocks belonging to a file are replicated for fault toler-

ance. The block size and replication factor are configurable

per file. In a distributed system even if we decide to deploy

dedicated high performance machines which are really cost-

ly, faults or disruptions are not frequent. So forerunners like

Google decided to use commodity hardware which is ubiq-

uitous and very cost effective , but to use such hardware they

have to make a design choice of treating faults or disruptions

as regular situation and system should able to recover from

such failures. Hadoop developed on similar design choices

to handle faults. So comparing luster, pvfs which system

assumes faults are infrequent and needs manual intervention

to ensure continued services on other hand Hadoop turns out

to be very robust and fault tolerant option. Hadoop ensures

that few failures in the system won’t disrupt continued ser-

vice of data through automatic replication and transfer of

responsibilities from failed machines to live machines in

Hadoop farm transparently. Though it’s mentioned that GFS

has same capabilities since its not available to other compa-

nies those capabilities cannot be availed.

A. Meaning of hadoop

Hadoop is an Open Source implementation of a

large-scale batch processing system. Hadoop is a top-level

Apache project being built and used by a global community

of contributors, written in the Java programming language. It

provides a distributed file system and a framework for the

analysis and transformation of very large data sets using the

Map Reduce paradigm. Hadoop framework is written in

Java, it allows developers to deploy custom- written pro-

grams coded in Java or any other language to process data in

a parallel fashion across hundreds or thousands of commodi-

ty servers An important characteristic of Hadoop is the parti-

tioning of data and computation across many (thousands) of

hosts, and executing application computations in parallel

close to their data.

International Journal of Innovatory research in Engineering and Technology - IJIRET

ISSN: XXXX-XXXX Volume X, Issue X, Month Year 22

Fig1.Hadoop systems[6]

The table 1 shows the components of hadoop. Hadoop is an

Apache project; all components are available via the Apache

open source license. Yahoo! has developed and contributed

to 80% of the core of Hadoop (HDFS and Map Reduce).

HBase was originally developed at

Table1 .Hadoop project components [4]

Power set, now a department at Microsoft.

Hive was originated and developed at Facebook. Pig,

Zookeeper, and Chukwa were originated and developed at

Yahoo! Avro was originated at Yahoo! and is being co-

developed with Cloudera. HDFS is the file system compo-

nent of Hadoop. While the interface to HDFS is patterned

after the UNIX file system, faithfulness to standards was

sacrificed in favor of improved performance for the applica-

tions at hand.

B.HDFS architecture

The Hadoop Distributed File System [3] is a distributed file

system designed to run on commodity hardware. It has many

similarities with existing distributed file systems. However,

the differences from other distributed file systems are signif-

icant. HDFS is highly fault-tolerant and is designed to be

deployed on low-cost hardware. HDFS provides high

throughput access to application data and is suitable for ap-

plications that have large data sets. HDFS stores file system

metadata and application data separately. A normal file sys-

tem is separated into several pieces called blocks, which are

the smallest units that can be read or written.

Normally the default size is a few kilobytes. HDFS also has

blocks, but of a much larger size, 64 MB bytes default. The

reason for that is to minimize the costs of seeks for finding

the start of the block. With the abstraction of blocks it is

possible to create files that are larger than any single disk in

the network. HDFS architecture consists of NameNode,

DataNode, and HDFS Client.

Fig2: HDFS architecture[3]

C. Name node

The HDFS namespace is a hierarchy of files and directories.

Files and directories are represented on the NameNode[3] by

inodes, which record attributes like permissions, modifica-

tion and access times, namespace and disk space quotas. The

file content is split into large blocks and each block of the

file is independently replicated at multiple DataNodes. The

NameNode maintains the namespace tree and the mapping

of file blocks to DataNodes. An HDFS client wanting to read

a file first contacts the NameNode for the locations of data

blocks comprising the file and then reads block contents

from the DataNode closest to the client. When writing data,

the client requests the NameNode to nominate a suite of

three DataNodes to host the block replicas. The client then

writes data to the DataNodes in a pipeline fashion. The cur-

rent design has a single NameNode for each cluster. The

cluster can have thousands of DataNodes and tens of thou-

sands of HDFS clients per cluster, as each DataNode may

execute multiple application tasks concurrently. HDFS keeps

the entire namespace in RAM. The inode data and the list of

blocks belonging to each file comprise the metadata of the

name system called the image. The persistent record of the

International Journal of Innovatory research in Engineering and Technology - IJIRET

ISSN: XXXX-XXXX Volume X, Issue X, Month Year 23

image stored in the local host’s native files system is called a

checkpoint. The NameNode also stores the modification log

of the image called the journal in the local host’s native file

system. For improved durability, redundant copies of the

checkpoint and journal can be made at other servers. During

restarts the NameNode restores the namespace by reading

the namespace and replaying the journal. The locations of

block replicas may change over time and are not part of the

persistent checkpoint.

D.DATA NODE

Each block replica on a DataNode is represented by two files

in the local host’s native file system. The first file contains

the data itself and the second file is block’s metadata includ-

ing checksums for the block data and the block’s generation

stamp. The size of the data file equals the actual length of

the block and does not require extra space to round it up to

the nominal block size as in traditional file systems. Thus, if

a block is half full it needs only half of the space of the full

block on the local drive. During startup each DataNode con-

nects to the NameNode and performs a handshake. The pur-

pose of the handshake is to verify the namespace ID and the

software version of the DataNode. If either does not match

that of the NameNode the DataNode automatically shuts

down. The namespace ID is assigned to the file system in-

stance when it is formatted. The namespace ID is persistent-

ly stored on all nodes of the cluster. Nodes with a different

namespace ID will not be able to join the cluster, thus pre-

serving the integrity of the file system. The consistency of

software versions is important because incompatible version

may cause data corruption or loss, and on large clusters of

thousands of machines it is easy to overlook nodes that did

not shut down properly prior to the software upgrade or were

not available during the upgrade. A DataNode that is newly

initialized and without any namespace ID is permitted to join

the cluster and receive the cluster’s namespace ID. After the

handshake the DataNode registers with the NameNode.

DataNodes persistently store their unique storage IDs. The

storage ID is an internal identifier of the DataNode, which

makes it recognizable even if it is restarted with a different

IP address or port. The storage ID is assigned to the

DataNode when it registers with the NameNode for the first

time and never changes after that. A DataNode identifies

block replicas in its possession to the NameNode by sending

a block report. A block report contains the block id, the gen-

eration stamp and the length for each block replica the serv-

er hosts. The first block report is sent immediately after the

DataNode registration. Subsequent block reports are sent

every hour and provide the NameNode with an up-todate

view of where block replicas are located on the cluster.

D.HDFC CLIENT User applications access the file system using the HDFS

client, a code library that exports the HDFS file system inter-

face. HDFS supports operations to read, write and delete

files, and operations to create and delete directories. When

an application reads a file, the HDFS client first asks the

NameNode for the list of DataNodes that host replicas of the

blocks of the file. It then contacts a DataNode directly and

requests the transfer of the desired block. When a client

writes, it first asks the NameNode to choose DataNodes to

host replicas of the first block of the file. When the first

block is filled, the client requests new DataNodes to be cho-

sen to host replicas of the next block. The interactions

among the client, the NameNode and the DataNodes are

illustrated in Fig.2

HDFS cluster has a single name node that manages the file

system namespace. The current limitation that a cluster can

contain only a single name node results in the following is-

sues:

1. Scalability: Name node maintains the entire file system metadata in

memory. The size of the metadata is limited by the physical

memory available on the node. To address these issues one

encourages larger block sizes, creating a smaller number of

larger files and using tools like the hadoop archive (har).

2. Isolation:

No isolation for a multi‐tenant environment. An experi-

mental client application that puts high load on the central

name node can impact a production application.

3. Availability: While the design does not prevent building a failover mech-

anism, when a failure occurs the entire namespace and hence

the entire cluster is down.

E.ADVANTAGE 1. Distribute data and computation. The computation

local to data prevents the network overload.

2. Simple programming model. The end user pro-

grammer only writes map-reduce tasks.

3. HDFS store large amount of information.

4. HDFS is simple and robust coherency model.

5. Data will be written to the HDFS once and then read

several times.

6. Fault tolerance by detecting faults and applying

quick, automatic recovery.

7. Ability to rapidly process large amounts of data in

parallel

8. Can be offered as an on-demand service, for exam-

ple as part of Amazon’s EC2 cluster computing service

E.LIMITATIONS

1. Rough manner:- Hadoop Map-reduce and HDFS

are rough in manner. Because the software under active de-

velopment.

2. Programming model is very restrictive:- Lack of

central data can be preventive.

3. Still single master which requires care and may

limit scaling.

4. Managing job flow isn’t trivial when intermediate

data should be kept.

5.Cluster management is hard:- In the cluster, opera-

tions like debugging, distributing software, collection logs

etc. are too hard.

F.CONCLUSION

We have seen the components of hadoop and the hadoop

distributed file system in brief. As compare to other file sys-

International Journal of Innovatory research in Engineering and Technology - IJIRET

ISSN: XXXX-XXXX Volume X, Issue X, Month Year 24

tem HDFS is a highly fault tolerance system. HDFS was its

single NameNode which handles all metadata operations.

G. REFERANCES

[1] Apache Hadoop. http://hadoop.apache.org/

[2] S.Ghemawat, H Gobioff, S. Leung. “The Google file

system,” In Proc. of ACM Symposium on Operating Sys-

tems Principles, Lake George, NY, Oct 2003, pp. 29–43.

[3]Konstantin Shvachko, et al. “The Hadoop Distributed File

System,” Mass Storage Systems and Technologies (MSST),

IEEE 26th Symposium on

IEEE,2010,http://storageconference.org/2010/Papers/MSST/

Shvachko.pdf.

[4] P.H. Carns, W. B. Ligon III, R. B. Ross, and R. Thakur.

“PVFS: A parallel file system for Linux clusters,” in Proc. of

4th Annual Linux Showcase and Conference, 2000, pp. 317–

327.

[5] J. Venner, Pro Hadoop. Apress, June 22, 2009.

[6].http://hadoop.apache.org/docs/r0.20.0/hdfs_design.html.