day7.hdfs & architecture

http://www.excelonlineclasses.co.nr/


[email protected]



mailto:[email protected]


Online Training Development Testing Job support Technical Guidance Job Consultancy Any needs of IT Sector

Excel Online Classes offers following services:


HDFS

- Nagarjuna K


HDFS Distributed FS designed to run on

Commodity Hardware

Provides high throughput access to application data , suitable for applications having large datasets


Assumptions & Goals Hardware Failure Streaming Data Access Large Datasets Simple coherency Model Moving Computation cheaper than

moving data


Hardware Failure Assumptions & Goals

HDFS instance many machines Each storing part of the data

Chances that any machine goes down can’t be avoided

Detection of faults, auto recovery is core architectural goal of HDFS


Streaming Data Access Assumptions & Goals

HDFS is designed fro batch processing rather than interactive usage by users.

Emphasis on Data throughput Not on low Latency data access.


Streaming Data Access Assumptions & Goals

HDFS built on !dea “Write once , Read many times pattern”

Overtime data set generated and placed in HDFS Analysis is done one large part of data , rather

than on first few records Time to read whole data set is more than

retrieving first or the last record.


Large Datasets Assumptions & Goals

A typical file ranges from GB to TB


Simple Coherency Model Assumptions & Goals

HDFS built on !dea “Write once , Read many times pattern”

The assumption enables high through put access


Moving Computation OR Data ? Assumptions & Goals

Computation intensive porgraming

Data intensive programing


Where HDFS doesn’t fit Low latency data access

Lots of small files

Multiple writers, arbitrary file modifications


Where HDFS doesn’t fit Low latency data access

Lots of small files High latency time Each file (say 10 KB of size) takes up a block

in HDFS Compress All the metadata is stored in HDFS memory


Where HDFS doesn’t fit Multiple writers, arbitrary file

modifications Single user writes files in HDFS.

Appending only at the end. Multiple sources of writing into a same file or writing at arbitrary offset is not supported (currently)


Blocks disc has block size

minimum amount of data that is read/write

512 bytes FileSystem blocks are few multiple of

disc block size few KB


Blocks In classical FS, single block may

contain data of only single file Leads to internal fragmentation.

Newer file systems, solves this problem by block suballocation tail merging


Blocks HDFS also has a block size

64 MB

Unlike normal FS , if file is less than 64 MB it doesn’t occupy underlying storage of 64MB.


Why BIG BLOCK size ? Throughput vs Latency

time to seek start of block Reading the whole block


Why BIG BLOCK size ? seek time = 10ms transfer rate (throughput) = 100MBPS

make seek time 1% of transfer rate , block size = 100MB

Default is 64 MB As the transfer rate increases , Block

size can be increased


hadoop fsck / -files -blocks Gives information about all the files and

blocks in the file system Replication

▪ under▪ over etc.,

corrupt ? etc.,


File Permissions on HDFS Client’s identity determined

user name and groups from which it operates.

Sharing of FS shouldn’t be used hostile environment

Going forward Kerberos authentication


Hadoop File Systems HDFS is just one implementation of

Hadoop FileSystems. org.apache.hadoop.fs.FileSystem

represents a FileSystem in hadoop


Hadoop File Systems

day7.hdfs & architecture

Documents