building a hadoop connector
Post on 19-Jun-2015
59 Views
Preview:
DESCRIPTION
TRANSCRIPT
pastiaro.wordpress.com
@rpastia
Building a connector – The Wrong Way
Mapper Reducer
Building a connector – The Right Way
Mapper ReducerPartitioner
InputSplit
InputFormat
RecordReader
RecordWriter
OutputFormat
The InputFormat: From Input to Mapper--range 2014-09-01;2014-09-20
--number_of_mappers 4
2014-09-01 2014-09-022014-09-03
2014-09-04
2014-09-05
… … …
2014-09-06
2014-09-20
2014-09-01
2014-09-02
2014-09-05
.
.
.
Input Split 1
(2014-09-01-A; record A)
(2014-09-01-B; record B)
(2014-09-01-…; record …)
(2014-09-02-A; record A)
(2014-09-02-B; record B)
(2014-09-02-…; record …)
(2014-09-05-A; record A)
(2014-09-05-B; record B)
(2014-09-05-…; record …)
Record Reader 1
Mapper
The InputFormat: From Input to Mapper
--range 2014-09-01;2014-09-20
--number_of_mappers 4
2014-09-01 2014-09-022014-09-03
2014-09-04
2014-09-05
… … …
2014-09-06
2014-09-20
2014-09-01
2014-09-02
2014-09-05
.
.
.
Input Split 1
(2014-09-01-A; record A)
(2014-09-01-B; record B)
(2014-09-01-…; record …)
(2014-09-02-A; record A)
(2014-09-02-B; record B)
(2014-09-02-…; record …)
(2014-09-05-A; record A)
(2014-09-05-B; record B)
(2014-09-05-…; record …)
Record Reader 1
Mapper
top related