apache apex - hadoop users group
TRANSCRIPT
![Page 1: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/1.jpg)
Pramod Immaneni <[email protected]>PPMC Member, Architect @DataTorrent IncDec 2nd, 2015
Stream Processing Architecture and ApplicationsApache Apex (incubating)
![Page 2: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/2.jpg)
© 2015 DataTorrent2
Apex Platform Overview
![Page 3: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/3.jpg)
© 2015 DataTorrent3
Apache Malhar Library
![Page 4: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/4.jpg)
© 2015 DataTorrent4
Native Hadoop Integration
• YARN is the resource manager
• HDFS used for storing any persistent state
![Page 5: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/5.jpg)
© 2015 DataTorrent5
Application Programming Model
A Stream is a sequence of data tuplesAn Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library• Operator has many instances that run in parallel and each instance is single-threaded
Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Filtered Stream
Output StreamTuple Tuple
Filtered Stream
Enriched Stream
Enriched
Stream
er
Operator
er
Operator
er
Operator
er
Operator
![Page 6: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/6.jpg)
© 2015 DataTorrent6
Advanced Windowing Support
Application window Sliding window and tumbling window
Checkpoint window No artificial latency
![Page 7: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/7.jpg)
© 2015 DataTorrent7
Application Specification
![Page 8: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/8.jpg)
© 2015 DataTorrent8
Partitioning and unification
![Page 9: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/9.jpg)
© 2015 DataTorrent9
Advanced Partitioning
![Page 10: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/10.jpg)
© 2015 DataTorrent10
Dynamic Partitioning
• Partitioning change while application is runningᵒ Change number of partitions at runtime based on statsᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitionsᵒ Supports re-distribution of state when number of partitions changeᵒ API for custom scaler or partitioner
unifiers not shown
1a 2a
1b 2b
3
2b
1b 2c
3
2a
2d
1a 2b
1b 2c 3b
2a
2d
3a1a
![Page 11: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/11.jpg)
© 2015 DataTorrent11
How tuples are partitioned• Tuple hashcode and mask used to determine destination
partitionᵒ Mask picks the last n bits of the hashcode of the tupleᵒ hashcode method can be overridden
• StreamCodec can be used to specify custom hashcode for tuplesᵒ Can also be used for specifying custom serialization
tuple: {Name, 24204842, San Jose}
Hashcode: 001010100010101
Mask (0x11)
Partition
00 1
01 2
10 3
11 4
![Page 12: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/12.jpg)
© 2015 DataTorrent12
Custom partitioning• Custom distribution of tuples
ᵒ E.g.. Broadcast
tuple:{Name, 24204842, San Jose}
Hashcode: 001010100010101
Mask (0x00)
Partition
00 1
00 2
00 3
00 4
![Page 13: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/13.jpg)
© 2015 DataTorrent13
Fault Tolerance• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint stateᵒ Frequency configurable per operatorᵒ Asynchronous and distributed by defaultᵒ Default store is HDFS
• Automatic detection and recovery of failed operatorsᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so that there is no loss of data
• Application master state checkpointed
![Page 14: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/14.jpg)
© 2015 DataTorrent14
Processing GuaranteesAtleast once• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lostᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following to ensure data is written once to store in case of fault recoveryᵒ Transactions with meta information, Rewinding output, Feedback from
external entity, Idempotent operationsAtmost once• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is sufficient
Exactly once• Operators checkpointed every window
ᵒ Can be combined with transactional mechanisms to ensure end-to-end exactly once behavior
![Page 15: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/15.jpg)
© 2015 DataTorrent15
Stream Locality• By default operators are deployed in containers (processes)
randomly on different nodes across the Hadoop cluster• Custom locality for streams
ᵒ Rack local: Data does not traverse network switchesᵒ Node local: Data is passed via loopback interface and frees up
network bandwidthᵒ Container local: Messages are passed via in memory queues
between operators and does not require serializationᵒ Thread local: Messages are passed between operators in a same
thread equivalent to calling a subsequent function on the message
![Page 16: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/16.jpg)
© 2015 DataTorrent16
Data Processing Pipeline ExampleApp Builder
![Page 17: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/17.jpg)
© 2015 DataTorrent17
Monitoring ConsoleLogical View
![Page 18: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/18.jpg)
© 2015 DataTorrent18
Monitoring ConsolePhysical View
![Page 19: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/19.jpg)
© 2015 DataTorrent19
Real-Time DashboardsReal Time Visualization
![Page 20: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/20.jpg)
© 2015 DataTorrent20
ResourcesApache Apex Community Page - http://apex.incubator.apache.org/
![Page 21: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/21.jpg)
© 2015 DataTorrent21
We Need Your Vote (Today)
Introducing Apache Apex - Not Just Another Stream Processing PlatformNext Gen Big Data Analytics with Apache ApexEnterprise-grade streaming under 2ms on Hadoop
![Page 22: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/22.jpg)
End
22
![Page 23: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/23.jpg)
© 2015 DataTorrent
Extra Slides
![Page 24: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/24.jpg)
© 2015 DataTorrent24
Partitioning and Scaling Out
• Operators can be dynamically scaled• Flexible Streams split• Parallel partitioning
• MxN partitioning • Unifiers
![Page 25: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/25.jpg)
© 2015 DataTorrent25
Fault Tolerance OverviewStateful Fault Tolerance Processing Semantics Data Locality
Supported out of the box– Application state– Application master state– No data loss
Automatic recovery Lunch test Buffer server
At least once At most once Exactly once
Stream locality for placement of operators
Rack local – Distributed deployment
Node local – Data does not traverse NIC
Container local – Data doesn’t need to be serialized
Thread local – Operators run in same thread
Data locality
![Page 26: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/26.jpg)
© 2015 DataTorrent26
Machine Data ApplicationLogical View
![Page 27: Apache Apex - Hadoop Users Group](https://reader034.vdocuments.site/reader034/viewer/2022042706/589a8b521a28abae648b471b/html5/thumbnails/27.jpg)
© 2015 DataTorrent27
Machine Data ApplicationPhysical View