implementation of a jni wrapper & mapreduce compatibility layer for streammine3g … ·...
TRANSCRIPT
![Page 1: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/1.jpg)
Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G
by Benjamin Schmidt
Faculty of Computer Science Institute of Systems Architecture, Chair of Systems Engineering
Dresden, 25.11.13
![Page 2: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/2.jpg)
Dresden, 25.11.13 Folie 2 von 25
Outline
• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
![Page 3: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/3.jpg)
Dresden, 25.11.13 Folie 3 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
![Page 4: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/4.jpg)
Dresden, 25.11.13 Folie 4 von 25
Introduction to StreamMine3G
![Page 5: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/5.jpg)
Dresden, 25.11.13 Folie 5 von 25
Introduction to StreamMine3G
![Page 6: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/6.jpg)
Dresden, 25.11.13 Folie 6 von 25
Introduction to StreamMine3G
...
...
![Page 7: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/7.jpg)
Dresden, 25.11.13 Folie 7 von 25
Introduction to StreamMine3G
• Event processing engine• Written in C++• Provides fault tolerance at low cost• Supports stateful operators
![Page 8: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/8.jpg)
Dresden, 25.11.13 Folie 8 von 25
Introduction to StreamMine3G
![Page 9: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/9.jpg)
Dresden, 25.11.13 Folie 9 von 25
Introduction to StreamMine3G
• Streams – Sequence of tuples/events• Topology – Pattern of communication
between Operators
• Slices/Partitions – Can be deployed on different nodes
![Page 10: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/10.jpg)
Dresden, 25.11.13 Folie 10 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
![Page 11: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/11.jpg)
Dresden, 25.11.13 Folie 11 von 25
JNI Wrapper : Introduction
Java
![Page 12: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/12.jpg)
Dresden, 25.11.13 Folie 12 von 25
JNI Wrapper : Introduction
![Page 13: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/13.jpg)
Dresden, 25.11.13 Folie 13 von 25
![Page 14: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/14.jpg)
Dresden, 25.11.13 Folie 14 von 25
JNI Wrapper : Bottlenecks
• C++ more efficient execution• Additional JNI Redirections• Datatype conversion• Generation of Java Objects during
De-/Serialization C++
![Page 15: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/15.jpg)
Dresden, 25.11.13 Folie 15 von 25
JNI Wrapper : Reasons for Java
• C++ rather difficult to implement• Most Pupils and Students learn Java from
the Beginning• Already implemented solutions• Possibly easier to debug
![Page 16: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/16.jpg)
Dresden, 25.11.13 Folie 16 von 25
JNI Wrapper: Problems
• Datatype conversion (Pointer & String)• Thread handling• Optimization (Bytearrays)
![Page 17: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/17.jpg)
Dresden, 25.11.13 Folie 17 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
![Page 18: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/18.jpg)
Dresden, 25.11.13 Folie 18 von 25
Hadoop MapReduce
• Written in Java• Cluster of Nodes• Needs Mapper & Reducer Operator• Before Reducer phase Mapper phase must →
be finished• Batch Processing
![Page 19: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/19.jpg)
Dresden, 25.11.13 Folie 19 von 25
MapReduce Compatibility Layer
JNI Wrapper
MapReduce CL
![Page 20: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/20.jpg)
Dresden, 25.11.13 Folie 20 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
![Page 21: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/21.jpg)
Dresden, 25.11.13 Folie 21 von 25
Future Tasks : Apache Pig
• Abstraction build on top of Hadoop• consists of a domain-specific dataflow
language, Pig Latin• Generates and compiles MapReduce
programs on the fly
![Page 22: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/22.jpg)
Dresden, 25.11.13 Folie 22 von 25
Future Tasks : Apache Pig
Apache Pig Compatability Layer
Hadoop MapReduceCompatability Layer
![Page 23: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/23.jpg)
Dresden, 25.11.13 Folie 23 von 25
Future Task : Storm
• Real-time processing• Topologies run forever• Scalable and fault-tolerant• Consists of spout, bolts, event streams and
Nimbus
![Page 24: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/24.jpg)
Dresden, 25.11.13 Folie 24 von 25
Future Tasks : Storm
STORM OperatorCompatability Layer
STORM
PerformanceEvaluation
?
STORMOperator
STORMOperator
![Page 25: Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G … · 2013-11-25 · Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G](https://reader030.vdocuments.site/reader030/viewer/2022040608/5ec573d475aebe15262d3f80/html5/thumbnails/25.jpg)
Dresden, 25.11.13 Folie 25 von 25
Thank you for your attention!