drone data flowing through apache nifi
TRANSCRIPT
1 ©HortonworksInc.2011– 2016.AllRightsReserved
DroneDataFlowingThroughApacheNiFiTimothySpann2016FutureofData– PrincetonMeetup
2 ©HortonworksInc.2011– 2016.AllRightsReserved
3 ©HortonworksInc.2011– 2016.AllRightsReserved
4 ©HortonworksInc.2011– 2016.AllRightsReserved
hdfs dfs -get/drone/meta/Bebop2_20160920085308-0400.json{"date":"2016-09-20T08:53:08","Compression":"JPEG","ExifVersion":"2.10","ComponentsConfiguration":"YCbCr","file.group":"root","CompressionType":"Baseline","Image Description":"{\"product_id\":\"090C\",\"uuid\":\"CE112B4276E13339AFBAE10E9ED794E3\",\"run_date\":\"2016-09-20T083537-0400\",\"filename\":\"Bebop_2_2016-09-20T085308-0400_CE112B.jpg\",\"media_date\":\"2016-09-20T085308-0400\"}","NumberofComponents":"3","Component2":"Cbcomponent:Quantizationtable1,Samplingfactors1horiz/1vert","Focal Length":"1.83mm","Component 1":"Ycomponent:Quantizationtable0,Samplingfactors2horiz/2vert","YCbCr Positioning":"Center ofpixelarray","tiff:ResolutionUnit":"Inch","uuid":"9576a956-ec32-4314-bc8b-bd5bb43af4f8","Date/TimeOriginal":"2016:09:2008:53:08","ShutterSpeedValue":"1/249sec","X Resolution":"72dotsperinch","tiff:Make":"PARROT","path":"/","PhotometricInterpretation":"YCbCr","Component3":"Crcomponent:Quantizationtable1,Samplingfactors1horiz/1vert","Unique ImageID":"[32bytes]","F-Number":"F2.3","modified":"2016-09-20T08:53:08","FocalLength35":"6mm","tiff:BitsPerSample":"8","ExposureProgram":"Program action(high-speedprogram)","GPSVersionID":"2.200","GPSLatitudeRef":"N","meta:creation-date":"2016-09-20T08:53:08","exif:FNumber":"2.2999999166745724","GPSAltitudeRef":"Sea level","Exposure Time":"8597231/2147483647sec","GPS Longitude":"-73° 4'50.41\"","Creation-Date":"2016-09-20T08:53:08","ISOSpeedRatings":"426","Make":"PARROT","Orientation":"Top,leftside(Horizontal/normal)","MeteringMode":"(Other)","tiff:Orientation":"1","GPSLongitudeRef":"W","tiff:Software":"Dragon3.9.0","exif:FocalLength":"1.8300001180412324","filename":"Bebop2_20160920085308-0400.jpg","XMPValueCount":"9","geo:long":"-73.080669","file.owner":"root","Software":"Dragon3.9.0","ExifImageHeight":"1088pixels","tiff:YResolution":"72.0","YResolution":"72dotsperinch","GPSLatitude":"40° 45'16.51\"","dc:description":"{\"product_id\":\"090C\",\"uuid\":\"CE112B4276E13339AFBAE10E9ED794E3\",\"run_date\":\"2016-09-20T083537-0400\",\"filename\":\"Bebop_2_2016-09-20T085308-0400_CE112B.jpg\",\"media_date\":\"2016-09-20T085308-0400\"}","geo:lat":"40.754585","FlashPixVersion":"1.00","DataPrecision":"8bits","White Balance":"Flash","tiff:ImageLength":"1088","description":"{\"product_id\":\"090C\",\"uuid\":\"CE112B4276E13339AFBAE10E9ED794E3\",\"run_date\":\"2016-09-20T083537-0400\",\"filename\":\"Bebop_2_2016-09-20T085308-0400_CE112B.jpg\",\"media_date\":\"2016-09-20T085308-0400\"}","dcterms:created":"2016-09-20T08:53:08","dcterms:modified":"2016-09-20T08:53:08","Last-Modified":"2016-09-20T08:53:08","file.permissions":"rwxrwxrwx","exif:ExposureTime":"0.00400339765660623","Last-Save-Date":"2016-09-20T08:53:08","GPSAltitude":"21metres","absolute.path":"/opt/demo/dronedata/","ColorSpace":"Undefined","File Size":"853159bytes","meta:save-date":"2016-09-20T08:53:08","file.creationTime":"2016-09-20T12:53:10+0000","Date/TimeDigitized":"2016:09:2008:53:08","FileName":"apache-tika-941370357006559178.tmp","Content-Type":"image/jpeg","Aperture Value":"F2.3","X-Parsed-By":"org.apache.tika.parser.DefaultParser,org.apache.tika.parser.jpeg.JpegParser","FileModifiedDate":"Tue Sep2023:33:24UTC2016","tiff:XResolution":"72.0","file.lastModifiedTime":"2016-09-20T12:53:10+0000","exif:DateTimeOriginal":"2016-09-20T08:53:08","Date/Time":"2016:09:2008:53:08","ExifImageWidth":"1920pixels","Image Height":"1088pixels","Image Width":"1920pixels","Unknown tag(0xc62f)":"[19bytes]","ResolutionUnit":"Inch","tiff:Model":"Bebop2","exif:IsoSpeedRatings":"426","MaxApertureValue":"F2.3","ExposureMode":"Auto exposure","Model":"Bebop 2","file.lastAccessTime":"2016-09-20T23:33:24+0000","tiff:ImageWidth":"1920","WhiteBalanceMode":"Auto whitebalance"}
5 ©HortonworksInc.2011– 2016.AllRightsReserved
pythonclassify_image.py --image_file /opt/demo/dronedataold/Bebop2_20160920083655-0400.jpgsolardish,solarcollector,solarfurnace(score=0.98316)windowscreen(score=0.00196)manholecover(score=0.00070)radiator(score=0.00041)doormat,welcomemat(score=0.00041)
bazel-bin/tensorflow/examples/label_image/label_image --image=/opt/demo/dronedataold/Bebop2_20160920083655-0400.jpgtensorflow/examples/label_image/main.cc:204]solardish(577):0.983162Itensorflow/examples/label_image/main.cc:204]windowscreen(912):0.00196204Itensorflow/examples/label_image/main.cc:204]manholecover(763):0.000704005Itensorflow/examples/label_image/main.cc:204]radiator(571):0.000408321Itensorflow/examples/label_image/main.cc:204]doormat(972):0.000406186
Local TensorFlow via Python or C++ Binary
6 ©HortonworksInc.2011– 2016.AllRightsReserved
Google Vision
7 ©HortonworksInc.2011– 2016.AllRightsReserved
Google Vision
8 ©HortonworksInc.2011– 2016.AllRightsReserved
Google VisionGoogle Vision
9 ©HortonworksInc.2011– 2016.AllRightsReserved
IBM
10 ©HortonworksInc.2011– 2016.AllRightsReserved
Microsoft Computer Vision API
11 ©HortonworksInc.2011– 2016.AllRightsReserved
Microsoft Computer Vision API
12 ©HortonworksInc.2011– 2016.AllRightsReserved
NIFITimothy Spann
13 ©HortonworksInc.2011– 2016.AllRightsReserved
Installation
Download the binary from here: http://hortonworks.com/downloads/#dataflowOr here:https://nifi.apache.org/download.htmlOr on Mac:brew install nifi
https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#starting-nifi
bin/nifi.sh start
14 ©HortonworksInc.2011– 2016.AllRightsReserved
TheFutureofData:ActionableIntelligence
D A T A I N M O T I O N
STO
RA
GE
STO
RA
GE
GROUP 2GROUP 1
GROUP 4GROUP 3
D A T A A T R E S T
INTERNETOF
ANYTHING
Hortonworks’uniqueapproachtodata-in-motionanddata-at-restpowersActionableIntelligence
15 ©HortonworksInc.2011– 2016.AllRightsReserved
DATAATREST
DATAINMOTION
ACTIONABLEINTELLIGENCE
MODERNDATAAPPLICATIONS
ActionableIntelligencefromConnectedDataPlatforms
Capturingperishableinsightsfromdatainmotion
Ensuringrich,historicalinsightsondataatrest
Necessaryformoderndataapplications
HortonworksDataFlow
HortonworksDataPlatform
16 ©HortonworksInc.2011– 2016.AllRightsReserved
HDFProvides“DataPlanofControl”byManagingIoT Dataflows
ConstrainedHigh-Latency
Localized Context
Hybrid – Cloud/On-PremiseLow-Latency
Global Context
Datasourceagnosticcollectionofdataacrossheterogeneousenvironments
17 ©HortonworksInc.2011– 2016.AllRightsReserved
RapidEcosystemAdoption:170+Processors
HTTP
Syslog
HTML
Image
Hash Encrypt
Extract
TailMerge
Evaluate
Duplicate Execute
Scan
GeoEnrich
Replace
ConvertSplit
Translate
HL7
FTP
UDP
XML
SFTP
RouteContent
RouteContext
RouteText
ControlRate
DistributeLoadAMQP
18 ©HortonworksInc.2011– 2016.AllRightsReserved
Data-in-motion:HortonworksDataFlowPoweredbyApacheNiFi
Collect,conductandcuratereal-timedata
End-to-endsecuritywithencryptionandrules
Traceabilityandreal-timeprovenance
DeliversInstant,PerishableInsights
HortonworksDataFlow
Data-in-motion
19 ©HortonworksInc.2011– 2016.AllRightsReserved
Accumulate,Analyze,ActonAllData
CentralizedArchitectureforMulti-tenancy
EnterpriseOperations,GovernanceandSecurity
DeliversRichHistoricalInsights
Data-at-rest:HortonworksDataPlatformPoweredbyApacheHadoop
HortonworksDataPlatform
Data-at-rest
20 ©HortonworksInc.2011– 2016.AllRightsReserved
à https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/
à https://github.com/jfrazee/awesome-nifi
à https://dzone.com/articles/getting-started-with-apache-nifi-and-hdf
à https://nifi.apache.org/docs.html
à https://community.hortonworks.com/articles/4356/getting-started-with-nifi-expression-language-and.html
LearningMore
21 ©HortonworksInc.2011– 2016.AllRightsReserved
Contact:
[email protected]/futureofdata-princeton
community.hortonworks.com/users/9304/tspann.html
22 ©HortonworksInc.2011– 2016.AllRightsReserved
HortonworksCommunityConnection
Read access for everyone, join to participate and be recognized
• FullQ&APlatform(likeStackOverflow)
• KnowledgeBaseArticles
• CodeSamplesandRepositories
23 ©HortonworksInc.2011– 2016.AllRightsReserved
CommunityEngagement
Participate now at: community.hortonworks.com©HortonworksInc.2011– 2015.AllRightsReserved
4,000+RegisteredUsers
10,000+Answers
15,000+TechnicalAssets
One Website!