you can't search without data

YouCan’tSearchWithoutDataBryanBende– StaffSoftwareEngineer@HortonworksNYCSolr/LuceneMeetup– December7th 2017

2 ©HortonworksInc.2011– 2016.AllRightsReserved

Agenda

Ã TheProblem

Ã ApacheNiFi Overview

Ã IntegrationbetweenNiFi &Solr

Ã Recent&FutureWork

Ã DemoCoolStuff!

Ã Q&A

AboutMe

Ã StaffSoftwareEngineer@Hortonworks

Ã ApacheNiFi PMC&Committer

Ã ContributedSolr processorsinMarch2015– https://issues.apache.org/jira/browse/NIFI-461

Ã bbende@hortonworks.com /Twitter@bbende /bryanbende.com

TheProblem

Itstartsoutsosimple…

Hey!Wehavesomeimportantdatato

sendyou!

Cool!Yourdataisreallyimportantto

Thisshouldbeeasyright?...

Butwhataboutformats&protocols?

WecanpublishAvrorecordstoaKafkatopic,does

thatwork?

Oh,wellwehaveaRESTservicethataccepts

JSON…

Andwhataboutsecurity&authentication?

Hmmwhataboutsecurity?Wecanauthenticatevia

Kerberos

Sorry,weonlysupport2-Way

TLSwithcertificates

Andwhataboutallthesedevicesattheedge?

Wealsoneedtograbdatafromallthesedevices,howarewegoingtodo

Wouldn’titbeniceiftherewasatoolthatcouldhelptheseteams?

EnterApacheNiFi…

Apache NiFi

• Created to address the challenges of global enterprise dataflow• Key features:

– VisualCommandandControl

– DataLineage(Provenance)

– DataPrioritization

– DataBuffering/Back-Pressure

– ControlLatencyvs.Throughput

– SecureControlPlane/DataPlane

– ScaleOutClustering

– Extensibility

NiFi Core Concepts

FBPTerm NiFi Term DescriptionInformationPacket

FlowFile Each objectmovingthroughthesystem.

Black Box FlowFileProcessor

Performsthework, doingsomecombinationofdatarouting,transformation,ormediationbetweensystems.

BoundedBuffer

Connection Thelinkage betweenprocessors, actingasqueuesandallowingvariousprocessestointeractatdifferingrates.

Scheduler FlowController

Maintainstheknowledgeofhowprocessesareconnected, andmanagesthethreadsandallocationsthereofwhichallprocessesuse.

Subnet ProcessGroup

Asetofprocessesandtheirconnections,whichcanreceiveandsenddataviaports.Aprocess groupallowscreationofentirelynewcomponentsimplybycompositionofits components.

VisualCommand&Control

• Drag& dropprocessorstobuildaflow

• Start,stop,&configurecomponentsinreal-time

• Viewerrors& correspondingmessages

• Viewstatistics& healthof thedataflow

• Create shareable templatesofcommonflows

Provenance/Lineage

• Tracksdataateachpointasitflowsthroughthesystem

• Records,indexes,andmakeseventsavailablefordisplay

• Handlesfan-in/fan-out,i.e.mergingandsplittingdata

• Viewattributesandcontentatgivenpointsintime

Prioritization

• Configureaprioritizer perconnection

• Determinewhatisimportantforyourdata– timebased,arrivalorder,importanceofadataset

• Funnelmanyconnectionsdowntoasingleconnectiontoprioritizeacrossdatasets

Back-Pressure

• Configureback-pressureperconnection

• BasedonnumberofFlowFiles ortotalsizeofFlowFiles

• Upstreamprocessornolongerscheduledtorununtilbelowthreshold

Latencyvs.Throughput

• Choosebetweenlowerlatency,orhigherthroughputoneachprocessor

• Higherthroughputallowsframeworktobatchtogetheralloperationsfortheselectedamountoftimeforimprovedperformance

• Processordeveloperdetermineswhethertosupportthisbyusing@SupportsBatchingannotation

Architecture- Standalone

OS/Host

FlowController

WebServer

Processor1 ExtensionN

FlowFileRepository

ContentRepository

ProvenanceRepository

LocalStorage

Ã FlowFile Repository– WriteAheadLog– StateofeveryFlowFile– Pointerstocontentrepository

(pass-by-reference)

Ã ContentRepository– FlowFile content– Copy-on-write

Ã ProvenanceRepository– WriteAheadLog+Lucene Indexes– Store&searchlineageevents

OS/Host

FlowController

WebServer

FlowFileRepository

ContentRepository

LocalStorage

OS/Host

FlowController

WebServer

FlowFileRepository

ContentRepository

LocalStorage

Architecture- Cluster

OS/Host

FlowController

WebServer

FlowFileRepository

ContentRepository

LocalStorage

ZooKeeper

Ã Samedataflowoneachnode,datapartitionedacrosscluster

Ã AccesstheUIfromanynodeÃ ZooKeeper forauto-electionof

ClusterCoordinator&PrimaryNode

Ã ClusterCoordinatorreceivesheartbeatsfromothernodes,managesjoining/disconnecting

Ã PrimaryNodeforschedulingprocessorsonasinglenode

NiFi &Solr

NiFi Solr Processors

Ã SupportSolr Cloudandstand-aloneSolr instances

Ã LeverageSolrJ (CloudSolrClient &HttpSolrClient)

Ã GetSolr – Extractnewdocuments

Ã PutSolrContentStream – Streamflowfilecontenttoanupdatehandler

PutSolrContentStream

Ã ChooseSolr Type– CloudorStandard

Ã SpecifyZooKeeper hosts,ortheSolr URLwithcore

Ã SpecifytheSolr pathfortheupdatehandler

Ã DynamicPropertiessentaskey/valuepairsonrequest

Ã Relationshipsforsuccess,failure,andconnectionfailure

GetSolr

Ã Incrementallyextractnewdocuments

Ã Mainqueryis*:*,Solr Queryisoptionalfilterquery

Ã DateFieldusedasfilterquery,fromlastexecutionorinitialvalue

Ã Sortedbydatefieldanduniquekey

Ã Cursormarkusedbehindthescenes

Ã Specifyreturnfields,orallifblank

Ã OutputSolr XML,orRecords

InteractingWithaSecureSolr

Ã BasicAuth– Providerusername/password

Ã Kerberos– SetJAASsystempropertyinbootstrap.conf– ProvidenameofJAASentryforprocessortouse

Ã TLS/SSL– ProvideanSSLContextService– One-wayTLSwithTruststore only– Two-wayTLSwithKeystore +Truststore

Recent&FutureWork

Problem– ConversionBetweenDataFormats

Ã Specializedprocessorstooperateondifferentdatatypes

Ã Sometimesmissingconversions

Ã Sometimesmissingaspecificfunctionforadatatype

Ã Sometimesimplementedwithdifferentlibrariescausinginconsistencies

Solution– RecordProcessing

Ã Introducetheconceptofa”record”– ReleasedinApacheNiFi 1.2.0(May2017),improvementsin1.3.0and1.4.0

Ã Centralizethelogicforreading/writingrecordsintocontrollerservices– Readers/WritersforCSV,Json,Avro,etc.

Ã Providestandardprocessorsthatoperateonrecords– ConvertRecord,QueryRecord,PartitionRecord,UpdateRecord,etc.

Ã Provideintegrationwithschemaregistries– LocalSchemaRegistry,HortonworksSchemaRegistry,ConfluentSchemaRegistry

Ã Canstillhandlearbitrarydata,butprocessrecordswhenappropriate

Problem– VariableHandling

Ã Needtoparametrizevaluesintheflowperenvironment– Connectionstrings,URLs,FileSystempaths,etc.

Ã Cansetvariablesinbootstrap.conf– -Dmy.var=foo

Ã Cansetapropertiesfileinnifi.properties– nifi.variable.registry.properties=production.properties

Ã Bothrequirecommandlineaccess

Ã Bothrequirerestarttopickupchanges

Solution– FirstClassVariableRegistry

Ã Variablesassociatedwithaprocessgroup,releasedin1.4.0

Ã Right-clickoncanvastoviewvariablesforcurrentgroup

Ã Hierarchicalorderofprecedence,resolveclosestreferencetocomponent

Ã Editingvariablesautomaticallyrestartsanycomponentsreferencingthevariables

Problem– HowdoIdeploymyflow?

Ã Mostorganizationswanttheclassicdevelopmentlifecycle(dev->int ->prod)

Ã Cancopyflow.xml.gz betweenenvironments– Requirescopyingentiredataflow– Can’ttellwhatchanged,hardtodiffifyouputinversioncontrol– Requiresallenvironmentsusethesameencryptionkeyforsensitiveproperties

Ã Canmaketemplatesforportionsoftheflow– Scriptcreationoftemplateanddeploymenttonextenvironment– Requiresstoppingflowandremovingcomponents,thenre-instantiatingtemplate– Noeasywaytoseechanges,hardtorollback

Solution– NiFi Registry

Ã DISCLAIMER- UNDERDEVELOPMENT&NOTRELEASEDYET!

Ã Complimentaryapplication,sub-projectofApacheNiFi– https://github.com/apache/nifi-registry– https://issues.apache.org/jira/projects/NIFIREG

Ã Centrallocationforstorage/managementofsharedresourcesacrossNiFi instances

Ã Initialcapabilitytostoreandretrieve“versionedflows”

Ã Aversionedflowisasnapshotofaprocessgroupatagivenpointintime

Ã Potentiallystoreextensions,shareddatasets,andmoreinthefuture

DEMO!!

ExampleScenario

Ã Userdata– https://randomuser.me

Ã InitiallyinCSVformat– name.title,name.first,name.last,email,registered– mr,dennis,reyes,dennis.reyes@example.com,2012-04-10 01:54:19– miss,carole,gomez,carole.gomez@example.com,2002-12-17 22:15:49

Ã Requirements– ConvertCSVtoJSON– Addafull_name fieldwithfirstname+lastname– Addagenderfieldbasedontitle(i.e.iftitle==mr thenMALE)– IngesttodifferentSolr collectionsdependingonenvironment

Questions?

Learnmoreandjoinus!

Apache NiFi sitehttp://nifi.apache.org

Subscribe to and collaborate atdev@nifi.apache.orgusers@nifi.apache.org

Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFI

Follow us on Twitter@apachenifi

Thankyou!

you can't search without data

Software

10 wordpress plugins you can't do without!

you can't spell "archie" without "acr" part 1

can't live without guide

i can't live without my radio

ten features of jetpack you can't live without

project10 newspaper (sept) ’can't live without’ issue

you can't spell "archie" without "acr" part two

can't be imagine diwali without crackers

40 taiwanese foods we can't live without

can't live with them, can't live without them

can't live without it! --- the 'audience-media relationship

7 hr & recruiting metrics you can't live without

you can't leave without visiting this places!

business process - can't live with them, can't live without...

you can't spell "archie" without "acr" part four

you can't fly, without supply

5 things baltimoreans can't do without

wordpress plugins you can't live without

12 anchors small business leaders can't lead without!

4 aca reports you can't do without