machine data 101 workshop audienceversion

96
Copyright © 2014 Splunk Inc. Machine Data 101: Turning Data into Insight Audience Version

Upload: splunk

Post on 16-Apr-2017

18 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Machine data 101 workshop audienceversion

Copyright©2014SplunkInc.

MachineData101:TurningDataintoInsight

AudienceVersion

Page 2: Machine data 101 workshop audienceversion

Agenda

§ Non-TraditionalDataSources

§ DataEnrichment

§ LevelUponSearchandReportingCommands

§ DataModelsandPivot

§ AdvancedVisualizationsandtheWebFramework

2

Page 3: Machine data 101 workshop audienceversion

Non-TraditionalDataSources

Page 4: Machine data 101 workshop audienceversion

Non-TraditionalDataSources

§ NetworkInputs

§ HTTPEventCollector

§ LogEventAlertAction

§ SplunkAppforStream

§ ScriptedInputs

§ DatabaseInputs

§ SplunkODBCDriver

§ ModularInputs

§ zLinux Forwarder

§ MINT

§ Non-SplunkDatastores

4

Page 5: Machine data 101 workshop audienceversion

TraditionalDataSources§ Captureseventsfromlogfilesinrealtime

§ Runsscriptstogathersystemmetrics,connecttoAPIsanddatabases

§ Listenstosyslog andgathersWindowsevents

§ Universallyindexesanydataformatsoitdoesn’tneedadapters

5

Windows• Registry• Eventlogs• Filesystem• sysinternals

Linux/Unix• Configurations• Syslog• Filesystem• Ps,iostat,top

Virtualization• Hypervisor• GuestOS• GuestApps

Applications• Weblogs• Log4J,JMS,JMX• .NETevents• Codeandscripts

Databases• Configurations• Audit/querylogs• Tables• Schemas

Network• Configurations• syslog• SNMP• netflow

Page 6: Machine data 101 workshop audienceversion

NetworkInputs

§ CollectdataoveranyUDPorTCPport§ Somedevicesonlysenddataoveranetworkport

§ BestPractice:usesyslog-ng orrsyslog§ Offerspersistence§ Categorizesdatabyhost

6

Page 7: Machine data 101 workshop audienceversion

HTTPEventCollector(HEC)

§ CollectdataoverHTTPorHTTPSdirectlytoSplunk§ ApplicationDeveloperfocus– fewlinesofcodeinapp

tosenddata§ HECFeaturesInclude:

§ Token-based,notcredentialbased§ IndexerAcknowledgements– guaranteesdataindexing§ RawandJSONformattedeventpayloads§ SSL,CORS(CrossOrigion access),andNetworkRestrictions

7

Page 8: Machine data 101 workshop audienceversion

LogEventAlertAction

§ UseSplunkalertingtoindexacustomlogevent§ Splunksearchableindexofcustomalertevents

§ ConfigurableFeaturesInclude:§ Host§ Source§ Sourcetype§ Index§ Eventtext– constructtheexactsyntaxofthelogevent,

includinganytext,tokens,orotherinformation

8

Page 9: Machine data 101 workshop audienceversion

TheSplunkAppforStream

WireDataEnhancesthePlatformforOperationalIntelligence

Efficient,Cloud-readyWireDataCollection

SimpleDeploymentSupportsFastTimetoValue

9

Page 10: Machine data 101 workshop audienceversion

Stream=BetterInsightsfor*

SolutionArea ContextualData WireData Enriched View

ApplicationManagement

applicationlogs,monitoringdata,metrics,events

protocolconversationsondatabaseperformance,DNSlookups,clientdata,businesstransactionpaths…

Measureapplicationresponsetimes,deeperinsightsforroot-causediagnostics,tracetxpaths,establishbaselines…

IT Operations applicationlogs,monitoringdata,metrics,events

payloaddataincludingprocesstimes,errors,transactiontraces,ICAlatency,SQLstatements,DNSrecords…

Analyzetrafficvolume,speedandpacketstoidentifyinfrastructureperformanceissues,capacityconstraints,changes;establishbaselines…

10

Page 11: Machine data 101 workshop audienceversion

Stream=BetterInsightsfor*SolutionArea ContextualData WireData Enriched View

Security app+infralogs,monitoringdata,events

protocolidentification,protocolheaders,contentandpayloadinformation,flowrecords

Buildanalyticsandcontextforincidentresponse,threatdetection,monitoringandcompliance

DigitalIntelligence

websiteactivity,clickstreamdata,metrics

browser-levelcustomerinteractions

CustomerExperience – analyzewebsiteandapplicationbottleneckstoimprovecustomerexperienceandonlinerevenues

CustomerSupport(online,callcenter)– fasterrootcauseanalysisandresolutionofcustomerissueswithwebsiteorapps

11

Page 12: Machine data 101 workshop audienceversion

ScriptedInputs

12

§ SenddatatoSplunkviaacustomscript§ Splunkindexesanythingwrittentostdout§ Splunkhandlesscheduling§ Supportsshell,Pythonscripts,WINbatch,PowerShell§ Anyotherutilitythatcanformatandstreamdata

StreamingMode§ Splunkexecutesscriptandindexesstdout

§ Checksforanyrunninginstances

WritetoFileMode§ Splunklaunchesscriptwhichproducesoutputfile,noneedforexternalscheduler

§ Splunkmonitorsoutputfile

Page 13: Machine data 101 workshop audienceversion

UseCasesforScriptedInputs

13

§ Alternativetofile-baseornetwork-basedinputs§ Streamdatafromcommand-linetools,suchasvmstat andiostat§ Pollawebservice,APIordatabaseandprocesstheresults§ Reformatcomplexorbinarydataforeasierparsingintoeventsandfields§ Maintaindatasourceswithsloworresource-intensivestartup

procedures§ Providespecialorcomplexhandlingfortransientorunstableinputs§ Scriptsthatmanagepasswordsandcredentials§ Wrapperscriptsforcommandlineinputsthatcontainspecialcharacters

Page 14: Machine data 101 workshop audienceversion

DatabaseInputs

§ Createvaluewithstructureddata§ Enrichsearchresultswithadditionalbusinesscontext

§ Easilyimportdatafordeeperanalysis§ IntegratemultipleDBsconcurrently§ Simpleset-up,non-invasiveandsecure

DBConnectprovidesreliable,scalable,real-timeintegrationbetweenSplunkandtraditionalrelationaldatabases

14

Page 15: Machine data 101 workshop audienceversion

ConfigureDatabaseInputs

15

§ DBConnectApp§ Real-time,scalableintegrationwithrelationalDBs§ Browseandnavigateschemasandtablesbeforedataimport§ Reliablescheduledimport§ SeamlessinstallationandUIconfiguration§ Supportsconnectionpoolingandcaching

§ “Tail”tablesorimportentiretables§ Detectandimportnew/updatedrowsusingtimestampsoruniqueIDs

§ SupportsmanyRDBMSflavors§ AWSRDSAurora,AWSRedShift,IBMDB2forLinux,Informix,MemSQL,MSSQL,MySQL,

Oracle,PostgreSQL,SAPSQLAnywhere(akaSybaseSA),SybaseASEandIQ,Teradata

Page 16: Machine data 101 workshop audienceversion

SplunkODBCDriver

16

§ Interactwith,manipulateandvisualizemachinedatainSplunkEnterpriseusingbusinesssoftwaretools

§ LeverageanalyticsfromSplunkalongsideMicrosoftExcel,TableauDesktoporMicrostrategy AnalyticsDesktop

§ Industry-standardconnectivitytoSplunkEnterprise§ Empowersbusinessuserswithdirectandsecureaccesstomachinedata

§ Combinemachinedatawithstructureddataforbetteroperationalcontext

Page 17: Machine data 101 workshop audienceversion

ODBC:HowitWorks

17

Page 18: Machine data 101 workshop audienceversion

ModularInputs

18

§ Createyourowncustominputs§ Scriptedinputwithstructureandintelligence§ FirstclasscitizenintheSplunkmanagementinterface§ AppearsunderSettings>DataInputs

§ Benefitsoversimplescriptedinput§ Instancecontrol:launchasingleormultipleinstances§ Inputvalidation§ Supportmultipleplatforms§ StreamdataastextorXML§ SecureaccesstomodinputscriptsviaRESTendpoints

Page 19: Machine data 101 workshop audienceversion

ExampleModularInputs

19

Twitter§ StreamJSONdatafromaTwittersourcetoSplunkusingTweepy

AmazonS3OnlineStorage§ IndexdatafromtheAmazonS3onlinestoragewebservice

JavaMessagingService(JMS)§ PollmessagequeuesandtopicsthroughJMSMessagingAPI§ Talkstomultipleproviders:MQSeries (Websphere MQ),ActiveMQ,TibcoEMS,HornetQ,RabbitMQ,NativeJMS,WebLogic JMS,SonicMQ

SplunkWindowsInputs§ RetrieveWINeventlogs,registrykeys,perfmon counters

Page 20: Machine data 101 workshop audienceversion

MoreModularInputs

20

Page 21: Machine data 101 workshop audienceversion

zLinux Forwarder

21

§ EasilycollectandindexdataonIBMmainframes

§ Collectapplicationandplatformdata

§ DownloadasnewForwarderdistributionfors390xLinux

Page 22: Machine data 101 workshop audienceversion

ExtendOperationalIntelligencetoMobileApps

22

DeliverBetterPerforming,MoreReliableApps

DeliverReal-TimeOmni-Channel

Analytics

End-to-EndPerformanceandCapacityInsights

Page 23: Machine data 101 workshop audienceversion

MonitorAppUsageandPerformance

• Improveuserretentionbyquicklyidentifyingcrashesandperformanceissues

• Establishwhetherissuesarecausedbyanapporthenetwork(s)

• Correlateapp,OSanddevicetypetodiagnosecrashandnetworkperformanceissues

23

Page 24: Machine data 101 workshop audienceversion

IntegratedAnalyticsPlatformforDiverseDataStoresFull-featured,IntegratedProduct

FastInsightsforEveryone

WorkswithWhatYouHaveToday

Explore Visualize Dashboards

ShareAnalyze

HadoopClusters NoSQLandOtherDataStores

Hadoop ClientLibraries StreamingResourceLibraries

Bi-directionalIntegrationwithHadoop

Page 25: Machine data 101 workshop audienceversion

ConnecttoNoSQLandOtherDataStores

• Buildcustomstreamingresourcelibraries

• SearchandanalyzedatafromotherdatastoresinHunk

• InpartnershipwithleadingNoSQLvendors

• UseinconjunctionwithDBConnectforrelationaldatabaselookups

Page 26: Machine data 101 workshop audienceversion

VirtualIndexes

§ EnablesseamlessuseofalmosttheentireSplunkstackondata

§ AutomaticallyhandlesMapReduce

§ Technologyispatentpending

Page 27: Machine data 101 workshop audienceversion

DataEnrichment

Page 28: Machine data 101 workshop audienceversion

Agenda

§ Tags – categorizeandaddmeaningtodata

§ FieldAliases – simplifysearchandcorrelation

§ CalculatedFields – shortcutcomplex/repetitivecomputations

§ EventTypes – groupcommoneventsandshareknowledge

§ Lookups – augmentdatawithadditionalexternalfields

28

Page 29: Machine data 101 workshop audienceversion

§ Addsinlinemeaning/context/specificitytorawdata

§ Usedtonormalizemetadataorrawdata

§ Simplifiescorrelationofmultipledatasources

§ CreatedinSplunk

§ Transferredfromexternalsources

WhatisDataEnrichment?

29

Page 30: Machine data 101 workshop audienceversion

§ Addmeaning/context/specificitytorawdata

§ Labelsdescribingteam,category,platform,geography

§ Appliedtofield-valuecombination

§ Multipletagscanbeappliedforeachfield-value

§ Casesensitive

Tags

30

Page 31: Machine data 101 workshop audienceversion

CreateTags

31

Page 32: Machine data 101 workshop audienceversion

§ Searcheventswithtaginanyfield

§ Searcheventswithtaginaspecificfield

§ Searcheventswithtagusingwildcards

FindtheWebServersTagsinAction

32

tag=webserver

tag::host=webserver

tag=web*

§ Tagthehostaswebserver

§ Tagthesourcetypeasweb

1

2

3

4

5

Page 33: Machine data 101 workshop audienceversion

§ Normalizefieldlabelstosimplifysearchandcorrelation§ Applymultiplealiasestoasinglefield

§ Example:Username|cs_username |Userà user§ Example:c_ip |client|client_ipà clientip

§ Processedafterfieldextractions+beforelookups

§ Canapplytolookups

§ Aliasesappearalongsideoriginalfields

FieldAliases

33

Page 34: Machine data 101 workshop audienceversion

Re-LabelFieldtoIntuitiveNameCreateFieldAlias

34

1

2

3

Page 35: Machine data 101 workshop audienceversion

§ Createfieldaliasofclientip=customer

§ Searcheventsinlast15minutes,findcustomerfield

§ Fieldalias(customer)andoriginalfield(clientip)arebothdisplayed

SearchusinganIntuitiveFieldNameFieldAliasinAction

35

1

3

2

sourcetype=access_combined

Page 36: Machine data 101 workshop audienceversion

§ Shortcutforperformingrepetitive/long/complextransformationsusingevalcommand

§ Basedonextractedordiscoveredfieldsonly

§ Donotapplytolookuporgeneratedfields

CalculatedFields

36

Page 37: Machine data 101 workshop audienceversion

ComputeKilobytesfromBytesCreateCalculatedField

37

1

21

2

3

Page 38: Machine data 101 workshop audienceversion

§ Createkilobytes=bytes/1024

§ Searcheventsinlast15minutesforkilobytesandbytes

SearchUsingKilobytesinsteadofBytesCalculatedFieldsinAction

38

1

2

sourcetype=access_combined

Page 39: Machine data 101 workshop audienceversion

§ Classifyandgroupcommonevents

§ Captureandshareknowledge

§ Basedonsearch

§ Useincombinationwithfieldsandtagstodefineeventtopography

EventTypes

39

Page 40: Machine data 101 workshop audienceversion

§ BestPractice:Usepunctfield§ Defaultmetadatafielddescribingeventstructure§ Builtoninterestingcharacters:",;-#$%&+./:=?@\\'|*\n\r\"(){}<>[]^! »§ Canusewildcards

CreateEventTypes

40

event punct

####<Jun3,20145:38:22PMMDT><Notice><WebLogicServer><bea03><asiAdminServer><WrapperStartStopAppMain><>WLSKernel<><><BEA-000360><ServerstartedinRUNNINGmode>

####<_,__::__>_<>_<>_<>_<>_<>_

172.26.34.223- - [01/Jul/2005:12:05:27-0700]"GET/trade/app?action=logoutHTTP/1.1"2002953

..._-_-_[:::_-]_\"_?=_/.\"__

Page 41: Machine data 101 workshop audienceversion

§ Showpunctforsourcetype=access_combined

§ Pickapunct,thenwildcarditafterthetimestamp

§ AddNOTstatus=200

§ Saveas“bad”eventtype+Color:red+Priority:1(shiftreloadinbrowsertoshowcoloring)

ClassifyEventsasKnownBadCreateEventType

41

eventtype=bad

sourcetype="access_combined" punct="..._-_-_[//_:::]*" NOT status=200

1

2

3

4

Page 42: Machine data 101 workshop audienceversion

LookupstoEnrichRawData

LDAPAD

WatchLists

CRM/ERP

CMDB

ExternalDataSources

Insightcomesout

DatagoesinCreateadditionalfieldsfromtherawdatawithalookuptoanexternaldatasource

Page 43: Machine data 101 workshop audienceversion

§ Augmentraweventswithadditionalfields§ Providecontextorsupportingdetails

§ Translatefieldvaluestomoredescriptivedata§ Example:addtextdescriptionsforerrorcodes,IDs§ Example:addcontactdetailstousernamesorIDs§ Example:adddescriptionstoHTTPstatuscodes

§ File-basedorscriptedlookups

Lookups

43

Page 44: Machine data 101 workshop audienceversion

44

1.Upload/createtable

2.Assigntabletolookupobject

3.Maplookuptodataset

Convert a Code into a DescriptionConfigure a Static Lookup

Page 45: Machine data 101 workshop audienceversion

§ GetthelookupfromtheSplunkWiki(saveto.csv file)http://wiki.splunk.com/Http_status.csv

§ Lookuptablefiles>Addnew§ Name:http_status.csv (musthave.csv fileextension)§ Upload:<pathto.csv>

§ Verifylookupwascreatedsuccessfully

1.CreateHTTPStatusTable

45

| inputlookup http_status.csv

1

2

3

Page 46: Machine data 101 workshop audienceversion

§ Lookupdefinitions>Addnew§ Name:http_status§ Type:File-based§ Lookupfile:http_status.csv

§ Invokethelookupmanually

2.AddLookupDefinition

46

1

2

sourcetype=access_combined | lookup http_status status OUTPUT status_description

Page 47: Machine data 101 workshop audienceversion

§ Automaticlookups>Addnew§ Name:http_status (cannothavespaces)§ Lookuptable:http_status§ Applyto:sourcetype=access_combined§ Lookupinputfield:status§ Lookupoutputfield:status_description

§ Verifylookupisinvokedautomatically

3.ConfigureAutomaticLookup

47

1

2

sourcetype=access_combined

Page 48: Machine data 101 workshop audienceversion

§ Temporallookupsfortime-basedlookups§ Example:IdentifyusersonyournetworkbasedontheirIPaddress

andthetimestampinDHCPlogs

§ Usesearchresultstopopulatealookuptable§ … | outputlookup <tablename|filename>

§ Callanexternalcommandorscript§ Pythonscriptsonly§ Example:DNSlookupforIPßà Host

§ Createalookuptableusingarelationaldatabase§ ReviewmatchesagainstadatabasecolumnorSQLquery

FancyLookups

48

Page 49: Machine data 101 workshop audienceversion

§ CreatingandManagingAlerts(JobInspector)

§ Macros

§ WorkflowActions

MoreDataEnrichment

49

Page 50: Machine data 101 workshop audienceversion

LevelUponSearch&ReportingCommands

Page 51: Machine data 101 workshop audienceversion

Agenda

§ Doingmorewithbasicsearchcommands

§ Advancedsearchcommands

§ Doingmorewithbasicreportingcommands

51

Page 52: Machine data 101 workshop audienceversion

SearchSyntaxComponents

52

Page 53: Machine data 101 workshop audienceversion

AnatomyofaSearch

53

Disk

Page 54: Machine data 101 workshop audienceversion

§ top– limit§ rare– sameoptionsastop§ timechart– parameters§ stats– functions(sum,avg,list,values,sparkline)§ sort– inlineascendingordescending§ addcoltotals§ addtotals

DoingMorewithBasicSearchCommands

54

Page 55: Machine data 101 workshop audienceversion

§ Commandshaveparametersorqualifiers

§ topandrarehavesimilarsyntax

§ Eachsearchcommandhasitsownsyntax– showinlinehelp

FindMostandLeastActiveCustomersUsingthetop+rareCommands

... | top limit=20 clientip

... | rare limit=20 clientip

IPswiththemostvisits

IPswiththeleastvisits

Page 56: Machine data 101 workshop audienceversion

§ Sortinlinedescendingorascending

56

... | stats count by clientip | sort - count

... | stats count by clientip | sort + count

Numberofrequestsbycustomer- descending

Numberofrequestsbycustomer- ascending

SorttheNumberofCustomerRequestsUsingthesortCommand

Page 57: Machine data 101 workshop audienceversion

§ ShowSearchCommandReferenceDocs§ Functionsforeval+where§ Functionsforstats+chartandtimechart

§ Invokeafunction

§ Renameinline

57

... | stats sum(bytes) by clientip | sort - sum(bytes)

... | stats sum(bytes) as totalbytes by clientip | sort - totalbytes

Totalpayloadbycustomer- descending

Totalpayloadbycustomer- ascending

DetermineTotalCustomerPayloadUsingfunctions+renamecommand

Page 58: Machine data 101 workshop audienceversion

§ Listallvaluesofafield

§ Listonlydistinctvaluesofafield

58

... | stats values(action) by clientip

... | stats list(action) by clientip

Activitybycustomer

Distinctactionsbycustomer

ObserveCustomerActivityUsingthelist+valuesFunctions

Page 59: Machine data 101 workshop audienceversion

§ Showdistinctactionsandcardinalityofeachaction

59

sourcetype=access_combined| stats count(action) as value by clientip, action| eval pair=action + " (" + value + ")"| stats list(pair) as values by clientip

AnalyzeCustomerActivityCombinelist+valuesFunctions

Page 60: Machine data 101 workshop audienceversion

§ Addcolumns

§ Sumspecificcolumns

60

... | stats count by clientip, action

2cols:clientip +action

... | stats sum(bytes) as totalbytes, avg(bytes) as avgbytes, count as totalevents by clientip | addcoltotals totalbytes, totalevents

Sumtotalbytesandtotaleventscolums

BuildingaTableofCustomerActivityAddColumnsandSumColumns

Page 61: Machine data 101 workshop audienceversion

61

... | stats sum(bytes) as totalbytes, sum(other) as totalother by clientip | addtotals fieldname=totalstuff

Foreachrow,addtotalbytes+totalother

Abetterexample:physicalmemory+virtualmemory=

totalmemory

BuildingaTableofCustomerActivitySumAcrossRows

Page 62: Machine data 101 workshop audienceversion

62

... | stats sparkline(count) as trendline by clientip

Incontextoflargereventset

... | stats sparkline(count) as trendline sum(bytes) by clientip

Inlineintables

TrendIndividualCustomerActivitySparklinesinAction

Page 63: Machine data 101 workshop audienceversion

AdvancedSearchCommandsCommand ShortDescription Hints

transaction Groupeventsbyacommonfieldvalue. Convenient,but resourceintensive.cluster Clustersimilareventstogether. Canbeusedon_raworfield.associate Identifiescorrelationsbetweenfields. Calculatesentropybtn fieldvalues.correlate Calculatesthecorrelationbetween

differentfields.Evaluatesrelationshipof allfieldsinaresultset.

contingency Buildsacontingencytablefortwofields. Computesco-occurrence,or%twofieldsexistinsameevents.

anomalies Computesanunexpectednessscoreforanevent.

Computessimilarityofevent(X)toasetofpreviousevents(P).

anomalousvalue Findsandsummarizesirregular,oruncommon,searchresults.

Considers frequencyofoccurrenceornumberofstdev fromthemean

Page 64: Machine data 101 workshop audienceversion

§ Seweventstogether+createsduration+eventcount

§ Sparklinesinlineintables

64

... | transaction JSESSIONID | table JSESSIONID, action, product_id

GroupbyJSESSIONID

ViewCustomerActivitybySessionUsingthetransactionCommand

Page 65: Machine data 101 workshop audienceversion

§ Intelligentgroup(createscluster_countandcluster_label)

65

... | cluster showcount=1 | table _raw, cluster_count, cluster_label

AutomaticallyGroupCustomerActivityUsingtheclusterCommand

Page 66: Machine data 101 workshop audienceversion

§ Predictovertime

§ ChartOverlaywithandwithoutstreamstats

§ Mapswithiplocation+geostats

§ Singlevalue

§ Meteredvisualswithgauge

DoMorewithBasicReportingCommands

66

Page 67: Machine data 101 workshop audienceversion

§ Predictfuturevaluesusinglower/upperbounds– singleandmultipleseries

67

... | timechart count as traffic | predict traffic

PredictWebsiteTrafficUsingthepredictCommand

Page 68: Machine data 101 workshop audienceversion

68

sourcetype=access_combined (action=view OR action=purchase)| timechart span=10m count(eval(action="view")) as Viewed,

count(eval(action="purchase")) as Purchased

CompareBrowsingvs.BuyingActivitySimpleChartOverlay

Page 69: Machine data 101 workshop audienceversion

69

... | iplocation clientip | geostats count by clientip

CombineIPlookupwithgeomapping

MapCustomerActivity GeographicallyGeolocation inAction

Page 70: Machine data 101 workshop audienceversion

70

... | stats count

DisplayaSimpleCountofEventsSingleValueinAction

Page 71: Machine data 101 workshop audienceversion

DisplayCountsUsingGaugesSingleValue,RadialandFillerGaugesinAction

71

... | stats count | gauge count 10000 20000 30000 40000 50000

Page 72: Machine data 101 workshop audienceversion

DataModelandPivot

Page 73: Machine data 101 workshop audienceversion

Agenda

§ Whatisadatamodel?

§ Buildadatamodel

§ PivotInterface

§ Accelerateadatamodel

73

Page 74: Machine data 101 workshop audienceversion

PowerfulAnalyticsAnyoneCanUse

Enablesnon-technicaluserstobuildcomplexreportswithoutthesearchlanguage

Providesmoremeaningfulrepresentationofunderlyingrawmachinedata

Accelerationtechnologydeliversupto1000xfasteranalyticsoverSplunk5

74

Pivot

DataModel

AnalyticsStore

Page 75: Machine data 101 workshop audienceversion

DefineRelationshipsinMachineDataDataModel• Describeshowunderlyingmachinedataisrepresentedandaccessed

• Definesmeaningfulrelationshipsinthedata

• Enablessingleauthoritativeviewofunderlyingrawdata

Hierarchicalobjectviewofunderlyingdata

Addconstraintstofilteroutevents

Page 76: Machine data 101 workshop audienceversion

TransparentAcceleration

• Automaticallycollected– Handlestimingissues,

backfill…• Automaticallymaintained– Usesaccelerationwindow

• Storedontheindexers– Peertothebuckets

• Faulttolerantcollection

Timewindowofdatathatisaccelerated

Checktoenableaccelerationofdatamodel

HighPerformanceAnalyticsStore

Page 77: Machine data 101 workshop audienceversion

Easy-to-UseAnalytics

• Drag-and-dropinterfaceenablesanyusertoanalyzedata

• Createcomplexqueriesandreportswithoutlearningsearchlanguage

• Clicktovisualizeanycharttype;reportsdynamicallyupdatewhenfieldschange

Selectfieldsfromdatamodel

Timewindow

Allcharttypesavailableinthecharttoolbox

Savereporttoshare

Pivot

Page 78: Machine data 101 workshop audienceversion

§ Definesleastcommondenominatorforadatadomain

§ Standardmethodtoparse,categorize,normalizedata

§ Setoffieldnamesandtagsbydomain§ PackagedasaDataModelsinaSplunkApp

§ Domains:security,web,inventory,JVM,performance,networksessions,andmore

§ MinimalsetuptousePivotinterface

CommonInformationModel(CIM)App

78

Page 79: Machine data 101 workshop audienceversion

§ Apps>FindMoreApps>

§ Search:“CommonInformationModel”

§ Installfree

§ Showfieldsforweb+WebDataModel

DownloadCIMApp

79

1

2

3

4

Page 80: Machine data 101 workshop audienceversion

DataModel&PivotTutorial

http://docs.splunk.com/Documentation/Splunk/latest/PivotTuto

rial/WelcometothePivotTutorial

80

Page 81: Machine data 101 workshop audienceversion

CustomVisualizationsandtheWebFrameworkToolkit

Page 82: Machine data 101 workshop audienceversion

Agenda

§ DeveloperPlatform

§ WebFrameworkToolkit(WFT)

§ RESTAPIandSDKs

§ GetaFlyingStart

82

Page 83: Machine data 101 workshop audienceversion

OptimizingtheAnalyticsProcess

83

Focusonthedata– intuitivetoolstoenabletheanalyst

Nosinglevisualizationexiststohandlealldatasets.

Neverlosesightoftherawdata

SplunkAnalytics

Explore

Context

Visualize

Algorithms

Page 84: Machine data 101 workshop audienceversion

6.0+6.1:Simple,Interactive,andExtensible

84

VISUALIZATIONEXPLORATION

CUSTOMIZABLEFRAMEWORK

POWERFULANALYTICS

PivotDataModels

InteractiveFormsContextualDrilldown

DashboardEditorWebFramework

Page 85: Machine data 101 workshop audienceversion

TheSplunkEnterprisePlatform

Collection

Indexing

SearchProcessingLanguage

CoreFunctions

Inputs,Apps,OtherContent

SDKContent

CoreEngine

UserandDeveloperInterfaces

WebFramework

RESTAPI

Page 86: Machine data 101 workshop audienceversion

What’sPossiblewiththeSplunkEnterprisePlatform?

PowerMobileApps

LogDirectly

ExtractData

CustomerDashboards

IntegrateBITools

IntegratePlatformServices

Developer Platform

Page 87: Machine data 101 workshop audienceversion

PowerfulPlatformforEnterpriseDevelopersDevelopersCanCustomizeandExtend

RESTAPI

BuildSplunkApps ExtendandIntegrateSplunk

SimpleXML

JavaScript

HTML5

WebFramework

JavaJavaScriptPython

RubyC#PHP

DataModels

SearchExtensibility

ModularInputs

SDKs

Page 88: Machine data 101 workshop audienceversion

SplunkSoftwareforDevelopers

GainApplicationIntelligence

BuildSplunkApps

IntegrateandExtendSplunk

Page 89: Machine data 101 workshop audienceversion

AWealthofSplunk AppsOver1,100appsavailableontheSplunkappssite

APISDKs UI

Server, Storage, Network

Server Virtualization

Operating Systems

Custom Applications

Business Applications

Cloud Services

App Performance MonitoringTicketing/ and

Other

WebIntelligence

Mobile Applications

Stream

Page 90: Machine data 101 workshop audienceversion

§ Interactive,cut/pasteexamplesfrompopularsourcerepositories:D3,GitHub,jQuery

§ Splunk6.xDashboardExamplesApphttps://apps.splunk.com/app/1603

§ CustomSimpleXML ExtensionsApphttps://apps.splunk.com/app/1772

§ SplunkWebFrameworkToolkitApphttps://apps.splunk.com/app/1613

ExampleAdvancedVisualizations

90

Page 91: Machine data 101 workshop audienceversion

91

http://www.d3js.org

Page 92: Machine data 101 workshop audienceversion

AddaD3BubbleChart

92

1. GotoFindMoreAppsandInstalltheSplunk6.xDashboardExamplesApp

2. EntertheApp3. GotoExamples>CustomVisualizations>

D3BubbleChart4. Copyautodiscover.js (file)+components/bubblechart (dir)

from:$SH/etc/apps/simple_xml_examples/appserver/staticto:$SH/apps/search/appserver/static

5. CopyandpastesimpleXMLtonewdashboard

Page 93: Machine data 101 workshop audienceversion

Resources

Page 94: Machine data 101 workshop audienceversion

SplunkDocumentation

94

• http://docs.splunk.com• OfficialProductDocs• Wikiandcommunitytopics• Updateddaily• Canbeprintedto.PDF

Page 95: Machine data 101 workshop audienceversion

SplunkAnswers

95

• http://answers.splunk.com• Communitydriven• Splunksupported• Knowledgeexchange• Q&A

Page 96: Machine data 101 workshop audienceversion

SplunkEducation

96

• RecommendedforUsers– UsingSplunk– Searching&Reporting

• RecommendedforUI/DashboardDevelopers– DevelopingApps

• Instructor-LedCourses– Web– Onsite