Copyright©2016SplunkInc.

ExperianMikeSclimentiSeniorSystemsEngineer

2

DisclaimerDuringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.For

importantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmade

asofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesorfunctionality

describedortoincludeanysuchfeatureorfunctionalityinafuturerelease.

ItshouldalsobenotedthattheviewsexpressedinthispresentationaresolelythoseoftheauthorinhisprivatecapacityanddonotinanywayrepresenttheviewsofConsumerInfo.com,Inc.(aka:ExperianConsumerServices),

anyotherentityofExperian,oritsAffiliates.

Alllogosusedinthispresentationarepropertyoftheirrespectivecompanies.

3

AboutMe• MikeSclimenti,SeniorSystemsEngineer

• ExperianConsumerServices– ITSystemsAdministration/Engineeringfor20+years

ê HighlyScalableInfrastructureDeployments&DisasterRecoveryê LargeScaleVMware&Symantec(Veritas)NetBackup Environmentsê ApplicationDeployments,SystemsManagement,ActiveDirectory,etc.

– MonitoringSystems2+years

• Splunkcustomer– Userfor8years– Adminfor2years(Splunk6.1,6.3)

• FavoriteSplunktee-shirt:“Becauseninjasaretoobusy”

4

Agenda• Architecture&LessonsLearneddeployingSplunkCloud:

– S3viatheSplunkAppforAWS– Kinesis– LambdaFunctions– TheHTTPEventCollector

• Howwewentfrom15minutesoflatencyonproductiondashboardsto…– Sub-5secondsoflatencysendinglogsdirectlyfromKinesis(viaLambda)tothe

HTTPEventCollector

5

CloudServicesArchitecture

6

Splunk’s S3ConnectorTheS3Connectorisefficientfor:

⏤ CloudFront

⏤ ELB(ElasticLoadBalancer)

⏤ CloudWatch &CloudWatch Logs

⏤ Cloudtrail

⏤ Billing

7

TheS3ConnectorWasWorking,But…• ThenIwentto.conf 2015• So,asIwassittingintheKeynotesessiononDay1,Ithought:

– CouldIgodirectlytotheHTTPEventCollectorfromtheapplication?ê NomoreUniversalForwarderstoinstallorupdateê FeweragentsrunningontheEC2instances

– WouldloggingtoKinesisandthentotheHTTPEventCollectorbemoreefficient?

AmazonEC2 AmazonKinesis AmazonLambda

8

TheHTTPEventCollector

Applications IoT Devices

Agentless,directdataonboardingviaastandarddeveloperAPI

curl -k https://<host>:8080/services/collector -H ‘Authorization: Splunk <token>’ -d ‘{”event”:”Hello Event Collector”}’

9

TheHTTPEventCollector(cont.)• Gotbacktotheoffice,begandoingfurtherresearch

• StartedplanningmigrationfromS3ConnectortotheHTTPEventCollector

• Beganseeinglatencyissuesw/theingestfromS3whilerunningsomeloadtests

• Timelineformigrationacceleratedduetolatencyof15minutesingestinglogsfromS3

Butthen…IrealizedHOUSTONWEHAVEAPROBLEM!

10

TheHTTPEventCollector(cont.)

10

• WewererunningSplunkCloudversion6.2• TheHTTPEventCollectordidnotexistinSplunkCloudversion6.2• InstalledtheHTTPEventCollectoronaHeavyForwarderrunningtheSplunkEnterprise6.3.

AmazonEC2 AmazonKinesis AmazonLambda SplunkEnterprise6.3HeavyForwarder

SplunkCloud6.2

11

TheHTTPEventCollector(cont.)

• Everythingwasrunninggreatuntiluntilwecrankedupourtraffic…

• LuckilySplunkCloudmadeversion6.3availableforproduction

• SplunkCloud6.2wasupgradedto6.3

• HTTPEventCollectorwasenabledonindexers

• Lambdafunctionsupdated

• Tuningbegan...

12

LambdaConfiguration

13

LambdaConfiguration

14

LambdaConfiguration

15

LambdaBatchSize• BatchsizeisthemaxnumberofeventsthatsentforsingleinvocationoftheLambdafunction

• Increaseditfrom100to1000to5000to10000thenbackto5000• 646bytesaverageeventsizebutthenHTTPeventcollectorstartedtoerrorsometimesbecauseofthedefaultmax_content_length =1,000,000bytes

• 1,000,000/646=1548eventsinbatch

sourcetype=applogs host=http-inputs.splunkcloud.com earliest=-24h latest=now | eval event_size=len(_raw) | stats avg(event_size

16

TuningtheHTTPEventCollector

17

HTTPEventCollectorScalingLimits.conf[http_input]max_content_length =1000000(bytes)

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf

Increasethemax_content_length =5,000,000bytes(~5MB)Batchsize=5000,memoryfortheLambdaat512MB

18

LambdaTuning• Makesureyouusehttps/SSLbetweenLambdaandHTTPEventCollector

• Setanappropriatebatchsize!“1000”isbetterthan“100”

• SetLambdaFunctionto“Latest”NOT “TrimHorizon”

• GiveyourLambdafunctiontherightamountofmemory• Changethetimeoutfrom“10”to“30”

19

AWSKinesisShards• Eachshardcansupport:

– Upto5transactionspersecondforreads– Uptoamaxtotaldatareadrateof2MB/sec– Upto1Krecordspersecondforwrites– Uptoamaxtotaldatawriterateof1MB/secê 2MB/secpershardê Planforpeaks

• MakesureyousplitKinesisintoenoughshardssothatitcanhandle:– Inboundstreamsfromyourapplication– OutboundstreamstoS3and/ortheHTTPEventCollector

20

MeasuringOurProgress• LatencySearch

sourcetype=applogs host=http-inputs.splunkcloud.com earliest=-2mlatest=now|eval latency_in_seconds=(_indextime - _time)|statsperc80(latency_in_seconds)as80th_percentile_latency_in_seconds

21

SampleNOCMonitoringDashboard

22

ThingstoRemember• S3worksbuttheHTTPEventCollectorisfaster

• YoumustbeusingSplunk CloudORSplunk Enterprise6.3(orhigher)

• TuneyourLambdafunction(mayimpactyourfunction$$$)

• ScaleupyourHTTPEventCollector

• MakesureyouhaveenoughKinesisshards(mayimpactyourKinesis$$$)

• Measureyourprogressthroughdashboardsandalerts

23

Resources• .conf2015“TheGreatShakeOff”

– http://www.ustream.tv/recorded/73893599 (startsatthe22minmark)

• Splunk’s HTTPEventCollector– http://dev.splunk.com/view/event-collector/SP-CAAAE6M

• AWSLambda– http://docs.aws.amazon.com/lambda/latest/dg/welcome.html

• AWSKinesisShardLimits– http://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html

ThankYou