Growing SplunkTyler Rutschman - Garmin International

Tuesday, May 15, 12

About Me

Linux System Administrator

Husband and Father of 2 Kids

DevOps, Productivity Hacks and Tools, The Big Lebowski

Tuesday, May 15, 12

OH: (during an outage)I don’t want to live in a world without Splunk.

Tuesday, May 15, 12

Backstory

Free instance installed in 2009 by the Network TeamSingle Instance on Central Log serverUpgrade to Enterprise

Tuesday, May 15, 12I started as an IT intern my mentor had a free copy of version 2.x running on the log server.

I was tasked with finding a solution for SOX & PCI requirements. (Which was mind expanding for an intern, to say the least)

Worked with purchasing to get a small license for the enterprise features.

My project ended up piping Splunk output into a python program that no one but I understood that printed out a text report that (I felt at least) was superior to the one in place at the time. (Big surprise, didn’t end up using it).

Building Blocks

Split Splunk off onto dedicated instanceLicense overwhelmed by single appLimited visibility and use

Tuesday, May 15, 12When I came back there was some cursory interest in the app, but no major users and no project champion.

Welcome, back Tyler... Splunk Expert (by Default).

I was also attached to Garmin Connect, which is our awesome fitness tracking site, after getting more comfortable in my settings, I began to integrate the site logs into Splunk

IF YOU HAVE MORE INPUTS THAN LICENSE

YOU’RE GONNA HAVE A BAD TIME

Tuesday, May 15, 12Obvious, but this was my experience during the first dedicated instance. We had a small license and it was all being used by Garmin Connect. It really wasn’t taking hold like I knew it could.

Plan for Expansion

Decided to make application more robustRead the DocumentationPlanned roll out

Multiple Applications

License Increase

Scalable Architecture

Tuesday, May 15, 12After I became more comfortable in my position, I felt impelled to make the application more robust and widespread.

I went to .conf last year, attended some training sessions and read up on the Administration documentation.

Enterprise ArchitectureElements

Puppet DeployInfrastructure LayoutGotchasFuture Plans

(so far)

Tuesday, May 15, 12Overview of the Current Architecture Elements, will then go in depth a bit more on each subject.

Puppet

Search, Indexer and Forwarder are “turn-key”ex: include splunk::indexer ...done

Really Awesome for Forwarders

Tuesday, May 15, 12Puppet makes deployment simple. Servers are built with one include statement.

Forwarders are split up based on role and inputs. Customize the inputs a bit if necessary and include the splunk forwarder class in the puppet node definition.

Infrastructure

Tuesday, May 15, 12Describe layers and functions.

Search is load balanced.

Search, Index and Forwarders are horizontally scalable.

Network/Taiwan instances aren’t pictured but are separate dedicated instances. Will move the network index into the main infrastructure real soon now.

How We Use SplunkWeb Access Logs

Service Usage Metrics

Feature Tracking

Diagnosing Problems in Production

Internal Application AuditsWindows Security Events

Tuesday, May 15, 12We don’t have a wide variety if inputs into Splunk at the moment.

We currently use it on all of the major IT web applications to obtain service metrics, track new features and diagnosing issues in Production.

The developers are also starting to cater their applications to output Splunk friendly logs

Windows security events are queried via WMI ad filtered to specific IDs, this helps keep the volume down while delivering value for the Windows guys.

Why I like Splunk

Makes Users HappyReal Time DataNo Alternatives

Tuesday, May 15, 12Ease of configuration, having the one stop shop for user-land configs. LDAP integration is super simple.

Able to generate detailed reports and drill into the data on the fly is a killer feature and something that you simply won't find with any other application.

User community and Documentation.

There are no real alternatives to Splunk. Some tools touch on some of the features gained with the app, but there is no offering that matches what splunk can give you. I’ve tried SEC, logwatch, Logstash, and Spiceworks. None were as user friendly and robust as Splunk.

Gotchas

Don’t Index a lot of data over NFSShared Knowledge Bundle Time SyncTag and Search permissions

Tuesday, May 15, 12Keeping up with the demand. From a license and user request perspective, I limited amount of time to handle the requests at hand. Familiar position for me at least, but a good problem to have.

Mounted Bundles must have the same time across the board.

Watch your permissions on saved searches and tags. They are usually private when I share them with another user and they cannot access.

Future Plans

Fix Central LoggingCheck Out Deployment ServerMore InputsTraining

Tuesday, May 15, 12Currently only one centralized syslog server, want to scale it out and put a farm of syslog servers behind a load balancer. Splunk will be the defininitive timeline for syslog events.

Read about Deployment Server but passed on it at the time. Would like to pick it back up and see how it could be beneficial.

Add additional inputs to the application

I’ve been tasked with training my coworkers on how to use the application. Once they pick it up and figure it out, they can do awesome things.

Tips and Advice

WMI Event Filter for Windows EventsSplunkbase (stack overflow engine)

Tuesday, May 15, 12

Questions & Feedback

Tuesday, May 15, 12