mentor graphics customer presentation

Copyright © 2015 Splunk Inc.

Lee Melvin & Shawn RobertsMentor Graphics

Hellomy name is:

Lee Melvin

Mentor Graphics

Technical Architect - Storage

Hellomy name is:

Shawn Roberts

Mentor Graphics

Technical Architect - Clients

Mentor Graphics® provides

software and hardware design

solutions that enable companies

to develop better electronic

products faster and more cost-effectively.

How I got started with Splunk

• Dual vendor network-attached storage strategy

• 100+ filers across 30+ sites, used by 10+ divisions

• Needed ways to• Provide per-division summary reporting, accessible to engineers on demand

• Implement performance service level objectives

• Provide storage administrators with actionable issues, not just graphs and metrics

• Provide reporting that spans vendors

• Vendor reporting tools expensive and inadequate for these needs• Previously, layered a fragile and manual scripting process on top

• Initial focus on capacity and performance analysis

Lee

Our “Storage” Solution

Forwarder/DataCollection Node

Lee

Custom Vendor SpecificAdd-on

Splunk’s plugin

Indexer/search head

How I got started with Splunk

• We had ~8,500 systems around the world that were not being monitored

• No visibility into what could be causing occasional crashes

• No ability to:• Add another agent

• “increase the IT tax”

• Negatively impact performance

• Track changes over time

• People were asking questions…

Shawn

Those 8,500 systems…

Shawn

Those 8,500 systems…

Shawn

Shawn

Shawn

Shawn

WMI?

Event Logs?

Event Forwarding?

8,500 Systems

Performance hit to network and clients

8,500 Systems

8,500 Systems!

Event Viewer <> Reporting

Shawn

Shawn

Shawn

Our “Client” Solution

EventForwarding

Shawn

CollectionServer

Doesn’t add another agent

Doesn’t “increase the IT tax”

No impact to performance

Track changes over time

Shawn

Tough client reports now possible

Boot Time

Average aka What’s normal?

Last 30 vs 365

Top/Bottom 10

Error Events

Top 10

Errors by category

Startup impacting

Application crashes

!

Shawn

Shawn

Storage reporting enhanced!

• Tagging filesystems by division, product group, performance tier, or other abstract (via Splunk lookup table)

• Custom technology add-on to ingest data from Hitachi HNAS

• Capacity/Performance reports that span both NetApp and Hitachi

• Division summary and per-filesystem views

• Actionable items for storage administrators

• Outliers view – slow, busy, near-full

• Stale data reporting – big chunks of unused data

Lee

Lee

Lee

Lee

Lee

Shawnhttp://www.flickr.com/people/15291928@N00

No good deed, etc…what’s next?

• Lookup/compare by:• Location

• AD OU

• Manager

• Type

• Model

• Manufacturer

Shawn

Client dashboard

• Measurements• Make, Model, CPU, Memory, Disks,

Encryption status, etc.

• Age of Windows installation

• Most common errors

• Last N errors

• SCCM Data• Installed software• Update status• Changes!

• Hardware• Software

• AV Data• Versions• Activity

Shawn

Client dashboard with Splunk

• Measurements• Make, Model, CPU, Memory, Disks,

Encryption status, etc.

• Age of Windows installation

• Most common errors

• Last N errors

• SCCM Data• Installed software• Update status• Changes!

• Hardware• Software

• AV Data• Versions• ActivityX vs Normal

X over Time

Shawn

Storage reporting

• Reports• Filer configuration auditing

• Geographic visualization

• Long term (5-year) reports leveraging summary indexes

• Matching workloads to appropriate performance tier

• Alerting• Alert filesystem owners on near-

capacity using lookup table

• Infrastructure• Web frontend to edit tags in

lookup table• Division-specific view roles to

limit access/confusion• Ingest additional HDS metrics• Ingest Symantec NetBackup logs• Round the learning curve on

data models, simplify

Lee

Lessons learned

• You have data in more places than you might think.

• You don’t always need a huge license purchase to answer hard questions.

• Use the provided apps for inspiration (and for cut/paste). Don’t treat them as an end state.

Shawn

Lee

Copyright © 2015 Splunk Inc.

Thank You!

Shawn Roberts

shawn_roberts@mentor.com

+ShawnRobertsNerd

Lee Melvin

lee_melvin@mentor.com

linkedin.com/in/lmelvin