splunksummit 2015 - security ninjitsu

Copyright © 2015 Splunk Inc.

Original talk by David Veuve Senior SE, Security SME, Splunk

Security Ninjitsu

Andrew Phillips Senior SE, Splunk

Disclaimer

2

During the course of this presentaKon, we may make forward looking statements regarding future events or the expected performance of the company. We cauKon you that such statements reflect our current expectaKons and esKmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in the this presentaKon are being made as of the Kme and date of its live presentaKon. If reviewed aTer its live presentaKon, this presentaKon may not contain current or

accurate informaKon. We do not assume any obligaKon to update any forward looking statements we may make.

In addiKon, any informaKon about our roadmap outlines our general product direcKon and is subject to change at any Kme without noKce. It is for informaKonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaKon either to develop the features

or funcKonality described or to include any such feature or funcKonality in a future release.

4

Check the Non-‐PresentaKon Version and the Security Ninjitsu App

3200 Words 1800 Words

Personal introducKon

5

  David Veuve – Senior Sales Engineer for Major Accounts in Northern California

  Security SME, Former customer, author of Search AcKvity app [email protected]

Who Are You? 1.  Someone technical who cares about security 2.  All Splunk skill levels 3.  No Enterprise Security required

6

Who is this session for? 1.  Someone technical who cares about security 2.  All Splunk skill levels 3.  No Enterprise Security required

7

Agenda

Four types of security correlaKon rules you probably want 1.  CorrelaKon across many sourcetypes and events 2.  Privileged user monitoring 3.  Conquering alert faKgue 4.  Threat Intel hits

All driven by customer requirements / requests

8

What Experience Are You About to Have?

9

  | eval state=If(SplunkExperience<Ninja, "InformaKon Overload", "Neato") | eval state=mvappend(state, "Excitement??")

  Don’t fear – the Security Ninjitsu app is available on SplunkBase.

  Feedback welcome!

Security CorrelaKon In Splunk

Mainframe Data

VMware

Plakorm for Machine Data

Splunk Solu0ons > Easy to Adopt

Exchange PCI Security

RelaKonal Databases

Mobile Forwarders Syslog / TCP / Other

Sensors & Control Systems

Across Data Sources, Use Cases & Consump0on Models

Wire Data

11

Mobile Intel

MINT

CIM

Mainframe Data

VMware




RelaKonal Databases




Wire Data

12

Mobile Intel

MINT

CIM

Mainframe Data

VMware




RelaKonal Databases




Wire Data

13

Mobile Intel

MINT

CIM

●  Easy in Enterprise Security

●  In ES or Core Splunk, any search can: –  Send an email –  Trigger ServiceNow / etc –  Run a script –  Add FW Blocks, Increase Logging, etc.

●  CorrelaKon in Splunk is just searching

Splunk CorrelaKon Rules

16

Security-‐relevant data models from Common InformaKon Model

Common Informa0on Model Standard Language

17

CIM Compliant!

Comparison

18

  Without Common InformaKon Model (Sourcetype=WinSecurity EventID=…) OR (sourcetype=linux_secure password OR key) OR sourcetype=… | eval user=coalesce(Windows_Account, user, Webstore_Admin_User…)   With Common InformaKon Model tag=authenKcaKon

•  AcceleraKon facilitates beser and broader analysis

•  Splunk has a few ways of acceleraKng content: •  Report AcceleraKon •  Data Model AcceleraKon •  Summary Indexing •  TSCollect •  Pre-‐Processing of logs

•  Go View Last Year’s talk: Security Ninjitsu (conf.splunk.com, 2014 Sessions)

How To Accelerate

19

Search Example

20

Raw Search   71 Seconds

With Data Model AcceleraKon   9.8 Seconds

CorrelaKon Across MulKple Sourcetypes

CorrelaKon Across MulKple Sourcetypes •  CorrelaKon is easy in Splunk. •  Easy: –  Across many auth log types –  Across auth logs and event logs –  Complex scenarios

• Now, some techniques!

22

Technique 1 – Common InformaKon Model

23

tag=authenKcaKon | chart count over src by acKon | where success>0 AND failure>10   If you leverage Splunk’s Common InformaKon Model you can write one search across many products.

  The above search could cover twenty different products, all with matching field extracKons

  Most searches in this session will be based on the common informaKon model

  Try with the ES Sandbox!

Techniques – Common InformaKon Model

24

tag=authenKcaKon | chart count over src by acKon | where success>0 AND failure>10   Many sourcetypes with one search!

Technique 2 – Flexible Stats

25

Example: | stats count(eval(acKon="success")) as successes count(eval(acKon="failure")) as failures by user

•  Almost anything from eval works in stats eval

Technique 2 – Flexible Stats

26

Great Techniques: •  If statements (use null for non-‐valid results) •  values(eval(if(acKon="success",user,null))) as "Successful Users" •  vs.. values(eval(acKon="success")) as "# of Successful Users"

•  Searchmatch and match for flexible matching •  AND OR NOT •  If(searchmatch("sudo") AND user!="service" AND (host="emailserver" OR host="webserver")…)

Techniques 3 – Expand Base Search

27

  Joins are computaKonally expensive, and limited Subsearches are beser, but not by a lot –  Super sparse (rare) search as subsearch – good!

  Both limited to 60 seconds and 10k results   Best to expand your base search

Technique 3 – Expand Base Search

28

  Bad: tag=malware …… | join host [search tag=proxy ……. ]

  Good: tag=malware OR tag=proxy | stats count(eval(tag="malware")) as malware count(eval(tag="proxy")) as proxy by host

  AccounKng for Host SubtleKes: | eval mydest=if(tag="malware", dest, src) | stats … by mydest

Technique 3 – Expand Base Search

29

  Incorrect (10k results!) – Join Version   Maybe Incorrect (400 seconds, 10k malware hits) – Subsearch Version

  Beser (72 seconds) – Expanded Base Search   Best (14 seconds) – tstats Search

Technique 4 – The other stats

30

  SomeKmes you need more flexibility   TransacKon is powerful, but expensive   Consider: –  streamstats – ordered processing –  eventstats – addiKve (non-‐destrucKve) stats processing –  geostats – be world aware

Techniques – Breaking Subsearch Limits

31

  Common Usage: [search index=malware | table host] index=proxy   Interpreted as: (host=vicKm1 OR host=vicKm2) index=proxy   Easy specificity creates huge performance improvements   (Did you know you can do | eval myhost=[search tag=malware | return dest]) Subsearches limited to 10,000 results and 60 seconds by default

  You can also return a literally interpreted search string: [search tag=malware | stats values(dest) as search | eval search=“(dest=“ . mvjoin(search, “ OR dest=“) . “)”] •  Can’t break 60 second limit without limits.conf change

Techniques – Higher Confidence

32

  Trigger your components and register to a summary index –  Hey, ES does that already!

  Example: Find sources or desKnaKons of brute force, vicKms of IDS hits, or malware events (clean or not) and determine if those hosts have new uncategorized web proxy acKvity

  We’ll look at that later

Core Use Case

33

  New Process Launch and uncategorized proxy acKvity within 15 minutes of anK-‐virus alert (successful or failed)

  High Probability C&C AcKvity

  Advanced use case, simple search

Core Use Case

34

  [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ]

  earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized)

  | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src

  | where proxy_events > 0 AND endpoint_events > 0

First, find our infected hosts.

Core Use Case

35





Pull endpoint + proxy data for those hosts

Core Use Case

36





See how many proxy and endpoint events per host

Core Use Case

37





Filter to just hosts that have the known bad events

Core Use Case

38

  [search tag=malware earliest=-‐20m@m latest=-‐15m@m | table dest | rename dest as src ]   earliest=-‐20m@m (sourcetype=sysmon OR sourcetype=carbon_black evensype=process_launch) OR (sourcetype=proxy category=uncategorized)   | stats count(eval(sourcetype="proxy")) as proxy_events count(eval(sourcetype="carbon_black" OR sourcetype="sysmon")) as endpoint_events by src   | where proxy_events > 0 AND endpoint_events > 0

Four Lines, but not hard

Scalability Improvements

39

  Raw Search: 21 seconds Tstats: 2.76 seconds

About Endpoint Logs

40

  Curious about Endpoint Monitoring? Check out the epic talk from Splunk Rockstar James Brodsky:

Splunking The Endpoint hJp://conf.splunk.com/session/2015/recordings/2015-‐splunk-‐119.mp4

Privileged User Monitoring

Privileged User Monitoring 1.  Start by detecKng something bad 2.  Focus on highly visible or highly privileged users

Our use case: Alert for users who log into way more systems than normal

42

How to Build StaKsKcal Analysis in Splunk

43

  Understand Your Use Cases   Begin by pulling your data –  Establish the base dataset tag=authenKcaKon | bucket _Kme span=1d | stats count by user, host, _Kme

–  Pull trend per host | stats avg(count) as avg first(count) as recent by user, host

–  Pull overall trends | eventstats dc(host) as NumServers by user

  Apply your business logic

Techniques in Analysis

44

  Understand Normal versus Now: | eval isRecent=if(_Kme>relaKve_Kme(now(),"-‐1d"), "yes", "no")

  Report on Causes for Analysis | eval Cause=if(NumServersHistorically*3 < NumServersRecently, "SubstanKal increase in the number of servers logged on to","") | where Cause!=""

AcceleraKon Analysis

45

  Raw Searching can be slow over big datasets tag=authenKcaKon earliest=-‐30d@d| bucket _Kme span=1d | stats count by user, host, _Kme

  Accelerated searching is fast! | tstats count from datamodel=AuthenKcaKon where earliest=-‐30d@d groupby AuthenKcaKon.dest AuthenKcaKon.user _Kme span=1d | rename AuthenKcaKon.dest as dest AuthenKcaKon.user as user

tag=authenKcaKon earliest=-‐30d@d| bucket _Kme span=1d | stats count by user, host, _Kme | eval isRecent=if(_Kme>relaKve_Kme(now(),"-‐1d"), "yes", "no") | stats avg(eval(if(isRecent="no",count,null))) as avg first(count) as recent by user, host | eventstats count(eval(if(avg>0,"yes",null))) as NumServersHistorically dc(eval(if(recent>0,"yes",null))) as NumServersRecently by user | eval Cause=if(isnull(avg) AND NumServersHistorically>0, "This is the first logon to this server", "") | eval Cause=if(NumServersHistorically*3 < NumServersRecently, mvappend(Cause,"SubstanKal increase in the number of servers logged on to"), Cause) | where Cause!=""

•  AcceleraKon facilitates beser and broader analysis

•  Splunk has a few ways of acceleraKng content: •  Report AcceleraKon •  Data Model AcceleraKon via Pivot •  Summary Indexing •  TSCollect •  Pre-‐Processing of logs

•  Go View Last Year’s talk: Security Ninjitsu (conf.splunk.com, 2014 Sessions)

How To Accelerate

46

Analysis – Part Two

47

  You know high risk, high exposure users –  Sys Admins –  ExecuKves –  Contractors –  First 3 months of employment, last 3 months of employment

  Sources: –  AD Group Membership –  AD Title –  HRIS Employment Status

Analysis – Part Two -‐ Example

48

| inputlookup LDAPSearch | eval risk = 1 | eval risk = case(NumWhoReportIn>100, risk+10, risk) | eval risk = case(like(Groups, "%OU=Groups,OU=IT Security,%"), risk + 10, risk) | fields risk sAMAccountName | outputlookup RiskPerUser

IdenKty Data IniKalize Risk

Business Logic

New Lookup

Analysis – Pu�ng it Together

49

[… insert your Privileged User AcKvity Search …] | stats count by user | lookup RiskPerUser sAMAccountName as user | eval AggRisk = risk * count | eval DescripKveRisk = case(AggRisk > 100, "very high", AggRisk>30, "medium", AggRisk>5, "low", 1=1, "very low")

Summarize Per User

Add Org-‐wide Risk

Create a new Lookup

Describe Risk

Analysis – Pu�ng it Together

50

Oh Yeah. Jack Bauer has gone rogue.

AcKon (ES Specific)

51

  In ES, pass the following overrides in your search: –  severity –  risk_score –  risk_object –  risk_object_type

  Beser yet, use the built-‐in ES IdenKty Framework!

Conquering Alert FaKgue

Conquering Alert FaKgue •  Typical Ker one analyst: one event per 10-‐15 min. –  Only 50 events per shiT.

•  You will always have more alert data than you have staff •  Many great techniques for managing this •  Let’s dig into my favorite five

53

(1/5) Analysis Technique – Risk-‐Based

54

•  Great for general purpose events •  Increase the risk associated with an enKty (user, system, signature, etc.)

•  Focus acKvity on high risk enKKes

•  Out of the box with ES (index=risk) •  Consider building your own by chaining | collect

(2/5) Analysis Technique -‐ StaKsKcal

55

  Understand Your Environment   Begin by pulling your data –  Establish the base dataset | bucket _Kme span=1d | stats sum(param1) as sum count(param1) as count by host, _Kme

–  Pull trend per host | stats avg(sum) as avg stdev(sum) as stdev first(sum) as recent by host

–  Pull overall trends | eventstats avg(avg) as overallavg …..

  Apply your business logic

(2/5) Analysis Technique – StaKsKcal – Part Two

56

Example Where Clause | where (avg_earliest > relaKve_Kme(now(), "-‐1d")) OR (earliest > relaKve_Kme(now(), "-‐1d") OR priority>3 )) …..

Most of the hosts infected in last day

High Priority host infected in the last day

(3/5) Analysis Technique – Combine MulKple Vectors

57

  With mulKple correlaKon searches, do a meta analysis on events. –  ES: index=notable –  Alert Manager: | rest "/services/alerts/fired_alerts" –  TickeKng system: API or DBConnect

  Search for hosts with mulKple alerts to create a high confidence high severity event.

(3/5) Analysis Technique – Combine MulKple Vectors

58

  Example:

index=notable | stats dc(search_name) as NumRules by dest   More Powerful Example: (index=notable AnKvirus OR ids) OR (tag=proxy category=uncategorized) [… use Stats Eval example for correlaKon …]

  In ES >= 3.2, search index=risk for correlaKons w/o notables

(4/5) Analysis Technique – Increase Logging

59

  Increase logging on suspect hosts   With ES, use Splunk Stream.   Also use your ETDR soluKon.   Leverage panblock, expect scripts to add to increased logging groups

  Write new correlaKon rules based on that increased logging –  Higher confidence, higher severity

(5/5) Analysis Techniques – Machine Learning

60

  With Machine Learning, you can build extremely powerful models and techniques for finding outliers programmaKcally.

  Look at Splunk UBA – this is what they do. –  Ask your SE!

  Look at the new ML App! –  Ask your SE! (Watch him look bewildered!)

Threat Feeds

Threat Feeds •  You know enough to build a threat intel engine •  Don’t

62

Great Threat Feed Tools

63

  ES is officially supported with nine types of threat intel

  Without ES, look at SA-‐Splice on Splunkbase – not supported, but works for many customers.

  Please, please don’t build it yourself!

IPs Domains User Names

Process Names Hashs CerKficate Hashes

CerKficate Common Names Email Addresses File Names

But that’s not all for Threat Intel

64

  Lots of things you can do with Threat Intel –  Turning Indica0ons of Compromise into Tangible Protec0on

hsp://conf.splunk.com/session/2015/recordings/2015-‐splunk-‐94.mp4

–  Managed Threat Intelligence in Splunk ES Splunk’s Brian Luger (ES Developer)

–  hsp://conf.splunk.com/session/2015/recordings/2015-‐splunk-‐148b.mp4

  Generate it yourself (go ask your SE and tell them Andrew Phillips sent you)

Demo the Security Ninjitsu App

Wrap Up

How to Be Successful

67

  Install the app on a non-‐producKon instance! –  Example Searches for Every Use Case/Technique –  One enKrely new use case

  Check out security sessions on hsp://conf.splunk.com   Post to hsp://answers.splunk.com with tag "correlaKonsearch"!   Talk to the person next to you!   Hunt down your nearest Splunk Security SME SE!

Give Me Feedback!

68

  Rate it in the app   $50 Amazon GiT Card will be randomly given to those who also submit feedback here: hsp://www.davidveuve.com/go/conf2015

  Download the app, play around with it, and give me feedback. hsp://www.davidveuve.com/go/conf2015

  Another $50 Amazon GiT Card will be randomly given!

THANK YOU

QuesKons

hsp://www.davidveuve.com/go/conf2015

THANK YOU

splunksummit 2015 - security ninjitsu

Data & Analytics