![Page 1: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/1.jpg)
Splunk Enterprise SecurityFor Proactive Monitoring
![Page 2: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/2.jpg)
AKA: Enterprise SecurityTips, Tricks, and Analytics
![Page 3: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/3.jpg)
Purpose
● Describe Tips for a Clean Setup of ES
● Provide Tricks “From the Field” in Setup/Mgmt
● Demonstrate Analysis With and Without Training Wheels
![Page 4: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/4.jpg)
![Page 5: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/5.jpg)
![Page 6: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/6.jpg)
Who Am I?
● Sean Wilkerson, Partner/Consultant, Aplura
● Speaker at SANS Log Mgmt Summits
● Splunk Pro Serv Partner Since 2008
![Page 7: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/7.jpg)
Splunk/ES Experience
● 20+ ES Engagements
● Dozens of Different Federal Entities
● Many Commercial Customers
● My 4th .conf
● 5+ Years of Splunk Pro Serv.
● 6+ Years using Splunk
● 12+ Years Of Logs (Shell, Scripts, SIM, Splunk)
● 14+ Years of Network --> Systems --> InfoSec
![Page 8: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/8.jpg)
![Page 9: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/9.jpg)
Who Are You?
● You Know a Handful of Splunk Search Cmds
● You Have Worked With Splunk Conf Files
● You Know Generally What ES is demos/talks
● You May be a Splunk/ES User/Administrator
● You Are Analysts!! <--- Really Important
![Page 10: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/10.jpg)
Content Available Now!
aplura.com/splunkconf2013
![Page 11: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/11.jpg)
Splunk App for Enterprise Security
Scalability to manage multi-terabytes of real-time and historical data
Pre-built security correlation rules, reports, and dashboards
Statistical analysis for defining ‘normal’
Incident investigation and management framework
Solution with out-of-the-box content to manage known and unknown threats.
Security AnalystsSOC Staff Security Execs/Mgrs
Security Auditors
![Page 12: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/12.jpg)
vocab(ES)
● ES = Enterprise Security
● TA = Technology Add-on (fields and tags)
● SA = Security Add-on (searches and corr logic)
● DA = Domain Add-on (dashboards)
● Macros = Shortcut to Splunk search string
● Correlation = Notable Event Searches
● Onboard = Inputs and TAs
● CIM = Common Information Model
![Page 13: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/13.jpg)
Splunk Enterprise Security
Tips
![Page 14: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/14.jpg)
ES From 10,001 Feet
● ES = Well-Organized Deployment
● Good Organization = Free Correlation
● CIM = Babelfish (One Language)
● Allows Fewer and Clearer Correlations
● Intelligently Doing More with Less...Overhead
● Improves the Speed to Root Cause Analysis
![Page 15: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/15.jpg)
Deployment Steps
● First, Solidify Architecture
● Install: DS, SH->IDX; Validate Storage, etc.
● Ensure ES Storage Supports TSIDX (100GB is 340G/yr w/ ES-2.x)
● Onboard at least: Firewall, WEL (AD), IDS, AV
● Start ES and Validate TAs
● Enable and Schedule Desired Correlations
● Integrate Assets and Identities
● Onboard Other Supported Data-Sources
● Onboard Custom Data-Sources
● Tune and Optimize
![Page 16: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/16.jpg)
General Housekeeping
● ASAP Start Defining Assets
● Time-Audit Before It's Too Late
● RT > index=* | eval timeDiff=_indextime-_time | timechart span=10m avg(timeDiff) by sourcetype
![Page 17: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/17.jpg)
Leverage ES Strengths
● ES Reports on Security-related Decisions
● Information is Grouped into Three Domains:
● Access (Logins, Admin Activity)
● Endpoint (Malware, Systems, Time)
● Network (Firewall, IDS, VA, WebProxy)
● Some Data Doesn't Need ES
● ES Assumes a Framework, So Should You
![Page 18: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/18.jpg)
**Hazards Ahead**
The journey is profitable; however,fumbled steps can land you in peril.
![Page 19: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/19.jpg)
Hazard: Underpowered Hardware
● Splunk (like DBs) Can Run on An Old Laptop
● It doesn't mean that it should!!
● Meet or Exceed the “Reference“ “Architecture“
● Don't Skimp on Hardware!!
● Until [ $IOPS >= 1200 ]; do storage++; done;
![Page 20: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/20.jpg)
Hazard: No EventGen in Production
● EventGen Creates Fake Data for DEMOS
● Do Not Enable This in Production!!
● Really? Do I Have to Say This?
● Yes, I do!!
![Page 21: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/21.jpg)
Hazard: App Isolation● “App Isolation” allows apps to play nice
with each other with little to no regard for precedence.
● For testing: Edit any SA-$NAME/medatadata/local.meta to add your custom app
● For permanence: Edit app SplunkEnterpriseSecuritySuite default/inputs.conf
● Be Mindful of App Isolation – It Can Bite Hard
![Page 22: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/22.jpg)
Hazard: Asset Formatting
● Assets provide the context between the data and correlations. VALUE++
● ES-2.2+ - “Asset Expander” - Validate/Format
● > index=_internal source=*lookup_expander.log
● Temporarily Adjust Input to Shorten Test Cycle
![Page 23: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/23.jpg)
Hazard: RealTime Correlations
● Many of the Correlations are “RealTime”
● Switch these to scheduled (generally speaking)
![Page 24: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/24.jpg)
Hazard: Customizations
● In Splunk – Custom = Immortalized
● This is the local vs default thing...
● Customizations Can Effect the Mechanics of ES
● Leverage the Customizations Encouraged in ES but Don't Make Your Own
● Customizations = Difficult Upgrades
● Customize Correlations With Care
● Do Not Customize Views, Assets-fields, or Scripts
![Page 25: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/25.jpg)
Splunk Enterprise Security
Tricks
![Page 26: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/26.jpg)
Tricks: Dynamic Lists
● This May Look Unexciting, but it is What I Get Asked for the Most
● Dynamic Assets/Identities (via SavedSearch) Whenever Possible
● Use SA-ldapsearch for Both, such as this:
| ldapsearch domain=$domain$ search="(&(objectClass=user)(!(objectClass=computer))(!(displayName=SystemMailbox*)))" attrs="cn,userPrincipalName,sAMAccountName,personalTitle,displayName,givenName,sn,suffix,mail,telephoneNumber,mobile,manager,priority,department,category,watchlist,whenCreated,accountExpires" | fields - _* | rename sAMAccountName as identity, personalTitle as prefix, displayName as nick, givenName as first, sn as last, mail as email, telephoneNumber as phone, mobile as phone2, manager as managedBy, department as bunit, whenCreated as startDate, accountExpires as endDate | table identity, prefix, nick, first, last, suffix, email, phone, phone2, managedBy, priority, bunit, category, watchlist, startDate, endDate | outputlookup simple_identity_lookup
![Page 27: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/27.jpg)
Tricks: Nice Assets
● Create Asset Categories In SavedSearch
| `assets` | mvexpand category | dedup category | sort category | table category | outputlookup category_lookup
● Use CIDR Blocks in Assets
● This Allows for Inclusion/Exclusion of Network by category reference. This is big!
– E.G. All IDS alerts by category=oracle_cluster
![Page 28: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/28.jpg)
Tricks: Nice Assets 2
● Plan Asset Categories and Benefit
● Use Built-ins When Available (e.g. email_servers)
● Don't Make More Granularity Than You Can Use
● Plan Supportive Naming Scheme For CIDR...
– foonet_nyc_dmz
– foonet_nyc_users
– foonet_chg_dmz
Note: Critical Point
![Page 29: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/29.jpg)
Tricks: Upgrade !SNAFU
● Read ReleaseNotes
● Unpack ES, Extract TAs
● Sync Upgraded TAs to DS:
● Do them one at a time
● Watch for default changes and lookup overwrites
● Push TAs Out to Search/Parsing Tiers
● Use UI and Do ES Upgrade
● Remove Unnecessary TAs and Ensure Yours are Pushed
![Page 30: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/30.jpg)
Tricks: Create a Custom TA
● It Looks Harder Than it is, but Don't Rush
● Have Your DataSource Manual Ready
● Prepare a regex parser too, as needed
● Work in Dev Environment Whenever Possible
● Copy a Similar TA
● Input the Data (Apply Necessary Parse-time Confs)
● Ensure Necessary Fields Are Present
● Ensure Necessary Tags/Eventtypes are There
● Validate Your TA (See Next Slide)
![Page 31: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/31.jpg)
Tricks: Validate TAs
Use search or macros to verify TAs
![Page 32: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/32.jpg)
Splunk Enterprise Security
Analytics (Unchained)
![Page 33: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/33.jpg)
Drilldown Gets You Started (Demo)
![Page 34: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/34.jpg)
ES Macros (Demo)
● `authentication`
● `ids_attack`
● `communicate`
● `malware`
● `proxy`
● `vulnerability`
![Page 35: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/35.jpg)
ES Lookups (Demo)
● Assets
● | `assets` ; | inputlookup simple_asset_lookup
● | `categories` ; | inputlookup category_lookup
● Identities
● | `identities` ; | inputlookup simple_identity_lookup
● Trackers (on my)
● | `access_tracker`
● | `port_protocol_tracker`
● | `ids_attack_tracker`
![Page 36: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/36.jpg)
Custom Analysis (Demo)
● `proxy` | search `get_subject(src, "10.11.36.23")`
● `proxy` | search NOT action="tcp_denied" [ search `proxy` | search action="tcp_denied" | dedup src | table src] | top dest by src
● `ids_attack` | search (severity="critical" OR severity=”high”) signature="dos*" `get_subject(src, "125.17.14.100")` category="dos"
![Page 37: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/37.jpg)
Additional Resources
● docs.splunk.com - General Manuals
● docs.splunk.com/Documentation/ES - ES
● splunk-base.splunk.com - User forums
● Cheatsheet - duh!
![Page 38: Splunk Enterprise Security For Proactive Monitoring · Tips, Tricks, and Analytics. Purpose Describe Tips for a Clean Setup of ES Provide Tricks “From the Field” in Setup/Mgmt](https://reader034.vdocuments.site/reader034/viewer/2022042122/5e9c74bf005178659659029c/html5/thumbnails/38.jpg)
Thank You!
ES: Trips, Tricks, Analytics (This Talk)
aplura.com/splunkconf2013
● Also:
● Best Practice PDF: aplura.com/splunkbp
● Talk: Security Analysis: aplura.com/splunklive2013
● Talk: Best Practice: aplura.com/splunklive2012
● Talk: SIEM Fails: aplura.com/lookbeforeyousim