© 2 0 1 9 S P L U N K I N C .

© 2 0 1 9 S P L U N K I N C .

DB Connect: Automating theH-E-Double Hockey Sticks Out Of It

Ryan MossPrincipal Security Engineer | Verizon

During the course of this presentation, we may make forward‐lookingstatements regarding

future events or plans of the company. We caution you that such statements reflect our

current expectations and estimates based on factors currently known to us and that actual

events or results may differ materially. The forward-lookingstatements made in the this

presentation are being made as of the time and date of its live presentation. If reviewed after

its live presentation, it may not contain current or accurate information. We do not assume

any obligation to update any forward‐lookingstatements made herein.

In addition, any information about our roadmap outlines our general product direction and is

subject to change at any time without notice. It is for informational purposes only, and shall

not be incorporated into any contract or other commitment.Splunk undertakes no obligation

either to develop the features or functionalities described or to include any such feature or

functionality in a future release.

Splunk, Splunk>, Turn Data Into Doing, The Engine for Machine Data, Splunk Cloud, Splunk

Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States

and other countries. All other brand names, product names, or trademarks belong to their

respective owners. © 2019 Splunk Inc. All rights reserved.

Forward-LookingStatements

© 2 0 1 9 S P L U N K I N C .

© 2 0 1 9 S P L U N K I N C .

Why are You Here?

Why this will benefit you.

© 2 0 1 9 S P L U N K I N C .

Why are You Here?

Talk about specific use cases involving the audit logs as well as ad-hoc queries

Makes new inputs and connection creation more simplistic

Allows you to focus on other areas of Splunk

Make your DBA’s happier

Why this will benefit you

© 2 0 1 9 S P L U N K I N C .

“ I absolutely LOVE it when I

get to manually input

connections and inputs in

DB Connect!”

Said no one. Ever.

© 2 0 1 9 S P L U N K I N C .

Introductions

Who is this guy, and what is he going to talk about?

© 2 0 1 9 S P L U N K I N C .

Introductions

Principal Security Engineer

Working on Splunk for the past 6+ years

Worked on on-prem and Cloud deployments

Had one of the fastest cloud deployments

Not a DBA (sorry)

Who is this guy?

© 2 0 1 9 S P L U N K I N C .

Agenda 1) What are the use cases?Possible use cases

2) How do I do it?Splunk HF

DB Repo

SPL

Scripts

3) Demo

4) Key Takeaways

5) Q & A

© 2 0 1 9 S P L U N K I N C .

What Can I Automate?

When and what can you automate

© 2 0 1 9 S P L U N K I N C .

What are the Use Cases?

DB Audit logs for Oracle and MSSQL

• Can automate the connections and the inputs

Ad-hoc queries (connections)

• Can automate the connections

When and what can you automate

© 2 0 1 9 S P L U N K I N C .

That’s Nice, But How Do I Do It?

How to automate DB Connect

© 2 0 1 9 S P L U N K I N C .

That’s Nice, But How Do I Do It?

Need to have some things in place before automation

• Splunk Heavy Forwarder

• DB to be used as a repo for DB information (server name and instance)

• Standardized port configuration

• Good naming convention

Utilize SPL to make your scripting easier

• Create scheduled reports to pull DB information from repo

• Use SPL to preformat the data

Use scripts to combine all these into usable connections and inputs

How to automate DB Connect

© 2 0 1 9 S P L U N K I N C .

That’s Nice, But How Do I Do It?

Size the HF Accordingly

• We use 12 core 16 GB RAM virtual host (works with ~650 connections pulling every 3 min)

Need to change the default sockets and threads

• Settings found in $SPLUNK_HOME\etc.\system\local\server.conf

• Need to change maxSockets and maxThreads under [httpsServer] stanza

• Splunk defaults to “0”. If set to “0”, Splunk automatically sets [maxSockets | maxThreads] to one third of the maximum allowable [open files | threads] on the host

• Can set it to unlimited by setting it to -1 (BE CAREFUL!)

Splunk HF

© 2 0 1 9 S P L U N K I N C .

1. Splunk HFSplunk HF

That’s Nice, But How Do I Do It?

© 2 0 1 9 S P L U N K I N C .

That’s Nice, But How Do I Do It?

Need to have a database repository that you can pull from

• Can be pulled from a CMDB

Need a standardized naming convention

• Naming convention includes the hostname as well as the instance name

• End of instance name include last digits of port

– Example: myhostisawsome\sqlprod84

DON’T USE DEFAULT PORT

• Create a port standard (specific ports for dev/qa/prod)

• Use high port numbers that are not common

– Example 489 for Prod, 488 for QA, 487 for Dev

Last two digits of instance are port number

• Example: myhostisawesome\sqlprod84 – 84 are last two digits of port

• Complete port for connection would be 48984

DB Repo

© 2 0 1 9 S P L U N K I N C .

DB Repo

That’s Nice, But How Do I Do It?

Examples Names:

Hosts\Instances Port Number

awesomesauce\sqlprod33 48933

coolbeans\sqlqa55 48855

winser01\sqldev84 48784

coolserver\sqlprod99 48999

pitattack\sqlqa22 48822

beandip\sqldev7 48707

© 2 0 1 9 S P L U N K I N C .

That’s Nice, But How Do I Do It?

Use the dbxquery on your HF

Utilize eval commands to pre-process the data

Use outputlookup to output the results to a CSV

Save the search as a scheduled report

Utilize SPL

© 2 0 1 9 S P L U N K I N C .

Utilize SPL

That’s Nice, But How Do I Do It?

Example query:

| dbxquery query="select * FROM [dbrepo].[dbo].[sqlinstances]"

connection="mssql_winser01selxa_sqldev84"

| eval instance_name_modified = replace(sqlinstances,"\\\\","_")

| eval instance_name_modified = "mssql_" +

instance_name_modified

| eval instance_name_modified = lower(instance_name_modified)

| rex field=instance_name_modified

"(?<server_name>mssql_\w+_sql[qa|dev|prod]*)(?<old_port>\d{1,2})“

| eval initialLength = len(tostring(old_port))

| eval port = "0".tostring(old_port)

| eval port = substr(port,initialLength,2)

| eval instance_name = server_name + port

| table instance_name

| outputlookup sql_server_instances.csv

© 2 0 1 9 S P L U N K I N C .

That’s Nice, but How Do I Do it?

Example output

Utilize SPL

© 2 0 1 9 S P L U N K I N C .

That’s Nice, but How Do I Do It?

Script will combine everything you did previously

Automates the connection, inputs as well as creating the checkpoint file

Utilize scripts

© 2 0 1 9 S P L U N K I N C .

Utilize Scripts

That’s Nice, But How Do I Do It?

Example Script:

#!/bin/bash

##### Shutdown Splunk Service in Prep to Update Files #####

#/opt/splunk/bin/splunk stop

sudo systemctl stop Splunkd.service

##### Remove Quotes and instance_name from file and output to

txt file #####

sed 's/\"//g'

/opt/splunk/etc/apps/search/lookups/sql_server_instances.csv >

sqlserver_list.txt

sed -i '/instance_name/d' sqlserver_list.txt

© 2 0 1 9 S P L U N K I N C .

Example Script (cont’d):

##### Separate out DEV QA PROD #####

cat sqlserver_list.txt | grep sqldev > mssql_server_dev_instance.txt

cat sqlserver_list.txt | grep sqlqa > mssql_server_qa_instance.txtcat sqlserver_list.txt | grep sqlprod > mssql_server_prod_instance.txt

##### Read contents of instance file and put it in a variable #####

mapfile -t devStanza < mssql_server_dev_instance.txt

mapfile -t qaStanza < mssql_server_qa_instance.txtmapfile -t prodStanza < mssql_server_prod_instance.txt

##### Copy the DB Connections to search for new additions #####

cp

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf db_connections.txt

##### Copy the DB Inputs file to search for new additions #####

cp /opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf

db_inputs.txt

Utilize Scripts

That’s Nice, But How Do I Do It?

© 2 0 1 9 S P L U N K I N C .

Example Script (cont’d):

##### Create the DEV connection file #####

for dstanza in ${devStanza[@]}; do

if grep -Fq "$dstanza" db_connections.txtthen

echo -n ""

else

echo "

[$dstanza]connection_type = generic_mssql

database = master

disabled = 0

fetch_size = 10000

host = $dstanzaidentity = sql_service_account

jdbcUseSSL = true

port = $dstanza

readonly = true

timezone = UTC"fi

done > dev_connection.txt

Utilize Scripts

That’s Nice, But How Do I Do It?

© 2 0 1 9 S P L U N K I N C .

Example Script (cont’d):

##### Only show correct Hostname and Port #####

sed -i 's/host = mssql_/host = /g' prod_connection.txt

sed -i 's/port = mssql_[^_]*_sqlprod/port = 489/g' prod_connection.txtcat prod_connection.txt | sed -r 's/(host\s+=\s+\w+)_.*/\1/' >

prod_connection_test.txt; mv prod_connection_test.txt

prod_connection.txt

##### Write the connections to the db_connections file #####cat prod_connection.txt >>

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf

cat qa_connection.txt >>

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf

cat dev_connection.txt >> /opt/splunk/etc/apps/splunk_app_db_connect/local/db_connections.conf

Utilize Scripts

That’s Nice, But How Do I Do It?

© 2 0 1 9 S P L U N K I N C .

Example Script (cont’d):

##### Create the DEV inputs file #####

for dstanza in ${devStanza[@]}; do

if grep -Fq "$dstanza" db_inputs.txt

then

echo -n ""

else

echo "

[$dstanza]

batch_upload_size = 1000

connection = $dstanza

disabled = 0

fetch_size = 300

index = mssql

index_time_mode = dbColumn

input_timestamp_column_number = 1

interval = 2-59/5 0,2-23 * * *

max_rows = 0

max_single_checkpoint_file_size = 1048576

mode = rising

query = SELECT *\\

FROM sys.fn_get_audit_file ('C:\\\\SplunkAudit\\\\SplunkAudit*',default,default)) \\

WHERE event_time > ?\\

ORDER BY event_time ASC

query_timeout = 300

sourcetype = mssql:audit

tail_rising_column_number = 1"

fi

done > dev_inputs.txt

Utilize Scripts

That’s Nice, But How Do I Do It?

© 2 0 1 9 S P L U N K I N C .

Example Script (cont’d):

##### Write inputs to SQL inputs file #####

cat prod_inputs.txt >>

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.confcat qa_inputs.txt >>

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf

cat dev_inputs.txt >>

/opt/splunk/etc/apps/splunk_app_db_connect/local/db_inputs.conf

Utilize Scripts

That’s Nice, But How Do I Do It?

© 2 0 1 9 S P L U N K I N C .

Utilize Scripts

That’s Nice, But How Do I Do It?

Example Script (cont’d):

##### Create the PROD checkpoint files #####

for stanza in ${prodStanza[@]}; do

if [ ! -e /opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect/$st

anza ]

then

echo '{"value":"1970-01-01

00:00:00.00","appVersion":"3.1.4","columnType":93,"timestamp":"1970-01-01T00:00:00.000-04:00"}' >

/opt/splunk/var/lib/splunk/modinputs/server/splunk_app_db_connect/$st

anza

fi

done

##### Start up Splunk #####

#/opt/splunk/bin/splunk start

sudo systemctl start Splunkd.service

© 2 0 1 9 S P L U N K I N C .

Demo

© 2 0 1 9 S P L U N K I N C .

© 2 0 1 9 S P L U N K I N C .

1. DB Connect doesn’t have to be a manual process

2. Create a DB Repo for your hosts\instances

3. Utilize SPL and scripts to automate the process

4. Can be used in Oracle as well as MSSQL

Key Takeaways

RATE THIS SESSION

Go to the .conf19 mobile app to

© 2 0 1 9 S P L U N K I N C .

You!

Thank

© 2 0 1 9 S P L U N K I N C .

Q&A

Ryan Moss | Principal Security Engineer