azure event hub connector user guide - solsys · 2019-07-31 · 3- install azure event hub...

2019

Azure Event Hub Connector User Guide

Solsys Azure EventHub Connector and Application Installation Guide V 1.3

pg. 1

Table of Contents

Introduction .................................................................................................................................................. 3

Overview ....................................................................................................................................................... 4

Azure EventHub Connector requirement ..................................................................................................... 6

Requirements on the Heavy Forwarder ................................................................................................ 6

Before installing add-on and connector ............................................................................................... 7

Azure requirements for the connector ......................................................................................................... 7

Azure Event Hub requirements ............................................................................................................. 7

Sizing guidelines .................................................................................................................................... 7

Source Type for Azure Event Hubs TA Splunk ............................................................................................... 8

Installation walkthrough ............................................................................................................................... 9

Installation Order ...................................................................................................................................... 9

1 – Install Azure Event Hub TA on Splunk Components ........................................................................... 9

2- Install Pre-requisites on Splunk Heavy Forwarder ................................................................................ 9

Install Python3 ...................................................................................................................................... 9

Install Azure EventHub SDK................................................................................................................. 10

Install Splunk SDK for Python .............................................................................................................. 11

3 - Install Azure Event Hub Connector on Splunk Heavy Forwarder ...................................................... 11

One Last Check .................................................................................................................................... 11

Azure Event Hub Connector Download Workflow ............................................................................. 11

Azure EventHub Connector Installation & Configuration Steps ......................................................... 11

Splunk Configuration GUI .................................................................................................................... 13

Wrapper Script Readme.txt ................................................................................................................ 15

4 - Configure Azure EventHub TA Splunk on Heavy Forwarder .............................................................. 15

Splunk Scripted Inputs – Pull Data from Event Hub ............................................................................ 15

Splunk Monitoring Inputs – Onboard Data in Splunk ......................................................................... 16

Selecting your Sourcetype .................................................................................................................. 17

Best Practices .............................................................................................................................................. 18

Troubleshooting EventHub Connector Script Errors .................................................................................. 18

Minimum Viable Product ............................................................................................................................ 20

Release notes for the Splunk Add-on for Azure Event Hubs ...................................................................... 21

Compatibility ........................................................................................................................................... 21


pg. 2

New features ........................................................................................................................................... 21

Fixed issues ............................................................................................................................................. 21

Known issues ........................................................................................................................................... 21

Third-party software attributions ........................................................................................................... 21

References .................................................................................................................................................. 22

Using Shared Access Signature ........................................................................................................... 22

Service Bus Access with SAS................................................................................................................ 22

Microsoft Azure SDK for Event Hubs .................................................................................................. 22

Splunk Wrapper .................................................................................................................................. 22

Appendix A .................................................................................................................................................. 23

Find Executables and Creating SymLink .............................................................................................. 23

Appendix B .................................................................................................................................................. 24

Updating $PATH .................................................................................................................................. 24

Appendix C.1 ............................................................................................................................................... 24

Installing CMAKE and C++ Complier.................................................................................................... 24

Appendix C.2 ............................................................................................................................................... 25

Troubleshooting Steps For Connecter ................................................................................................ 25

Appendix D: ................................................................................................................................................. 26

The Azure Monitor Splunk Add-on on Splunkbase ............................................................................. 26

Appendix E: Reference Critical Design Elements & Options ...................................................................... 27


pg. 3

Introduction

Version 1.3

Vendor Products Azure Event Hubs

Visible External Document

This document introduces three applications, Azure Event Hub Connector for Splunk, Azure Event Hub Insights Application and the Azure Event Hub Connector. The Connector allows the Splunk platform to consume topic messages from Azure Event Hubs using scripts. The Technical Add-on provides the inputs to invoke the scripts and forward the data to Splunk, it also providers CIM-compatible knowledge objects to use with other Splunk apps. The Insights Application provides dashboards that leverages the operational-insights-logs eventhub from Azure.

Download the Azure Event Hub Technical Add-on from Splunkbase:

https://splunkbase.splunk.com/app/4532/

Download the Azure Event Hub Insights App for Connector from Splunkbase:


For a summary of new features, fixed issues, and known issues, see Release Notes for the Splunk Add-on for Azure EventHubs.

For information about installing and configuring the Add-ons for EventHubs, see Installation and configuration overview for the Splunk Add-on for Azure EventHubs.

https://docs.splunk.com/Splexicon:CommonInformationModel



http://solsys.ca/

http://solsys.ca/

http://docs.splunk.com/Documentation/AddOns/released/Kafka/Installationoverview

http://docs.splunk.com/Documentation/AddOns/released/Kafka/Installationoverview


pg. 4

Overview Solsys is delivering a Splunk Add-on for Azure Event Hubs as part of the PLATO Splunk Operation Transition SOW.

Detailed Logical Overview

Heavy

Forwarder

Indexer

Azure Event Hub SA

Event Hubs

Add On

Event Hubs

Event Hubs

Azure Key Vault

//[my namespace].servicebus.windows.net/[event hub

name]/publishers/[my publisher name]





An authorization rule has a name, is associated with specific rights, and

carries a pair of cryptographic keys. You use the rule's name and key via the

Service Bus SDK or in your own code to generate a SAS token. A client can

then pass the token to Service Bus to prove authorization for the requested

operation.

Search

Heads

Application Logs

Operational Logs

Azure VM Metrics

Add On


pg. 5

Simple Logical Overview

Search Head

Indexer

Heavy Forwarder

Microsoft Azure

Event Hub - Namespace 1

Event Hub – Namespace 2

Event Hub 1

Event Hub 2

Event Hub 1

Event Hub 2

Event Hub 3 Splunk SDK

Python 3

Azure Event Hub SDK

Solsys – Event Hub Connector

Splunk – Azure EventHub TA



VMs

Machine Components:

Components Description

Splunk Search Head A Splunk Search Head (SH) is required for the end-user to search the eventhub index, view search-time field extractions and use SPL operators to extract intelligence from the data

Splunk Indexer A Splunk Indexer will apply the index-time extractions, parses data and then stores them to the Splunk DB

Splunk Heavy Forwarder A Splunk Heavy Forward is a necessary component of this architecture. It will host the Eventhub add-on and all its prerequisites

Azure Event Hub(s) Azure Event Hubs with namespaces configured to collect logs. At least one configured Event Hub and namespace is required for this add-on to start collecting data


pg. 6

Software Components:

Components Description Location of Installation

Azure Event Hub TA Splunk The Azure Event Hub TA for Splunk contains configuration to monitor the output created by the Connector, inputs to invoke the script for

Splunk: Search Head, Indexer, Heavy Forwarder

Azure Event Hub Connector The Azure Event Hub Connector for Splunk contains the installation scripts required for Splunk to communicate with the Azure SDK

Splunk: Heavy Forwarder

Python3 The Splunk Heavy Forwarder Server needs to have python3 installed with its necessary pre-requisites libraries. Python3 needs to be installed at the system level.


Splunk SDK for Python Splunk SDK for Python is a pre-requisite for the Connector to work


Azure Event Hub SDK for Python Splunk SDK for Python is a pre-requisite for the Connector to work. It allows communication with remote Azure Event Hubs


Azure EventHub Connector requirement

Requirements on the Heavy Forwarder

The following components need to be installed on the Heavy Forwarder Server:

- Azure Event Hub SDK, Splunk SDK for Python, Python3 - Azure Event Hub Connector for Splunk - Azure Event Hub TA

To install and configure the SDK, the user may need to be a member of the root group. Ensure that once the SDK’s are installed Splunk can access the SDK libraries.

For the Connector to successfully run, an ‘admin’ user needs to be created on the Splunk Heavy Forwarder.

Since this add-on runs on the Splunk platform, all general Splunk system and sizing requirements apply on running this add-on. For specific scenarios and unique cases please reach out to a Solsys Splunk Consultant for more information.


pg. 7

Before installing add-on and connector

1. Get the Azure Event Hub Technical and Supporting Add-on by downloading it from Splunk base

2. Determine where and how to install this add-on in your deployment, using the tables on this page.

3. Perform any prerequisite steps before installing, if required and specified in the tables below.

4. Complete your installation.

If you need step-by-step instructions on how to install an add-on in your specific deployment environment, see Installation walkthroughs on this page for links to installation instructions specific to a single-instance deployment or contact a Solsys Splunk Consultant.

Azure requirements for the connector

Azure Event Hub requirements SAS Authorization Policy Name & Key to generate SAS Token used for Event Hub authorization

• Event Hub Namespaces contain individual Event Hubs.

• An access policy is necessary to access the data in the Event Hubs.

• Access policies can be defined on the namespace (which the hubs will inherit), or on the individual hub.

• The access policy used to connect to the hubs is stored in Azure in a key vault.

• The access policy is stored as a name/value pair in the key vault

• The Azure Event Hubs Add-on connects to the vault in order to get the access policy (this puts the security in the hands of the Azure admin).

• The Azure Event Hubs Add-on uses different polices to access each individual hub.

Sizing guidelines There are no sizing guidelines for the monitor inputs.

For the modular inputs that collect Azure topic messages, your sizing requirements depend on how much data per second producers push to the Azure topics from which you want to collect data. The Azure Event Hub Add-ons can handle millions of records per minute.

Indexing the data depends on splunkd capacity and how the environment is architected. If splunkd can process N records per second, and there are M records per second flowing into the Azure topics, you need a capacity of M divided by N in your heavy forwarders.

Disclaimer: This deployment guide does not cover Splunk or Azure architecture capacity or disk size planning.

http://docs.splunk.com/Documentation/AddOns/released/Kafka/Install#Installation_walkthroughs


pg. 8

Source Type for Azure Event Hubs TA Splunk

The Azure EventHub Connector pulls in data from the actual eventhub, whether it is custom application logs,

system logs, server metrics etc. The connector can connect to that hub in a namespace provided the

service/provider is subscribed to the EventHub. Once the connector is setup there are 3 pre-defined sourcetypes in

the Azure_Event_Hub_TA_Splunk to assist you with your data onboarding into Splunk:

Note: The Azure EventHub Connector does not collect telemetry information, it collects the raw events/incidents

that subscribed to the eventhub.

Predefined Sourcetypes:

[eventhub:operational:json]

description = Format of default operational insights eventhub entity

BREAK_ONLY_BEFORE_DATE =

DATETIME_CONFIG =

KV_MODE = JSON

LINE_BREAKER = ([\r\n]+)

MAX_EVENTS = 100000

MAX_TIMESTAMP_LOOKAHEAD = 27

NO_BINARY_CHECK = true

SHOULD_LINEMERGE = false

TIME_FORMAT = %Y-%m-%dT%H:%M:%S

TIME_PREFIX = \{\"records\"\:\s\[\{\s+\"time\"\:\s+\"

TRUNCATE = 100000

category = Custom

disabled = false

pulldown_type = true

TZ=UTC

[eventhub:json]

description = Generic eventhub json sourcetype

BREAK_ONLY_BEFORE_DATE=null

CHARSET=UTF-8

LINE_BREAKER=([\r\n]+)

MAX_EVENTS=100000

MAX_TIMESTAMP_LOOKAHEAD=27

NO_BINARY_CHECK=true

SHOULD_LINEMERGE=false

TIME_FORMAT=%Y-%m-%dT%H:%M:%S

TIME_PREFIX=.+\s\[\{\s+\"time\"\:\s+\"

TRUNCATE=100000

category=Cloud

disabled=false

pulldown_type=true

INDEXED_EXTRACTIONS=json

TZ=UTC

AUTO_KV_JSON=false

[eventhub:log]

description = Format like linux log files. Utilizes Splunks default syslog sourcetype transformations

DATETIME_CONFIG = /etc/datetime.xml

LEARN_SOURCETYPE = true

REPORT-syslog = syslog-extractions

CHARSET = UTF-8

category=Cloud

BREAK_ONLY_BEFORE_DATE = true

TRANSFORMS = syslog-host

TIME_FORMAT = %b %d %H:%M:%S

DATETIME_CONFIG =

MAX_TIMESTAMP_LOOKAHEAD = 27

LINE_BREAKER = ([\r\n]+)

MAX_EVENTS = 5000

Example of eventhub:operation:json

{"records": [{ "time": "2019-05-14T15:48:19.1726705Z", "resourceId"…}]}


pg. 9

Installation walkthrough

Installation Order The installation order is a general guideline on

1- Install Azure Event Hub TA on Splunk Components

2- Install Pre-requisites on Splunk Heavy Forwarder Server

3- Install Azure Event Hub Connector on Splunk Heavy Forwarder

4- Configure Azure Event Hub TA on Heavy Forwarder

5- Search on the Splunk Search Head!!!

1 – Install Azure Event Hub TA on Splunk Components The Azure Event Hub TA Splunk can be downloaded on Splunkbase. Once the file is downloaded extract and place

the TA the following directories:

- $SPLUNK_HOME/etc/apps/Azure_Event_Hub_TA_Splunk on the Search Head

- $SPLUNK_HOME/etc/apps/Azure_Event_Hub_TA_Splunk on the Indexer

- $SPLUNK_HOME/etc/apps/Azure_Event_Hub_TA_Splunk on the Heavy Forwarder

The method to install the TA in the Splunk directories should follow Splunk Best practices, the method

may also change depending on a clustered/non-clustered environment.

2- Install Pre-requisites on Splunk Heavy Forwarder The following prerequisites would need to be completed before installing the Solsys Azure Event Hub

Connector Splunk. The pre-requistiess are needed to ensure that Splunk is successfully able to work

seamless

Install Python3 There are multiple ways of installing Python3 in your RHEL environment, here are two ways tried and

test in our lab:

Example on Fedora:

1) $sudo yum install python3 (to download latest version from yum repository)

2) Use Python Installation guide https://docs.python-guide.org/starting/install3/linux/

a. Ensure that these third-party libraries are installed: setuptools,pip, openssl, cmake

b. The Splunk SDK and Event Hub SDK are both installed using setuptools

Minimum requirement of python 3 is version 3.6 and onwards.

https://docs.python-guide.org/starting/install3/linux/


pg. 10

Example in Red Hat:

Using the Yum command in red hat here are the libraries you need to install on your server

Libraries to install in red hat:

- rh-python36-python.x86_64

- rh-python36-python-libs.x86_64

- rh-python36-python-devel.x86_64

- pip(3) install cmake

Troubleshooting Steps:

- Verify that you and the Splunk user can execute python3 command by typing it in the CLI. If not

please proceed to Appendix A for troubleshooting steps.

- Once python3 is installed verify that the executable exists in /usr/bin/ or in /usr/local/bin. If not

create a symmlink or edit bash_profile to add the path of the executable.

Install Azure EventHub SDK

To install the Azure Event SDK you would root permission or add Splunk into the ‘wheel’ user group. You

can add Splunk in this group by using the following command “usermod -aG wheel splunk”

- $sudo wget https://github.com/Azure/azure-event-hubs-python/archive/v1.3.1.tar.gz -O

eventhub.tgz

- $splunk tar -xvzf eventhub.tgz

- $splunk cd azure-event-hubs-python-1.3.1

- $splunk python3 setup.py install --user

There may be warning messages during the installation process for the Connector, however, if the there

is no Fatal error, the installation will succeed.

Notes: If the installation fails, take note of any python dependencies that are missing in the output of

the installation.

Troubleshooting Steps:

- If you are encountering errors in your Azure SDK installation process, please make note of the

first error that shows in your installation output.

- It may mention a library or command e.g ‘CMAKE’ that is unavailable. If that is the case,

troubleshoot by downloading the dependencies.

- Appendix C1 contains the libraries that are needed for CMAKE and the C++ dependencies in

redhat

https://github.com/Azure/azure-event-hubs-python/archive/v1.3.1.tar.gz%20-O%20eventhub.tgz

https://github.com/Azure/azure-event-hubs-python/archive/v1.3.1.tar.gz%20-O%20eventhub.tgz


pg. 11

Install Splunk SDK for Python

- $sudo wget https://github.com/splunk/splunk-sdk-python/archive/1.6.6.tar.gz -O splunksdk.tgz

- $splunk tar -xvzf splunksdk.tgz

- $splunk cd splunk-sdk-python-1.6.6

As Splunk User run the following command:

- $splunk python3 setup.py install --user

Note: If the installation fails, take note of any python dependencies that are missing in the output of the

installation.

3 - Install Azure Event Hub Connector on Splunk Heavy Forwarder Prior to installing the connector ensure that you have installing the Azure Event Hub TA for Splunk. The Azure

Event Hub TA for Splunk can be downloaded on Splunkbase. Once the file is downloaded extract and place the TA

the following directory:

- $SPLUNK_HOME/etc/apps/Azure_Event_Hub_TA_Splunk on the Heavy Forwarder

We can now finally move onto the installing for the Connector.

One Last Check The Azure Event Hub Connector requires Python 3 for both the Event Hub Processor Host module and the async

send/receive functionality. Pleas ensure that the below have been installed correctly.

Prerequisites:

- Python 3.6+

- Azure Event Hub SDK

- Splunk SDK

Azure Event Hub Connector Download Workflow

The Connector can be download via a Solsys Inc. Sales representative, please reach out to

[email protected] for further instructions on how to download the connector.

Azure EventHub Connector Installation & Configuration Steps

Before we can setup or install the Connector we have to ensure that the Splunk user is added to the wheel group

on the server. You can achieve this by running this command “usermod -aG wheel splunk” as root. This allows the

Splunk user certain admin privileges essential to running some python libraries.

Installation Once receive the Connector from Solsys, extract the file in a location Splunk can access (and ensure splunk has

ownership of the entire folder)

- Cd $DirectoryOfEventHubConnector

https://github.com/splunk/splunk-sdk-python/archive/1.6.6.tar.gz%20-O%20splunksdk.tgz


mailto:[email protected]


pg. 12

- $splunk python3 setup.py install - -user (note two dashes before user ’-‘-‘)

Note: There may be warning messages during the installation process for the Connector, however, if the there is no

Fatal error, the installation will succeed.

Validate Installation

Once your install the Connector, the result of the installation should create an eventhubconsumer executable.

Verify that you able to run the ‘eventhubconsumer’, executable. If not, please proceed to Appendix A and B to

troubleshoot.

Configuration

Configuration involves setting up the environment variables that the Connector users to functions. Each of the

variables used by the connector are defined in a table at the end of this section.

These steps can we used to setup the variables.

We first start with the EVENTHUB_CONSUMER_LOG_DIR variable for log output (by default, logs are written to

/tmp)

- $ echo "export EVENTHUB_CONSUMER_LOG_DIR=/tmp" >> .bash_profile

o Depending on your environment you may have to update .bashrc instead in your user home

directory

Optional variables and default paths

- $ echo "export EVENTHUB_CONSUMER_EXECUTABLE=/usr/local/bin/eventhubconsumer" >>

.bash_profile

- $ echo "export EVENTHUB_CONSUMER_OFFSET_DIR=/tmp" >> .bash_profile

- $ echo "export EVENTHUB_CONSUMER_PYTHONPATH=/usr/bin/python3" >> .bash_profile

Once the variables have been defined update your bash profile

- source ~/.bash_profile

o Refresh your bash profiles once the paths are defined

Note: it's important that this line is added to .bash_profile/.bashrc for the user that Splunk will run as


pg. 13

To test if the variables you just defined are picked up by the splunk user run the “echo ” command against that

variable. E.g “echo EVENTHUB_CONSUMER_LOG_DIR” to print out the path you defined in the bash profile. The default

paths of the variables have been listed below the in the table.

Table of variables:

Variable Definition Default Path EVENTHUB_CONSUMER_LOG_DIR Directory to write log files to disk /tmp

EVENTHUB_CONSUMER_EXECUTABLE Path for eventhub executable file /usr/local/bin/eventhubconsumer

EVENTHUB_CONSUMER_OFFSET_DIR Directory contains offset file for each partition. The offset tracks the location of each partition where it is reading data

/tmp

EVENTHUB_CONSUMER_PYTHONPATH Path of pythong3 executable /usr/bin/python3

Once the variables have been defined, copy wrapper.py into $SPLUNK_HOME/bin/scripts (or another Splunk script

directory)

- $ cp wrapper.py $SPLUNKHOME/etc/apps/Azure_EventHub_TA_Splunk/bin/

Note that the wrapper script is written in Python 2.x in order to be compatible with the Splunk internal Python

environment.

Splunk Configuration GUI From Splunkbase download the “REST storage/password manager” app (don't set a realm).

- Once downloaded and installed please navigate to the app and click on “create”

- Add EventHub Credentials from Azure Portal


pg. 14

- The username for the storage password entry should be the SAS policy name and the password should be

the SAS policy key. Save this in the “Azure_EventHub_TA_Splunk” App context

- When creating the stored credential, you can optionally set a realm, app, and user scope. These values

can later be passed to the script as flags to reduce the visibility of the key within Splunk.

- The Realm, Owner, and App Scope fields during credential creation relate to the --realm, --owner, and --

app flags passed to the wrapper script

Configuration Table for App

Configuration Parameter Value

Username Your SAS Policy Name from Azure Portal

Password Your SAS Policy Primary Key from Azure Portal

Realm Realm

Owner Admin

Read Users All (*)

Write Users Admin

App Scope Azure_EventHub_TA_Splunk

Sharing Global


pg. 15

Wrapper Script Readme.txt

When configuring the wrapper script, the following positional arguments are required. Only use these

configuration options if you would like to change the default settings.

namespace - the Azure Event Hub namespace/service bus

event_hub - the Azure Event Hub name

sas_policy - the Event Hub consumer SAS policy

The following optional arguments can be specified if needed:

-r, --realm - Splunk realm associated with stored SAS policy key

-a, --app - App context for Splunk service connection namespace

-o, --owner - User context for Splunk service connection namespace

-s, --storage - Storage backend for partition offsets. Currently only the file backend is implemented

-g, --consumer_group - Event Hub consumer group. Defaults to $Default

-p, --prefetch - Number of messages to fetch from each partition. Default is 100

-t, --timeout - Read timeout value for consumer socket in seconds. Default is 10

-l, --log_backup_count - Number of log backups to keep. Rotates hourly. Default is 23

Usage: wrapper.py [-h] [-r REALM] [-a APP] [-s STORAGE] …. namespace event_hub sas_policy

Restart Splunk as Splunk user

These arguments are passed onto the script if the Splunk GUI and Splunk CLI section have been configured

properly.

Example of Scripted Inputs:

[script://$SPLUNK_HOME/etc/apps/Azure_EventHub_TA_Splunk/bin/scripts/wrapper.py <eventhubnamespace> insights-operational-logs action] disabled = false interval = 60.0 passAuth = admin

4 - Configure Azure EventHub TA Splunk on Heavy Forwarder The last steps in onboarding data to Splunk is using the command line, to 1) update the scripted inputs to start

using the wrapper.py that is packaged with the Event Hub Connector and 2) create a monitor inputs for

onboarding the data into Splunk

To create a scripted input, the preferred method is to do it via the CLI. Once it is setup, you can enable/disable via

Splunk GUI.

Splunk Scripted Inputs – Pull Data from Event Hub Navigate to the event hub connector home directory. Locate the wrapper.py script and copy it to:

- $SPLUNKHOME/etc/apps/Azure_EventHub_TA_Splunk/bin

- Change ownership to Splunk by using chown splunk:splunk <path of script>

The next step is to create an inputs.conf in Azure_EventHub_TA_Splunk/local.


pg. 16

1) Create a script input for wrapper.py, and set passAuth to an administrative user (or one with access to

passwords.conf). Pass the service bus (namespace), event hub name, and SAS policy name to the script as

arguments. An example of creating a scripted input is as follows:

[script://$SPLUNK_HOME/bin/scripts/wrapper.py <namespace> <eventhub> activity]

disabled = 0

interval = 60.0

passAuth = admin #You must include this parameter for the scripted inputs to work

E.g

[script://$SPLUNK_HOME/bin/scripts/wrapper.py Solsys insights-operational-logs consumer]

disabled = 0

interval = 60.0

passAuth = admin

2) Once configured restart Splunk

3) You logs will now be written to the location defined by the “EVENTHUB_CONSUMER_LOG_DIR”

environment variable. By default, it will go to /tmp directory unless you have specified the variable to go

to a different directory. To see the directory, just use ‘echo $EVENTHUB_CONSUMER_LOG_DIR’

4) Once Splunk has restart navigate to the ‘$EVENTHUB_CONSUMER_LOG_DIR’ directory to verify if the logs

have started to pull from the eventhub.

5) You can always go to settings-> Data/Data inputs and enable/disable the inputs if needed

If your are experiencing any issues pulling data form the Hub, please navigate to the Troubleshooting EventHub

Connector Script Errors section.

Note: The default inputs for the Operational Insights Logs has been added in the TA. The data from this eventhub

is expected to me in JSON format. For data in other formations, new inputs and sourcetypes may need to be

created.

Splunk Monitoring Inputs – Onboard Data in Splunk The Azure_EventHub_TA_Splunk comes with examples of sample monitoring inputs for the Splunk admin to view

and create. The monitoring inputs will allow Splunk to read the log files and send them to your Splunk indexer and

then eventually let you start search your data.

Once you have verified that data is now being pulled from the eventhub, before creating a monitoring input.

Here is what a sample of your inputs.conf should look like in Azure_EventHub_TA_Splunk/local/inputs.conf.

Please adjust the configuration according to your environment.

#[monitor://$EVENTHUB_CONSUMER_LOG_DIR/<namespace>_<eventhub>*.log]

#disabled = true

#index = eventhub

#sourcetype = eventhub:operational:json

#whitelist = solsys_insights-operational-logs_[\d]+\.log

#blacklist = \d+$

#ignoreOlderThan = 1d

#[monitor:///tmp/solsys_insights-operational-logs*.log]

#disabled = true

#index = eventhub

#sourcetype = eventhub:operational:json


pg. 17

#whitelist = solsys_insights-operational-logs_[\d]+\.log

#blacklist = \d+$

#ignoreOlderThan = 1d

Selecting your Sourcetype The Azure_EventHub_TA_Splunk comes with 3 predefined sourcetypes:

- eventhub:operational:json

- eventhub:json

- eventhub:log

Eventhub:operational:json Sourcetype

This sourcetype is exclusive for the operational-insights-logs eventhub from Azure. Please only onboard

data from the Operational-insights-log eventhub with this sourcetype. This will allow you to take

advantage of the pre-build dashboards in the ‘Azure Event Hub Insights App for Connector’ Application.

Eventhub:json Sourcetype

This is a generic sourcetype for any other JSON type data that is onboarded via the Connector. This

sourcetype can be used as a template to create newer sourcetypes. Create a new sourcetype with the following naming convention: Eventhub:<data_type> OR Eventhub:<data_type>:json

When using this ST as a base, adjust the TIME_FORMAT, TIME_PREFIX and TZ(TIMEZONE) according to the data.

This User Guide does not cover the steps to creating sourcetypes via GUI. That is covered in Splunk training as a Splunk Admin.

Eventhub:json Log

This is a generic sourcetype for any other syslog RFID 3614 type data that is onboarded via the

Connector. This sourcetype can be used as a template to create newer sourcetypes following the naming

convention. Create a new sourcetype with the following naming convention: Eventhub:<data_type>

Eventhub:<data_type>:log

When using this ST as a base, adjust the TIME_FORMAT, TIME_PREFIX and TZ(TIMEZONE) according to

the data.

This User Guide does not cover the steps to creating sourcetypes via GUI. That is covered in Splunk

training as a Splunk Admin.


pg. 18

Best Practices Some of the best practices for onboarding data include:

- Saving passwords.conf from the Rest Storage/Manager app in context of the

Azure_EventHub_TA_Splunk application

- Create a new index (e.g eventhub) to onboard data to. Adjust the `azure_eventhub_index` from

the ‘Azure Event Hub Insights App for Connector’ app to reflect the index that the data is

forwarded to

- Ensure that data that is not from the operational-insights-logs eventhub is onboarded with the

sourcetype eventhub:<data_type>. This will make sure that we can use the analytics

- Ensure that all scripted inputs and monitoring inputs are saved in the correct/same app context,

Azure_EventHub_TA_Splunk.

- Since there is vast amount and types of data that can be sent to event hub, as a Splunk Admin, it

is imperative that you identify different eventtypes, create them and follow CIM compliance

Troubleshooting EventHub Connector Script Errors

Once you complete the “Splunk Scripted Inputs – Pull Data from EventHub” and are unable to pull logs

from the eventhub, this section will help you.

If the script is invoked properly, the errors should be logged to index=_internal sourcetype=splunkd. In

this index/sourcetype, search for “wrapper.py”. This will show you all errors related to the EventHub

Connector Application.

Alternatively, you can download our ‘App for Event Hub Insights’ application, published on Splunkbase.

The app contains a dashboard on displaying all errors related to the wrapper.py scripts and helps your

view the volume of the data and the latest errors. The app can be download from this link:




pg. 19

Here is a snippet of what the errors dashboard looks like:


pg. 20

Minimum Viable Product Python 3.6+ script that

1. Generates SAS tokens for each event hub 2. Uses Event Processor Hosts to set up tasks to pull down Event Hub data feeds from each partition 3. Stores Event Hub feed checkpoint data in an external data blob 4. Ingests the raw data feeds into Splunk

On the Azure side

1. Custom consumer group for each Event Hub 2. One or more policies/SAS keys to use for SAS token generation

https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-features#event-consumers


pg. 21

Release notes for the Splunk Add-on for Azure Event Hubs

Compatibility Version 1.0.0 of the Splunk Add-on is compatible with the following software, CIM versions, and platforms.

Splunk platform versions 6.2 or later

CIM 4.2 or later

Platforms Unix and Linux

Vendor Products Microsoft Azure EventHub

New features Version 1.0.0 of the Splunk Add-on has the following new features.

• Supports requirement a third-party tool being able authenticate against individual event hub instead of namespace.

• Event Hubs consumers need to authenticate using SAS Token which gets generated using SAS key.

• Supports scaling through use of Event Hub Processor Host module

• Installation wrapper abstraction

Fixed issues Version 1.0.0 of the Splunk Add-on fixes the following issues.

Known issues Version 1.0.0 of the Splunk Add-on

Third-party software attributions Version 1.0.0 of the Splunk Add-on incorporates the following third-party software or libraries.

• The Azure Event Hub SDK requires Python 3 for both the Event Hub Processor Host module and the async

send/receive functionality.


pg. 22

References Using Shared Access Signature https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-

part-1

Service Bus Access with SAS https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-sas

Microsoft Azure SDK for Event Hubs

SDK Repo: https://github.com/Azure/azure-event-hubs-python

https://docs.microsoft.com/en-us/python/api/azure-

eventhub/azure.eventprocessorhost.eph.eventprocessorhost?view=azure-python

Splunk Wrapper Splunk Answers Wrapper for Env Python

https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1

https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-sas

https://github.com/Azure/azure-event-hubs-python

https://docs.microsoft.com/en-us/python/api/azure-eventhub/azure.eventprocessorhost.eph.eventprocessorhost?view=azure-python

https://docs.microsoft.com/en-us/python/api/azure-eventhub/azure.eventprocessorhost.eph.eventprocessorhost?view=azure-python

https://answers.splunk.com/answers/8/can-i-add-python-modules-to-the-splunk-environment.html


pg. 23

Appendix A Find Executables and Creating SymLink

In red hat when we install python3 or any other software from our repository, sometimes the

executable may not be immediately accessible once your install the software. To find out if you can run

python3 after using yum command, run ‘python3’ in your CLI.

If you get an error “python3 command not found” proceed to the steps below:

There are two ways to try and find the executable.

1) Use whereis command:

• Instructions on using this command can be found on this link:

• https://kb.iu.edu/d/acec

• If the result of the command is path to the executable proceed to create a

symlink

2) Use find command:

• This is an inefficient way to search for the executable but a proven one.

• Execute this command as root to search all directories:

• Find <directory> -name ‘text-to-search’ eg. Find / -name ’python3’

• The result of the find command should show your the directory of where the

python3 executable is located

Create a SymLink

Once we create the link below shows how you can create symbolic link in linux.

https://kb.iu.edu/d/abbe

ln -s source_file myfile

In your instance once you locate the executable, you would need to create the symbolic link in your

/usr/local/bin directory. The command may look like this

$sudo ln -s /opt/rh/python36/lib/bin/python3 /usr/local/bin/python3

When you navigate to the /usr/local/bin directory and run ‘ls’ you should see something similar to the

screenshot below:

Now if you run python3 in your CLI, you should be taken to the python3 interface. If not please check if

you have the correct file/executable for python3 AND if your PATH is updated to include the

/usr/local/bin directory.

https://kb.iu.edu/d/acec

https://kb.iu.edu/d/abbe


pg. 24

Appendix B

Updating $PATH Path is an environment variable that contains all your environments executable directories. The below

link can be used to update your $PATH in linux. The steps may vary slightly based on your flavour of

linux.

There are multiple ways of updating the PATH. If the result of the update will let you run executables in

the /usr/local/bin directory the method does not matter.

Here is an example of how to update your PATH.

https://www.cyberciti.biz/faq/unix-linux-adding-path/

I have summarized some of the steps here.

1) To see what directories are defined under PATH, run echo $PATH. The result may be something

like this:

2) To modify the path navigate to your home directory ‘cd $home’

3) Modify your .bash_profile.

4) Update PATH by appending it with ‘:/usr/local/bin’

a. E.g PATH=$PATH:$HOME/bin:/usr/local/bin

5) Refresh your .bash_profile

a. Source ~/.bash_profile

6) Run echo $PATH to check if the new directory has been added to your PATH.

You should now be able to run python3 executable successfully!

Appendix C.1

Installing CMAKE and C++ Complier This step may be required if you are experiencing CMAKE fatal errors during the installation.

The yum libraries that are required for this are:

- gcc-c++

- kdebase-workspace-devel

- dbusmenu-qt-devel

You can install these in red hat by using the ‘yum install commmand’ as root.

yum install gcc-c++ kdebase-workspace-devel dbusmenu-qt-devel

https://www.cyberciti.biz/faq/unix-linux-adding-path/


pg. 25

These libraries may require a cffi python module as well when installing the Azure SDK. That can be

installed using pip/pip3 as follows:

pip install cffi

Note: If you installed python3 dev tools, it is highly likely you have pip/pip3 available. You just have to

find the executable and create a symlink as shown in Appendix A

Appendix C.2

Troubleshooting Steps For Connecter

Some general troubleshooting steps to ensure that things are working as expected:

1) Add splunk user to wheel group on your linux server:

• This is important so that Splunk can utilize the various SDKs

• usermod -aG wheel splunk

2) Verify that ```which eventhubconsumer``` can locate the eventhub executable from the

Solsys Azure Eventhub Consumer Connector as the Splunk user

• If not, locate the eventhubconsumer in the

<eventhubconnectordirecotry>/.local/bin directory and create a symmlink for it

in the /usr/bin or /usr/local/bin directory.

• Once the symmlink is created run ‘which eventhubconsumer’ again to make

sure that Splunk can detect it

3) If step 2 does not work, make sure that the $PATH environmental variable is updated

• Run echo $PATH as the Splunk user to check if the executable path contains

/usr/bin or /usr/local/bin

• If the $PATH is not updated, you need to modify the users .bash_profile or

.bashrc with the updated path.

• An example of what .bash_profile may look like is shown below:

• Once updated run ‘source ~/.bash-profile’ or ‘source ~/.bashrc`

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin:/usr/local/bin

export PATH

export EVENTHUB_CONSUMER_OFFSET_DIR=/var/log/splunk/offset

export EVENTHUB_CONSUMER_LOG_DIR=/var/log/splunk/log_dir

4) Ensure that Splunk can write to the EVENTHUB_CONSUMER_LOG_DIR


pg. 26

Appendix D: The Azure Monitor Splunk Add-on on Splunkbase This add-on was developed by Microsoft with Splunk. Azure Monitor Add-on For Splunk (https://splunkbase.splunk.com/app/3534/) to pulls data from Event Hubs. The Azure Monitor Add-on was developed by Microsoft with Splunk. High-lights include

• Event Hub Namespaces contain individual Event Hubs.

• An access policy is necessary to access the data in the Event Hubs.

• Access policies can be defined on the namespace (which the hubs will inherit), or on the individual hub.

• The access policy used to connect to the hubs is stored in Azure in something called a key vault.

• The access policy is stored as a name/value pair in the key vault

• The Azure Monitor Add-on connects to the vault in order to get the access policy (this puts the security in the hands of the Azure admin).

• The Azure Monitor Add-on uses the same policy to access each individual hub in the namespace. Azure Monitor Add-on Constraints Security Design

1. Authentication Coarseness: The existing plugin seems to be only supporting authentication against namespace instead of individual event hubs contained in the namespace. This is going to be a big issue, as latest suggested security design requires a third-party tool to be able authenticate against individual event hub instead of namespace. Are there any suggested workarounds?

2. Authentication Method: The current plugin only seems to be supporting authentication with SAS key (primary or secondary) but that doesn't work for many financial institutions as any tools that require access to Event Hubs need to authenticate using SAS Token which gets generated using SAS key.



pg. 27

Appendix E: Reference Critical Design Elements & Options

1. Events can be pushed from a hub to Splunk via an Azure Function. This negates the need for the Azure Monitor Add-on altogether. However, this method does not support indexer acknowledgement, so proceed with caution.

2. The add-on loops though a set of pre-defined hubs and uses the same access policy for each. a. The code can be modified to retrieve a separate access policy from the key vault for each

individual hub. I can think of a couple of ways to do this. 3. Only use the event hub namespace for this add-on (meaning other sources do not write to the

namespace) and see if a security exception can be made

4. Leverage SAS key and SAS policies to create SAS Token for authentication

azure event hub connector user guide - solsys · 2019-07-31 · 3- install azure event hub...

Documents