qlean for ibm security qradar siem · due to qradar extensions architecture, an application data...

QLean for IBM Security QRadar SIEM: Admin Guide

© 2018 ScienceSoft™ | from 17

www.scnsoft.com

QLEAN FOR IBM SECURITY

QRADAR SIEM

ADMIN GUIDE

http://www.scnsoft.com/



Table of Contents

Overview ........................................................................................................................................ 3

QLean Installation ......................................................................................................................... 4

Download QLean ........................................................................................................................................... 4

Install QLean ................................................................................................................................................. 4

Request license key ...................................................................................................................................... 5

High Availability license ................................................................................................................................. 5

Install license key .......................................................................................................................................... 5

Execution Parameters .................................................................................................................. 6

Configuration ................................................................................................................................ 7

Manual Execution ......................................................................................................................... 7

Scheduling for Periodical Monitoring .......................................................................................... 8

Managing QLean data ................................................................................................................... 9

Creating a backup ......................................................................................................................................... 9

Restoring a backup ....................................................................................................................................... 9

Email Reporting .......................................................................................................................... 10

Health Markers ............................................................................................................................ 11

Custom Logo ............................................................................................................................... 13

Troubleshooting .......................................................................................................................... 13

Appendix A: Monitoring metrics ................................................................................................ 14



Overview QLean, previously known as Health Check Framework (HCF) for

IBM Security QRadar SIEM, is a tool that allows QRadar users,

administrators and security officers to perform periodical and on-

demand monitoring of a range of statistical, performance and

behavioral parameters of QRadar deployment including All-in-

One and distributed environments.

Supported QRadar versions:

7.2.8

7.3.0

7.3.1

QLean gathers and analyzes more than 50 different parameters (metrics) and produces an Excel report that

can be delivered to one or more recipients via email. This report reflects system health statistics in a tabular

and graphical representation. For complete list of supported metrics please refer to Appendix A: Monitoring

metrics section.

NOTE: QLean is a commercial software and requires a license key to run. Free demo mode with limited

functionality is also available. No license key required for running QLean in this mode.

NOTE: QLean is developed by ScienceSoft Inc. and not supported by IBM.



QLean Installation QLean is distributed as a QRadar extension and can be installed in two ways:

All-in-One (AiO): All QLean components run within QRadar extension container

Distributed: User interface in QRadar extension and back-end on a separate Linux server

NOTE: Please do not confuse QLean installation types with QRadar deployments of the same names.

AiO installation is the preferable one and requires just a few steps to set up.

NOTE: Please back up QLean data periodically as described in Managing QLean data section of this

document.

Distributed installation can be used to minimize performance impact on QRadar Console, simplify

administration and avoid the risk of reporting data loss in case of possible extension container failure.

If you want to create a distributed QLean installation, please contact technical support to get instructions.

In order to prepare All-in-One QLean deployment please follow the steps below.

Download QLean

Go to https://exchange.xforce.ibmcloud.com/hub

Login using your IBMid

Filter by Type: Application

Select QLean extension

Click Download button at the top right corner

Save the extension zip file

Install QLean

Login to QRadar UI

Go to Admin tab

Open Extensions Management

Click Add button

Select Install immediately checkbox, click Browse button, locate the extension file downloaded

from IBM App Exchange and click Add button

Confirm on all steps and wait for installation to finish. This may take a while.

Close Extensions Management window, press Ctrl+F5 to fully reload QRadar UI. New QLean tab

will be added.

Go to QLean tab. A message asking for QLean server deployment will be shown.

Click on the ‘Please click here’ link. QLean configuration page will be opened in a new window.

Wait for installation to finish.

Select All-in-One deployment type, enter your QRadar Console root password and click Connect

button. The page will reload in a few seconds.

NOTE: For more details on using IBM App Exchange and Extension Management tool, please refer to official

IBM documentation:

https://www.ibm.com/support/knowledgecenter/en/SS42VS_7.3.0/com.ibm.apps.doc/c_Qapps_MngExts.htm

l

https://exchange.xforce.ibmcloud.com/hub

https://www.ibm.com/support/knowledgecenter/en/SS42VS_7.3.0/com.ibm.apps.doc/c_Qapps_MngExts.html

https://www.ibm.com/support/knowledgecenter/en/SS42VS_7.3.0/com.ibm.apps.doc/c_Qapps_MngExts.html



Request license key

In order to generate Excel reports, QLean requires a license key. You can request a commercial license by

sending request to [email protected].

The following information required for the license key to be created:

Company name

Contact person

Contact person position

Contact email

Contact phone

QRadar version

UUID code of QRadar Console/AiO

Total licensed EPS capacity

In order to obtain UUID code of your QRadar Console/AiO follow the steps below:

Login as root user to QRadar Console/AiO via SSH

Execute command: dmidecode -s system-uuid

Copy the generated alpha-numeric code to use it in your license request.

You will be sent an email with a license key once you complete your purchase.

High Availability license

If you have a High Availability option for QRadar Console/AiO, additional license key is required to get QLean

reports when the secondary instance is active.

In this case, make sure to include in your request both primary and secondary UUIDs. Note that a standby

instance is only accessible via SSH from an active one.

Install license key

In QRadar UI, navigate to QLean tab and click on the gear button at the top right corner

Open QLean deployment section

Click Upload license file button, locate the ZIP file received from ScienceSoft and click Open

button

mailto:[email protected]



Execution Parameters

QLean

configuration UI Description

Enable debug Debug mode. Enhanced execution log will be created. Commands output will be

written to /opt/scnsoft/hcf/reports/HCF-YYYY-mm-DD-HH-MM-DebugInfo.zip file for

further review. Some information about your deployment and connected Log

Sources will be extracted from configuration database.

Disable Data

Quality analysis

Disable Data Quality analysis in order to reduce QLean execution time.

Disable Advanced

JMX metrics

Disable Advanced Metrics in order to reduce QLean execution time.

Rules performance

interval

Rules performance check duration (in seconds). If not defined, will run with default

600 seconds interval (10 minutes).

Offense Analysis:

exclude inactive

When set, inactive offenses will not present in Offense Analysis selection

Offense Analysis:

include dismissed

When set, Closed and Hidden offenses will be included in Offense Analysis

selection

Time range for Ariel

queries

Time range for Ariel queries (in hours). If not defined, will run with default 24 hours

range.

Define custom

FROM address for

email reports

Define custom FROM address for email reports.

URL for custom

logo

Define custom URL for logo in XLS report.

Top Offenses Count Number of offenses to display for Top Unique Offenses metric. Default is 10.

Top Assets Count Number of assets to display for Top Risky Assets metric. Default is 10.

Backups number Number of backups to display for Last backups metric. Default is 5.

Log Source Types

Count

Number of Log Source Types to display for EPS per Log Source Type metric.

Default is 10.

Log Source Actions

Count

Number of Log Sources to display for Last inactive, disabled, added, deleted,

modified Log sources and Protocol Configuration Errors metrics. Default is 10.

Rules Performance

Count

Number of rules to display for Rules Performance metrics. Default is 10.

Top Reports Number of reports to display for Top Heavy Reports metric. Default is 10.

Sys Notification

Count

Number of entries to display for Last Warnings and Errors from System

Notification metrics. Default is 10.

Autoupdate Errors

Count

Number of entries to display for Last Autoupdate Errors metric. Default is 10.



Configuration In order to override default parameters, open QLean configuration page through the gear button at the top

right corner of QLean tab in QRadar UI.

Set desired values in Execution parameters fields and click Save configuration button. All further on-

demand QLean reports will be generated using this configuration, unless overridden via parameters set in

QLean configuration page.

To revert to out-of-the-box settings, click Reset to default button.

Manual Execution Go to QLean tab in QRadar UI

Click on the gear icon in the top right corner

Go to Execution parameters section

Define execution parameters as described in the previous section

Click Run QLean button

Execution status will be displayed at the top of the window. Once finished, reports list will be updated.



Scheduling for Periodical Monitoring In QRadar UI, go to QLean tab and click on the gear button at the top right corner

Define required execution parameters in Execution parameters section (1)

Click Add to schedule button (2)

Define the schedule using drop-down lists or enter manually in the text field (3)

Click Schedule button (4)

Click Edit QLean tasks button to review and/or change existing crontab entries



Managing QLean data Due to QRadar extensions architecture, an application data may be possibly lost during application update, or

because of the platform’s failures.

Thus, taking periodic backups of your QLean reports and settings is highly recommended.

The following data can be backed up and restored:

XLS Reports

UI reports

License keys

Scheduled jobs

Execution parameters configuration

Health Markers configuration

Execution logs

Debug information

The following data cannot be backed up and restored via QLean Configuration interface, and therefore should

be re-applied manually:

Custom logo

The list of email recipients is configured in QRadar itself and will not be affected by any application changes

or failures.

Creating a backup

Login to QRadar UI

Navigate to QLean tab and click on the gear button at the top right corner


Click on Backup QLean data button

Download the output ZIP file and save it to a safe place

Restoring a backup

Login to QRadar UI



Click on Restore QLean data button

Locate and upload the backup ZIP file



Email Reporting After each run QLean can send reports via email. By default, the report will be sent to the email address

specified under System Settings – Administrative Email Address in QRadar Admin tab. If you want to send

QLean reports via email to other addresses, refer to steps below.

Login to QRadar UI



Click Create report recipients list button. QRadar Reference Set will be created and the button will

change to Report recipients which is intended to manage email addresses via QRadar standard

Reference Set editor.

Update Reference Set content

o Press Add button

o Add one or several email addresses to the list

NOTE: In order to temporary disable email reports without removing the existing recipients, add nomail item

to the Reference Set. Delete this item once you need email reports again.



Health Markers QLean provides user with extended email reports which contain 25 “OK/Failed” Health Markers in order to

indicate important metrics changes in your QRadar deployment. In case of marker fire you’ll receive a warning

with description and some basic recommendations.

Health Markers fire on the following metrics:

Console Disk Usage: if used disk space on the Console/AiO appliance exceeds the given threshold

(95% by default).

Deleted Log Sources: if at least one Log Source was deleted during the last days (3 days by

default).

Modified Log Sources: if at least one Log Source was modified during the last days (3 days by

default).

Autoupdate Errors: if at least one Autoupdate failed during the last days (3 days by default).

Asset Risk Level: if at least one Asset reached Risk level, which exceeds the top-10 average level

on more than given threshold (70% by default).

Offense Types: if at least one Offence type occurs more often (80% by default) than the top-10

average periodicity.

Nightly Backups: if at least this many (0 by default) failures occurred among last 5 backups.

System Notifications: if at least one error/warning was detected in System Notifications journal

during the last days (3 days by default).

Inactive Log Sources: if at least one Log Source became inactive during the last days (3 days by

default).

Disabled Log Sources: if at least one Log Source was disabled during the last days (3 days by

default).

Protocol Errors: if at least one Log Source has protocol configuration errors.

Modified Searches: if at least one Search was modified during the last days (3 days by default).



Data Integrity: if at least one event/flow data file corrupted or integrity check failed.

Rules Execution Time: if at least one correlation rule executes longer than the top-10 average rules

execution time on more than given threshold (70% by default).

Rules Response Time: if at least one correlation rule responses longer than the top-10 average

rules response time on more than given threshold (70% by default).

Reports Execution Time: if at least one report executes longer than the average execution time

among top-10 heaviest reports on more than given threshold (70% by default).

Distributed EPS: if at least one managed host reached EPS utilization more than the given

threshold (95% by default).

Distributed FPI: if at least one managed host reached FPI utilization more than the given threshold

(95% by default).

Managed hosts RAM: if at least one managed host runs below the given amount (10% by default)

of free RAM.

Managed hosts CPU: if at least one managed host has CPU load over the given threshold (95% by

default) in the last 15 minutes.

Managed hosts /store partition: if at least one managed host has used /store partition space over

the given threshold (90% by default).

Managed Hosts Status: if at least one managed host is in state different than Active or Standby

(normal operation of HA appliances).

Generic DSM: if at least one SIM Generic DSM Log Source generates more than given number of

events (50 by default).

Unknown Events: if at least one Log Source generates more than given threshold of unknown

events (90% by default).

Default thresholds can be modified through QLean configuration page by defining required values per

marker:



Custom Logo For branding purposes, QLean allows to use a custom logo picture in report headers.

The following requirements must be met:

Image format: PNG

Color depth: 24 bit

Image size: 296x59 or less

Image resolution: 72 dpi

Report header background color is RGB 22, 54, 92 (HEX #16365C). For better logo readability use transparent

image background or contrast colors.

In order to add a custom logo follow the steps below:

Prepare your logo file according to the requirements above

Login to QRadar UI


Open Reports section

Click Select logo file button and locate your logo file

Click Upload button. A warning message will be shown if some requirements are not met.

Click Delete custom logo button to remove your custom logo when necessary.

NOTE: only one logo file can be stored at one time. Any existing logo file will be overwritten after pressing

Upload button.

Troubleshooting If you have problems with QLean execution or reports generation, run it with Enable debug checkbox selected.

HCF-YYYY-mm-DD-HH-MM-DebugInfo.zip file will be generated and attached to the report email.

Get the container and application logs via Download QLean Logs button under QLean Configuration -

QLean deployment. Save the output qlean_debug_log.zip.

Forward these files and your Excel report to the following address for investigation: [email protected]

mailto:[email protected]



Appendix A: Monitoring metrics The following metrics are monitored with HCF:

CONSOLE

Console IP address

Console UUID

QRadar software version

Version history

QRadar users

DEPLOYMENT: QRADAR HOSTS

Managed hosts

IP address

HA IP address

Hostname

HA host role

Is Console

Uptime (in days)

Average CPU load, %

Total RAM, Mb

Free RAM, %

Total /store space, Gb

Free /store space, %

Appliance type

Disk usage details

DEPLOYMENT: SYSTEM HEALTH

Recent backups

Name

Date

Status

Size, Mb

Integrity of events (recent 24h)

Number of correct files

Number of failed files

Number of corrupted files

Integrity of flows (recent 24h)

Number of correct files

Number of failed files

Number of corrupted files

Last warnings and errors from System

Notification

QRadar host IP address

Date

Description

Last auto-update errors

Date

Package name

Description

ENVIRONMENT: LOG SOURCES

Number of active Log Sources

Number of Log Source groups

Last inactive Log Sources

Log Source name

Date last seen

User performed the last action

Last disabled Log Sources

Log Source name

Date last edited


Protocol configuration errors

Log Source name

Date last seen


Last added Log Sources

Log Source name

Date added


Last modified Log Sources

Log Source name

Date modified


Last deleted Log Sources

Log Source name

Date deleted


Log Sources list

Log Source name

Log Source identifier

Log Source activity status

Last seen event date

Average EPS

Peak EPS

Peak EPS date

Protocol

Log Source type

Log Source extension

Date added

Addition type

Is bulk added

Status

Assigned Log Source groups

Description

Date modified

Collector



Log Source ID

ENVIRONMENT: EPS/FPM STATISTICS

EPS/FPM per Managed host

Managed host name

Managed host IP address

Average EPS for the last period (24h by default)

Average FPM for the last period (24h by default)

EPS license limit

FPM license limit

EPS utilization, %

FPM utilization, %

Average EPS from qradar.log

Average FPM from qradar.log

EPS per Log Source type

Log Source type

Total EPS consumption, %

Average EPS

Peak EPS

ENVIRONMENT: RAW INBOUND EPS PER

MANAGED HOST

Event Collector hostname and IP address

Destination Processor IP address

Average EPS

Peak EPS

EPS license limit

ENVIRONMENT: RAW INBOUND FPM PER

MANAGED HOST

Flow Collector hostname and IP address

Destination Processor IP address

Average FPM

FPM license limit

ENVIRONMENT: DATA QUALITY BY

DEVICE TYPE

Log Source type

List of missing categories

Category coverage, % (percentage of seen event categories against the total number of supported categories)

Category name

Average event severity

Number of event types seen

Number of event types supported

Event count

Event coverage, % (percentage of seen event types against the number of supported event types within particular category)

ENVIRONMENT: DATA QUALITY BY LOG

SOURCE

Log Source type

Log Source name

Average event severity

Number of event types seen

Number of event types supported by DSM

Event count

Coverage, % (percentage of seen event types against the number of event types supported by particular DSM)

ENVIRONMENT: UNKNOWN EVENTS AND

SOURCES

Unknown events

Log Source IP address

Log Source name

Log Source type

Number of received events

Number of Unknown events

Percentage of Unknown events

SIM Generic Log DSM

Source IP address

Number of events received

ENVIRONMENT: RUNTIME STATISTICS

JMX runtime metrics

Event average payload size

Event average rate

Event average record size

Event records dropped (no connection)

Event records dropped count

Flow average payload size

Flow average rate

Flow average record size

Flow records dropped (queue full)

Flow records dropped count

DSM Information

DSM name

Number of received events

Events parsed, %

Events normalized properly, %

Events unrecognized, %

Events aborted with exception, %

Average event parsing time, ms

ENVIRONMENT: ASSETS

Number of assets

Assets with the highest Risk level and

Vulnerabilities count

CORRELATION: OFFENSES



Top unique offenses

Offense name

Number of events/flows involved

Top offense closing reasons

Closing reason (reason name + 80 characters of a note)

Number of offenses closed by reason

Offense analysis

Rule name

Rule description

Rule tests

IDs of generated offenses

Offense indexes

Number of entries involved (events/flows/common/offenses)

CORRELATION: RULES

Number of enabled rules

Number of disabled rules

Number of Building Blocks

Number of custom rules

Number of modified rules

Rules performance

Fired rules count

Average rule execution time, sec

Average rule action time, sec

Average rule response time, sec

CORRELATION: REPORTS

The most time expensive reports

Report name

Estimated execution time, sec

Last execution time, sec

Last modified searches

Search name

Modification date

User performed last modification

SOC KPI

Incident resolution time

Number of offenses closed in 4h, 12h, 24h, 3d, 7d, >7d

Incident response time

Number of offense actions in 4h, 12h, 24h, 3d, 7d, >7d

Incident closed per user

Number of offenses closed by QRadar user

Incident detected

Number of new offenses over time

Incident severity

Average offense severity level over time

System tuning actions

Number of reference data and CRE rules modification actions over time

FINE TUNING

Untuned default Building Blocks

Building Block name

Creation date

Modification date

Untuned default Network Hierarchy elements

Element name

Element CIDR

Untuned Network Hierarchy correlation rules

Rule name

NH elements addressed in a rule

Custom DSM Unknown events

Log Source name

Number of unknown events

Total number of received events

Unknown events, %

Flow sources

Flow interface

Number of received flows

Unassigned Log Sources

Percentage of Log Sources assigned to Log Source groups

ADVANCED PERFORMANCE

Global views performance

Speed of search

Search type

Search name

Search ID

Regex relative performance

Relative performance gap

Custom property name

Regex string

User

Update time



Appendix B: Release notes

1.1.0

Fixed: scheduled tasks run using proper timezone settings of QRadar Console

Fixed: flow collectors selection for Raw FPM

Improved: results sorting in Data Quality

New: missing event categories list in Data Quality by Device Type

New: drill-down functionality in Data Quality: Unknown&Stored and SIM Generic Log Sources

1.0.0

Initial version

qlean for ibm security qradar siem · due to qradar extensions architecture, an application data...

Documents