1040 bennacer complete

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Samir Bennacer & Nathan Tisdale ArcSight Advanced Support Engineers September 19, 2012

Managing an Arcsight Express 3.0 appliance


Agenda

Overview Application health

Events in Events out

Managing event archives Hardware health

Streamlining the RMA process


Managing ArcSight Express 3.0: Overview of appliance

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4

ESM 101: Concepts for ESM + CORR-Engine provides a full overview of event flow.

Data to information

Hierarchy

Scale with additional Appliances to accomodatee volume and reporting needs

Report

Dashboards Traditional

reports

Use case

Customizing rules, data monitors, lists, to fit your business needs

ArcSight Solution Packages

Network model

Details regarding networks, assets, zones etc.

Connectors

Smart Connectors collect & normalize events form environment


Connecting to Express 3.0

ArcSight Console Interactive Web Base Monitoring Traditional Console also available Management Console Manage Users accounts, storage, connectors, notifications and license HP Integrated Lights-Out (iLO) Secure remote management regardless of server status or location. See KM1272897 for details regarding setup of iLO.


Event Storage is for daily events that are younger than the retention period. Archives are used to preserve events offline, beyond the retention period.

ArcSight Express 3.0 Management Console

Storage management


You can add and delete users and groups, and perform other user management functions


Administration


Dashboards in the Management Console appear as layouts of dashboard data using a browser-based runtime environment. You can see all the dashboards that appear in the ESM Console and you can rearrange the layouts and save them


Dashboards


AE 3.0 Advanced Administration

https://ipadress:8443/arcsight/web/manage.jsp


ArcSight Express 3.0 Directory Layout

/opt partition, a separate partition, 64bit XFS, total space 1.5TB All ArcSight software & data under single directory: /opt/arcsight This directory is owned by user arcsight. All arcsight operations should be run as arcsight, not root

ESM manager: /opt/arcsight/manager ESM web: /opt/arcsight/web Appliance process management: /opt/arcsight/services Logger: /opt/arcsight/logger MySQL: /opt/arcsight/logger/data/mysql Event archive directory: /opt/arcsight/logger/data/archives 1 directory per day

Event storage directory: /opt/arcsight/logger/data/logger 1GB per file


ArcSight Express 3.0 Storage Allocation

Total storage: 1.6TB Root partition: 100GB Event Storage: 919GB None event storage(InnoDB): 200GB Event archival storage: 200GB The rest: ESM binary/log files/Posgres DB/MySQL temporary sort area


ArcSight Express 3.0 Storage Engines

The CORR-Engine relies on MySQLs pluggable storage engine architecture Allows for different types of data handling ArcSights high performance event storage and retrieval InnoDB Built-in transactional support, allowing updates and deletes Multiversion concurrency control (same as Oracle) Used for ESM resources(rules, channels, ) & trend data, active / session list data, annotations MySQL seamlessly handles the joins (e.g.: events and cases, actors) Patent-pending technology superstore (single database with Row and Column store)


CORR-Engine

CORR-Engine: Overview

Logger Server

Logger Storage Engine

Event Store

ESM Manager

Events

Events

Resources and data

CORR-

Engine

Comm Layer

ArcSight Event Store

MySQL InnoDB Storage Engine


ArcSight Express 3.0 Events

All event fields are indexed No penalty for accessing any event fields Compare with previous generation having only 16 indexed arc_event fields

Remove side tables completely No more need to manage side table cache sizes No more descriptor side table flooding issues Fold arc_event_ [agent, category, device, label, geo_location, correlation] into arc_event Better query performance due to elimination of event join with side tables


ArcSight Express 3.0 DB Management

No partition compressor needed No partition table needed System built-in optimizer No database statistic job No query tuner needed No user level query hint required(table/index/different join type etc)

No event time correction(clamping) required


Managing ArcSight Express 3.0 Application health


Troubleshooting feature behavioral issues

Typical Steps Identify Event Flow System/Services Offline Error Unexpected Behavior Wish it did

Collect Data Model Version Logs

Engage support Support Center

Process remains same ESM troubleshooting of issues with same nature


Command line application check

All services are running web service is available manager service is available execprocsvc service is available logger service is available mysql service is available postgresql service is available

Manager is not running web service is mixed_statuses manager service is unavailable execprocsvc service is unavailable logger service is unavailable mysql service is available postgresql service is unavailable

/etc/init.d/arcsight_services status | start | stop


Logs & more

Need to investigate application behavior?

Writing events (caching?)

Manager logs

Thread dumps

DB sessions

Manage.Jsp

Database logs

Reading events (channels/reports)

Manager logs

Database logs

Behavior (startup / content)

Manager logs

System tables

Collect and upload logs to Service Center


Collecting application logs

SENDLOGS Send Logs wizard remembers most of the choices you make when you run it for the first time Local logs only

Time Range Including other components

Connectors CORR-Engine Diagnostics (runsql, session-waits, threaddumps) Time Range

Sanitize Incident number (for naming file)


Collecting database logs

ARCDT This diagnostic tool can be used as part of sendlogs or run separately as needed. runsql Create sql file [ e.g. select count(*) from arc_resource where resource_type=1; ] /opt/arcsight/manager/bin/arcsight arcdt runsql f usercount.sql

session-waits /opt/arcsight/manager/bin/arcsight arcdt session-waits

thread-dumps /opt/arcsight/manager/bin/arcsight arcdt thread-dumps


Summary of Express 3.0 changes

Single script to Manage all ArcSight services CORR-Engine improves read and write performance Compression of events when persisted Less need to use Trends to boost reporting performance Elimination of descriptor Side Tables boosts performance Web Management Interface Collect logs with sendlogs Archival without Partition Management


Managing ArcSight Express 3.0 Managing event archives


Event retention

Max event storage size: 919GB MRT(manager-receipt-time) based data retention(not end time) Oldest events will be overwrite first(FIFO) Events pruning based on the age of events Email notifications can be configured to alert people During First Boot Wizard Via management console (web UI)


Configuring event retention


Storage layout

Event storage layout

1-Jul

2-Jul

3-Jul

4-Jul

5-Jul

6-Jul


The events portion of the CORR-Engine storage management system consists of two major parts: The active Retention period Defined by max age or space. This defines the on-line events. Archives Data files containing events of one day, which have been copied to the archive location, with two additional files containing metadata related to these data files.


Event flow


Archiving process

A subdirectory with the format YYYYMMDD is created for each day to store the archive. If the directory is already present, then it is removed first.

Two metadata files, an XML and CSV, for the archival information are created These files are compressed and copied to the archiving location. Finally, the data files containing the events are copied to the archive location.

Note : The archiving is continual and is copying the events files offline as opposed to what used to be in ESM where the same partition was taken offline. Does not affect events in online retention


Archiving modes

There are two modes of archiving: Scheduled - Runs on a daily basis, archiving the events from the day before. Manual (user-driven)

We recommend to use scheduled mode and use the manual mode for retrying an unsuccessful scheduled one.


Where the archives reside

Changing the size and the location of the archive folder is is not supported in AE 3.0 If there is no more space the archiving will fail. We can monitor the space used in the management-ui Mounting external drive to the archive directory is not supported.


Time to start the archive operation for the current day's events as well as any days manually marked for "retry":

Configuring archive schedule


Manual mode for retrying an unsuccessful scheduled one.

Manual archive (user-driven)


Archive jobs


...Still in active storage

Each archive will represent one day's worth of events The number of archives in this list will be the number of days that fit in the configured retention policy

constrained by both a time-dimension and a space-dimension e.g. a retention policy of 30 days will have up to 30 items in the list .. less if there is not enough space.

For each item(archive) in the list there will be identifying information: Date, Archive ID The different states that these archives can be in are Pending, In-progress, Archived, Not-Archived.


Archives


2. No longer in active storage

Initially will be in the Deactivated state thus events are not accessible. Can be in one of the following states Deactivated, Activating, and Activated


Auditing and logs

Device event class ID Audit event description

archive:100 Archive created

archive:101 Archive deleted

archive:102 Event archive settings updated

archive:103 Event archive disk space used

archive:110 Archive activated

archive:111 Archive activation cancelled

archive:112 Archive activation failed

archive:120 Archive operation succeeded

archive:121 Archive operation cancelled

archive:122 Archive operation failed

archive:130 Archive deactivated

archive:131 Archive deactivation cancelled

archive:132 Archive deactivation failed

archive:140 Archive scheduled

archive:141 Archive schedule cancelled

archive:142 Archive schedule failed


We can see the Archive auditing in the Server.std.log and server.log

{fileType=Archive, cat=/Monitor/Archive/Create, cs2=20110913, severity=1, msg=Created archive 20110913, fileId=0504403158265495552, deviceEventClassId=archive:100, start=1315952153371, name=Archive created, rt=1315952153371, fname=20110913, cs2Label=Archive Name, end=1315952153371, fpath=/opt/arcsight/logger/data/archives/20110913} INFO | jvm 1 | 2011/09/13 15:22:35 | {fileType=Archive, cat=/Monitor/Archive/Configuration/Scheduling/Success, cs2=20110913, severity=1, msg=Successful scheduling of archive 20110913, fileId=0504403158265495552, deviceEventClassId=archive:140, start=1315952153648, name=Archive scheduled, rt=1315952153648, fname=20110913, cs2Label=Archive Name, end=1315952153648, fpath=/opt/arcsight/logger/data/archives/20110913


Look for Service:EventArchiveManager in the server.status.log

Service:EventArchiveManager

[2011-10-03 03:55:22,456] Service:EventArchiveManager

[2011-10-03 03:55:22,456] ObjectName:Arcsight:service=EventArchiveManager

[2011-10-03 03:55:22,539] ArchiveEnabled="true"

[2011-10-03 03:55:22,623] ArchiveSchedule4Display="Wed Dec 31 01:00:00 PST 1969"

[2011-10-03 03:55:22,623] ConfiguredDaysInRetentionPolicy="0"

[2011-10-03 03:55:22,666] DiskspaceFree="210453397504"

[2011-10-03 03:55:22,708] LocationOnDisk="/opt/arcsight/logger/data/archives"

[2011-10-03 03:55:23,559] OfflineArchives="[

archiveDate=[Fri Sep 30 00:00:00 PDT 2011 --> 1317366000000millis]

archiveID=[0504403158265495564]

STATE=[ACTIVE] ; online=[false] ; eventCount=[0] ; diskspaceConsumed=[0]

startDateOfEvents=[Fri Sep 30 00:00:00 PDT 2011 --> 1317366000000millis]

endDateOfEvents=[Fri Sep 30 23:59:59 PDT 2011 --> 1317452399999millis]

AdditionalInfo=[null]

dateActivated=[Sun Oct 02 08:17:30 PDT 2011 --> 1317568650860millis]

dateDeactivated=[Sun Oct 02 03:35:24 PDT 2011 --> 1317551724859millis]

dateArchiveStarted=[Wed Jan 01 00:00:00 PST 9500 --> -361932940800000millis]

dateArchiveCompleted=[Wed Jan 01 00:00:00 PST 9500 --> -361932940800000millis]

scheduledStartDateOfArchiveOperation=[Sun Oct 02 08:17:30 PDT 2011 --> 1317568650833millis], ]


How to restore archive backups to a new system after a system failure?

Always Stop logger service /sbin/service arcsight_services stop logger To test how many archives will be restored and see if any are unreadable. Run /opt/arcsight/logger/current/arcsight/logger/bin/arcsight restorearchives -t To clear any existing events from the system. and then register all the backup archives you placed in

/opt/arcsight/logger/data/archives Run /opt/arcsight/logger/current/arcsight/logger/bin/arcsight restorearchives C To register all the backup archives you placed in /opt/arcsight/logger/data/archives. Run /opt/arcsight/logger/current/arcsight/logger/bin/arcsight restorearchives -Start Logger Service /sbin/service arcsight_services start logger


Managing ArcSight Express 3.0 Hardware health


Appliance fails to boot

System or OS level failure Following these steps and contact support for analysis. Connect to the appliance using iLO, KVM switch or using keyboard/monitor physically connected to the

back Get screen shot or photo showing the boot-up failure

e.g. errors, warnings, failures to start services, etc.) Errors should indicate if the problem is software or hardware

In case of File System inconsistency the system prompts to run fsck (file system check) to fix it. Refer to KM1272371 for more details

Check /var/log/messages and dmesg in single user mode for errors Retrieve logs


Diagnosing tools

Commands to validate hardware failure: HP Appliances (RAID Controller and disks) hpacucli ctrl all diag file=hp-raid-diag-output.zip ris=off xml=off zip=on hpacucli ctrl all show config detail hpacucli help HP iLO webUI

ipmitool sel ipmitool sensor dmesg cat /var/log/messages


Summary of resources for diagnosing hardware

Memory testing for errors with memtest86+ as per KM1271513 Disk errors on console, in dmesg, /var/log/messages, chkdsk, iDRAC/iLO RAID errors using commands in previous slide CPU errors on console, in dmesg, /var/log/messages, ipmitool M/B errors on console, in dmesg, /var/log/messages, ipmitool PSU errors ipmitool LCD front panel often displays error messages from SEL


General system performance issues

Slow UI response log in via ssh (putty under Windows) as per KM1271560 check duplex mode and speed of the network interfaces using ethtool verify that the Appliances own name is in the /etc/hosts file. diagnose the processes consuming most CPU, memory and disk I/O with commands such as top, ps and

vmstat. check disk space usage using commands such as df and du check for errors in /var/log/messages and using the dmesg command


Types of RMA

Four key scenarios of RMA: Single Failed Hard Drive replace drive Multiple Failed Hard Drives replace entire appliance Failed PSU (one or more) replace PSU Internal hardware failure (CPU, RAM, M/B, RAID controller, etc) replace entire appliance


Filing an RMA

To avoid delays, be sure to provide all information below: Contact Full Name Company Name Address (no PO boxes) City, State, Postal Code Country (if outside of US, must include) Contact Phone Number: VAT/Tax ID# (if outside of US, must include) Serial Number of Defective Appliance: Model: ArcSight Version: Technical Reason:

Work with ArcSight Support to review all data and file RMA with the above information.


After the event Visit these demos

Find out more

Attend these sessions

1045, The continuing evolution of the HP Arcsight CORR-Engine, Tuesday 10:00 a.m.-10:50 p.m.

1034, From HP Arcsight ESM to express migration , Tuesday, 4:00 p.m. 5:30 p.m.

Solution building by example, TT206

SIEM now what?, TT211

One size doesnt fit all: customized training, TT205

Contact your sales rep

Visit the Protect 724 website https://protect724.arcsight.com

Download the Concepts whitepaper at: https://protect724.arcsight.com/docs/DOC-2174

Your feedback is important to us. Please take a few minutes to complete the session survey.


Thank you

Managing an ArcsightExpress 3.0 applianceAgendaManaging ArcSight Express 3.0:Overview of applianceData to informationConnecting to Express 3.0ArcSight Express 3.0 Management ConsoleArcSight Express 3.0 Management ConsoleArcSight Express 3.0 Management ConsoleAE 3.0 Advanced AdministrationArcSight Express 3.0 Process ManagementArcSight Express 3.0 Directory LayoutArcSight Express 3.0 Storage AllocationArcSight Express 3.0 Storage EnginesCORR-Engine: OverviewArcSight Express 3.0 EventsArcSight Express 3.0 DB ManagementManaging ArcSight Express 3.0Application healthTroubleshooting feature behavioral issues Command line application checkLogs & moreCollecting application logsCollecting database logsSummary of Express 3.0 changesManaging ArcSight Express 3.0 Managing event archivesEvent retentionConfiguring event retentionStorage layoutThe events portion of the CORR-Engine storage management system consists of two major parts: Event flowArchiving processArchiving modesWhere the archives resideConfiguring archive scheduleManual archive (user-driven) Archive jobs...Still in active storageArchives2. No longer in active storageAuditing and logsSlide Number 40 Service:EventArchiveManager How to restore archive backups to a new system after a system failure?Managing ArcSight Express 3.0 Hardware healthAppliance fails to bootDiagnosing toolsSummary of resources for diagnosing hardware General system performance issuesTypes of RMAFiling an RMAFind out moreThank you

1040 bennacer complete

Documents

available management

esm console

rma process copyright

information hierarchy

ilo secure remote management

daily events

event storage

connectors smart connectors