the importance of consolidation, correlation, and …...virtualization with the mainstream arrival...

Managing the Virtualized Enterprise The Importance of Consolidation, Correlation, and Detection – Enterprise Security Series

White Paper

White Paper

Managing The Virtualized Enterprise

Abstract The benefits of employing virtualization in the corporate data center are compelling – lower operating

costs, better resource utilization, increased availability of critical infrastructure to name just a few. It is an

apparent “no brainer’ which explains why so many organizations are jumping on the bandwagon. Industry

analysts estimate that between 60 and 80 percent of IT departments are actively working on server

consolidation projects using virtualization. But what are the challenges for operations and security staff

when it comes to management and ensuring the security of the new virtual enterprise? With new

technology, complexity and invariably new management challenges generally follow.

Over the last 18 months, Prism Microsystems, a leading security information and event management

(SIEM) vendor, working closely with a set of early adopter customers and prospects, has been working on

extending the capability of EventTracker to provide deep support for virtualization, enabling our

customers to get the same level of security for the virtualized enterprise as they have for their non-

virtualized enterprise. This White Paper examines the technology and management challenges that result

from virtualization, and how EventTracker addresses them.

The information contained in this document represents the current view of EventTracker on the

issues discussed as of the date of publication. Because EventTracker must respond to changing

market conditions, it should not be interpreted to be a commitment on the part of EventTracker,

and EventTracker cannot guarantee the accuracy of any information presented after the date of

publication.

This document is for informational purposes only. EventTracker MAKES NO WARRANTIES, EXPRESS

OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the

rights under copyright, this paper may be freely distributed without permission from EventTracker,

if its content is unaltered, nothing is added to the content and credit to EventTracker is provided.

EventTracker may have patents, patent applications, trademarks, copyrights, or other intellectual

property rights covering subject matter in this document. Except as expressly provided in any

written license agreement from EventTracker, the furnishing of this document does not give you

any license to these patents, trademarks, copyrights, or other intellectual property.

The example companies, organizations, products, people and events depicted herein are fictitious.

No association with any real company, organization, product, person or event is intended or should

be inferred.

© 2017 EventTracker Security LLC. All rights reserved. The names of actual companies and products

mentioned herein may be the trademarks of their respective owners.

White Paper


New Complexity, New Challenges

The introduction of virtualization has changed the playing field when it comes to managing the security

and operations of the corporate enterprise.

Until virtualization there had always existed a fairly close relationship between the hardware and software

layers of a computing infrastructure. A server machine was typically a “box’, .i.e. a self-contained machine

consisting of a chassis, CPU’s, an operating system (UNIX, Linux or Windows typically) with some

applications installed and some disk spaced mapped. Network equipment were other “boxes’ that

managed the network traffic between servers and desktops. Once provisioned the server and the network

equipment became fairly static and straightforward to manage.

Over the last ten years this relationship at least on the server side has been complicated by the move to

specialized storage devices and rack and blade systems. Despite this growth in complexity, it was still

relatively manageable overall. To provide visibility into the workings of the server you monitored the

Operating System and by doing this you got limited, but adequate, visibility into the underlying hardware

layer, as well as the application layer. The network produced management information that provided

visibility into the information flowing between machines. From a management standpoint you had a set

of trusted users or administrators that were responsible for the machines, a different network team and

in bigger companies, occasionally some storage specialists and a security group. Everyone had distinct and

fairly well defined duties. It was not perfect, but the complexity could be managed.

Virtualization

With the mainstream arrival of virtualization, the close relationship between the physical and the software

layer is now completely severed. Now at best there is a loose coupling of the OS instance with the platform

it runs on and there is an entirely new, virtualized layer that separates the two as well. The close

relationship of OS to physical infrastructure has been replaced by the virtualization layer – the hypervisors

and management tools that manage the setup and deployment of the virtual machines. The host OS still

has control over the application layer, but the hardware is allocated through the VM management layer.

The Hypervisors also support network communication between virtual machines which side-steps the

classic network group that traditionally controlled traffic on the wire. Further complicating the equation

is that with virtual networking, network traffic sometimes never gets onto the wire which renders most

network security tools ineffective.

Systems Management

Many organizations are also deploying systems management applications in the form of Dell OpenManage

or HP Insight Manager to manage large scale server farms. These have become important as enterprises

move to “rack and stack’, where virtual servers are often dependent on shared infrastructure to operate.

With potentially many servers dependant on shared infrastructure it becomes important to monitor the

White Paper


hardware state, as a small hardware failure can have a catastrophic impact on service. These management

applications can help manage at the hardware layer and at the OS or software layer, but typically do not

provide the richness of a specialty virtualization management product and most experts in the field

caution against using such solutions for the virtual layer.

The addition of this new virtualization layer compounds the complexity of management and monitoring.

There are different and sometimes more critical points of failure, and there are entirely new systems and

applications that need to be monitored. Prior to racks and virtualization, if a machine failed it would take

out a couple (at most) of critical applications. Today if a rack fails it might take out 10 physical servers. A

single physical server could be running 8-10 guest Operating Systems, with each of those running critical

applications and services, so even a single physical server machine failure can be catastrophic. In addition,

if the management application for the virtual infrastructure is successfully attacked or hijacked there is

potential for operational carnage. Server sprawl was messy and inefficient to manage, with lots of points

of failure, but there were few points of failure that could literally take an entire company off-line. In the

new Virtualized enterprise there are more, different and even more critical services to monitor.

Organizational Change

For separation of duties and operational efficiency in many organizations that have adopted virtualization

there is now an admin team that is responsible for management of the virtual layer – the provisioning and

creation of virtual machines. But the clear separation of duties that existed pre-virtualization has blurred

– the virtual team might, for instance, have to worry about networks if they are using virtualization for

communication between guest machines.

Imagine the simplest of examples from this new paradigm – prior to virtualization you turned a machine

on and an OS typically booted up. Done. Now you switch on a machine and the virtualization layer takes

over. It then manages the creation of potentially multiple virtual machines running different Operating

Systems with distinct network configurations. Virtual Machines start and stop, they can move dynamically

from physical machine to physical machine. Even the disk space and often the network are mapped in the

virtual world.

This discussion is not to imply that virtualization is inherently insecure in any way, it is simply changing the

way businesses need to operate and think about their security. There are new and different critical

applications and infrastructure that need to be monitored and brand new threats – and consequently the

approaches to monitoring and prevention must adapt.

SIEM in the Virtual Enterprise

Security Information and Event Management solutions have three real purposes in life. First to help

prevent attacks and security breaches from either internal or external bad actors. With virtualization the

attack service changes. Before virtualization you could attack at the hardware layer or hijack a machine

during the boot process. The other option was to attack at the OS/software layer. Now a hacker can attack

White Paper


the VM layer as well. Once in the VM layer, the hacker can reconfigure machines and potentially traverse

into a guest OS. Since VMs can all be running on the same physical machine the hacker can then traverse

from machine to machine in the host without the network traffic ever being visible on the wire.

The second purpose of SIEM solutions is to help companies meet compliance by tracking user and

administrator activity and access. With virtualization there is an entirely new set of power users that are

acting in the enterprise - administrators that manage the virtual layer. They need to be audited as well.

One of the best ways to secure a VM infrastructure is by enforcing strict separation of duties – for example

the persons responsible for the virtual infrastructure (provisioning etc.) and the virtual machine instances

themselves (OS and applications) should not be the same if at all possible, and the network, server and

virtual management teams should have policy-based segregation of duties.

Finally, the third purpose is to ensure smooth continuing operations. Having a consolidated view of all the

events happening in the enterprise increases the overall availability of IT service. In an increasingly

complex infrastructure automating these tasks with a SIEM solution is the only way to detect the small

signs of impending problems in advance.

In order to ensure security and smooth operations, enterprise visibility must be maintained and collection

of logs from all distinct layers must be performed. In the next pages we look at several important

technologies that need to be monitored as they have become important layers of the system

infrastructure in a virtualized enterprise, and offer a hacker new attack vectors. In order to keep this

manageable, we have focused on the “machines’ – the racks, the servers, the storage devices and the

software that controls them. We will look at the types of events generated by Dell OpenManage and both

VMware and Microsoft Hyper-V events. The ability to manage the network, application and OS elements

are an assumption, and are already supported with existing SIEM solutions.

White Paper


Unified Server Management

Unified Server Management offerings or what HP refers to as “Unified Infrastructure Management’ are a

series of management products that are designed to manage the entire IT infrastructure – from the

Chassis to the Network Attached Storage and from the OS level down to the bare-bones hypervisor. These

applications can collect IPMI information that provides rich, low level information on the state of the

hardware, and as they are provided by the server vendors (Dell OpenManage, HP Insight Manager, IBM),

they provide a great deal of information on the state of the SAN devices if a company has standardized on

a single vendor for both storage and systems. These systems also provide a rich set of commands to

configure, patch and operate the hardware, OS and storage of the infrastructure.

With Blades and Racks and shared resource pools of hardware components it is advisable to collect and

monitor logs coming from these applications. In large scale virtualized enterprises these applications are

often used side by side with a Vcenter.

Pre-OS Events

Once, it was safe to assume that when you powered a machine off, it became unreachable. Now with a

combination of UPS, networks and IPMI, even machines that are powered off are still potentially

accessible.

The Intelligent Platform Management Interface (IPMI) standard has existed since 1998 with the majority

of the major chip set vendors such as Intel and AMD, and Server Vendors such as Dell and HP, supporting

the Standard. IPMI runs on the Baseboard Management Controller and allows Administrators to remotely

manage a system before an OS is even booted or the power switched on. This powerful combination of

capabilities enables an IT organization to substantially reduce the cost of server maintenance, however it

also opens a potential path for hackers to get in and cause damage. In IPMI 2.0, for example, a person

remotely accessing the interface is able to discover all the commands available to them and perform

inventory on the underlying platform, as well as change hardware settings on the machine. In addition,

once the OS has been booted, the BMC and IPMI can continue to run if provided a power source enabling

another entry point into the device, outside of the operating system.

With this capability, monitoring access through IPMI is a must. Unfortunately a single standard for IPMI

trap generation does not exist and the platform vendors have integrated the IPMI functionality into the

Server Management Systems. Information can be generated from various sources including the BIOS, OS

Bootstrap Loader, Network Interface Card, System Alert ASIC, System Management Micro-controller,

System Management Software and the Alert Proxy Software. A great deal of useful operational data with

regards to the state of the system hardware, memory and disks becomes available. In addition important

security and audit events are generated for IPMI user-logon failures, system reconfiguration or the turning

off of logging in IPMI.

White Paper


OpenManage Events

Array Disk Events

2106 Smart FPT (predictive failure) exceeded. The disk is likely to fail in the near future.

2107 Smart configuration change. The disk is likely to fail in the near future.

2108 Smart warning. The disk is likely to fail in the near future.

2109 SMART warning temperature. The disk is likely to fail in the near future.

2110 SMART warning degraded. The disk is likely to fail in the near future.

2111 Failure prediction threshold exceeded due to test - No action needed.

2094 Predictive Failure reported. The disk is likely to fail in the near future.

2095 SCSI sense data. A SCSI device experienced an error, but may have recovered.

Automatic System Recovery

1006 Automatic System Recovery (ASR) action was performed. The Operating System was hung.

Battery Sensor Events

1700 Battery sensor has failed.

1701 Battery sensor value unknown.

1702 Battery sensor returned to a normal value.

1703 Battery sensor detected a warning value.

1704 Battery sensor detected a failure value.

1705 Battery sensor detected a non-recoverable value.

2104 Controller battery is reconditioning.

2105 Controller battery recondition is completed.

2169 The controller battery needs to be replaced.

2170 The controller battery charge level is normal.

White Paper


Battery Sensor Events

2171 The controller battery temperature is above normal.

2172 The controller battery temperature is normal.

2174 The controller battery has been removed.

2175 The controller battery has been replaced.

2176 The controller battery Learn cycle has started.

2177 The controller battery Learn cycle has completed.

2178 The controller battery Learn cycle has timed out.

2179 The controller battery Learn cycle has been postponed.

2180 The controller battery learn cycle will start in %1 days.

2181 The controller battery Learn cycle will start in %1 hours.

2215 Battery charge process interrupted

2216 The battery learn mode has changed to auto.

2217 The battery learn mode has changed to warn.

BIOS Update Schedule Events

1002 A system BIOS update has been scheduled for the next reboot.

1003 A previously scheduled system BIOS update has been canceled.

Chassis Intrusion

1250 Chassis intrusion sensor has failed

1251 Chassis intrusion sensor value unknown

1252 Chassis intrusion returned to normal

1253 Chassis intrusion in progress

1254 Chassis intrusion detected

1255 Chassis intrusion sensor detected a non-recoverable value

White Paper


Chassis Management Controller (CMC) Events

2000 CMC generated a test trap

2002 CMC reported a return-to-normal or informational

2003 CMC reported a warning

2004 CMC reported a critical event

2005 CMC reported a non-recoverable event

Cooling Device Events

1100 Fan sensor has failed

1101 Fan sensor value unknown

1102 Fan sensor returned to a normal value

1103 Fan sensor detected a warning value

1104 Fan sensor detected a failure value

1105 Fan sensor detected a non-recoverable value

Current Sensor Events

1200 Current sensor has failed

1201 Current sensor value unknown

1202 Current sensor returned to a normal value

1203 Current sensor detected a warning value

1204 Current sensor detected a failure value

1205 Current sensor detected a non-recoverable value

White Paper


Disk Error

2273 A block on the physical disk has been punctured by the controller

2306 Bad block table is 80% full.

2307 Bad block table is full. Unable to log block

2331 A bad disk block has been reassigned.

2340 The BGI completed with uncorrectable errors.

2349 A bad disk block could not be reassigned during a write operation.

Enclosure Events

2138 Enclosure alarm enabled

2139 Enclosure alarm disabled

2151 Asset tag changed

2152 Asset name changed

2153 Service tag changed

2162 Communication with enclosure regained

2173 Unsupported configuration n detected. The SCSI rate of the enclosure management modules (EMMs) is not the same.

2190 The controller has detected a hot plugged enclosure.

2191 Multiple enclosures are attached to the controller. Unsupported configuration.

Firmware

2120 Enclosure firmware mismatch

2128 BGI cancelled

2131 Firmware version mismatch

2165 The RAID controller firmware and driver validation was not performed. The configuration file cannot be opened.

2166 The RAID controller firmware and driver validation was not performed. The configuration file is out of date or corrupted.

2311 The firmware on the EMMs is not the same version.

White Paper


Hardware Log Sensor

1550 Log monitoring has been disabled

1551 Log status is unknown

1552 Log size is no longer near or at capacity

1553 Log size is near or at capacity

1554 Log size is full

1555 Log sensor has failed

Log Backup Clear

0000 Log was cleared

0001 Log backup created

Memory Device

1403 Memory device status warning. Correction rate exceeded acceptable value.

1404 Memory device status warning. A memory device correction rate exceeded an acceptable value, a memory spare bank was activated, or a multibit ECC error occurred.

Physical Disk

2049 Physical disk removed

2050 Physical disk offline

2051 Physical disk degraded

2052 Physical disk inserted

2060 Copy of data started on physical disk %1 from physical disk %2.

2062 Physical disk initialization started

2065 Physical disk rebuilds started

2074 Physical disk rebuilds cancelled

White Paper


Physical Disk

2075 Copy of data completed on physical disk %2 from physical disk %1

2080 Physical disk initializes failed

2083 Physical disk rebuilds failed

2087 Copy of data resumed from physical disk %2 to physical disk %1

2089 Physical disk initializes completed

2092 Physical disk rebuilds completed

2141 Physical disk dead segments recovered

2146 Bad block replacement error. A portion of a physical disk is damaged.

2147 Bad block sense error. A portion of a physical disk is damaged.

2148 Bad block medium error. A portion of a physical disk is damaged.

2149 Bad block extended sense error. A portion of a physical disk is damaged.

2150 Bad block extended medium error. A portion of a physical disk is damaged.

2158 Physical disk online

2195 Dedicated hot spare assigned. Physical disk %1

2196 Dedicated hot spare unassigned. Physical disk %1

2198 The physical disk is too small to be used for Replace member operation

2211 The physical disk is not supported.

2183 Replace member operation failed on physical disk %1

2184 Replace member operation cancelled on physical disk

2185 Replace member operation stopped for rebuild of hot spare on physical disk

1650 Unknown device plug event type received.

1651 Device added to system

1652 Device removed from system

1653 Device configuration error detected

1500 AC power cord sensor has failed

1501 AC power cord is not being monitored

White Paper


Physical Disk

1502 AC power has been restored

1503 AC power has been lost



1350 Power supply sensor has failed

1351 Power supply sensor value unknown

1352 Power supply returned to normal

1353 Power supply detected a warning

1354 Power supply detected a failure

1355 Power supply sensor detected a non-recoverable value

1600 Processor sensor has failed

1601 Processor sensor value unknown

1602 Processor sensor returned to a normal value

1603 Processor sensor detected a warning value

1604 Processor sensor detected a failure value

1605 Processor sensor detected a non-recoverable value

2048 Device failed

2056 Virtual disk failed

2076 Virtual disk check consistency failed

2077 Virtual disk format failed

2079 Virtual disk initialization failed

2080 Physical disk initializes failed

2081 Virtual disk reconfiguration failed

2082 Virtual disk rebuilds failed

2083 Physical disk rebuilds failed

2094 Predictive disk failure reported.

White Paper


Physical Disk

2101 Temperature dropped below the minimum warning threshold

2102 Temperature exceeded the maximum failure threshold

2103 Temperature dropped below the minimum failure threshold

2106 Smart FPT (predictive failure) exceeded. The disk is likely to fail in the near future.

2107 Smart configuration change. The disk is likely to fail in the near future.

2108 Smart warning. The disk is likely to fail in the near future.

2109 SMART warning temperature. The disk is likely to fail in the near future.

2110 SMART warning degraded. The disk is likely to fail in the near future.

2112 Enclosure was shut down. The physical disk enclosure is either hotter or cooler than the maximum or minimum allowable temperature range.

2123 Redundancy lost

2125 Controller cache preserved for missing or offline virtual disk

2129 Virtual disk BGI failed

2131 Firmware version mismatch

2132 Driver version mismatch

2137 Communication timeout

2146 Bad block replacement error

2148 Bad block medium error

2149 Bad block extended sense error

2150 Bad block extended medium error

2163 Rebuild completed with errors

2165 The RAID controller firmware and driver validation was not performed. The configuration file cannot be opened.


2167 The current kernel version and the non- RAID SCSI driver version are older than the minimum required levels.

2168 The non- RAID SCSI driver version is older than the minimum required level.

White Paper


Physical Disk

2169 The controller battery needs to be replaced.

2182 An invalid SAS configuration has been detected.

2183 Replace member operation failed on physical disk %1. The physical disk being replaced has failed.

2191 Multiple enclosures are attached to the controller. This is an unsupported configuration.

2201 A global hot spare failed.

2250 Redundant Path is broken

2264 A device is missing.

2265 A device is in an unknown state.

2268 Storage Management has lost communication with the controller.

2270 The physical disk clear operation failed.

2272 Patrol Read found an uncorrectable media error.

2282 Hot spare SMART polling failed.

2283 A redundant path is broken.

2289 Multi-bit ECC error.

2292 Communication with the enclosure has been lost.

2293 The EMM has failed.

2295 A device has been removed.

2297 An EMM has been removed.

2299 Bad physical connection

2300 The enclosure is unstable.

2301 The enclosure has a hardware error.

2302 The enclosure is not responding.

2307 Bad block table is full. Unable to log block

2310 A virtual disk is permanently degraded.

White Paper


Physical Disk

2314 The initialization sequence of SAS components failed during system startup. SAS management and monitoring is not possible.

2316 Diagnostic test failed.

2319 Single-bit ECC error. The DIMM is degrading.

2320 Single-bit ECC error. The DIMM is critically degraded.

2321 Single-bit ECC error. The DIMM is critically degraded. There will be no further reporting.

2322 The DC power supply is switched off.

2336 Controller event log: %1. Controller generated event log while Storage Management was not running

2337 The controller is unable to recover cached data from the battery backup unit (BBU).

2340 The BGI completed with uncorrectable errors.

2346 Physical device error occurred.

2347 The rebuild failed due to errors on the source physical disk.

2348 The rebuild failed due to errors on the target physical disk.

2349 A bad disk block could not be reassigned during a write operation.

2350 There was an unrecoverable disk media error during the rebuild.

2356 SAS SMP communications error.

2357 SAS expander error.

2373 Attempted import of unsupported Virtual Disk type

Redundancy Unit

1300 Redundancy sensor has failed

1301 Redundancy sensor value unknown

1302 Redundancy not applicable

1303 Redundancy is offline

1304 Redundancy regained

White Paper


Redundancy Unit

1305 Redundancy degraded


2098 Global hot spare assigned

2099 Global hot spare unassigned

2122 Redundancy degraded


2124 Redundancy normal

2163 Rebuild completed with errors


2167 The current kernel version and the non- RAID SCSI driver version are older than the minimum required levels.

2168 The non- RAID SCSI driver version is older than the minimum required level.

2197 Replace member operation has stopped for rebuild.

2200 Replace member operation is not possible as combination of SAS and SATA physical disks is not supported in the same virtual disk.

1000 Server Administrator starting

1001 Server Administrator startup complete

1050 Temperature sensor has failed

1051 Temperature sensor value unknown

1052 Temperature sensor returned to a normal value

1053 Temperature sensor detected a warning value

1054 Temperature sensor detected a failure value

1055 Temperature sensor detected a non-recoverable value

2100 Temperature exceeded the maximum warning threshold

2101 Temperature dropped below the minimum warning threshold

2102 Temperature exceeded the maximum failure threshold

White Paper


Redundancy Unit

2103 Temperature dropped below the minimum failure threshold

2154 Maximum temperature probe warning threshold value changed

2155 Minimum temperature probe warning threshold value changed

Virtual Disk Events

2053 Virtual disk created

2054 Virtual disk deleted

2055 Virtual disk configuration changed

2056 Virtual disk failed

2057 Virtual disk degraded

2058 Virtual disk check consistency started

2059 Virtual disk format started

2061 Virtual disk initialization started

2063 Virtual disk reconfiguration started

2064 Virtual disk rebuilds started

2067 Virtual disk check consistency cancelled

2070 Virtual disk initialization cancelled

2076 Virtual disk Check Consistency failed

2077 Virtual disk format failed

2079 Virtual disk initialization failed

2081 Virtual disk reconfiguration failed

2082 Virtual disk rebuilds failed

2085 Virtual disk check consistency completed

2086 Virtual disk format completed

2088 Virtual disk initialization completed

2090 Virtual disk reconfiguration completed

White Paper


Virtual Disk Events

2091 Virtual disk rebuilds completed

2114 A consistency check on a virtual disk has been paused (suspended)

2115 A consistency check on a virtual disk has been resumed

2116 A virtual disk and its mirror have been split

2117 A mirrored virtual disk has been un-mirrored

2118 The write policy change write policy

2125 Controller cache preserved for missing or offline virtual disk

2127 Background initialization (BGI) started

2129 BGI failed

2130 BGI completed

2136 Virtual disk initialization OK / Normal

2159 Virtual disk renamed

2192 The virtual disk Check Consistency has made corrections and completed.

2193 The virtual disk reconfiguration has resumed.

2194 The virtual disk read policy has changed.

2199 The virtual disk cache policy has changed.

Voltage Sensor Events

1150 Voltage sensor has failed

1151 Voltage sensor value unknown

1152 Voltage sensor returned to a normal value

1153 Voltage sensor detected a warning value

1154 Voltage sensor detected a failure value

1155 Voltage sensor detected a non-recoverable value

White Paper


Virtualization Management

Virtualization technology comes in several different forms. There is virtualization running as a software

application running on a host Operating System such as Microsoft’s Virtual Server 2005 or the

virtualization support included in Windows Server 2008. This approach has perceived disadvantages from

a security perspective as the attack service of the virtualization layer is a general purpose OS. Microsoft

also offers Hyper-V Server 2008 that strips the host OS to Windows Server Core, but still the footprint and

the attack surface is larger than an embedded hypervisor and once into the host OS, the guest OS’s can

be compromised. For the Microsoft virtualization solutions, the logs are all stored in the Applications and

Service Logs in the EventViewer of the host OS. EventTracker is able to collect all these logs through the

standard windows collection methods.

In the case of VMware the 2 hypervisors available are ESX and ESXi ESX is similar to the Hyper-V Server

2008 model, and is a bootable hypervisor. The Operating environment in the ESX case is a stripped down

Linux kernel. It is argued that it is more secure than a general purpose OS installation such as Server 2008

or even Server Core as it is more stripped and it is Linux. ESXi on the other hand, represents the other

popular type of virtualization technique, and is usually embedded directly on the server hardware and

operates more like firmware than software like ESX or Hyper-V. ESXi is very small, and offers access only

through defined and limited APIs.

In larger installations, ESXi combined with a management application like Vcenter is emerging as the

preferred choice. As the hypervisors and the management application have been pared down, it is

expected that these are inherently more secure as the attack surface has been reduced. From a security

perspective this approach has a completely different management layer outside of the Operating System.

Both the Hypervisor and the management applications fortunately produce logs and these logs should be

collected and stored in the Log Management SIEM solution.

EventTracker is able to collect logs directly from the bare-bones hypervisors such as Vmware ESXi, or the

management application in the case of Vcenter, or from ESX. The following diagram shows the collection

architecture.

White Paper


Hyper V Events Hyper-V is made up of distinctive services and each service generates an exhaustive list of events. The

events follow the general Microsoft approach to logging – log it all and log it in great detail. These events,

when normalized, provide a complete picture of what has occurred and when it occurred. Combine these

with login information provided by AD and you have a complete who/what/when picture of both manual

and automated changes in the virtual environment.

Hyper-V Hypervisor

1 Hyper-V successfully started.

5 Hyper-V launch aborted due to auto-launch being disabled in the registry.

6 Hyper-V failed Code Integrity check.

7 Hypervisor traces are corrupted

17 Hyper-V launch failed. The registry key could not be opened by the Hyper-V boot driver

18 Hyper-V launch failed. Registry value could not be read

19 Hyper-V launch failed; the registry value %2 of key %1 is not a string.

20 Hyper-V launch failed; sleep and hibernate could not be disabled (status %1).

26 Hyper-V launch failed. Hyper-V boot loader's internal logic failed

27 Hyper-V launch failed; the Hyper-V boot loader was unable to allocate sufficient resources to perform the launch.

28 Hyper-V launch failed. The Hyper-V boot loader does not support the vendor of at least one of the processors in the system.

29 Hyper-V launch failed. Processor does not appear to support the features required by Hyper-V

White Paper


Hyper-V Hypervisor

30 Hyper-V launch failed. The system's combination of processors is not supported.

31 Hyper-V launch failed. The system does not appear to have a sufficient level of ACPI support to launch Hyper-V.

32 Hyper-V launch failed. At least one of the processors in the system does not appear to provide a virtualization platform supported by Hyper-V.

33 Hyper-V launch failed. the Hyper-V image could not be accessed

34 Hyper-V launch failed. Hyper-V image could not be loaded

35 Hyper-V launch failed. The Hyper-V image could not be read

36 Hyper-V launch failed; the Hyper-V image failed code integrity checks

37 Hyper-V launch failed. The Hyper-V image does not contain the Hyper-V image description data structures

38 Hyper-V launch failed. At least one of the processors in the system was unable to launch Hyper-V

40 Hyper-V launch failed The Hyper-V image is not the correct revision

41 Hyper-V launch failed. Either VMX not present or not enabled in BIOS.

42 Hyper-V launch failed. Either SVM not present or not enabled in BIOS.

Virtual Machine Management Service

2000 Could not register service connection point

2001 Could not unregister service connection point

10000 SID Mapping Error

10001 Failed to create NT VIRTUAL MACHINE security identifier mappings

10010 10011 The security identifier S-1-5-83 is already mapped to another domain.

10020 10021 Failed to create security identifier mapping

10030 10031 Failed to create security identifier mapping

10104 Failed to revert to VSS snapshot on one or more virtual hard disks of the VM

10107 Corrupt or invalid configuration files

11900 VM configuration section is corrupt

White Paper



12242 Cannot mount the device read/write because the device is already mounted read-only

12243 Cannot mount the device

13000 User failed to create external configuration store

13001 Failed to create external configuration store at <location>

14030 14031 Failed to update the VM's saved state information

14040 14041 Failed to query domain information.

14050 Failed to register service principal name.

14060 14061 Failed to locate the default configuration store.

14062 14063 Failed to locate the default virtual hard disk directory

14072 Automatic restart has been disabled for VM because the VM stopped responding repeatedly

14073 VM stopped responding repeatedly.

14074 VM already running when the Hyper-V VM Management service started.

14080 14081 VM failed to automatically restart

14090 14091 Hyper-V VM Management service is shutting down while some VM's are running.

14092 14093 Service is shutting down.

14094 14095 Service started successfully.

14096 14097 Service failed to start.

14098 Required driver is not installed or is disabled.

14100 14101 Shutting down physical computer. Stopping/saving all VM’s .

14210 14211 Snapshot Operation failed to delete snapshot

14241 Cannot find the specified VM.

14270 VM unable to check user access rights

14330 14331 Failed to delete snapshot because it is specified as the automatic recovery snapshot for VM

15010 15011 Failed to create new VM

15040 15041 Failed to import VM

White Paper



15050 15051 Failed to export VM

15070 15071 Service failed to remove snapshot

15080 A new VM was added in a different location and the creation process never completed

15110 15111 Failed to modify service settings.

15120 15121 VM failed to initialize

15140 15141 VM failed to turn off

15150 15151 VM Save Operation failed

15170 15171 VM failed to pause

15180 15181 VM failed to resume

15190 15191 Snapshot Operation failed

15220 15221 VM failed to reset

15240 15241 VM failed to begin delayed startup

15300 Failed to access configuration store

15310 Created configuration store

15320 Failed to create configuration store

15330 VM Bus (VMBus) cannot start because the physical computer's PCI chipset does not properly support Message Signaled Interrupts.

15340 The VM bus is not running.

15500 15501 VM failed to start worker process

16000 16001 VM Management service encountered an unexpected error

16020 VM encountered an unexpected error. The system cannot find the path specified.

16040 Cannot get information about available space for path

16060 16061 VM paused due to insufficient disk space

16090 16100 Worker Process validation failed

16110 An error occurred while waiting to start VM

16120 VM startup error

White Paper



16140 VM cannot delete file

16150 Cannot delete directory

16160 Cannot delete snapshot file

16170 Cannot delete snapshot directory

16180 Service cannot update the snapshot list for deleted snapshot

16190 Service cannot update the parent for snapshot

16200 Service cannot update the instance of last applied snapshot

16330 Cannot load the snapshot configuration because it is corrupt

16370 Service cannot create the storage required for the snapshot

16371 Snapshot Operation failed

16430 Service timed out waiting for the worker process to exit

17010 Service assigned to an invalid authorization scope

17030 A VM is assigned to an authorization scope that is not defined in the policy store

17040 The authorization store could not be initialized

17050 Failed to initialize application in the current authorization store

17080 Updated the content of the authorization store successfully.

17090 Content of the authorization store could not be updated from the store persistent location

17100 Cannot open authorization store

18002 18003 Cannot take snapshot

18030 Import failed. Unable to create identifier while importing VM

18031 Import failed.

18080 18081 VM import failed

18160 Failed to get summary information for VM

18190 Worker process health is critical for VM

18200 Worker process health is now OK for VM

18240 18241 Unable to find virtual hard disk file

White Paper



18540 VM was reset because the guest operating system requested an operation that is not supported by Hyper-V

19000 19010 WMI namespace is not registered in the CIM repository.

19020 WMI provider has started.

19030 WMI provider failed to start

19040 WMI provider has shut down.

19060 19061 Failed to get saved state information for VM. It is assumed that the VM is in a saved state

20100 20101 Failed to register the configuration for the VM

20102 20103 Failed to unregister the configuration for the VM

20104 20105 Failed to verify that the configuration is registered for the VM

20106 20107 Service did not find the VM

20108 20109 Failed to start the VM

20110 20111 Failed to shut down the VM

20112 20113 Service failed to forcibly shut down the VM

20114 20115 Service failed to verify the running state of the VM

20132 20133 Failed to delete the configuration for the VM

14250 14251 Cannot find the specified snapshot

14320 14321 Cannot delete snapshot

15060 15061 Failed to apply snapshot

15130 15131 VM failed to start

15510 15511 The worker process for VM failed to respond within the startup timeout period and was restarted

16010 16011 Operation failed

16050 16051 VM is about to run out of disk space

16360 16361 Cannot access the folder where snapshots are stored

18040 18041 Unable to rename file or directory

White Paper



18050 18051 Failed to stop the rename of the file or directory

18060 18061 Import failed

18100 18101 Failed to create export directory.

18110 18111 Failed to copy file during export

18120 18121 An unknown device failed to import

18160 18161 Failed to get summary information for VM

18550 18560 VM was reset because an unrecoverable error occurred on a virtual processor

19050 19051 VM failed to perform operation. The VM is not in a valid state to perform the operation.

Virtual Hard Drive Management Service

12140 Failed to open attachment

12141 File extension is invalid

15050 The system successfully converted VHD

15051 The system successfully created VHD

15052 The Hyper-V Image Management Service started.

15053 The system is expanding VHD

15000 15001 Device mount failed. The device is already mounted read-only, and an attempt was made to mount it read/write

15100 Filename is invalid

15101 Failed to open attachment

15102 Invalid file extension

15103 The system is compacting VHD

15104 The system is merging VHD

15105 The system is converting VHD

15106 The system successfully compacted VHD

15107 The system successfully merged VHD

White Paper


Virtual Hard Drive Management Service

15108 The system mounted VHD

15109 The system successfully expanded VHD

15110 Invalid VHD

15111 Invalid file name. You cannot use the following names (LPTn, COMn, PRN, AUX, NUL, CON) as they are reserved by Windows.

15200 The Hyper-V Image Management Service stopped.

15201 The Hyper-V Image Management Service failed to start

15202 The system successfully un-mounted VHD

12242 12243 The system is creating VHD

Hyper-V High-Availability Service

21100 Missing or invalid VM ID resource property

21101 Missing or invalid VmStoreRoot resource property

21102 21203 VM failed to register

21103 21104 21502 VM failed to unregister

21105 VM configuration update failed

21106 VM failed to initiate startup

21107 VM failed to initiate shutdown

21108 VM failed to start

21109 21110 VM failed to terminate

21117 Virtual network switch port settings creation failed.

21118 VM update settings failed

21119 VM successfully started

21120 VM successfully registered

21200 System not found

21201 Missing or invalid VM ID resource property

21202 Virtual network switch port already exists

White Paper


Hyper-V Configuration

4096 Configuration no longer accessible. The system cannot find the path specified or configuration is deleted.

4097 Configuration no longer accessible.

4098 Configuration is now accessible.

Hyper-V SynthStore

12242 12243 Failed to mount device. The device is already mounted read-only, and an attempt was made to mount it read/write.

Hyper-V Network

14000 Switch created

14002 Switch deleted

14004 Switch port created.

14006 Switch port deleted

14008 Switch port connected

14010 Switch port disconnected

14012 Internal miniport created

14014 Internal miniport deleted

14016 External Ethernet port bound

14018 External Ethernet port unbound

14020 Switch set up

14022 Switch torn down

14050 Switch create failed

14052 Switch delete failed

14054 Switch port create failed

14056 Switch port delete failed

14058 Switch port connect failed

White Paper


Hyper-V Network

14060 Switch port disconnect failed

14062 Switch port create failed

14064 Switch port delete failed

14066 Ethernet port bind failed

14068 Ethernet port unbind failed

14070 Switch set up failed

14072 Switch tear down failed

14108 Unable to open handle to switch driver

14110 Network WMI provider service started successfully

14112 Network WMI provider service failed to start

14116 Timed out trying to acquire network configuration lock

14118 Unable to initialize network configuration

Hyper-V Image Management Service

12140 12141 Failed to open attachment

12242 12243 Failed to mount device. The device is already mounted read-only, and an attempt was made to mount it read/write

15000 15001 Invalid virtual hard disk

15051 Invalid file extension

15052 Invalid file extension. You cannot use the following names (LPTn, COMn, PRN, AUX, NUL, CON) as they are reserved by Windows.

15053 Invalid file name

15100 System is compacting Image

15101 The system successfully compacted Image

15102 The system is merging Image

15103 The system successfully merged Image

15104 The system is expanding Image

White Paper


Hyper-V Image Management Service

15105 The system successfully expanded Image

15106 The system is converting Image

15107 The system successfully converted Image

15108 The system mounted Image

15109 The system successfully un-mounted Image

15110 The system is creating Image

15111 The system successfully created Image

15200 Image Management service started.

15201 Image Management service stopped.

15202 Image Management service failed to start

Hyper-V Worker

3170 3171 Worker failed to initialize the virtual machine during reset

3200 3201 Worker failed to save, but ignored the error to allow the virtual machine to continue shutdown

3210 3211 Worker failed to save RAM contents during a snapshot operation

3220 3221 Unable to save RAM contents

3230 3231 Unable to restore RAM contents

3240 3241 Unable to save RAM block

3250 3251 Unable to restore RAM block because of an unexpected block data size.

3260 3261 Unable to restore RAM because some RAM blocks are missing.

3270 3271 Unable to restore RAM because some RAM block data is corrupt.

3280 3281 Failed to initiate a snapshot operation

3284 3285 VM was shutdown as a result of a failure to resume execution during a snapshot operation

3286 3287 VM was paused as a result of a failure to resume execution during a snapshot operation

White Paper


Hyper-V Worker

3290 3291 Unable to restore RAM and unable to create a restore buffer.

3310 3311 Failed to initialize restore operation

3320 3321 Failed to create memory contents file

3330 3331 Failed to access the snapshot folder.

3350 3351 Failed to create auto virtual hard disk

3360 3361 Unable to stop the virtual processors.

3370 3371 Unable to reset the virtual hard disk path as a result of a failure to create a snapshot

3432 3433 Could not set the processor affinity for the worker process

5110 Failed to start the worker process using the correct security context

11901 Configuration section is corrupt

11902 RC Vista Ultimate SP1 x86 (Device 'Microsoft Synthetic Display Controller'): An unrecoverable internal error has occurred.

12010 VM' Microsoft Emulated IDE Controller failed to power on with Error 'Incorrect function.'

12070 RC Vista Ultimate SP1 x86 Microsoft Synthetic Video failed to pause with error 'Catastrophic failure'

12200 12201 Virtual machine Out of Memory Error

12242 12243 Failed to mount device. The device is already mounted read-only, and an attempt was made to mount it read/write

12440 12441 Error while opening file during ethernet device startup. The Hyper-V Networking Management service provider may not be installed

12540 RC Vista Ultimate SP1 x86 device Microsoft Synthetic Display Controller experienced a protocol error indicative of a deep system problem.

15160 15161 Failed to restore virtual machine state.

17010 Hyper-V Service is assigned to an unsupported authorization scope

17030 VM is assigned to an authorization scope that is currently not defined in the policy store. The VM will be reassigned to the default authorization scope

17040 The authorization store could not be initialized

17050 Failed to initialize application in the current authorization store

White Paper


Hyper-V Worker

17080 The content of the authorization store has been updated

17090 The content of the authorization store could not be updated

18500 Virtual machine started successfully

18510 VM saved successfully

18520 Snapshot succeeded

VMware Events VMware generates far fewer raw events than Hyper-V but the events tend to focus on the types of

information that security personnel would need to know and less on general day to day health and status

messages. The following is a list of events emitted by VMware and included in the EventTracker Knowledge

Pack. Items marked "predefined alert" are included in the KP tested against VMware 3.x.

Virtual Center Events

Alarm created Datacenter renamed

Alarm removed High resource usage alarm (predefined alert)

Datacenter created Host added to datacenter

Datacenter removed Host removed from datacenter

Virtual Machine Management

Guest OS shutdown Virtual machine removed

VM resource allocation events Virtual machine renamed

Guest OS state changed Virtual machine reset

VM resource configuration updated Virtual machine relocated

Virtual machine cloned Virtual machine suspended

Virtual machine created Virtual machine switched off

Virtual machine powered on Virtual machine snapshot created

Virtual machine registered Virtual machine reverted

Virtual machine reconfigured

White Paper


User Management

Successful user login User permission rule added

Failed user login (predefined alert) User permission rule removed

User logout Task failed or canceled by user (predefined alert)

User permission rule changed

User Management

Remote console connected Remote console disconnected

Summary At its most basic, security management is about first “seeing” everything that is happening, and then

applying processes, tools and solutions that can help you make sense of all the information and make you

more secure. In IT, with each new added technology comes complexity – distributed systems, remote

access, the internet, virtualization all create significant new challenges for security teams. Virtualization is

no different.

Also the real security requirements i.e. what is most critical to monitor, are generally driven by corporate

structure, infrastructure and policy. Businesses have different technology vendors, different

organizational structures, different compliance mandates and rarely, if ever, does one size fit all or even

more than one.

With EventTracker, the challenge of visibility is solved. EventTracker provides the most comprehensive

support for virtual environments of any vendor on the market. Having all the data collected dependably

in one place gives an organization the ability to become secure. This data is categorized and available for

advanced real-time analysis where events from all the different technology layers can be monitored. For

example, an enterprise critical application can be assigned to a virtual machine. Using Vmware’s Vmotion,

that virtual machine can be reassigned different hardware based on performance or availability measures.

It becomes critical to know that if a disk error is being received from OpenManage that that disk is mapped

to that VM, and that VM is running the critical service. With centralized visibility all that becomes possible.

Plus, descriptions on all events are available on the EventTracker Knowledgebase, so security personnel

don’t have to worry about understanding hundreds of new events.

From there, with an understanding of the organizational structure and policies, rules can be quickly setup

to alert on violations of policy. For compliance, auditing is easily facilitated and no trusted user is able to

effect change in the enterprise without at least a record being created. Security starts from visibility – not

only the simple ability to see it, but understand it and make sense of it.

White Paper


The EventTracker Solution The EventTracker solution is a scalable, enterprise-class Security Information and Event Management

(SIEM) solution for Windows systems, Syslog/Syslog NG (UNIX and many networking devices), SNMP V1/2,

legacy systems, applications and databases. EventTracker enables “defense in depth”, where log data is

automatically collected, correlated and analyzed from the perimeter security devices down to the

applications and databases. To prevent security breaches, event log data becomes most useful when

interpreted in near real time and in context. Context is vitally important because often the critical

indications of impending problems and security violations can only be learned by watching patterns of

events across multiple systems. Complex rules can be run on the event stream to detect signs of such a

breach. EventTracker also provides real-time alerting capability in the form of an email, page or SNMP

message to proactively alert security personnel to an impending security breach.

The original event log data is also securely stored in a highly compressed event repository for compliance

purposes and later, forensic analysis. For compliance, EventTracker provides a powerful reporting

interface, scheduled or on-demand report generation, automated compliance workflows that prove to

auditors that reports are being reviewed and many other features. With pre-built auditor grade reports

included for most of the compliance standards (FISMA, HIPAA, SOX, GLBA, and others); EventTracker

represents a compliance solution that is second to none. EventTracker also provides advanced forensic

capability where all the stored logs can be quickly searched through a powerful Google-like search

interface to perform quick problem determination.

EventTracker lets users completely meet the logging requirements specified in NIST SP 800-92 Guide to

Computer Security Log Management, and additionally provides Host Based Intrusion Detection, Change

Monitoring and USB activity tracking on Windows systems, all in a turnkey, off the shelf, affordable,

software solution.

EventTracker provides the following benefits

A highly scalable, component-based architecture that consolidates all Windows, SNMP V1/V2,

legacy platforms, Syslog received from routers, switches, firewalls, critical UNIX servers (Red Hat

Linux, Solaris, AIX etc), Solaris BSM, workstations and various other SYSLOG generating devices.

Automated archival mechanism that stores activities over an extended period to meet auditing

requirements. The complete log is stored in a highly compressed (>90%), secured (Sealed with

SHA-1 checksum) archive that is limited only by the amount of available disk storage.

Real-time monitoring and parsing of all logs to analyze user activities such as logon failures and

failed attempts to access restricted information.

Full support for monitoring of virtualized enterprises.

Alerting interface that generates custom alert actions via email, pager, console message, etc.

White Paper


Event correlation modules to constantly monitor for malicious hacking activity. In conjunction with

alerts, this is used to inform network security officers and security administrators in real time. This

helps minimize the impact of breaches.

Various types of network activity reports, which can be scheduled or generated as required for any

investigation or meeting audit compliances.

Host-based Intrusion Detection (HIDS).

Role-based, secure event and reporting console for data analysis.

Change Monitoring on Windows machines

USB Tracking, including restricted use, insert/removal recording, and a complete audit trail of all

files copied to the removable device.

Built-in compliance workflows to allow inspection and annotation of the generated reports.

About EventTracker EventTracker’s advanced security solutions protect enterprises and small businesses from data breaches

and insider fraud, and streamline regulatory compliance. The company’s EventTracker platform comprises

SIEM, vulnerability scanning, intrusion detection, behavior analytics, a honeynet deception network and

other defense in-depth capabilities within a single management platform. The company complements its

state-of-the-art technology with 24/7 managed services from its global security operations center (SOC)

to ensure its customers achieve desired outcomes—safer networks, better endpoint security, earlier

detection of intrusion, and relevant and specific threat intelligence. The company serves the retail,

hospitality, healthcare, legal, banking and financial services, utilities and government sectors.

EventTracker is a division of Netsurion, a leader in remotely-managed IT security services that protect

multi-location businesses’ information, payment systems and on-premise public and private Wi-Fi

networks. www.eventtracker.com.

https://www.eventtracker.com/

the importance of consolidation, correlation, and …...virtualization with the mainstream arrival...

Documents