pan manager ped message reference -...

PAN Manager PED Message Reference

For BladeFrame® BF400 S2 and BladeFrame® BF200

Document Number 430-SB0077

August 2008

PM5.2_BF

CopyrightCopyright © 2008 Egenera, Inc. All rights reserved.

This document, and the product described in it, is furnished under license and may only be used in accordance with the terms of such license. The content of this document is furnished for information purposes only and is subject to change without notice.

Egenera, Egenera stylized logos, BladeFrame, BladeLatch, BladeMate, BladePlane, cBlade, Control Blade, PAN Manager, pBlade, Processing Blade, sBlade, and Switch Blade are either registered trademarks or trademarks of Egenera, Inc. in the United States and/or other countries.

PRIMERGY is a registered trademark of Fujitsu Siemens Computers.

AMD, AMD Opteron, and AMD Athlon are trademarks of Advanced Micro Devices, Inc..

EMC, CLARiiON, and Symmetrix are registered trademarks of EMC Corporation.

The IBM Developer Kit for Linux, Java 2 Technology Edition contains software which is copyright IBM Corporation, Sun Microsystems Inc., Hewlett-Packard Co., and X Consortium.

Intel and Itanium are registered trademarks and Xeon is a trademark of the Intel Corporation in the United States and other countries.

Sun, Sun Microsystems, the Sun Logo, Solaris, and the Java logo are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.

Linux is a registered trademark of Linus Torvalds.

Microsoft and Windows are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The virtual VGA console uses Microsoft Terminal Services Advanced Client (TSAC), which is a copyright of Microsoft Corporation.

MindTerm is copyright AppGate AB.

Nero is a trademark of Nero AG.

NetApp is a registered trademark and Network Appliance is a trademark of Network Appliance, Inc.

Oracle9i is a trademark of Oracle Corporation.

Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries.

SUSE is a registered trademark of SUSE LINUX Products GmbH, a Novell business.

VMware, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc.

Xen, XenSource, XenServer, and XenEnterprise are either registered trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other countries.

All other company and product names are trademarks or registered trademarks of their respective holders.

Printed in the United States of America.

Egenera, Inc., 165 Forest Street, Marlboro, Massachusetts 01752.

Contents

Preface

Customer Support .................................................................................................... viii

Document Conventions ........................................................................................... viii

Chapter 1: About the Precise Error Daemon

Reviewing Events and Errors .................................................................................. 1-1

Controlling the Size of the ped.log File .................................................................. 1-4Types of Thresholds ......................................................................................... 1-4Entering Throttle Mode .................................................................................... 1-5Exiting Throttle Mode ...................................................................................... 1-5Reducing the Size of the ped.log File............................................................... 1-6

Chapter 2: PED Event Filters

Filtering Blade Events ............................................................................................. 2-1

Filtering Switch Events ........................................................................................... 2-3

Filtering Fabric Events ............................................................................................ 2-5

Chapter 3: Error Message Descriptions

Error Message Format ............................................................................................. 3-2Message and Variables ..................................................................................... 3-2

PM5.2_BF iii


Severity............................................................................................................. 3-3Threshold.......................................................................................................... 3-3Causes and Actions........................................................................................... 3-4

Common Recovery Procedures ............................................................................... 3-4Determining Whether the Switch or cBlade Is the Cause ................................ 3-4Determining Whether the pBlade or Slot Is the Cause..................................... 3-5Diagnosing Switch Problems ........................................................................... 3-5For More Information....................................................................................... 3-6

EM16384 ................................................................................................................. 3-7EM16385 ................................................................................................................. 3-8EM16386 ............................................................................................................... 3-10EM16387 ............................................................................................................... 3-11EM16388 ............................................................................................................... 3-12EM16389 ............................................................................................................... 3-13EM16390 ............................................................................................................... 3-14EM16391 ............................................................................................................... 3-15EM16392 ............................................................................................................... 3-16EM16393 ............................................................................................................... 3-17EM16394 ............................................................................................................... 3-18EM16395 ............................................................................................................... 3-19EM16396 ............................................................................................................... 3-20EM16397 ............................................................................................................... 3-21EM16398 ............................................................................................................... 3-22EM16399 ............................................................................................................... 3-23EM16400 ............................................................................................................... 3-24EM16401 ............................................................................................................... 3-25EM16402 ............................................................................................................... 3-26EM16403 ............................................................................................................... 3-27EM24576 ............................................................................................................... 3-28EM24577 ............................................................................................................... 3-29EM24578 ............................................................................................................... 3-30EM24579 ............................................................................................................... 3-31EM24580 ............................................................................................................... 3-32EM24581 ............................................................................................................... 3-33EM24582 ............................................................................................................... 3-34EM24583 ............................................................................................................... 3-35EM24584 ............................................................................................................... 3-37EM24585 ............................................................................................................... 3-38EM24586 ............................................................................................................... 3-39

iv PM5.2_BF

Contents

EM24587 ............................................................................................................... 3-40EM24588 ............................................................................................................... 3-41EM24589 ............................................................................................................... 3-42EM24590 ............................................................................................................... 3-43EM24591 ............................................................................................................... 3-44EM24592 ............................................................................................................... 3-45EM24593 ............................................................................................................... 3-46EM24594 ............................................................................................................... 3-47EM24595 ............................................................................................................... 3-48EM24596 ............................................................................................................... 3-49EM24597 ............................................................................................................... 3-50EM24598 ............................................................................................................... 3-51EM24599 ............................................................................................................... 3-52EM24600 ............................................................................................................... 3-53EM33280 ............................................................................................................... 3-54EM49152 ............................................................................................................... 3-55EM49153 ............................................................................................................... 3-56EM49154 ............................................................................................................... 3-57EM49155 ............................................................................................................... 3-58EM49156 (BladeFrame BF200) ............................................................................ 3-59EM49157 ............................................................................................................... 3-60EM49158 ............................................................................................................... 3-61EM49159 ............................................................................................................... 3-62EM49160 ............................................................................................................... 3-63EM49161 ............................................................................................................... 3-65EM49162 ............................................................................................................... 3-66EM49163 ............................................................................................................... 3-67EM49164 and EM49165 ....................................................................................... 3-68EM49166 ............................................................................................................... 3-69EM49167 ............................................................................................................... 3-70EM49168 ............................................................................................................... 3-71EM49169 ............................................................................................................... 3-72EM49170 ............................................................................................................... 3-73EM49171 ............................................................................................................... 3-74EM49172 ............................................................................................................... 3-75EM49173 ............................................................................................................... 3-76EM49174 ............................................................................................................... 3-77EM49175 ............................................................................................................... 3-78

PM5.2_BF v


Index

vi PM5.2_BF

Preface

Welcome to PAN Manager PM5.2_BF. PAN Manager PED Message Reference is part of the PAN Manager documentation set. Its purpose is to explain the root cause of errors that occur in the fabric of the PAN hardware platform.

Audience — PAN Manager PED Message Reference is for PAN Administrators and LPAN Administrators.

Topics — Read this book to learn about the following:

• How the Precise Error Daemon (PED) filters error messages

• How to determine whether the switch, cBlade, pBlade, or slot caused the error

• How to diagnose switch problems

• Possible causes and recommended actions for each PED error

Documentation Set — To learn about the other documents available in the PAN Manager documentation set for your hardware platform, see Chapter 2, “Document List,” of PAN Manager Feature Summary.

Platform-Specific Product Features — To learn about the specific PAN Manager features available in the current release, see Chapter 3, “Feature List,” of PAN Manager Feature Summary.

PM5.2_BF vii

../featsum/feature_doc_list.html

../featsum/featlistchap.html


Customer Support

If you require customer support regarding this product, use the following contact information.

Fujitsu Siemens Computers customer support

Document Conventions

Internet http://www.fujitsu-siemens.com/support

Telephone See the Help Desk information at http://manuals.fujitsu-siemens.com/primergyservers.html under General Information, Warranty and Support.

Convention Description

> Directory-level delimiter used to navigate the left pane of the PAN Manager GUI.

Example: Resources > Ethernet Connections

Sans serif italics Variable text, such as a path, a filename, or an LPAN name.

Example: lpan -c lpanname

Sans serif Text that must be typed as shown.

Example: Type root at the login prompt.

viii PM5.2_BF

Preface

Bold The name of a field or window element appearing in a GUI. It also highlights default values in PAN Manager man pages.

Example: In the Users page...

Italics Text that is emphasized.

Example: Do not connect the power.

[text] Text that is optional to a command.

{text} A set of choices, one of which is required.

| Separation of mutually exclusive choices in syntax lines.

Example: lpan [-aD | -rD]{switch | SCSI_ID} lpanname

Note Information of importance or that may not fit in main text.

Caution Failure to heed a caution could result in loss of data.

Warning — Failure to heed a warning could result in physical harm to the user or the hardware.

Convention Description

!

PM5.2_BF ix


x PM5.2_BF

Chapter 1About the Precise Error

Daemon

This chapter describes how the precise error daemon filters events to reduce the number of messages reported to the /var/log/messages file on the cBlade. This chapter contains the following sections:

• Reviewing Events and Errors

• Controlling the Size of the ped.log File

Reviewing Events and Errors

PAN administrators should review the event and error logs periodically to address any issues or problems that occur. Administrators can use the following tools to review events and errors:

• The Events tab in PAN Manager displays each significant event that occurs in the PAN. Refer to PAN Manager Administrator’s Guide for more information.

PM5.2_BF 1-1


• The operating system and PAN Manager virtualization extensions log messages in the /var/log/messages file. Because the network fabric that connects the blades and other hardware components is complex, the precise error daemon filters fabric-related events and determines which subset merits logging in the /var/log/messages file.

The precise error daemon runs on each cBlade and carries out the following tasks, as shown in Figure 1.1:

• Receives events about state changes from the fabric-related drivers.

• Logs each state change as an EVnnnnn event (where nnnnn is the event number) to the /var/log/ped.log file. The ped.log file can grow very large and contains a complete history of all fabric-related events. The documentation set does not describe the EVnnnnn events.

• Builds a state model of the system, which includes the following:

• The status of every node (up, down, or unknown)

• The status of every network link (up, down, intermittent, or unknown)

• The expected state of every node (up, down, rebooting, or unknown)

• Uses event filters, which examine each event in context with the system state at the time of the event, to determine whether an event is truly an error. If the event is an error, the precise error daemon logs an EMnnnnn message (where nnnnn is the error number) to the /var/log/messages file. Chapter 2, "PED Event Filters," describes the event filters. Chapter 3, "Error Message Descriptions," describes each EMnnnnn message.

1-2 PM5.2_BF

Filters.html

Filters.html

Messages.html

Messages.html

About the Precise Error Daemon

Figure 1.1 The Precise Error Daemon

As the figure shows, the /var/log/messages file contains messages from the precise error daemon on behalf of the fabric-related drivers, and unfiltered messages from other drivers and operating system services. Because the fabric-related errors have ped as the process name and the EMnnnnn error number, they are easy to spot in the /var/log/messages file.

Precise Error Daemon

/var/log/messages

/var/log/ped.log

Fabric-Related Drivers

Other Drivers

Events

Events

Events

Mes

sage

s

Messages

Filters

Messages

EVnnnnn events

EMnnnnn messages

StateModel

PM5.2_BF 1-3


Controlling the Size of the ped.log File

This section describes how the precise error daemon controls the size of the /var/log/ped.log file using two types of thresholds, and how administrators can reduce the size of the ped.log file. This section describes the following topics:

• Types of Thresholds

• Entering Throttle Mode

• Exiting Throttle Mode

• Reducing the Size of the ped.log File

Types of Thresholds

The precise error daemon monitors the number of events that it writes to the ped.log file. The precise error daemon controls the size of the ped.log file using two different thresholds:

• The overall threshold for the ped.log file.

By default, the threshold for the ped.log file is 20,000 events in 2 hours. If the precise error daemon exceeds this threshold, it enters throttle mode, as described in “Entering Throttle Mode” on page 1-5. If you want, you can change the threshold, as described in “Reducing the Size of the ped.log File” on page 1-6.

• The individual threshold for each EVnnnnn event that can be logged to the ped.log file.

Every EVnnnnn event has its own threshold that cannot be changed. During normal operation, the precise error daemon ignores these thresholds and logs every event it receives to the ped.log file. When the precise error daemon is in throttle mode, however, it enforces the individual threshold for each event.

1-4 PM5.2_BF

About the Precise Error Daemon

Note: The precise error daemon also controls the size of the /var/log/messages file with individual thresholds for the EMnnnnn errors. These thresholds are completely different and independent from the thresholds for events. For more information, see the threshold topic in Chapter 3, "Error Message Descriptions."

Entering Throttle Mode

By default, if 20,000 events are generated within 2 hours, the precise error daemon does the following:

• Logs the EM16402 error to the /var/log/messages file to indicate that it has entered throttle mode.

• Runs in throttle mode for two hours.

If you have changed the threshold and period, the precise error daemon enters throttle mode when the new threshold is reached, and stays in throttle mode for the new period.

In throttle mode, the precise error daemon enforces the individual threshold and period of each EVnnnnn event. Any event that exceeds its individual threshold in throttle mode is suppressed for the remainder of throttle mode. For example, the EV43008 error has a threshold of 60 times in 500 minutes. When an EV43008 error occurs 61 times, the precise error daemon does the following:

• Suppresses the logging of the 61st instance and any additional instances of the EV43008 event.

• Logs the EV49152 event to indicate that the threshold for the EV43008 event was exceeded. The EV49152 event can be reported with every event that exceeds its threshold.

Exiting Throttle Mode

The precise error daemon remains in throttle mode until the number of events generated is less than the threshold and period (by default, 20,000 events within a 2-hour period). When the precise error daemon exits throttle mode, it takes the following actions:

• Logs the EM16403 error to the /var/log/messages file to indicate that it has exited throttle mode.

PM5.2_BF 1-5

Messages.html


• Begins logging every EVnnnnn event to ped.log again, and ignores the individual thresholds for events.

Reducing the Size of the ped.log File

The /var/log/ped.log file can become quite large. The /var/log directory is shared by other log files, so limited space can become an issue. To resolve this issue, you can limit the size of the ped.log file by lowering the threshold and period for the ped.log file, which reduces the number of events written to the ped.log file.

To adjust the threshold and period that causes the precise error daemon to enter throttle mode, use the following command:

# /opt/egenera/bin/ped -T threshold -P period

where:

• threshold — is the maximum number of messages allowed per period. Specifying a value of 0 disables throttle mode. Specifying a negative value reverts to the default value. Default value: 20,000 events.

• period — is the amount of time, in minutes, that the event rate is measured. Setting a value of 0 disables throttle mode. Setting a negative value reverts to the default value. Default value: 120 minutes.

The following example reduces the number of events reported to 10,000 events, which is half of the default of 20,000 events:

# /opt/egenera/bin/ped -T 10000 -P 120

1-6 PM5.2_BF

Chapter 2PED Event Filters

This chapter describes how the PED applies a set of filters to each event it receives. These filters analyze events in context with the current state of the system and prevent the reporting of unwanted messages to the /var/log/messages file. For example, the filters consolidate duplicate events from each blade, indicating a blade outage into a single message in the /var/log/messages file.

The filters can be divided into the following broad categories:

• Filtering Blade Events

• Filtering Switch Events

• Filtering Fabric Events

Filtering Blade Events

The precise error daemon uses a set of filters that analyze blade events and consolidate the messages logged. For example, these filters log a single message to report that a blade is down instead of

PM5.2_BF 2-1


a message from every blade reporting that a blade is down. These filters do not log a message to report that a blade is down when it is being rebooted.

Table 2.1 outlines the purpose and specific errors logged by each blade event filter. For descriptions of each error in the table, see Chapter 3, "Error Message Descriptions."

Table 2.1 Blade Event Filters

Filter Purpose Errors Logged to the /var/log/messages File

Node Up/Node Down

Consolidates each normal node up/node down event into a single message, reports a single message for each system crash, and does not report messages for reboots initiated by administrators.

EM49157 DstNode is up

EM16385 DstNode is down unexpectedly

EM49156 DstNode is down

EM49155 Communication has been restored between SrcNode and DstNode for subsystem Name

EM24576 SrcNode cannot communicate with DstNode for subsystem Name

Intent to Reboot

Logs a blade’s intent to reboot event to the /var/log/ped.log file.

None.

vEth Node Up/vEth Node Down

Does not log any vEth node up or vEth node down error messages that occur when a blade boots or reboots.

None.

Dump Messages

Logs an error when a pBlade or cBlade has crashed and is writing a dump file, and an error when the blade finishes writing the dump file.

EM16386 SrcNode detects that DstNode is dumping to /crash_dumps/Filename.dmp on

SrcNode

EM49159 SrcNode detects that DstNode is

finished dumping to SrcNode

Blade Insertion and Removal

Logs all blade insertion and removal errors.

EM49152 SrcNode detected DstNode serial

number xxxxxxxxxx

2-2 PM5.2_BF

Messages.html

PED Event Filters

Filtering Switch Events

The precise error daemon uses a set of filters that analyze switch events and consolidate the messages logged. For example, these filters log a single message when a switch is removed instead of a message from every blade when a switch is removed.

Table 2.2 outlines the purpose and specific errors logged by each switch event filter. For descriptions of each error in the table, see Chapter 3, "Error Message Descriptions."

Table 2.2 Switch Event Filters

Intelligent Platform Management Interface (IPMI) Commands

Logs all chassis commands.

EM49153 SrcNode pid xxxxx performed a

chassis Name command to the DstNode

Disk Mapping Commands

Logs all pBlade disk mapping commands to the /var/log/ped.log file.

None.



Switch Failure Logs one error when the switch fails and one error when the switch recovers.

EM16384 SrcNode detects that SwitchNode has failed

EM49154 DstNode has recovered

PM5.2_BF 2-3

Messages.html


clan Switch Logs errors involving hardware registers in the switch, including the ipc, inport dstore, vi, and atm cell registers.

Reports the module and port numbers, which are internal links between chips in the switch.

Does not log errors if the event occurs too frequently.

Does not log link deadlock avoidance errors when the blade is down or has crashed when the event occurs.

EM16387 SwitchNode detected ipc fatal

error for DstNode

EM16388 SwitchNode detected ipc fatal

error module = m port = p

EM16389 SwitchNode detected inport dstore error for DstNode

EM16390 SwitchNode detected inport dstore

error module = m port = p

EM16391 SwitchNode detected too many vis

error for DstNode

EM16392 SwitchNode detected too many vis error module = m port = p

EM24585 SwitchNode detected link deadlock avoidance error for DstNode

EM24586 SwitchNode detected link deadlock

avoidance error for module m port p

EM24587 SwitchNode detected dropped atm

cell error in module m port p

EM24590 SwitchNode detected dropped atm cell error for DstNode

EM24591 SwitchNode detected null vi error

for DstNode

EM24588 SwitchNode detected null vi error

in module m port p

Serial Bit and Floating Data Cell

Logs errors involving hardware registers in the switch, including the serial bit and floating data cell registers.

Does not log errors if the node is down or rebooting when the event occurred.

EM24577 SwitchNode detected n serial bit errors communicating with DstNode

EM24578 SwitchNode detected n serial bit errors in module m port p

EM24589 SwitchNode detected floating data error in module m port p

EM24592 SwitchNode detected floating data error for DstNode


2-4 PM5.2_BF

PED Event Filters

Filtering Fabric Events

The precise error daemon uses a set of filters that analyze fabric events and consolidate the messages logged. For example, these filters log a single message when a bad path to a specific node is found instead of a message from every blade that complains about the path.

Table 2.3 outlines the purpose and specific errors logged by each fabric event filter. For descriptions of each error in the table, see Chapter 3, "Error Message Descriptions."

Table 2.3 Fabric Event Filters


Fabric and Intermittent Fabric

Logs errors that describe the specific piece of fabric that is broken.

EM24580 Communication error between

SrcNode and DstNode through SwitchNode for subsystem Name

EM24579 Intermittent communication error

between SrcNode and DstNode through SwitchNode for subsystem Name

Fabric Recovered

Logs only the fabric recovered messages that correspond to a reported fabric error.

EM49155 Communication has been restored between SrcNode and DstNode for subsystem Name

EM49158 Communication has been restored

between SrcNode and DstNode through SwitchNode for subsystem Name

PM5.2_BF 2-5

Messages.html


rclan Logs all rclan error messages, including descriptor errors, interrupt errors, and sequence mismatch errors.

Does not log the error if the error occurs on a path that fails the rclan self test or if the error occurs too frequently.

EM24584 SrcNode detected receive descriptor error (0xErrorData) on path n of 2 to node DstNode subsystem Name rvi r

EM24583 SrcNode detected receive interrupt error (0xErrorData) on path n of 2 to node DstNode subsystem Name rvi r

EM24581 SrcNode detected sequence mismatch with node DstNode subsystem Name rvi r

EM16401 rclan: Node Error registering memory status = 0xStatus for subsystem Name


2-6 PM5.2_BF

Chapter 3Error Message

Descriptions

This chapter describes each error message that the precise error daemon can log to the /var/log/messages file. The chapter consists of the following sections:

• Error Message Format

• Common Recovery Procedures

• Error message descriptions (organized by error number)

PM5.2_BF 3-1


Error Message Format

The precise error daemon logs error messages in the following format to the /var/log/messages file:

month day time platform-node ped: EMnnnnn: SEVERITY message

where:

• month — The month.

• day — The day of the month.

• time — The time in 24-hour format.

• platform — The name of the platform.

• node — The cBlade, which is c1 for cBlade 1, or c2 for cBlade 2.

• ped — The name of the process that is running the precise error daemon.

• EMnnnnn — The error number.

• SEVERITY — The class of error: CRITICAL, ERROR, or INFO.

• message — The details of the message.

Message and Variables

The message describes the specific blades and software components that are effected by the error. For example:

SrcNode detects that SwitchNode has failedSwitchNode detected ipc fatal error for DstNode

In each message, the components are shown as variables (in italics). Common variables include the following:

• SrcNode — The blade that detects the error, or the blade that is the source for an operation that failed.

• DstNode — The blade where an error is occurring, or the blade that is the destination for an operation that failed.

3-2 PM5.2_BF

Error Message Descriptions

• SwitchNode — The component responsible for switching packets, such as an sBlade.

Note: For the BladeFrame BF200, SwitchNode refers to a switch in the cBlade ES instead of an sBlade.

Severity The precise error daemon classifies errors according to the severity levels outlined in Table 3.1:

Table 3.1 Severity of Error Messages

Threshold Each error has a threshold. The threshold is expressed as a number of times that the error can be logged in a number of minutes (the period) for each blade or each source/destination pair. Once the threshold is reached for a specific blade or source/destination, the precise error daemon suppresses reporting any additional instances of the error to the log for the remainder of the period and logs a single EM49160 error instead. At the end of the period, the precise error daemon resets the threshold for that blade or source/destination pair and begins logging the error again. For more information, see EM49160.

Severity Error Number Description

CRITICAL EM16nnn A component failure occurred, such as a blade going down unexpectedly. Investigate the message immediately, and follow the recommended actions.

ERROR EM24nnn A software error occurred, such as a communication error between blades. Investigate the message, and follow the recommended actions.

INFO EM49nnn An event of interest occurred, such as a blade being inserted. You do not need to investigate the message, and usually no action is required.

PM5.2_BF 3-3


Causes and Actions

Each error description provides a table that lists one or more possible causes of the error. For each cause, the table outlines the actions you can take to correct the problem. If necessary, refer to PAN Manager Administrator’s Guide for instructions on rebooting and replacing blades.

Common Recovery Procedures

This section describes the following common recovery procedures that are referenced by the specific error descriptions in this chapter:

• Determining Whether the Switch or cBlade Is the Cause

• Determining Whether the pBlade or Slot Is the Cause

• Diagnosing Switch Problems

• For More Information

Determining Whether the Switch or cBlade Is the Cause

For errors where it is unclear whether SwitchNode (a switch specified in the message) or SrcNode (a cBlade specified in the message) is at fault, complete the following steps:

1. Remove and insert SwitchNode.

Note: In some cases, removing and inserting SwitchNode can cause an EM24599 error, which indicates a hung redundant virtual interface (RVI). If the EM24599 error occurs, reboot SrcNode and skip the rest of these steps.

2. If Step 1 did not fix the problem and an EM24599 error did not occur as a result of Step 1, reboot SrcNode.

a. If this fixes the problem, replace SrcNode.

b. If the problem persists, replace SwitchNode.

3-4 PM5.2_BF


Determining Whether the pBlade or Slot Is the Cause

For errors where it is unclear whether Node (a pBlade specified in the message) or the slot (the slot in the chassis where Node is installed) is at fault, complete the following steps:

1. Swap Node to a slot located in a different zone in the BladeFrame BF400 S2 or to a different slot in the BladeFrame BF200.

The pBlades reside in four zones, as follows:

• Zone A — pBlades in slots 1 – 6

• Zone B — pBlades in slots 7 – 12

• Zone C — pBlades in slots 13 – 18

• Zone D — pBlades in slots 19 – 24

Thus, if the pBlade in slot 4 is experiencing problems, you can move it to a free slot that resides in Zones B through D. If the pBlade in slot 13 is experiencing problems, you can move it to a free slot in Zone A, B, or D.

2. If the error persists after the swap, the problem is with Node and not the slot; replace Node.

3. If the error goes away after the swap in Step 2, the problem is with the slot. Replace SwitchNode.

Diagnosing Switch Problems

In many situations, the precise error daemon automatically power cycles the switch to recover. There are some situations, however, where it does not make sense for the precise error daemon to automatically power cycle the switch. For example, if the other switch is broken, you must fix the problems with the broken switch before power cycling the current switch.

To diagnose a problem with a switch:

1. Determine if there are other errors that accompanied the switch error and fix those errors first.

2. If no other errors accompanied the switch error, verify that the other switch is running with the following PAN Manager command. If the other switch is broken, you must fix it first.bframe -l

PM5.2_BF 3-5


Caution: If you power cycle the only running switch, you interrupt all applications and services running on the platform.

3. If both switches are running, remove and insert SwitchNode. If this fixes the problem, skip the rest of these steps:

4. If removing and inserting SwitchNode does not fix the problem, replace SwitchNode.

For More Information

For instructions on rebooting and replacing blades, see PAN Manager Administrator’s Guide.

3-6 PM5.2_BF


EM16384

Message SrcNode detects that SwitchNode may have failed

where:

SrcNode — control blade 1 or 2

SwitchNode — switch blade 1 or 2

Note: The precise error daemon does not automatically power cycle the SwitchNode or reboot the cBlade when this error occurs.

Severity CRITICAL

RecoveryPossible Cause Recommended Action

The cBlade cannot communicate with SwitchNode. It is unclear whether the cBlade or SwitchNode is at fault.

Complete the steps in “Determining Whether the Switch or cBlade Is the Cause” on page 3-4.

If an EM49153 error preceded this EM16384 error, an administrator powered off SwitchNode.

Check the /var/log/messages file on both cBlades to be sure which errors accompanied this error. Power on SwitchNode.

If both an EM49153 and EM16399 error preceded this EM16384 error, the precise error daemon powered off SwitchNode because of serious recurring errors.

Check the /var/log/messages file on both cBlades to be sure which errors accompanied this error. Replace SwitchNode.

If one or more EM49162 errors preceded this EM16384 error, SwitchNode has been removed.

Replace SwitchNode.

PM5.2_BF 3-7


EM16385

Message DstNode is down unexpectedly

where:

DstNode — control blade 1 or 2, or processing blade 1 through n

Severity CRITICAL


If an EM16386 error preceded this EM16385 error, the cBlade or pBlade crashed and dumped.

Look for a crash dump file in the/crash_dumps directory on one of the cBlades, and send it to Fujitsu Siemens Computers customer support for analysis. The blade reboots automatically.

If an EM49170 error followed this EM16385 error, the pServer running on the pBlade crashed and dumped using the kdump facility.

Consult the pServer Guide for the specific pServer and the OS documentation to determine the configured location of the crash dump file. Send the crash dump file to Fujitsu Siemens Computers customer support for analysis. The blade reboots automatically.

If this EM16385 error is followed by an EM49153 error, an administrator powered off the cBlade or pBlade without shutting down first.

None. PAN Manager reboots the blade.

3-8 PM5.2_BF


If this EM16385 error is preceded by an EM49153 error, a nonmaskable interrupt (NMI) occurred that powered off the cBlade or pBlade without shutting down first.

See the section on saving cBlade state information with the save_state command in the PAN Manager Administrator’s Guide, and forward the results to Fujitsu Siemens Computers customer support.

If this EM16385 error is followed by an EM49162 error, an administrator removed the cBlade or pBlade from the BladeFrame.

Replace the cBlade or pBlade. If an administrator did not remove the blade, contact Fujitsu Siemens Computers customer support about upgrading the blade’s firmware.

A communication failure occurred on both paths to the cBlade or pBlade.

If this EM16385 followed one or more communication errors, fix those errors.

The precise error daemon restarted during the reboot of the cBlade or pBlade.

None.

Possible Cause Recommended Action

PM5.2_BF 3-9


EM16386

Message SrcNode detects that DstNode is dumping to /crash_dumps/Filename.dmp on SrcNode

where:


DstNode — control blade 1 or 2, processing blade 1 through n

Filename — the name of the crash dump file

Severity CRITICAL


The pBlade or cBlade crashed and is performing a crash dump because of an operating system panic.

Send the crash dump file to Fujitsu Siemens Computers customer support for analysis.

If an EM49153 (IPMI NMI error message) preceded this EM16386 error message, an administrator issued an IPMI NMI command to the pBlade.

Send the crash dump file to Fujitsu Siemens Computers customer support for analysis.

3-10 PM5.2_BF


EM16387

Message SwitchNode detected ipc fatal error for DstNode

where:


DstNode — control blade 1 or 2, processing blade 1 through n

Severity CRITICAL


A switch failure occurred involving the ipc hardware register in SwitchNode. SwitchNode will automatically be reset if it is safe to do so.

If you see a single instance of this error, no action is required.

Multiple instances of this error could indicate that SwitchNode may fail and need replacement. Contact Fujitsu Siemens Computers customer support.

PM5.2_BF 3-11


EM16388

Message SwitchNode detected ipc fatal error module = m port = p

where:


m — the number of the internal module in SwitchNode

p — the number of the internal port in SwitchNode, which is between port 0 and port 7

The internal module number and port number specify internal links between chips in SwitchNode. These component numbers do not affect the action taken but can be useful in tracking a pattern of errors.

Severity CRITICAL


A switch failure occurred involving the ipc hardware register in SwitchNode. SwitchNode will automatically be reset if it is safe to do so.



3-12 PM5.2_BF


EM16389

Message SwitchNode detected inport dstore error for DstNode

where:



Severity CRITICAL


A switch failure occurred involving the inport dstore hardware register of SwitchNode. SwitchNode will automatically be reset if it is safe to do so.



PM5.2_BF 3-13


EM16390

Message SwitchNode detected inport dstore error module = m port = p

where:





Severity CRITICAL


A switch failure occurred involving the inport dstore hardware register of SwitchNode. The SwitchNode will automatically be reset if it is safe to do so.



3-14 PM5.2_BF


EM16391

Message SwitchNode detected too many vis error for DstNode

where:



Severity CRITICAL


A switch failure occurred involving the vi hardware register in SwitchNode. SwitchNode will automatically be reset if it is safe to do so.



PM5.2_BF 3-15


EM16392

Message SwitchNode detected too many vis error module = m port = p

where:





Severity CRITICAL


A switch failure occurred involving the vi hardware register in SwitchNode. The switch will automatically be reset if it is safe to do so.



3-16 PM5.2_BF


EM16393

Message SwitchNode needs to be reset but cannot be because some paths through BrokenSwitchNode are not working

where:


BrokenSwitchNode — switch blade 1 or 2

Severity CRITICAL


Other errors, such as EM16396 errors, preceded this EM16393 error. It is not safe to replace SwitchNode.

Fix the other errors that preceded this EM16393 error to resolve the problems with the BrokenSwitchNode.

PM5.2_BF 3-17


EM16394

Message SwitchNode needs to be reset due to errors relating to Node but has been reset too frequently: SwitchNode will not be reset automatically

where:


Node — control blade 1 or 2, or processing blade 1 through n

Severity CRITICAL


There are hardware problems with either SwitchNode or Node.

If Node is a cBlade, complete the steps in “Determining Whether the Switch or cBlade Is the Cause” on page 3-4.

If Node is a pBlade, complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

There are hardware problems with a Giganet connection on the cBlade specified by Node.

Replace SwitchNode. If the problem persists, replace the cBlade.

3-18 PM5.2_BF


EM16395

Message SwitchNode cannot be reset because BrokenSwitchNode is down

where:


BrokenSwitchNode — switch blade 1 or 2

Severity CRITICAL


There are errors on the SwitchNode that might be fixed by reset. However, the BrokenSwitchNode is not working, so it would not be safe to reset SwitchNode at this time.

Fix the problem with the BrokenSwitchNode. When BrokenSwitchNode is working, the precise error daemon automatically power cycles SwitchNode.

PM5.2_BF 3-19


EM16396

Message path through SwitchNode is the only working path for DstNode

where:



Severity CRITICAL


There are errors on SwitchNode that might be fixed by performing a switch reset. However, the DstNode does not have a working path through the other switch, so it would not be safe to reset SwitchNode at this time.

If DstNode is a pBlade, complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

If DstNode is a cBlade, try to fix the path problem so DstNode has a working path through the other switch. If you cannot fix DstNode, find a time that the loss of DstNode can be tolerated. Complete the steps in “Determining Whether the Switch or cBlade Is the Cause” on page 3-4.

3-20 PM5.2_BF


EM16397

Message SwitchNode needs to be reset but cannot be because it is the only working switch

where:


Severity CRITICAL


There are errors on the SwitchNode that might be fixed by performing a switch reset. However, the other switch is not working, so it would not be safe to reset SwitchNode at this time.

Fix the problem with the other switch that is broken. When the broken switch is working again, the precise error daemon automatically power cycles SwitchNode.

PM5.2_BF 3-21


EM16398

Message SwitchNode has been reset due to previously reported errors

where:


Severity CRITICAL


The frequency or type of errors that preceded this EM16398 error caused the precise error daemon to reset SwitchNode.

None.

3-22 PM5.2_BF


EM16399

Message SwitchNode has been powered off due to repeated failures

where:


Severity CRITICAL


SwitchNode generated too many errors, so the precise error daemon turned off SwitchNode.

See the error messages in the /var/log/messages file that preceded this EM16399 error and fix them. This EM16399 error is not always caused by a failed switch.

PM5.2_BF 3-23


EM16400

Message SwitchNode has been reset due to previously reported errors relating to Node

where:


Node — processing blade 1 through n

Severity CRITICAL


The precise error daemon needed to reset SwitchNode because of error messages that preceded this error.

None.

If this error persists and is followed by an EM16399 error in the /var/log/messages file, SwitchNode detects errors caused by Node.

Complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

3-24 PM5.2_BF


EM16401

Message rclan: Node Error registering memory status = 0xStatus for subsystem Name

where:

Node — processing blade 1 through n

Status — the status of the memory, as a hexadecimal value

Name — the subsystem name: rpc, veth, network, or giscsci

In some cases, the subsystem Name can indicate the source of the error.

Severity CRITICAL


The precise error daemon detected a low memory condition or a software error on Node, which could be the result of a software application that is using an unusually large amount of memory or executing an infinite loop.

Check whether the system is still running.

• If the system is running normally, no action is required.

• If the system is not running normally or has crashed, contact Fujitsu Siemens Computers customer support. In some cases, the Status and Name can help narrow down the problem.

PM5.2_BF 3-25


EM16402

Message ped: SrcNode has entered throttle mode

where:


Severity CRITICAL


The precise error daemon detected events that exceeded the threshold during the specified time period. The precise error daemon has entered throttle mode, and is suppressing the reporting of some EVnnnnn events to the /var/log/ped.log file. This does not affect the reporting of EMnnnnn errors to the /var/log/messages file.


Multiple instances of this error followed by the EM16403 error could indicate that you need to change the threshold for throttle mode. For more information, see “Controlling the Size of the ped.log File” on page 1-4.

3-26 PM5.2_BF


EM16403

Message ped: SrcNode has left throttle mode

where:


Severity CRITICAL


The precise error daemon detected that the number of events is within the threshold during the specified time period. The precise error daemon is no longer in throttle mode and is reporting all EVnnnnn events to the /var/log/ped.log file. This does not affect the reporting of EMnnnnn errors to the /var/log/messages file.


Multiple instances of this error preceded by the EM16402 error could indicate that you need to change the threshold for throttle mode. For more information, see “Controlling the Size of the ped.log File” on page 1-4.

PM5.2_BF 3-27


EM24576

Message SrcNode cannot communicate with DstNode for subsystem Name

where:


DstNode — processing blade 1 through n



Severity ERROR


There are no working communication paths between SrcNode and DstNode. It is difficult to determine the hardware or software component that causes a communications-related error.


If the problem persists, complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

3-28 PM5.2_BF


EM24577

Message SwitchNode detected n serial bit errors communicating with DstNode

where:


n — the number of serial bit errors


Severity ERROR


DstNode is causing communication errors with SwitchNode.

If DstNode is a cBlade, complete the steps in “Determining Whether the Switch or cBlade Is the Cause” on page 3-4.

If DstNode is a pBlade, complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

SwitchNode is causing communication errors with DstNode because of problems with the serial bit hardware register in SwitchNode.

Replace SwitchNode.

There might be wiring problems in the BladePlane, although this is extremely rare.

Contact Fujitsu Siemens Computers customer support.

PM5.2_BF 3-29


EM24578

Message SwitchNode detected n serial bit errors in module m port p

where:


n — the number of serial bit errors



Note: The precise error daemon does not automatically reboot SwitchNode when this error occurs.

The internal module number and port number specify internal links between chips in SwitchNode. These component numbers do not affect the action taken, but can be useful in tracking a pattern of errors.

Severity ERROR


SwitchNode is having internal communication problems with its serial bit hardware register.

If this error occurs once, no action is required. If multiple instances of this error occur, complete the steps in “Diagnosing Switch Problems” on page 3-5.

3-30 PM5.2_BF


EM24579

Message Intermittent communication error between SrcNode and DstNode through SwitchNode for subsystem Name

where:




Name — The subsystem name: rpc, veth, network, or giscsci


Severity ERROR


Data corruption occurred as a result of communication between SrcNode and DstNode through SwitchNode.


If the error persists, complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

PM5.2_BF 3-31


EM24580

Message Communication error between SrcNode and DstNode through SwitchNode for subsystem Name

where:




Name — The subsystem name: rpc, veth, network, or giscsci


Severity ERROR


Communication was lost between SrcNode and DstNode through SwitchNode.



3-32 PM5.2_BF


EM24581

Message SrcNode detected sequence mismatch with node DstNode subsystem Name rvi r

where:

SrcNode — control blade 1 or 2, or processing blade 1 through n



r — the redundant virtual interface (RVI) number

In some cases, the subsystem Name can indicate the source of the error. The RVI number r does not affect the action taken, but can be useful in tracking a pattern of errors.

Severity ERROR


Data corruption occurred on the path between SrcNode and DstNode. The system has recovered from this transient error.

Monitor the system for more EM24581 error messages.

A software initialization problem occurred that does not require a reboot of the system.

If you see 10 or more of these EM24581 errors, follow the recommended actions for EM24579.

PM5.2_BF 3-33


EM24582

Message SrcNode detected illegal signal command 0xSignalData from node DstNode subsystem Name rvi r

where:


SignalData — the signal data transmitted from the Giganet Ethernet port, as a hexadecimal value





Severity ERROR


Data corruption occurred on the path from SrcNode to DstNode.



3-34 PM5.2_BF


EM24583

Message SrcNode detected receive interrupt error (0xErrorData) on path n of 2 to node DstNode subsystem Name rvi r

where:

SrcNode — control blade 1 or 2, or processing blade 1 throughn

ErrorData — the error data, as a hexadecimal value

n — the path (1 or 2)




In some cases, the Name and path number n can indicate the source of the error. The RVI number r does not affect the action taken, but can be useful in tracking a pattern of errors.

Severity ERROR

PM5.2_BF 3-35


Recovery This error can be caused by hardware or software, and the value displayed in the ErrorData field of the message determines the cause and recommended action.


A hardware error caused data corruption on the path from SrcNode to DstNode. The ErrorData field has bit 20 set (0x00100000).



A hardware error caused data corruption on the path from SrcNode to DstNode. The ErrorData field does not have bit 20 set (0x00100000), and other hardware errors accompanied this error.

A software error caused data corruption on the path from SrcNode to DstNode. The ErrorData field is set to 0x840000 or 0x940000.

A software error caused data corruption on the path from SrcNode to DstNode. The ErrorData field does not have bit 20 set (0x00100000), and no other errors accompanied this error.

This error is harmless, and no action is required.

3-36 PM5.2_BF


EM24584

Message SrcNode detected receive descriptor error (0xErrorData) on path n of 2 to node DstNode subsystem name rvi r

where:


ErrorData — the error data, as a hexadecimal value





In some cases, the subsystem Name and path number n can indicate the source of the error. The RVI number r does not affect the action taken, but can be useful in tracking a pattern of errors.

Severity ERROR


Data corruption occurred on the path from SrcNode to DstNode. This error indicates that the data was corrupted when transmitted to the bus and is usually caused by defective hardware.

Reboot SrcNode. If the error persists, reboot DstNode.



PM5.2_BF 3-37


EM24585

Message SwitchNode detected link deadlock avoidance error for DstNode

where:



Severity ERROR


The giganet chip on DstNode is unable to write data across the Peripheral Component Interconnect (PCI) bus. Typically, hardware problems with the pBlade cause this problem.

Complete the steps in “Determining Whether the pBlade or Slot Is the Cause” on page 3-5.

There are problems with SwitchNode.

Complete the steps in “Diagnosing Switch Problems” on page 3-5.

3-38 PM5.2_BF


EM24586

Message SwitchNode detected link deadlock avoidance error for module m port p

where:





Severity ERROR


There are problems with SwitchNode.


PM5.2_BF 3-39


EM24587

Message SwitchNode detected dropped atm cell error in module m port p

where:





Severity ERROR


There are problems with the atm cell hardware register in SwitchNode.


3-40 PM5.2_BF


EM24588

Message SwitchNode detected null vi error in module m port p

where:





Severity ERROR


There are problems with the vi hardware register in SwitchNode, or the programming on the switch is incomplete.


PM5.2_BF 3-41


EM24589

Message SwitchNode detected floating data error in module m port p

where:





Severity ERROR


There are problems with the floating data hardware register in SwitchNode.


3-42 PM5.2_BF


EM24590

Message SwitchNode detected dropped atm cell error for DstNode

where:



Severity ERROR


There are problems with the atm cell hardware register in SwitchNode.


PM5.2_BF 3-43


EM24591

Message SwitchNode detected null vi error for DstNode

where:



Severity ERROR


There are problems with the vi hardware register in SwitchNode


The programming on the switch is incomplete.

Power cycle the cBlade, which reprograms the switch.

3-44 PM5.2_BF


EM24592

Message SwitchNode detected floating data error for DstNode

where:



Severity ERROR


There are problems with the floating data hardware register in SwitchNode.


There are problems with DstNode. Reboot the DstNode. If the problem persists, replace DstNode.

PM5.2_BF 3-45


EM24593

Message Oneway communication error on path from SrcNode to DstNode through SwitchNode for subsystem Name

where:






Severity ERROR


The DstNode is not receiving data from the SrcNode through SwitchNode, even though SrcNode is receiving data from DstNode through SwitchNode. This error occurs when the pServer running on DstNode is an older software version that does not support the precise error daemon, and the SwitchNode was power cycled to recover from a switch error.

Contact Fujitsu Siemens Computers customer support about upgrading the pServer on DstNode to a release that supports the precise error daemon.


If the problem persists, reboot DstNode.

3-46 PM5.2_BF


EM24594

Message The paths between SrcNode and DstNode are cross-wired

where:



Severity ERROR


The network connections at the SrcNode are cross-wired (that is, wired incorrectly).

For all causes, the precise error daemon logs these EM24594 messages in groups. The failed blade is the one blade in common to all the EM24594 messages. Replace the failed blade.

The network connections at the DstNode are cross-wired.

The BladePlane is cross-wired (extremely unlikely).

PM5.2_BF 3-47


EM24595

Message kernel: Remote log message String

where:

String — a message format from a software release that precedes the introduction of the precise error daemon

Severity ERROR


A kernel that does not support the precise error daemon logged a message in an old message format. The precise error daemon wraps the old error message in String so it can apply a threshold to the message.

Interpret the old error message in String.

3-48 PM5.2_BF


EM24596

Message Precise error daemon on SrcNode could not handle event (EVnnnnn) from DstNode

where:


EVnnnnn — the event number


Severity ERROR


The version of the precise error daemon running on the cBlade is not compatible with the kernel running on DstNode. The system has not been configured correctly: a pBlade is running a newer version of software that is not supported by the current version of software running on the cBlade.


PM5.2_BF 3-49


EM24597

Message Precise error daemon on SrcNode could not find event (EVnnnnn) from DstNode in message table (Filename)

where:




Filename — the file that contains the message table

The precise error daemon uses the message table to determine the message associated with an event.

Severity ERROR




3-50 PM5.2_BF


EM24598

Message Precise error daemon on SrcNode could not find argument (Name) in event (EVnnnnn) from DstNode

where:


Name — the name of the argument



Severity ERROR




PM5.2_BF 3-51


EM24599

Message SrcNode detected hung rvi on path n of 2 to node DstNode subsystem Name rvi r: Reboot SrcNode to clear this error

where:


n — the number of the path (1 or 2)



r — the number of the redundant virtual interface (RVI)

The path number n, subsystem Name, and RVI number r can indicate which SwitchNode (switch blade 1 or 2) caused the hung RVI.

Severity ERROR


Powering off SwitchNode. Reboot SrcNode.

Powering on SwitchNode.

Power cycling SwitchNode.

Resetting SwitchNode.

3-52 PM5.2_BF


EM24600

Message SrcNode attempting to recover from sequence mismatch with node DstNode subsystem Name rvi r

where:






Severity INFO


At least 1000 instances of EM24581 were reported, and the system is attempting to correct the error.

If this message keeps recurring, contact Fujitsu Siemens Computers customer support.

PM5.2_BF 3-53


EM33280

Message The number of receive descriptor errors (errors) detected by DstNode exceeds the threshold of n in 24 hours

Severity WARNING

3-54 PM5.2_BF


EM49152

Message SrcNode detected DstNode serial number xxxxxxxxxx

where:


DstNode — control blade 1 or 2, switch blade 1 or 2, or processing blade 1 through n

xxxxxxxxxx — the serial number of DstNode

Severity INFO

Recovery

See Also The section on saving cBlade state information with the save_state command in the PAN Manager Administrator’s Guide


A user inserted a blade. None.

The cBlade performed a start-up operation.

None.

The cBlade specified by SrcNode detected the blade specified by DstNode through an IPMI operation. There will be additional messages indicating that the blade was removed. If the blade was not actually removed and reinserted, the fault is likely to be with the power supplies in the pBlade, with a slight chance that the fault is in the cBlade.

Use the save_state command to collect additional diagnostic information and forward the results to Fujitsu Siemens Computers customer support for analysis.

PM5.2_BF 3-55


EM49153

Message SrcNode pid xxxxx performed a chassis Name command to the DstNode

where:


xxxxx — the process ID (or PID) of the process that issues the IPMI chassis command

Name — the name of the IPMI chassis command: off, on, cycle, reset, or nmi


Severity INFO


PAN Manager issued an IPMI chassis command.

None.

An administrator issued an IPMI chassis command on DstNode from the cBlade at SrcNode.

None.

3-56 PM5.2_BF


EM49154

Message SwitchNode has recovered

where:


Severity INFO


SwitchNode recovered and has been reprogrammed.

None.

PM5.2_BF 3-57


EM49155

Message Communication has been restored between SrcNode and DstNode for subsystem Name

where:





Note: This EM49155 error is similar to the EM49158 error, except it does not provide the name of the switch.

Severity INFO


The blade is performing a boot operation.

None.

3-58 PM5.2_BF


EM49156 (BladeFrame BF200)

Message Node is down

where:

Node — control blade 1 or 2, the switch in control blade 1 or 2, or processing blade 1 through 6

Severity INFO


Node was rebooted. None.

Node was halted.

Node was reset.

Node was power cycled.

Node was powered off.

PM5.2_BF 3-59


EM49157

Message Node is up

where:

Node — control blade 1 or 2, switch blade 1 or 2, or processing blade 1 through n

Severity INFO


Node was booted. None.

3-60 PM5.2_BF


EM49158

Message Communication has been restored between SrcNode and DstNode through SwitchNode for subsystem Name

where:






Note: This EM49158 error is similar to the EM49155 error, except it provides the name of the switch.

Severity INFO


Communication is restored between SrcNode and DstNode.

None.

PM5.2_BF 3-61


EM49159

Message SrcNode detects that DstNode is finished dumping to SrcNode

where:



Severity INFO


The precise error daemon logs this informational message when DstNode completes its dump operation to SrcNode.

An operating system panic occurred on a blade, a dump file was written in the /crash_dumps directory. Forward the crash dump file to Fujitsu Siemens Computers customer support for analysis.

3-62 PM5.2_BF


EM49160

Message Message EMnnnnn exceeded threshold of n in m minutes for SrcNode to DstNode; will be disabled until time

where:

EMnnnnn — the error number

n, m — the number of times (n) the message can be logged in a time period (m minutes)



time — the time at which the specified error message’s threshold is reset

Threshold Example

The EM49160 error can be reported with any error that has a threshold. As an example, the EM16386 error has a threshold of 2 times in 60 minutes. If an EM16386 error occurs 4 times in 30 minutes, the precise error daemon does the following:

1. Logs the EM16386 error two times.

2. Suppresses the logging of the additional two instances of the EM16386 error.

3. Logs this EM49160 error to indicate that the threshold for the EM16386 error was exceeded.

4. Waits 30 minutes (the time remaining in the period), and then resets the threshold for the EM16386 error.

5. Begins logging the EM16386 message again until the threshold is reached.

Severity INFO

PM5.2_BF 3-63



The precise error daemon logs this error when the error message specified by EMnnnnn exceeds the specified threshold. For example, the precise error daemon would log error EM49160 if error EM24586 occurred more than 2 times in 60 minutes.

See the description of the EMnnnnn error message that was referenced by this EM49160 error to determine whether further action is required. When a threshold has been exceeded for an error, that error has been occurring for a period of time, and you might need to look further back in the /var/log/messages file to find the root cause.

3-64 PM5.2_BF


EM49161

Message Precise error daemon restarted on SrcNode

where:


Severity INFO


SrcNode has recently rebooted. None. If a blade was removed, you might want to reinsert it.

The precise error daemon on SrcNode crashed and restarted.

None.

The precise error daemon on SrcNode was upgraded.

None.

PM5.2_BF 3-65


EM49162

Message SrcNode detected DstNode serial number xxxxxxxxxx has been removed

where:



xxxxxxxxxx — the serial number of DstNode

Severity INFO


The DstNode was removed from the chassis.

None. You might want to reinsert DstNode.

The DstNode is experiencing IPMI failure.

Contact Fujitsu Siemens Computers customer support for assistance.

The SrcNode is experiencing IPMI failure.


3-66 PM5.2_BF


EM49163

Message SrcNode detected high interrupt latency on path n of 2 to node DstNode

where:




Severity INFO


SrcNode is experiencing high interrupt latency.


PM5.2_BF 3-67


EM49164 and EM49165

Message EM49164 SrcNode detected DstNode lost n eventsEM49165 SrcNode detected DstNode lost 1 event

where:



n — the number of events lost

Severity INFO


SrcNode is generating a lot of events, and DstNode lost one (EM49165) or more (EM49164) events.

Determine the reason for all the events on SrcNode by examining the /var/log/messages and /var/log/ped.log files, and fix any problems. Lost events may cause the precise error daemon to report incorrect errors about SrcNode.

3-68 PM5.2_BF


EM49166

Message SrcNode detected lost events from DstNode

where:



Severity INFO


SrcNode is generating a lot of events.

Determine the reason for all the events on SrcNode by examining the /var/log/messages and /var/log/ped.log files, and fix any problems. Lost events may cause the precise error daemon to report incorrect errors about SrcNode. If this error persists, contact Fujitsu Siemens Computers customer support for further assistance.

The precise error daemon on SrcNode was not running for some time.

No action is required. Lost events may cause the precise error daemon to report incorrect errors about SrcNode.

SrcNode is so busy that the precise error daemon cannot keep up with the normal event flow.

Determine why SrcNode is busy by examining what processes are running. Lost events may cause the precise error daemon to report incorrect errors about SrcNode.

PM5.2_BF 3-69


EM49167

Message SrcNode detected DstNode is using remote log interface. ped will use compatibility mode for DstNode

where:



Severity INFO


The pServer kernel does not support the precise error daemon.

None. If you upgrade the pServer to a kernel that supports the precise error daemon, you won’t see this message again.

3-70 PM5.2_BF


EM49168

Message SrcNode resetting send for path n of 2 to node DstNode

where:




Severity INFO


The rclan driver module timed out while attempting to send a message.

If a single instance of this error occurs, no action is required. If multiple instances of this error occur, contact Fujitsu Siemens Computers customer support.

PM5.2_BF 3-71


EM49169

Message SrcNode has detected that control blade version is not up to date

where:

SrcNode — processing blade 1 through n

Severity INFO


A Windows pServer detected that the cBlade is running an outdated version of PAN Manager that does not fully support the pServer release.

The Windows pServer continues running as long as it can. Upgrade the cBlade to a revision that fully supports the pServer release. See http://www.fujitsu-siemens.com/support for release compatibility information.

3-72 PM5.2_BF


EM49170

Message SrcNode has booted into a dump kernel

where:

SrcNode — processing blade 1 through n

Severity INFO


This EM49160 error is preceded by an EM16385 error and indicates that the pServer running on the pBlade crashed and dumped using the kdump facility.

Consult the pServer Guide for the specific pServer and the OS documentation to determine the configured location of the crash dump file. Send the crash dump file to Fujitsu Siemens Computers customer support for analysis. The blade reboots automatically.

PM5.2_BF 3-73


EM49171

Message SrcNode pid xxxxx action PR on device ID

where:


xxxxx — the PID of the process that accessed the SCSI-3 persistent reservation (PR)

action — the action performed for the PR, which is enabled, disabled, or cleared

ID — the unique ID (UID) for the LUN associated with the PR, such as EMC-SYMMETRIX-WWN-6006048000038794003753594d323537

Severity INFO


PAN Manager on SrcNode enabled, disabled, or cleared a PR for the specified LUN on behalf of the specified process. This message occurs in the following situations:

• When the administrator configures the SCSI-3 PR feature for a LUN.

• When SrcNode reboots (one message per LUN).

None.

3-74 PM5.2_BF


EM49172

Message SrcNode detected Local Director version n is state

where:


n — the version of the local virtual LUN (vLUN) director, expressed as an integer

state — the state of the local vLUN director, which is either master or member

The master vLUN director communicates with and manages the vLUNs that carry out SCSI-3 PRs. The master vLUN director could be running on SrcNode (the local vLUN director) or on the other cBlade (the remote vLUN director). Should the master vLUN director fail, the member vLUN director becomes the master.

Severity INFO


The vLUN director service successfully started on SrcNode. This message occurs whenever SrcNode boots.

None.

PM5.2_BF 3-75


EM49173

Message SrcNode detected Remote Director version n is state

where:


n — the version of the Remote Director, expressed as an integer

state — the state of the Remote Director, which is either master or member

The master vLUN director communicates with and manages the vLUNs that carry out SCSI-3 PRs. The master vLUN director could be running on SrcNode (the local vLUN director) or on the other cBlade (the remote vLUN director). Should the master vLUN director fail, the member vLUN director becomes the master.

Severity INFO


The virtual LUN (vLUN) director service successfully started on the other cBlade. This message occurs whenever SrcNode boots.

None.

3-76 PM5.2_BF


EM49174

Message SrcNode detected Remote Director is not responding

where:


Severity INFO


The local vLUN director service on SrcNode cannot communicate with the remote vLUN director service on the other cBlade. If this condition persists and SrcNode is the member vLUN director, this EM49174 error is followed by an EM49175 error.

None.

PM5.2_BF 3-77


EM49175

Message SrcNode detected Master Director is out of contact...promoting local vLUNs

where:


Severity INFO


A failover of the vLUN director service occurred. This EM49175 error is usually preceded by one or more EM49174 errors. SrcNode transitions from the member vLUN director to the master vLUN director, and the vLUNs on SrcNode transition from member status to master status.

None.

3-78 PM5.2_BF

Index

B

blade events, filtering 2-1BladePlane, diagnosing problems 3-5,

3-29, 3-47

C

cBladesdiagnosing errors 3-4

chassis commands 2-3, 3-10, 3-56, 3-66clan switch filter 2-4common recovery procedures 3-4controlling the size of ped.log 1-4crash dumps 2-2, 3-8, 3-10, 3-25, 3-62CRITICAL error messages 3-3

D

diagnosing errors 3-4disks

disk mapping commands filter 2-3DstNode variable 3-2dump messages filter 2-2

E

entering throttle mode 1-5, 3-26error message format 3-2ERROR messages 3-3events 1-2

filters 1-2, 2-1suppressing 1-5thresholds 1-4

exiting throttle mode 1-5, 3-27

F

fabric events 1-2, 2-5filters 1-2, 2-1floating data cell filter 2-4format of error messages 3-2

G

Giganet connections 3-47

I

INFO error messages 3-3

PM5.2_BF Index-1


inserting blade filter 2-2intent to reboot filter 2-2intermittent fabric filter 2-5IPMI commands 2-3, 3-10, 3-56, 3-66

L

limiting the size of ped.log 1-6

M

messages fileerror message format 3-2precise error daemon 1-2

N

network fabric 1-2node up/node down filter 2-2

P

pBladesdiagnosing errors 3-5

ped.log file 1-2controlling size 1-4reducing size 1-6threshold 1-4

power cycling switches 3-5power supplies, diagnosing problems

3-55precise error daemon 1-2

error message format 3-2filters 2-1power cycling switches 3-5startup 3-65

throttle mode 1-5, 3-26, 3-27

R

rclan filter 2-6recovery procedures 3-4reducing the size of ped.log 1-4, 1-6removing blade filter 2-2

S

sBladesdiagnosing errors 3-4, 3-5

serial bit filter 2-4severity of error messages 3-3slots, diagnosing errors 3-5software versions 3-46, 3-49, 3-50,

3-51, 3-70SrcNode variable 3-2state model 1-2suppressing

errors 3-63events 1-5, 3-26

switchesdiagnosing errors 3-4, 3-5event filters 2-3

SwitchNode variable 3-3

T

thresholds 1-4, 3-3, 3-26, 3-63throttle mode 1-4, 3-26, 3-27

V

/var/log/messages file 1-2, 3-2

Index-2 PM5.2_BF

Index

/var/log/ped.log file 1-2version mismatch 3-46, 3-49, 3-50,

3-51, 3-70vEth filter 2-2

Z

zones 3-5

PM5.2_BF Index-3


Index-4 PM5.2_BF

Information on this document On April 1, 2009, Fujitsu became the sole owner of Fujitsu Siemens Compu-ters. This new subsidiary of Fujitsu has been renamed Fujitsu Technology So-lutions.

This document from the document archive refers to a product version which was released a considerable time ago or which is no longer marketed.

Please note that all company references and copyrights in this document have been legally transferred to Fujitsu Technology Solutions.

Contact and support addresses will now be offered by Fujitsu Technology So-lutions and have the format …@ts.fujitsu.com.

The Internet pages of Fujitsu Technology Solutions are available at http://ts.fujitsu.com/... and the user documentation at http://manuals.ts.fujitsu.com.

Copyright Fujitsu Technology Solutions, 2009

Hinweise zum vorliegenden Dokument Zum 1. April 2009 ist Fujitsu Siemens Computers in den alleinigen Besitz von Fujitsu übergegangen. Diese neue Tochtergesellschaft von Fujitsu trägt seit-dem den Namen Fujitsu Technology Solutions.

Das vorliegende Dokument aus dem Dokumentenarchiv bezieht sich auf eine bereits vor längerer Zeit freigegebene oder nicht mehr im Vertrieb befindliche Produktversion.

Bitte beachten Sie, dass alle Firmenbezüge und Copyrights im vorliegenden Dokument rechtlich auf Fujitsu Technology Solutions übergegangen sind.

Kontakt- und Supportadressen werden nun von Fujitsu Technology Solutions angeboten und haben die Form …@ts.fujitsu.com.

Die Internetseiten von Fujitsu Technology Solutions finden Sie unter http://de.ts.fujitsu.com/..., und unter http://manuals.ts.fujitsu.com finden Sie die Benutzerdokumentation.

Copyright Fujitsu Technology Solutions, 2009

pan manager ped message reference -...

Documents