dellcert - blades training

1 | P a g e B l a d e S e r v e r C o n f i g u r a t i o n & T r o u b l e s h o o t i n g

DellTM Blade Server Configuration and Troubleshooting

Contents Introduction .................................................................................................................................................. 8

Course Safety ............................................................................................................................................ 8

Electrostatic Discharge .............................................................................................................................. 8

Module 1: Blade System Technology .......................................................................................................... 10

Introduction ............................................................................................................................................ 10

Overview of Blade Technology ............................................................................................................... 10

Blade or Blade Server .......................................................................................................................... 11

Blade System ....................................................................................................................................... 11

Common Terms ....................................................................................................................................... 12

Fabric ................................................................................................................................................... 12

Lane ..................................................................................................................................................... 12

Link ...................................................................................................................................................... 12

Port ...................................................................................................................................................... 13

Product Lines ........................................................................................................................................... 13

Summary ................................................................................................................................................. 13

Module 2: Dell Blade System Enclosures .................................................................................................... 14

Module Objectives .................................................................................................................................. 14

M1000e Modular Blade Enclosure: Enclosure Basics ......................................................................... 14

Enclosure Overview ................................................................................................................................ 14

Basic Configuration and Expansion ..................................................................................................... 15

The Front of the Enclosure .................................................................................................................. 16

The Back of the Enclosure ................................................................................................................... 16

Cabling Order ...................................................................................................................................... 17

Cooling ................................................................................................................................................ 18

Enclosure Components ........................................................................................................................... 19


Chassis Management Controller (CMC) .............................................................................................. 19

iDRAC .................................................................................................................................................. 20

iKVM .................................................................................................................................................... 20

IOMs .................................................................................................................................................... 21

Fans ..................................................................................................................................................... 21

Midplane ............................................................................................................................................. 23

PSUs .................................................................................................................................................... 24

Power Distribution Units ..................................................................................................................... 24

Hot‐Swappable Components .............................................................................................................. 25

I/O Modules ............................................................................................................................................ 26

Topic Objectives .................................................................................................................................. 26

Fabrics ................................................................................................................................................. 26

Pass‐through Modules ........................................................................................................................ 27

Choosing IOMs .................................................................................................................................... 28

Fabric Consistency Checking ............................................................................................................... 28

Stacking IOMs ..................................................................................................................................... 29

PowerConnectTM M6220 Switch .......................................................................................................... 30

Cisco Switch ........................................................................................................................................ 31

Redundancy ............................................................................................................................................ 32

CMC Redundancy ................................................................................................................................ 32

Fabric Redundancy .............................................................................................................................. 32

Fan Redundancy .................................................................................................................................. 33

PSU Redundancy ................................................................................................................................. 33

Summary ................................................................................................................................................. 34

Module 3: Key Components ....................................................................................................................... 37


Common Features ................................................................................................................................... 37

Blades .................................................................................................................................................. 37

Features .............................................................................................................................................. 37

Mezzanine Cards ..................................................................................................................................... 40

Hard Drive Options.................................................................................................................................. 40

Serial Advanced Technology Attachment (SATA) II ............................................................................ 40


Serial Attached SCSI (SAS) ................................................................................................................... 40

Embedded Ethernet Controllers ......................................................................................................... 41

iDRAC .................................................................................................................................................. 41

Differences .............................................................................................................................................. 42

M600 Unique Features ....................................................................................................................... 43

M805 and M905 Unique Features ...................................................................................................... 44

Drive Controllers ..................................................................................................................................... 45

Drive Controller Options ..................................................................................................................... 45

CERC 6i ................................................................................................................................................ 46

CERC 6i Cache Policy ........................................................................................................................... 46

CERC 6i Configuration Utility............................................................................................................... 47

Creating an Array on the SAS6/iR ....................................................................................................... 48

SAS6/iR Virtual Disk Management ...................................................................................................... 53

View virtual disk properties ................................................................................................................ 53

Synchronize a virtual disk .................................................................................................................... 54

Activate a virtual disk .......................................................................................................................... 54

Delete a virtual disk ............................................................................................................................ 54

Internal Storage ...................................................................................................................................... 55

Diskless Configurations ....................................................................................................................... 55

SATA Repeater Daughter Card ............................................................................................................ 55

SAS6/iR Modular Daughter Card ......................................................................................................... 56

CERC 6i Modular Daughter Card ......................................................................................................... 57

Storage Daughter Card Installation and Removal ................................................................................... 58

Blade Storage Daughter Card Installation ........................................................................................... 59

Blade Storage Daughter Card Removal ............................................................................................... 59

SAS/SATA HDD Backplane Components ................................................................................................. 60

SAS/SATA HDD Backplane ................................................................................................................... 60

HDD Indicator LEDs ............................................................................................................................. 61

Mechanical Issues ............................................................................................................................... 61

Clustering and Virtualization ................................................................................................................... 62

Cluster Computing .............................................................................................................................. 62

Virtualization ....................................................................................................................................... 62


Summary ............................................................................................................................................. 63

Module 4: Systems Management Hardware .............................................................................................. 65


Management Basics ................................................................................................................................ 65

Management Connections .................................................................................................................. 65

Management Features ........................................................................................................................ 66

Operator Panel and LCD ...................................................................................................................... 68

Interpreting the Operator Panel LED .................................................................................................. 68

LCD Menus .......................................................................................................................................... 69

Chassis Management Controller ............................................................................................................. 69

Features and Capabilities .................................................................................................................... 70

CMC Cabling ........................................................................................................................................ 71

Daisy‐chain Connection ...................................................................................................................... 71

Redundancy ........................................................................................................................................ 72

Initial Configuration ............................................................................................................................ 72

Integrated KVM ....................................................................................................................................... 73

Features .............................................................................................................................................. 73

Capabilities .......................................................................................................................................... 74

OSCAR Configuration .......................................................................................................................... 75

Integrated DRAC...................................................................................................................................... 75

Embedded but Separate ..................................................................................................................... 76

Management Features ........................................................................................................................ 76

Management Interface ....................................................................................................................... 77

Security Features ................................................................................................................................ 79

Out‐of‐band Communication .............................................................................................................. 79

Configuration Interfaces ..................................................................................................................... 80

iDRAC Configuration Utility ................................................................................................................. 81

iDRAC Web interface ........................................................................................................................... 81

CMC Web Interface ............................................................................................................................. 81

Chassis LCD Panel ................................................................................................................................ 81

Local racadm ....................................................................................................................................... 81

OpenManage Integration ........................................................................................................................ 82


OpenManage Server Administrator .................................................................................................... 82

IT Assistant Integration ‐ now referred to as DMC ‐ provides Power Monitoring for servers

supporting PMBus reports with the following types of data: ............................................................. 83

Summary ................................................................................................................................................. 84

Module 5: Dell OpenManage Software ...................................................................................................... 87


System Management Basics ................................................................................................................... 87

What Is Management? ........................................................................................................................ 88

Dell OpenManage System Overview .................................................................................................. 88

The Dell OpenManage Model ............................................................................................................. 89

Systems Build and Update Utility ........................................................................................................ 90

Deployment toolkit ............................................................................................................................. 91

Server Management: Remote Management ...................................................................................... 91

Server Management: In‐band Interfaces ............................................................................................ 92

Systems Management Protocols ........................................................................................................ 93

Review ..................................................................................................................................................... 93

Dell OpenManage Server Administrator ................................................................................................. 93

OMSA: Instrumentation Service ......................................................................................................... 94

OMSA Storage Management Service .................................................................................................. 94

OMSA: Remote Access Controller ....................................................................................................... 96

Monitoring Tools: IT Assistant ........................................................................................................... 97

Monitoring Tools: Baseboard Management Controller ...................................................................... 97

Maintenance Tools: Dell Server Update Utility .................................................................................. 98

Maintenance Tools: Dell Online Diagnostics ...................................................................................... 98

Summary ................................................................................................................................................. 99

Module 6: iDRAC ....................................................................................................................................... 100

iDRAC Management and Security Features .......................................................................................... 100

iDRAC Management Features ........................................................................................................... 101

iDRAC Security Features .................................................................................................................... 101

iDRAC Security Features .................................................................................................................... 102

Configuration Tasks ............................................................................................................................... 103

Configuring the Management Station .............................................................................................. 103


Configuring the iDRAC ....................................................................................................................... 105

Configure the Managed Server ......................................................................................................... 108

Configuring the iDRAC Using the Web Interface .............................................................................. 110

Configuring the iDRAC Using the Web Interface .............................................................................. 110

Using Console Redirection .................................................................................................................... 112

Console Redirection Considerations ................................................................................................. 112

Configuring Console Redirection in the iDRAC Web Interface ......................................................... 113

Using the Video Viewer ..................................................................................................................... 113

Using the Video Viewer ..................................................................................................................... 113

Summary ............................................................................................................................................... 114

Module 7: Network Fabrics and IOMs ...................................................................................................... 116

Chassis Overview .................................................................................................................................. 116

Fabric A Broadcom Ethernet Controllers .............................................................................................. 117

Fabric A Features .............................................................................................................................. 117

Dell LOM TOE License Key ................................................................................................................. 117

Expansion Cards Overview .................................................................................................................... 118

Mezzanine Cards ................................................................................................................................... 118

Dell BCM57086S ................................................................................................................................ 118

Emulex LPe1105‐M4 FC HBA ............................................................................................................. 118

QLogic QME2472 FC HBA .................................................................................................................. 119

Qlogic SANBlade QNE247M Fibre Channel HBA Technical Specifications ........................................ 120

IOMs ...................................................................................................................................................... 121

Dell/Emulex PT‐1016 Fibre Channel Pass‐Through ........................................................................... 121

Dell/Emulex PT‐1016 ......................................................................................................................... 121

Dell/Emulex PT‐1016 Fibre Channel Pass‐Through ........................................................................... 122

Key Features ...................................................................................................................................... 122

Technical Specifications .................................................................................................................... 122

Brocade 4424 Fibre Channel Switch ................................................................................................. 123

InfiniBand Technology .......................................................................................................................... 124

About InfiniBand ............................................................................................................................... 124

How InfiniBand Works ...................................................................................................................... 124

Advantages of InfiniBand .................................................................................................................. 125


Cabling............................................................................................................................................... 126

Link Speeds........................................................................................................................................ 126

InfiniBand Physical‐Layer Characteristics ......................................................................................... 127

Parallel Transmission in InfiniBand Links .......................................................................................... 127

Mellanox ConnectX IB MDI InfiniBand Host Channel Adapter (HCA) Mezzanine Card .................... 128

InfiniBand IOM: Cisco SFS M7000E IB Switch ................................................................................... 131

Technical Specifications .................................................................................................................... 133

Power‐On Indicators ......................................................................................................................... 134

Summary ........................................................................................................................................... 135

Module 8: Troubleshooting ...................................................................................................................... 137

Common Configuration Issues for Blades ............................................................................................. 137

Objectives.............................................................................................................................................. 137

Receive Message stating that the CMC’s are not redundant ............................................................... 137

Solutions............................................................................................................................................ 137

Update the Firmware ........................................................................................................................ 137

Message received in the iDRAC sel log ................................................................................................. 138

FC (QLGC or EMLX) Mezzanine would experience "F1/F2" during post. .......................................... 138

Server non responsive at power up ...................................................................................................... 138

Troubleshooting Resource .................................................................................................................... 139

Support.dell.com ................................................................................................................................... 140

Ftp.dell.com .......................................................................................................................................... 141

Module 9: Navigating Dell Information and Tools .................................................................................... 142

Dell System E‐Support Tool (DSET) ....................................................................................................... 142

Support.dell.com ................................................................................................................................... 142

Learndell.com ........................................................................................................................................ 143

Enterprise Support Services .................................................................................................................. 143

Thank You! ............................................................................................................................................ 143


Introduction

Course Safety

Throughout the course, you will be expected to observe strict safety procedures at all times. These

include (but are not limited to):

Always use and carry tools in the correct manner.

Never lift anything that you feel is too heavy.

Never leave parts in places where others may trip over them.

When disassembling systems, always think about other people near you.

If you see anything that you feel is a possible safety concern, please alert your instructor

immediately.

A lot of the equipment used during this course is extremely heavy. If you feel that any piece of

equipment is too heavy for you to easily carry, please seek assistance.

Electrostatic Discharge

Electrostatic discharge (ESD) is a major concern when handling components, especially expansion cards

and system boards. Very slight charges can damage circuits in ways that may not be obvious.

Intermittent problems or a shortened product lifespan can also result.

Static Electricity ‐ A charge stored in any body.

Electrostatic Discharge ‐ A sudden transfer of electrostatic charge between bodies at different

electrostatic potential, usually as a spark as the bodies approach one another.

ESD can damage electronic components immediately, or with delayed effect.

You must take precautions to prevent ESD.

A discharge of static electricity can damage the components of your computer. Electrostatic discharge

occurs when, by touching a component, you cause a sudden voltage surge that equalizes the electricity

between you and the component. By using the wrist‐grounding strap that comes in the ESD kit and

taking several precautions, you can minimize the chances of a discharge occurring.

If a wrist‐grounding strap is not available, you can discharge the static electricity in your body by

touching an unpainted metal surface, such as on the computer chassis.

Before you unplug the machine, perform the following steps: while the system is plugged into the Earth

circuit via the power socket, attach the ESD wrist strap to your wrist and clip the other end to a ground

object.

Then you can unplug the machine. Static‐sensitive components arrive wrapped in anti‐static packing

material.

Do the following when handling static‐sensitive components:


Use an ESD wrist‐grounding strap.

Handle all sensitive components in a static‐safe area.

If possible, use anti‐static floor mats and workbench pads.

When unpacking a static‐sensitive component from its shipping carton, do not remove the component

from the antistatic packing material until you are ready to install the component into your system.


Module 1: Blade System

Technology

Introduction

This module introduces the basic

terms and concepts that are

developed in this course. After

completing this module, you will

be able to:

Define the basic terms and

concepts used in this

course.

Explain the benefits of

blade technology.

List the components of the

M‐series blade servers.

Overview of Blade Technology

Blade servers can be viewed as a complete server chassis, a component on a system board, a type of

computer configuration, or simply as a type of server:

A blade server is a server chassis housing multiple thin, modular electronic circuit boards, known

as server blades. Each blade is a server in its own right, often dedicated to a single application.

The blades are literally servers on a card, containing processors, memory, integrated network

controllers, an optional Fibre channel host bus adaptor (HBA) and other input/output (IO) ports.

A blade server is a computer on a System board, which includes processor, memory, and

sometimes storage. The blade server is intended to address the needs of large‐scale computing

centers to reduce space requirements and lower costs.

Blade servers are part of a computer configuration where power, cooling, storage and

connectivity are largely provided by an outer housing or chassis. The chassis contains and

services a number of specialized, stripped down System board units ‐ the blade servers ‐ each

one a complete computer or service device containing only vital processing and storage

elements.

Blade servers are self‐contained computer servers, designed for high density.


Blade or Blade Server

Blade or Blade Server refers to the individual servers in a

blade enclosure. Dell blades include but are not limited

to:

PowerEdge 1955 9G blade

PowerEdge M600 10G Intel‐based blade

PowerEdge M605 10G AMD‐based blade





Blade Enclosure

Blade enclosure refers to the enclosure that provides

power, cooling, and connectivity to the blades. Dell

blade enclosures include:

PowerEdge 1955 enclosure

PowerEdge M1000e enclosure

Blade System

A blade system refers to the gestalt of the blade enclosure

and the blades it contains.

The enclosure provides power, cooling, and connectivity for

the blades. Within the enclosure, the individual blade servers

can be clustered (like disk arrays) or function independently.


Common Terms

Fabric

A method of encoding, transporting, and synchronizing data between devices. Examples of fabrics

include:

Fibre Channel (FC)

Gigabit Ethernet (GE)

InfiniBand® (IB)

Fabrics are carried inside the PowerEdge M1000e enclosure. They carry information between blades and

I/O modules (IOMs) through the midplane.

Fabrics are also carried outside the M1000e through the physical interfaces on the IOMs.

Lane

A single fabric data transport path between IOMs and end devices. Each lane comprises one transmit

and one receive circuit. Differential signaling reduces noise interference.

Physically, a lane is four wires in a cable or four copper traces on a printed circuit board, each of

which carries one of the following signals:

A positive transmit signal

A negative transmit signal

A positive receive signal

A negative receive signal

Specific terms for lanes vary by technology:

Ethernet: Link

Fibre Channel: Link

InfiniBand: Physical Lane

PCIe: Lane

In this course, link has a different definition.

Link

A collection of multiple fabric lanes used to form a single communication transport path between lOMs

and end devices. Examples include:

Two, four, or eight lane PCIe

Four lane 10Gb‐KX4 PCIe


Although Ethernet and Fibre channel use this term for both single and multiple lanes, this course uses

the terms distinctly to avoid confusion in the learning process. A link, as used in this course, provides

synchronization across multiple lanes so that they act as a single transport.

Port

The physical I/O end interface of a device to a link or lane. A port can connect to multiple lanes.

Product Lines

Dell currently offers two blade product lines.

The PowerEdge 1955 line, which packages both

blades and a blade enclosure

The M‐series, which comprises:

o The M1000e blade enclosure

o The M600 Intel®‐based blade

o The M605 AMD®‐based blade

o The M805 AMD‐based blade

o The M905 AMD‐based blade

.

Summary

Review the summary of this module before taking the self‐check to see how well you mastered

the topic.

Certain blade terminology can have different meanings in the industry. For the purposes of this

course, we define these terms as follows:

o Blade or Blade Server refers to the individual servers in a blade enclosure.

o Blade enclosure refers to the enclosure that provides power, cooling, and connectivity

to the blades.

o Blade system is a data center in a chassis.

o A fabric is a method of encoding, transporting, and synchronizing data between devices.

o A lane is a single fabric data transport path between IOMs and end devices.

o A link is a collection of multiple fabric lanes used to form a single communication

transport path between IOMs and end devices.

o A port is the physical I/O end interface of a device to a link or lane.


Module 2: Dell Blade System Enclosures

Module Objectives

M1000e Modular Blade Enclosure: Enclosure Basics

This module describes the PowerEdge M1000e Modular

Blade Enclosure and the components that make it work.

The M1000e integrates the latest in management, I/O,

power, and cooling technologies in a convenient

modular package. Designed to support current and

future generations of server, storage, networking, and

management technologies, the M1000e includes the

headroom necessary to scale your environment to meet

your future needs.

After completing this topic, you will be able to:

List the M1000e features and components.

List the minimal configuration of the M1000e.

Explain how mezzanine cards and IOMs interact

in fabrics.

List the order in which you should connect cables when installing the M1000e.

Describe the cooling paths of the M1000e.

Enclosure Overview

The M1000e is a 10U enclosure that holds up to 16 blades. That means you can have 64 blades in a 40U

space.

The M1000e solution comprises:

Network, storage, and cluster interconnect modules (switches and pass‐through modules)

A high performance and highly available passive mid‐plane that connects blade server modules

to the infrastructure components

Power supplies

Fans

Integrated KVM (iKVM) and Chassis Management Controllers (CMCs)

The PowerEdge M1000e uses redundant and hot‐pluggable components throughout to provide

maximum uptime.

Benefits:

Simplified operations, with expanded management features


Energy savings and increased compute capacity with efficient power supplies, fans and dynamic

power management

Increased computing density: 64 blades, 512 processor cores in a 42U rack (2U remain free)

Comparison of the PowerEdge 1955 and Ml000e Blade Enclosures 1955 Ml000e Form Factor 7U 10U

Blade Capacity 10 (60 in 42Us) 16 (64 in 4OUs) Multi‐slot Blades No Yes I/O Bays 4/2 fabrics 6/3 Fibre channel Switch or pass‐through Switch or pass‐through Ethernet Switch or pass‐through Switch or pass‐through InfiniBand No Yes Power Supplies 4 6 (3+3) Fans 4 (2 fans in 2 bays) 9 (3 fans in 3 zones) Chassis Management Modules

No Up to 2

KVM Yes Optional Simultaneous Virtual KVM

No Yes

Interactive LCD No Yes Power Management No Yes Power Monitoring No On CMC Internal Persistent Storage

Blade level Blade and chassis level

Basic Configuration and Expansion

The minimal configuration of the M1000e is:

One CMC: Adding a second CMC provides redundancy.

Three power supply units (PSUs): To make it easier to upgrade, Dell recommends adding an

additional package of three PSUs.

Nine fans: With all nine fans populated, you never have to worry about having sufficient cooling

for expansion.

You also need at least one blade to make the enclosure useful.


The Front of the Enclosure

Up to 16 blades are accessible from the front of the

M1000e. On the bottom front left corner of the

chassis there is a control panel to connect to the

integrated KVM (iKVM) on the back of the enclosure.

This connection allows you to connect a crash cart, if

needed.

The Back of the Enclosure

From the back of the enclosure, you can access:

One or two CMCs (one on each upper

right and left corner)

An integrated KVM (iKVM)

Up to six PSUs

Nine fans

Fabric connections (A, B, and C in two

banks)


Cabling Order

For optimal cable management when setting up the M1000e, connect the cables in this order:

1. CMC A

2. CMC B (if populated)

3. iKVM

4. Power Supplies

5. Fabric A IOMs

6. Fabric B IOMs

7. Fabric C IOMs


Cooling

The M1000e contains many

devices, all of which generate heat

and require power. Each of these

devices reports its temperature

and power status to the CMC,

which manages the PSUs and fans

to conserve energy and maximize

performance.

Air enters the front of the chassis

and exits through the rear of the

chassis. On its way, it passes:

Through the blades

Through the holes in the

midplane

Past the IOMs and other

rear components

Out of the back of the chassis

Plenums distribute the air among three

columns of fans. This arrangement breaks

the airflow into three zones. The CMC

directs the fans in each zone to pull as much

air as needed to cool that zone.

Input/output modules (IOMs), CMCs, and

the iKVM use a bypass duct to draw ambient

air from the top‐front of the enclosure. It

passes from the top of the IOM to the

bottom and exits into the plenum between

fans.


Each PSU also has its own fan. Air flows

through the PSU from front to back,

keeping the unit cool.

The CMC also throttles the utilization of

each PSU and distributes the workload

among all of the PSUs. When the CMC

throttles back the utilization of a PSU, the

PSU draws less current and generates less

heat.

Enclosure Components

Now that you know what the M1000e contains, we can explore the components in greater depth. This

topic provides more information about the M1000e components, including:

One or two CMCs (for redundancy)

Integrated Dell Remote Access Controller (iDRAC)

Nine fans

Three, four, or six PSUs

Optional iKVM


Describe the components in the M1000e.

List the hot‐swappable enclosure components.

Chassis Management Controller (CMC)

The CMC is a hot‐swappable systems management device

that has a hardware and a software element. It provides

remote management and power control for the M1000e

and installed components. You can configure the CMC to

send e‐mail alerts or SNMP trap alerts for warnings or

errors related to:

Temperature

Hardware misconfiguration

Power outage

Fan speed

Each CMC includes an integrated 24‐port Ethernet switch

for out‐of‐band internal use only. The ports are all


dedicated to specific devices:

iDRAC on the blades (16)

CMCs (2)

Fabric IOMs (6)

Gb connectors (2)

The M1000e must have at least one CMC. You can add a second CMC for redundancy. Each CMC has the

equivalent processing power of a small server.

You can initially configure your CMCs via:

A serial connection

The operator panel and LCD on the front of the enclosure

iDRAC

The M‐series enhances management with multiple DRACs:

DRAC functionality is built into the M1000e enclosure’s CMC.

An integrated DRAC (iDRAC) in each blade has full hardware management device functionality at

the blade level.

The M‐series supports the Simplify IT model:

The iDRAC rolls up to the CMC.

The CMC rolls up to the Dell OpenManage

systems management software.

The Dell OpenManage systems

management software rolls up to other

products.

iKVM

The KVM on the M1000e is optional. It is a card

that mounts in a special slot in the rear of the

enclosure, rather than a discrete unit. Dell calls it

an integrated KVM (iKVM) because, when

installed, it functions as part of the enclosure. The

connections are accessible from the rear of the

enclosure to provide access to all installed blades and CMCs. There are no cables beyond those that

connect the mouse, keyboard, and monitor to the KVM.

When installed and configured, the iKVM activates the VGA and two USB ports on the front of the

enclosure. The connections provide crash cart access to the enclosure.

Features of the iKVM include:


Fault indicator

Power indicator

ACI port

Two USB connectors for keyboard and mouse

Video connector

IOMs

IOMs provide access to the internal and external networks in the M1000e. The enclosure can house up

to six IOMs, two on each of three fabrics. This flexibility facilitates greater role diversity in the blades.

Fabric A

o Dual 1 Gb fabric for the enclosure’s internal network

Fabric B & C

o Dual 1 Gb fabric for the enclosure’s internal network

o Up to 2x 10Gb links per fabric

o Dedicated Ethernet and Serial Management connections

o Support for Fibre Channel and InfiniBand

Ethernet and Fibre Channel pass‐

through modules are also available.

More information is available in the

IOM topic of this module.

Fans

Each M1000e ships with all nine fans

installed. This provides the following

benefits:

Fully populated enclosures

have a redundant fan.

Cooling is already in place

when upgrading an

enclosure that is not fully

populated.

The fans install in three banks of

three fans to provide for zoned

cooling. The CMC controls fans in each zone to:

Maximize cooling effectiveness

Reduce power consumption

Reduce noise


Midplane

The nervous system of the M1000e is its midplane. The midplane connects all of the various

components and modules together and facilitates communication among them. Ports on the blades

connect to the midplane. So do the IOMs in the rear of the enclosure. Signals pass from the blades to

the midplane to the IOMS to the external network and back again.

The midplane routes traffic from the three fabrics through the IOMs on the back of the enclosure.

When designing the M1000e’s midplane, Dell was thinking beyond the next generation of technology.

The midplane is already certified for:

Fibre Channel 8 (FC8)

InfiniBand

Blades connect to the front of the midplane. Signal

and power connectors on the blades slide into

receptors on the midplane.


Because of the diversity of

devices, the rear of the midplane

looks more complex than the

front. All six PSUs connect to the

midplane to provide power for the

enclosure. In turn, the midplane

provides:

Power to nine fans

Power and

communications for the:

o CMCs

o iKVM

o LCD and operator

panel

o IOMs

PSUs

The M1000e ships with a minimum

of three PSUs in a non‐redundant

configuration. For redundancy, you

need a minimum of four PSUs, but

Dell recommends fully populating

the enclosure with six PSUs. If you

do not want to use all six power

supplies, you can:

Manually turn off power supplies

not in use

Automatically turn off unused PSUs

with dynamic power engagement

o Three PSUs running

o One is redundant and not running

o Two are in full standby

When using more than three PSUs, Dell recommends using a separate power distribution unit (PDU) for

each PSU. For more information, see PSU Redundancy and Grid Redundancy later in this module.

Power Distribution Units

The power supply units (PSUs) do not plug directly into a wall outlet. Instead a variety of power

distribution units (PDUs) provide 220V power to the PSUs.


Type Required Outlet Description Example Installation Single‐phase 20 amp (20A) (16A De‐Rated in US)

NEMA L6‐30R

Provides one 20A circuit per PSU.

The recommended configuration is one PDU per PSU.

Single‐phase 30A (24A De‐Rated in US)

NEMA L6‐30R

Two PDUs provide grid redundancy, though the recommended configuration is one PDU per PSU.

Total power consumption for the enclosure must not exceed 24 amps in the non‐recommended configuration.

Little headroom remains for other devices.

Three‐phase 30A (24A De‐Rated in US)

NEMA L15‐30R

One PDU supports up to three PSUs.

Two PDUs provide grid redundancy.

Each PDU breaks out three 20 amp single phase connections for the PSUs.

Each connector has its own 20 amp breaker.

Single‐phase 60A (48A De‐Rated in US)

Hubbell 363R6W

One PDU supports up to three PSUs.

Two PDUs provide grid redundancy.

Each PDU breaks out three 20 amp single phase connections for the PSUs.

Each connector has its own 20 amp breaker.

Hot‐Swappable Components

Components that you can add or remove without powering down the M1000e include:

CMC (if redundant with two CMCs)

iKVM

Individual blades

IOMs

Fans

PSUs


Before removing the blade or the

drives, you must shut down the

operating system on the blade.

I/O Modules

Topic Objectives

I/O Modules (IOMs) handle all of their fabric traffic between the blades in the enclosure and devices

outside the enclosure.


Explain the function of the three fabrics supported by the M1000e.

List fabric options for the M1000e.

Explain how mezzanine cards and IOMs interact in fabrics.

Fabrics

The M1000e supports three high‐speed fabrics, labeled A, B, and C. Blades in the enclosure

communicate with each fabric using:

Dedicated connections through the midplane

Mezzanine cards installed in the blade

IOMs installed in the back of the enclosure

Fabric A Fabric A supports Gigabit Ethernet for communication between devices in the enclosure. It can also

connect to external networks.

Ethernet controllers on the blades are based on Broadcom technology and support:

o TCP/IP Offload Engine (TOE)

In addition, full SCSI HBA capability with full protocol offload and a broader array of operating

system boot support is available.

While blades released in this series initially have a dual GbE LOM configuration, the system midplane is

designed to allow future support for up to four GbE LOMs in the half‐height form factor.

Fabrics B and C You can configure Fabrics B and C to support your unique needs by adding up to two dual‐port

mezzanine cards to each blade. These optional mezzanine cards include:


o Ethernet (including iSCSI and 10 Gb Ethernet)

o Fibre Channel

o InfiniBand

Like the connector, mezzanine card form factors are common to both fabric B (card 1) and C (card 2).

The optional mezzanine cards connect to the blade’s chipset via 8‐lane PCIe.

No matter which technology you choose for Fabrics B and C, the mezzanine cards installed in the blades

and the lOMs installed in the enclosure must be compatible.

Pass‐through Modules

A pass‐through module provides a transparent connection between the LAN and individual blades. Dell

currently offers Ethernet and Fibre Channel pass‐through modules.


Ethernet The Ethernet pass‐through module supports 10/100/1000Mb

connections as determined by the LOM firmware or blade

operating system.

Fibre Channel The Fibre Channel pass‐through module supports 1, 2, or 4

Gigabit per second (GBs) connections.

Choosing IOMs

The IOM installed in a fabric determines the types of

mezzanine cards that you can install in the blades that

attach to that fabric. For example, you cannot install an

Ethernet pass‐through module in A1 and an Ethernet

switch in A2. Fabrics expect the same type of IOM in

both slots. For best results, you should install two

modules of the same brand and model.

So, if you install a Fibre Channel IOM in Fabric B, blades

that attach to Fabric B must have a Fibre Channel

mezzanine card installed in the first mezzanine. If a

mezzanine card for a different technology were installed

in that slot, the blade would not boot.

You can mix the mezzanine cards in the blades, but not

the IOMs. So you could install a QLogic Fibre Channel

card in one blade and an Emulex Fibre Channel

controller in another, but you should install matching

IOMs in the fabric.

Also, Ethernet IOMs designed for Fabrics B and C are not interchangeable with Ethernet IOMs for

Fabric A.

Fabric Consistency Checking

The M1000e system management hardware and software includes Fabric Consistency Checking, which

prevents accidental activation of any misconfigured Fabric device on a blade.

Since mezzanine‐to‐IOM connectivity is hardwired yet fully flexible, it is possible to inadvertently hot

plug a blade server module with the wrong mezzanine into the system. So if Fibre Channel IOMs are

installed in Fabric C I/O slots, all blade server modules must have either:


No mezzanine in Fabric C.

Fibre Channel cards in Fabric C.

For example: If an Ethernet mezzanine card were in a C slot of a blade being swapped into the

enclosure, the Fabric Consistency Check would automatically detect this misconfiguration, and the CMC

would alert you of the error.

In this situation, no damage occurs to the system, and you can reconfigure the faulted module.

Blade Mezzanine Card Slot B

Blade Mezzanine Card Slot C

I/O Bay A1/A2 I/O Bay B1/B2 I/O Bay C1/C2

None None Ethernet switch or pass‐through module

None None

Ethernet None Ethernet switch or pass‐through module

Ethernet switch or pass‐through module

None

None Ethernet Ethernet switch or pass‐through module

None Ethernet switch or pass‐through module

Ethernet Ethernet Ethernet switch or pass‐through module



Fibre Channel None Ethernet switch or pass‐through module

Fibre Channel switch or pass‐through module

None

None Fibre Channel Ethernet switch or pass‐through module

None Fibre Channel switch or pass‐through module

Fibre Channel Fibre Channel Ethernet switch or pass‐through module



Ethernet Fibre Channel Ethernet switch or pass‐through module



Fibre Channel Ethernet Ethernet switch or pass‐through module



The relationship of the blade mezzanine cards to the chassis fabrics

Stacking IOMs

You stack switches differently, depending on what you want to accomplish.

Switches for the M1000e have optional 10GbE uplinks and/or stacking connectors. These let you:

Manage/configure multiple switches as one with stacking

Consolidate uplinks from multiple enclosures into 2‐4 x 10GbE ports


Reduce port consumption on external switches

Simplify cabling

Stacking IOMs

PowerConnectTM M6220 Switch

The Dell PowerConnect M6220 is a

key component of the FlexIO

architecture of the M1000e.

The PowerConnect M6220 packages a

high performance 24‐port stackable

Ethernet switch with modular bays

that can be populated with 10GbE or

stacking modules. Stacking switches

creates a single logical switch that can be managed and configured. This stacking capability is combined

with the ability to uplink to the Ethernet core via high bandwidth 10GbE connections.

The PowerConnect M6220 switch has four external Ethernet ports and two external option module bays

for 10GbE uplink or stacking modules. The four external gigabit Ethernet ports have standard RJ‐45

connectors, but the external option module bays require a 10GbE or stacking module. Each module bay

accepts a dual‐port 10GbE uplink module. Module bay 1 will also accept a dual‐port stacking module.


You can stack up to 12 PowerConnect M6220 switches for 240 1Gb Ethernet ports and 24 10Gb ports.

To stack the PowerConnect M6220, you must:

Install an optional 48Gb stacking module in bay 1 of each stacked switch.

Assign a separate unique ID for the stack.

Interconnect all the stacked switches.

The PowerConnect M6220 can be upgraded in place with stacking and 10GbE modules installed in the

option modules.

Cisco Switch

Cisco Ethernet switches can be purchased in one of

three base configurations:

Cisco Catalyst 3130X — 10G Rack Switch

Cisco Catalyst 3130G — GbE Rack Switch

Cisco Catalyst 3032 — Entry Level GbE Switch

These three switches cannot be differentiated by appearance.

Upgradability depends on the configuration option. For example, if you want to stack Cisco switches,

you need the Cisco Catalyst 3130G or 3130X.

You can stack up to nine CBS 3130 switches to create a single logical switch that:

Simplifies manageability

Consolidates uplinks

Lowers total cost of ownership (TCO)

Requires one of the following switches that allows Virtual Blade Switch interconnect:

o Cisco Catalyst 3130X — 10G Rack Switch

o Cisco Catalyst 3130G — GbE Rack Switch

Requires a Cisco proprietary cable

Some conditions apply:

You can stack PowerConnect switches with standalone PowerConnect switches.

You cannot stack Cisco Ethernet switches with standalone Cisco Ethernet switches.

You can stack both PowerConnect and Cisco switches between enclosures.

Cisco provides three base configurations. Upgrading requires a rip and replace of the existing base

configuration if it was not purchased with stacking and/or 10GbE options.


Redundancy

Redundancy is the inclusion of multiple devices

that perform the same function to reduce the

likelihood of downtime when one of the devices

fails. The M1000e supports the following

redundancies:

CMC

Fabric

Fans

PSU

Fabric redundancy typically relies on additional configuration, software, and technologies such as NIC teaming or clustering.

CMC Redundancy

Each M1000e can house two CMCs, even though only one CMC is active at

any given time, and only one is required. The standby CMC mirrors the active one. If the active CMC

fails, the standby CMC takes over. You can configure the CMCs to alert you when a failover occurs.

On boot, the CMCs check to see if another CMC is active in the enclosure. The first CMC to boot

becomes active. The second one automatically becomes the standby CMC

Fabric Redundancy

I/O port redundancy within a fabric

Each mezzanine card or LOM has two ports on each blade. If one of the ports fails, traffic can be rerouted by software through the other port.

IOM redundancy within a fabric

Each of the three fabrics has two IOMs in each enclosure. If either of the IOMs fails, traffic can be rerouted by software through the alternate IOM, depending on how the fabric is configured.

Redundancy between fabrics

To have fabric redundancy, Fabrics B and C must use the same protocol. If Fabrics B and C are Ethernet, failure of any of the three fabrics is covered by the remaining fabrics. If Fabrics B and C use the same protocol, failure of either fabric is covered by the remaining one.


Fan Redundancy

All nine fans are required within the M1000e enclosure to power up the chassis. If any of the fans fails,

the remaining eight fans can keep a fully populated enclosure cool until the failed fan can be replaced.

Fans are N+1 redundant, meaning that any single fan can fail without impacting system uptime or

reliability. If a fan fails, system behavior is dependent on the resultant temperatures of the system, as

monitored by the Server Module iDRAC and I/O Modules. The CMC continues to interpret the airflow

needs of each server and I/O module to control the fan speeds appropriately. The system will not ramp

the fans to full speed in the event of a fan failure unless deemed necessary by on‐board monitoring.

The M1000e is engineered for good sound quality in accordance with the Dell Enterprise acoustical

specification. Compared to previous generations of products, the fans have more levels of control,

allowing much finer tuning of the fan behavior. Firmware is optimized to choose the lowest fan speeds

and therefore the lowest acoustical output for any condition and configuration.

When the fans are running at normal rpm they fall just within unprotected hearing safety parameters.

However, when starting up, or during a components failure, the increase in fan speeds bring the noise

beyond unprotected regulations for noise, therefore hearing protection must be used.

If the enclosure is not fully populated, fewer fans may be required, but you should still keep all nine

working. The CMC can manage the fans so that only the requisite number runs at any time.

PSU Redundancy

The M1000e can house up to six PSUs but only requires a minimum of three. So installing any number

greater than three PSUs provides some degree of redundancy.

Four PSUs can cover the failure of any one PSU.

Six PSUs provides maximum redundancy.

No redundancy (N+0)

Power from all three power supplies on one AC circuit (grid) is used to power‐on the entire chassis.


Power supply redundancy (N+1)

The additional power supply is kept as a spare, ensuring that a failure of any one power supply will not cause the servers or chassis to power‐down. All six power supplies are never in use at the same time. Based on load and availability, from two to four of the available power supplies are used at any given time. The rest are held ready for failover.

Grid redundancy (N+N)

All six power supplies are used at 50% capacity. Each bank of three power supplies connects to a separate AC grid.

Summary

Review the summary of this module before taking the self‐check to see how well you mastered the

topic.

The PowerEdge M1000e comprises all of the components needed to support blade networking

and clustering.

o The minimum configuration of the M1000e includes a CMC, three PSUs, and nine fans.

For the enclosure to be usable, you must add at least one M‐series blade.

o You can access the blades, crash cart connection, and operator panel from the front of

the M1000e.

o The following components are accessible from the back of the enclosure:

One or two CMCs (one on each upper right and left corner)

An integrated KVM (iKVM)

Up to six PSUs

Nine fans

Fabric connections (A, B, and C in two banks)


You should connect the cables on the M1000e in a specific order.

The M1000e is an air‐cooled enclosure. Air moves from front to back through three distinctive

paths:

o Through the blades, past the midplane, into a plenum, and out an exhaust fan

o Past the CMC, iKVM, and IOMs, into a plenum, and out an exhaust fan

o Through the PSUs

The components of the M1000e include:

o At least one CMC communicates with all of the enclosure components and with the

iDRACs built into each blade. You can configure the CMC to send e‐mail alerts or SNMP

trap alerts for warnings or errors related to:

Temperature

Hardware misconfiguration

Power outage

Fan speed

o The M1000e’s optional iKVM integrates into the enclosure to provide access to all

blades.

o IOMs communicate with the three fabrics supported by the M1000e:

Fabric A is always Ethernet and supports the enclosure’s private network.

Fabrics B and C can be:

o Ethernet

o Fibre Channel

o InfiniBand

The M1000e ships with all nine fans populated. The fans install in three banks of three fans.

The midplane facilitates communication between all of the devices in the enclosure.

o Fabric A is a dual 1Gb fabric that supports the enclosure’s internal network.

o Fabrics B and C can support various fabrics, including Fibre Chanel and InfiniBand.

You can install three to six PSUs in the M1000e, but you need at least four for redundancy.

The M1000e supports both PDUs available for 9G servers and two 10G server PDUs.

Hot‐swappable components in the M1000e include:

o CMC (if redundant with two CMCs)

o iKVM

o Individual blades

o IOMs

o Fans

o PSUs

IOMs and mezzanine cards on a fabric must use the same technology. While you can mix

mezzanine cards of the same technology on a fabric, both IOMs should be identical.

o The Fabric Consistency Check prevents accidental activation of misconfigured fabric

devices on a blade.

o The M6220 is a stackable 24‐port Ethernet switch designed for Fabric A, but 16 of the

ports are reserved for blades.


Redundancy is the inclusion of multiple devices that perform the same function to reduce the

likelihood of downtime when one of the devices fails.

o When two CMCs are installed in the M1000e, the first to boot becomes the active CMC.

The other mirrors the active one and stands ready to take over if the active CMC fails.

o The M1000e supports:

I/O port redundancy within a fabric

IOM redundancy within a fabric

Redundancy between fabrics

o The M1000e ships with all nine fans installed.

o Four or more PSUs provide some degree of redundancy. For full redundancy, you should

install all six PSUs and attach them to two separate grids.


Module 3: Key Components

Module Objectives

This module discusses the modular servers (blades) that

are compatible with the M1000e Modular Blade

Enclosure (M1000e).

After completing this module, you will be able to:

Identify M‐series blades.

List the features of M‐series blades.

Describe the differences between the M600, M605, M805, and M905.

Explain the benchmarks used to measure M‐series blades.

Explain the difference between clustering and virtualization.

Common Features

Blades are the reason you have a blade enclosure. This topic introduces the blades that are compatible

with the M1000e.


Identify M‐series blades.

Describe the features and options of M‐series blades.

Blades

The M1000e supports the following blades:

M600, an Intel‐based single‐height blade

M605, an AMD‐based single‐height blade



M610, an Intel based single‐height blade

M710, an Intel‐based full‐height blade

Features

M‐series blades share the following features:

Two mezzanine card slots with dual porting

o Ethernet

o Fibre Channel

o InfiniBand


Midplane interface

Integrated Dell Remote Access Controller (iDRAC)

o IPMI 2.0

o Virtual media

o Virtual KVM

o Serial over LAN (SOL)

o Out‐of‐band GUI and CLI

Six I/O ports (with both mezzanine cards installed)

PCIe I/O technology

(M805 and M905) Up to 32GB DDR‐2 SDRAM or 64GB with qualified DIMMs

o Memory ECC, SDDC, Spare Bank, Mirroring

Up to two hot‐pluggable hard drives

o Static Disk Drive support

o SAS

o SATA

Integrated RAID

o CERC 6i RAID options for local drives

o SAS6/iR

Dual port embedded network interface card with TOE

The chart on this screen, shows the features of some of the Dell M‐Series Blade Servers

Features of Dell M‐series Blade Servers

Feature M600 M605 M805 M905 Form Factor Single‐height Single‐height Full‐height Full‐height Processors Intel AMD AMD AMD Processor wattage options

Standard and LV Standard and LV Standard, HE and SE

Standard, HE and SE

RAM slots 8 8 16 24 Total RAM To 64GB To 64GB To 128GB To 128GB Hard drive types SAS and SATA SAS and SATA SAS SAS LOMs 2 2 4 4 Mezzanine card slots

2 2 4 4

RAID card SAS/iR CERC 6i SAS/iR CERC 6i SAS/iR CERC 6i SAS/iR CERC 6i

In addition, both blades can:

Boot from iSCSI using Ethernet connections embedded on the system board


Boot from Fibre Channel SAN

Support enterprise solutions:

o Oracle and SQL database program integration

o VMware program integration

o Dell | EMC integration

o Dell | EqualLogic iSCSI storage

o Dell PowerVault storage

o High performance computing cluster (HPCC)


Mezzanine Cards

In addition to the two Ethernet

controllers embedded on the

system board, M‐series blades*

have two mezzanine card slots.

These slots enable you to install

mezzanine cards to communicate

with the fabrics.

For the blade to boot, the mezzanine card must match the fabric as determined by the I/O module

(IOM) installed in the enclosure.

If the card does not match the fabric, the Chassis Management Console (CMC) can be programmed to

send you an alert. Because the blade does not boot in this situation, you don’t have to worry about

causing any damage to the blade or the enclosure.

*The M710, M805 and M905 blade servers have four mezzanine slots.

Hard Drive Options

Although using local hard drives is optional, M‐series blades support the following internal storage

buses:

Serial Advanced Technology Attachment (SATA) II

SATA II (Serial Advanced Technology Attachment), a hot‐swappable, serial revision of the popular legacy

ATA bus, features:

A raw data transfer rate of 3.0 gigabits per second (Gb)

Significantly smaller connectors and cables

Dedicated bandwidth and cable for each device.

SATA II drives are less costly than their SAS counterparts and offer cost effective alternatives for

customers who have strict budgets and are less concerned with maximum mean time before failure

(MTBF) or performance.

SATA II drives operate at 7.2 RPM, considerably faster than previous generations. SATA II delivers a

blend of high performance and configuration convenience.

Serial Attached SCSI (SAS)

SAS combines elements of SCSI and SATA. It supports the traditional SCSI command structure, so SCSI

software tools continue to work.

Like SATA II, SAS’s raw data transfer rate is 3.0 Gb. SAS, however, is a full‐duplex bus, so data can be

routed in both directions, yielding higher simultaneous data transfers to and from multiple devices than

SATA II.


Finally, SAS supports more devices and longer external cable lengths than SATA II. SAS is recommended

for the most high‐end, high‐demand enterprise server deployments.

Both architectures improve speed and reliability over previous generations of hard drive technology.

Embedded Ethernet Controllers

Two embedded Broadcom Ethernet controllers attach to Fabric A. These controllers support the TCP

offload engine (TOE).

The controllers’ circuitry is embedded in the planar as independent Ethernet interface devices. The

following information details the features of the LAN devices.

PCI‐e x4 interface

SMB for network manageability only with LOM1:

o IPMI

o ASF 2.0

64KB on‐chip packet buffer

MAC and SERDES interface integrated

Wake‐on‐LAN (WOL) 10/100 speeds

PXE is supported

iSCSI Firmware Assisted Boot capability

iSCSI protocol offload option enabled via hardware license key

All NICs are capable of running at 10/100/1000Mb even with the pass‐through modules. This speed was

not possible in previous pass‐through technology.

iDRAC

The Integrated DellTM Remote Access Controller (iDRAC) is a systems management hardware and

software solution that provides:

Remote management capabilities (out‐of‐band)

Crashed system recovery

Power control functions

iDRACs are integrated in M‐series blades. The iDRAC hardware component provides support for:

Virtual Media (vMedia)

Virtual KVM (vKVM)

Out‐of‐band graphical user interface (GUI)

Out‐of‐band inventory

Out‐of‐band status messaging

The iDRAC uses an integrated system‐on‐chip for the remote monitoring and system control. The iDRAC

co‐exists on the system board with the managed M‐series blade servers.


All network connections to the iDRAC are through the CMC Gb1 port. The CMC routes traffic to the

iDRACs on its servers through the enclosure’s private network. This private management network is

between the CMC and the iDRAC. Private network connects all the networks and the CMCs plus the

iKVM.

Differences

Although M‐series blades have many common features and can be used interchangeably—assuming

they have the same mezzanine cards—significant differences between models exist. This topic explores

those differences.


Identify differences between the M600, M605, M805, and M905.


M600 Unique Features

The M600 has the following features:

Up to two dual or quad core

Intel processors

Intel 5000P chipset

Fully buffered 567MHz RAM

To 64GB with qualified 8GB

DIMMs

M605 Unique Features

The M605 has the following features:

Up to two dual or quad core

AMD processors

NVIDIA MPC55 chipset

667 MHz, single/dual/quad‐

ranked DIMMs

To 64GB with qualified 9GB

DIMMs


M805 and M905 Unique Features

Leveraged Design

The system board on the PowerEdge M905 is the same

system board that is used on the PowerEdge M805. The

number of processors installed during the initial P.O.S.T.

determines the system name displayed upon P.O.S.T.

The only difference between the two systems

cosmetically is that the M805 has two processors and

two hyper‐transport bridge cards in processor sockets 3 and 4. Also, because of the processor reduction

from 4 to 2 processors, the total amount of memory supported is 128 GB of RAM on the PowerEdge

M805 versus the 196 GB supported on the PowerEdge M905.

Feature PowerEdge M805 (“Full Height” AMD Blade)

PowerEdge M905 (“Full Height” AMD Blade)

Chipset Dual Nvidia MCP 55 Dual Nvidia MCP 55

Processor Opteron 2000 series

(split plane) Dual or quad core

Dual core (non‐split plane)

AMD

Memory 16 X DDR2 667/800Mhz DIMMS (128 GB total)

24 X DDR2 667/800Mhz DIMMS (192 GB total)

Hard drives Hot‐pluggable 2 x 2.5” SAS Hot‐pluggable 2 x 2.5” SAS

SAS/RAID controllers Daughter card Options for: 1. LSI 1068e w/IR functionality

(SAS6/ir) 2. CERC6

Daughter card Options for: 1. LSI 1068e w/IR

functionality (SAS6/ir) 2. CERC6

Embedded LOMs 4 x GB Ethernet w/hardware TCP/IP Offload Engine, iSCSI Offload accelerator and iSCSI Firmware Boot

4 x GB Ethernet w/hardware TCP/IP Offload Engine, iSCSI Offload accelerator and iSCSI Firmware Boot

Other I/O Four x8 PCIe mezzanine cards Four x8 PCIe mezzanine cards

Embedded Management iDRAC iDRAC


External Ports 3 x USB 2.0 ports 3 x USB 2.0 ports

Internal Persistent Storage 1. CMC based 2. Dedicated Internal SC Slot

per blade for FI embedded hypervisor (same implementation as 805)

1. CMC based 2. Dedicated Internal SC Slot per blade for F1 embedded hypervisor (same implementation as 805)

Drive Controllers

The M‐series blade servers support up to two optional hard drives. The type of drives you can install

depends on the controller.


List the drive controllers available for M‐series blades.

Identify the features of the M‐series drive controllers.

Describe how to configure RAID options on the drive controllers.

Drive Controller Options

Optional storage cards provide internal RAID capabilities for the M‐series blade servers as described in

the following table:

These cards attach into a card slot under the hard drive bays in the front of the blade, not into one of

the I/O mezzanine (card slots).

CERC 6i is for specific applications for optimal performance (for example, Citrix). Read‐only applications

are best because the 128MB cache is not currently battery‐backed.

Dell also supports the option for no hard drives. In this configuration, no card would be installed.


CERC 6i

The CERC 6i is an internal RAID card for customers who require basic RAID functionality. Features

include:

Host‐based daughter card to connect to internal drives

PCI Express (PCIe) Host Interface

128MB cache

Support for RAID‐0 and RAID‐1

This version of the CERC 6i card does not provide a battery to back up the cache. So, it is possible to lose

data during a power outage.

CERC 6i Cache Policy

The cache controller writes a block of data to cache memory, which is much faster than writing to the

physical disk. The cache controller sends an acknowledgement of data transfer completion to the host

system.

The CERC 6i can use two types of caching:

With write‐back caching, the controller sends a data transfer completion signal to the host when

the controller cache has received all the data in a

transaction. The controller then writes the cached data

to the storage device when system activity is low or

when the write buffer approaches capacity. The cached

data is not written to the storage device immediately.

With write‐through caching, the controller sends a data

transfer completion signal to the host system when the

disk subsystem has received all the data in a

transaction.

The default cache setting is write‐through caching.

The risk of using write‐back cache is that the cached data can

be lost if there is a power failure before it is written to the

storage device. Write‐back caching has a performance

advantage over write‐through caching.

Write‐Back mode is available when the user selects Force WB with no battery in the CERC 6i interface

while booting the server. When Forced Write‐Back mode is selected, the virtual disk is in Write‐Back

mode even if the battery is not present.


The CERC 6i does not have a battery backup as of this writing. Because of the possibility of data loss, Dell

recommends that you use write‐through caching with the CERC 6i.

CERC 6i Configuration Utility

The CERC 6i's BIOS configuration utility can be accessed by pressing the <Ctrl><R> keys when booting

the blade. It is a storage management application embedded in the controller firmware that performs

the following actions independently of the operating system:

Configures and maintains RAID disk groups and virtual disks

Manages the RAID system

CTRL+R is to be used for initial setup and disaster recovery. Advanced features can be set through

OpenManageTM Storage Manager.

You can apply CTRL+R using only the keyboard. Other keystroke functionality:

The arrow keys move the cursor in a specific direction.

Pressing <Alt> and an underlined letter acts as a menu shortcut.

Pressing <Enter> activates the highlighted option.

Pressing <Tab> moves to the next control on the page.

Pressing <Esc> closes a pop‐up.


Creating an Array on the SAS6/iR

This procedure configures the SAS6/iR to use RAID‐0 integrated striping (IS) or RAID‐1 integrated

mirroring (IM) on the blade’s hard drives.

Step 1 When the system is booting, press

<Ctrl><C> to enter the SAS6/iR

controller BIOS.

The system prompts you to press

<Ctrl><C>, then the text changes to

indicate that the SAS6/iR user

interface is loading.

If the system did not recognize the

<Ctrl><C> keystroke, it displays a list

of attached devices.


Step 2 When the user interface loads, use

the Arrow keys to highlight and then

press <Enter> to select the desired

controller.

In this example, both the SAS5 and

SAS6 are listed. Notice that the SAS6

is listed with its chipset 1068E.

The controller indicates that it is

reading non‐volatile settings.


Step 3

The Adapter Properties screen can be

manipulated by using the Arrow keys to

highlight RAID Properties and press

<Enter>.

Step 4 On the Select New Array Type screen,

use the arrow keys to highlight the type

of array you want to create and press

<Enter>.

Choose Create IM Volume for a

mirror or RAID‐1 array.

Choose Create IS Volume for a

striped or RAID‐0 array.


Step 5 Create the striped volume.

1) For each drive to be included:

a) Use arrow keys to highlight the

Yellow text.

Disks that do not display yellow

text are not selectable. They

may already be used in

another configuration—either

another volume or a hot spare

configuration.

b) Press the spacebar to change

the RAID Disk selection to Yes.

2) Press <C> to create the striped

volume.

If the space bar is pressed on any other

column than the RAID disk column the

following message should display.

After selecting the disk drives and

pressing <C>, the SAS6/iR BIOS asks you

to create and save the new array.

Step 6 Choose Save Changes, then exit by

navigating with the arrow keys and

pressing <Enter>.

The interface gives you one last chance

to abandon the changes.


Step 7 Press <F3> to finalize the new array.

The volume is now created.

The interface returns to the Adapter

Properties screen.

Step 8 Use the arrow keys to highlight SAS

Topology and press <Enter>.


Step 9 Make sure the new virtual disk appears in

the list.

Step 10 Notice that the drives in Bay 0 and Bay 1

are listed as RAID physical disks, because

they are part of an array. The rest of the

bays show the drive manufacturer and

product number as well as version

number. This information can be obtained

for the drives in Bay 0 and Bay 1 by

expanding the IM Volume to show the

disks that are in that volume.

Step 11 Press <ESC> to exit the user interface.

SAS6/iR Virtual Disk Management

You can perform the following virtual disk management tasks with the SAS6/iR:

View virtual disk properties

Follow these steps to view the properties of IS and IM virtual disks:

1. Select a controller from the Adapter List in the Configuration Utility.

2. Select the RAID Properties option.

If there are no existing virtual disks, you will be prompted to create an IM or an IS virtual

disk.

If there is one existing virtual disk, select View Existing Array.

If there are two existing virtual disks, press <Alt><N> to view the next virtual disk.

3. Press <Enter> when the Manage Virtual Disk item is selected to manage the current virtual disk.


Synchronize a virtual disk

Synchronizing a virtual disk means that the firmware synchronizes the data on the secondary disk(s) of

the mirror. Follow these steps to start synchronization for an IM virtual disk:

1. Select Synchronize Mirror.

2. Press <Y> to start the synchronization or <N> to cancel it.

This process can take several hours to complete.

Activate a virtual disk

A virtual disk can become inactive if, for example, it is removed from one SASS/iR controller or computer

and moved to another one. The Activate Mirror option lets you reactivate an inactive virtual disk that

has been added to a system. This option is only available when the selected virtual disk is currently

inactive.

1. Select Activate Mirror.

2. Press <Y> to proceed with the activation or press <N> to abandon it.

After a pause, the virtual disk will become active.

Activation of migrated virtual disks is only supported when the migrated virtual disk(s) is in an optimal

state and contains all the physical disks.

Delete a virtual disk

Before deleting a virtual disk, be sure to back up all data on the virtual disk that the customer wants to

keep.

Follow these steps to delete a selected virtual disk:

1. Select Delete Virtual Disk.

2. Press <Y> to delete the virtual disk or press <N> to abandon the deletion.

3. Press <F3> to confirm deletion of the virtual disk. After a pause, the firmware deletes the virtual

disk.

If the physical disks of a virtual disk are removed and the virtual disk's configuration is subsequently deleted from the SASS/iR controller, the physical disks show up only as simple disks with no RAID association if they are placed back onto the same SASS/iR controller. Once the virtual disk is removed from a SAS6/iR controller using the BIOS Configuration Utility (regardless of whether the physical disks are present), the virtual disk cannot be restored.


Internal Storage

Dell blade server storage can be configured in the following ways:

Diskless Configuration

SATA Repeater Daughter Card

SAS6/iR Modular Daughter Card

CERC 6i Modular Daughter Card


Enumerate the types of blade server storage devices.

List the features of each type of storage.

Diskless Configurations

Characteristics of diskless configurations include:

No storage daughter card populated

Two HDD blanks populated

SAS/SATA backplane present (for cooling purposes)


Features of the SATA Repeater Daughter Card include:

Provides standalone hard‐disk drive support

No hot‐plug support

No RAID configurations supported

Two SATA HDDs supported

System BIOS configurable (Press F2 during POST)

BIOS Option: SATA Configuration is present only if a SATA controller card is installed



Features of the SAS6/iR Modular Daughter Card include:

Up to two SAS or SATA hot‐pluggable HDDs supported

SAS and SATA HDD mixing is not supported

RAID‐0 or RAID‐1 supported

Press <Ctrl><C> to enter the SAS6/iR Configuration Utility



Features of the CERC 6i Modular Daughter Card include:

Up to two SAS hot‐pluggable HDDs supported

RAID‐0 or RAID‐1 supported

Read and write cache memory for improved performance

Cache is not battery backed up


Storage Daughter Card Installation and Removal

This topic covers installing and removing blade storage daughter cards in Dell PowerEdge blade servers.


Explain how to add a storage card to a blade enclosure.

Explain how to remove a storage card from a blade enclosure.


Blade Storage Daughter Card

Installation

Installation and removal of blade

storage daughter cards is

straightforward.

To install a blade storage daughter

card:

1. Align the controller board so that

the tabs on the metal system board

tray fit through the notches on the

edges of the controller board.

2. Slide the board towards the

connector on the system board

until the board's edge connector is

fully seated.

Blade Storage Daughter Card

Removal

The blade storage daughter card is

located underneath the hard drive

bays of the blade enclosure. To

remove the card:

1. Open the release lever to

disengage the controller

board edge connector

from the system board

connector.

2. Lift the controller board

straight up from the

system board.


SAS/SATA HDD Backplane Components

This section describes components of the SAS/SATA HDD backplane and the functions of HDD blanks in

Dell blade servers.


List the components of the SAS/SATA HDD backplane.

Describe the functionality of the SAS/SATA HDD Indicator LEDs.

Explain the function and convenience factors of HDD blanks in Dell blade servers.

List requirements for populating the HDD slots in a Dell blade server.

SAS/SATA HDD Backplane

Components of the backplane include the following:

Two SAS/SATA HDD Connectors for support of two SAS/SATA 2.5” HDDs

Cable‐less card edge interface to blade planar

Tool‐less HDD backplane installation/removal via latch and hook retention method

Common HDD backplane mechanical/electrical/ board solution shared between the M‐series

blade servers

A common HDD backplane mechanical/electrical/board solution shared between the M805 and

M905

For proper system cooling, the blade must have an HDD or an HDD blank installed in each HDD bay.


HDD Indicator LEDs

Features of the HDD status indicator

LEDs, found on the front of the

enclosure, include:

The HDD status indicator

LED is only functional for

RAID HDD configurations.

For non‐RAID

configurations, only the

drive‐activity indicator is

active.

o The HDD status

indicator LED is not

supported for SATA

Repeater SATA HDD

configurations.

Mechanical Issues

This topic covers HDD blanks and cooling

issues.

Dell PowerEdge blade servers leverage the

existing 2.5” HDD Blank, common to already

shipping platforms. To maintain proper

system thermals, HDD Blanks must be

installed for HDD slots that do not have a

2.5” SAS or SATA HDD populated.


Clustering and Virtualization

Blades lend themselves to two popular server technologies: cluster computing and virtualization. In a

sense, clustering and virtualization are opposite ends of the server spectrum.

Clustering is grouping two or more servers together so that they function as a single unit.

Virtualization is causing a single server to behave as if it were two or more servers.

After completing this topic, you will be able to explain the difference between clustering and

virtualization.

Cluster Computing

Clustering is grouping two or more servers together so that they function as a single unit. Clustered

computers communicate with each other over a private local area network.

Different types of cluster architecture serve different computing needs.

High‐performance computing clusters (HPCCs): A high‐performance computing cluster (HPCC) is

a powerful and economical solution for enhancing your research environments. An HPCC is a

group of commercial off‐the‐shelf computers interlinked via high‐speed interconnects to create

a robust, high‐speed computing resource that rivals the performance of traditional mainframe

supercomputers at a fraction of the cost.

Grid computing: Grids work much like HPCCs, except that the nodes each retain their identities.

SETI@home is one example of a very large grid.

High availability (HA) clusters: Also known as failover clusters, these clusters improve the

availability of critical resources. Most HA clusters consist of two nodes; one active and one

redundant, The redundant node mirrors the active node and takes over its functionality in the

event of a failure.

Load‐balancing clusters: A front‐end controller allocates workload among the nodes in the

cluster to achieve maximum throughput and balance the workload of each node.

Blade servers give you the advantage of high‐density computing, reduced cabling, reduced cooling

requirements and optimized power.

Virtualization

Virtualization allows you to run multiple applications and operating systems independently on a single

server. Administrators can quickly move workloads from one virtual workspace to another—easily

prioritizing business needs while maximizing server resources. Flexibility in allocating computing

capacity allows the consolidation of applications and servers, easing IT management requirements and

lowering expenses.

Dell offers a range of virtualization solutions, including:

Microsoft® Virtual Server

VMware® ESX ServerTM


Citrix® XenServerTM

Getting the most out of any of these solutions requires in‐depth training that goes beyond the scope of

this course.

Summary

The M1000e supports the following blades:

o M600, an Intel‐based single‐height blade

o M605, an AMD‐based single‐height blade

o M805 and M905, AMD‐based full‐height blades

M‐series blades:

o Support the same midplane interface

o Have an integrated iDRAC

o Have six I/O ports when both mezzanine cards are installed

o Have bays for two hot‐swappable hard drives

o Include Broadcom Ethernet controllers that attach to Fabric A

o Support two mezzanine cards to communicate with Fabrics B and C

o Support two internal hard drives that can be configured in a two‐drive RAID array

Differences between the M‐series blade servers include:

o The M600 is an Intel‐based, single‐height blade that is designed for heavy use in data

centers.

o The M605 is an AMD‐based, single‐height blade that supports TPM.

You have a choice of 2 RAID drive controllers.

o The CERC 6i provides RAID‐0 and RAID‐1 functionality and 128MB RAID caching.

Although you can set the CERC 6i to use write‐back caching, you should avoid this

option, because the lack of battery backup places your data at risk to power loss. The

CERC 6i’s configuration utility is commonly known as CTRL+R, because you press those

keys when booting the blade to access the utility.

o The SAS6/iR is an entry‐level RAID‐capable drive controller for cost sensitive customers.

Administrators can perform RAID configuration on the SAS6/iR by pressing <Ctrl><C>

during POST. The SAS6/iR configuration utility provides an option to configure the

Virtual disks. The user can activate or delete the virtual disks.

Internal Storage consists of a variety of daughter card configurations.

o Types of blade storage configurations:

Diskless Configuration, where there are no disks, and HDD blanks are used to

populate the backplane




o SAS/SATA HDD Backplane:

Has connectors for two SAS/SATA 2.5” HDDs

Has Cable‐less card edge interface to Blade Planar


Has Tool‐less HDD backplane installation/removal

Common HDD Backplane mechanical/electrical/board solution shared between

models

LEDs for drive activity and HDD status, but HDD status LED is only functional in

RAID HDD configuration

The 2.5” HDD blank is common to existing platforms

HDD blanks must be installed for any slot without a SAS/SATA HDD populated

o Blade storage daughter card installation/removal:

To install, align the controller board so that the tabs on the metal system board

tray fit through the notches in the edges of the controller board, then slide the

board towards the connector on the system board until the board’s edge

connector is fully seated.

To remove, open the release lever to disengage the controller board edge

connector from the system board connector, then lift the controller board

straight up from the system board.

Different types of cluster architecture serve different computing needs.

Blade servers give you the advantage of high‐density computing, reduced cabling, reduced

cooling requirements and optimized power.


Module 4: Systems Management Hardware

Module Objectives

This module outlines the systems management options available with the M‐series blade servers.

After completing this module, you will be able to describe the management features of M1000e blade

servers.

Management Basics

Dell’s M1000e modular server delivers major enhancements in management features. These

enhancements derive from direct customer feedback and assist customers deploying modular (blade)

servers. Each subsystem has been reviewed and adjusted to:

Optimize efficiencies

Minimize impact to existing management tools and processes

Provide future growth opportunities to standards‐based management


Explain how management information is communicated through the enclosure.

Describe the M1000e enclosure management features.

Management Connections

Management connections in the M1000e enclosure transfer health and control traffic throughout the

enclosure. The system management fabric uses 100 Mb Ethernet over differential pairs routed to each

module.

Two 100 Mb interfaces, one switched and one unswitched, communicate between Chassis

Management Controllers (CMCs).

Every Integrated DRAC (iDRAC) module has a management network link to each CMC.


Redundancy is at the controller level. Failure of any individual link triggers failover to the redundant

CMC, as long as the system is configured with a redundant CMC.

Management Features

Efficiency is the core of the blade philosophy, and easy, secure management is at the heart of efficiency

in a blade environment. Management features available on M‐series blade servers include:

Chassis Management Controller (CMC) The CMC is the central point for infrastructure monitoring, inventory, and control. It provides dedicated

Ethernet management connectivity to iDRACs and I/O switches. Adding an optional second CMC

provides redundancy. Failure of the active CMC will trigger failover to the standby CMC if a second CMC

is installed.

A 24‐port Ethernet switch is reserved for internal communication between the iDR4C on the blades to

the CMC. It also provides communication between the CMC and the external management network.

If two CMCs are installed, the heartbeat for CMC redundancy is also present and CMC redundancy is

supported over the internal network. The internal network is separate from the networks used by the

host LOMs and the mezzanine cards in the blades.

The CMC provides internal persistent storage with a flash card.


iDRAC

The M1000e supports the Integrated Dell Remote Access Controller (iDRAC) that is integrated on each M‐series blade.

All network connections to the iDRAC are through the CMC network interface (Gb1). The CMC routes traffic to the iDRACs on its servers through a private, internal network. This private management network is outside of the servers’ data path and outside of the operating system's control (out‐of‐band). The managed servers' in‐band network interfaces are accessed through I/O modules (lOMs) installed in the chassis.

Each blade server module's iDRAC has its own IP address and can be accessed, if security settings allow, directly through a supported browser, Telnet, SSH, or IPMI client on the management station. The internal network is separate from the networks used by the host LOM.

iKVM

The iKVM:

Switch local KVM (front or rear)

Seamlessly tier into Dell and Avocent KVM infrastructure

Provides local console access to all the

blades at the CMC (one at a time)

LCD operator panel Supports simplified initial chassis setup

wizard

Provides enhanced local troubleshooting

and information

Displays configuration, status information,

and system health for each blade’s iDRAC

and IOM, respectively


Operator Panel and LCD

The operator panel contains the local user

interface that provides blade and enclosure

health and information. Functions include:

Chassis level diagnostic LEDs

LCD

Power button

The operator panel is always powered, even in

chassis standby mode.

The operator panel pivots so that you can adjust it for easy viewing, wherever the enclosure is in the

rack. It also retracts fully into the chassis when it is not needed.

Interpreting the Operator Panel LED

The LED in the operator panel changes color to indicate

the enclosure’s status:

Blue — All devices in the enclosure are

functioning properly.

Orange — Check the status of the devices using:

o The LCD menus

o Remote chassis management


LCD Menus

The LCD shows chassis information, chassis

status and faults, and major blade server

module faults. It also provides an interface for

blade server module and CMC network

configuration. Included is an error message

when no CMC is present in the chassis. Service

Tag, Asset Tag and IP Address are easily visible

on every M1000e modular system through the

LCD display.

Health: Displays alerts, status information, and health information for:

Enclosure

Blades

IOM

Information:

Module model/type and user defined name

Hardware configurations

Service tag

IP Address

You can initially set the CMC’s IP address using the LCD Configuration Wizard.

Chassis Management Controller

The Chassis Management Controller

(CMC) is a hot‐pluggable systems

management hardware and software

solution. The M1000e always has at least

one CMC, but may have two. Redundancy

is provided in an Active–Standby pairing

of the modules in the enclosure. Failover

occurs when the active module fails or

degrades.


Describe the features and capabilities of the CMC.

Explain how to access the CMC and how it communicates with other devices in the enclosure.

Explain redundancy in M‐series CMCs.


Features and Capabilities

The CMC has its own microprocessor and memory but receives power from the enclosure. It interfaces

through dual stacking Ethernet ports and one serial port. The CMC’s serial port interface provides

common management of up to six I/O modules (IOMs) through a single connection.

The CMC provides secure remote management access and power control functions for the chassis and

installed modules. It manages or facilitates the management of the following:

Status, Inventory and Alerting for blade server modules, chassis infrastructure and I/O Modules

Centralized configuration for iDRAC, I/O Modules, CMC, and iKVM

Power sequencing of modules in conjunction with the defined chassis power states

Power budget management and allocation

Configuration of the embedded management switch, which facilitates external access to

manageable modules

Remote user management

User interface entry point (Web, Telnet, SSH, serial)

Monitoring and alerting for chassis environmental conditions or component health thresholds

This includes but is not limited to the following:

o Real time power consumption

o Power supplies

o Fans

o Power allocation

o Temperature

o CMC redundancy

o I/O fabric consistency

Ethernet traffic management (firewall)

Private VLAN

Private DHCP service

Private TFTP service

Virtual Media

Virtual KVM (vKVM) interface between the external IP network and the blade server module

iDRAC

Firmware management of CMC, IOM, iDRAC, iKVM

All serial port configurations of IOMs go through the serial port of the CMC

The CMC can also be configured to send e‐mail alerts or SNMP trap alerts for warnings or errors related

to:

Temperatures

Hardware configuration errors

Power outages

Fan speeds


Protocols and other tools supported by the CMC include:

SMASH‐CLP features providing chassis blade server module power control functions

WSMAN

CIM XML

SNMP

racadm

SSL/SSH

CMC Cabling

At least one CMC must be installed in the primary

CMC bay for the enclosure to power up. If a second,

optional CMC module is installed, failover protection and hot‐plug replacement is available.

Each CMC has two Ethernet ports:

Gb1 is used to connect to the external management network.

Gb2 will allow CMCs in adjacent enclosures to be daisy‐chained.

With basic cabling, you connect the Gb1 port to the management network and leave the Gb2 port

unused. DO NOT connect both ports to your network. Gb2 is only for

daisy‐chaining CMCs.

A 24‐port Ethernet switch provides internal communication between:

The iDRAC on each blade

IOMs

Optional KVM

Optional second, redundant CMC

Daisy‐chain Connection

If you have more than one enclosure in a rack, you can reduce the

number of connections to the management network by daisy‐

chaining up to four chassis together.

If each of four chassis contains a redundant CMC, by daisy‐

chaining you reduce the number of management network

connections required from eight to two.

If each chassis has only one CMC, you reduce the connections

required from four to one.

When daisy‐chaining chassis together, Gb1 is the uplink port and Gb2

is the stacking port. A Gb1 port must connect to the management

network or to the Gb2 port of the CMC in an enclosure that is closer

to network. The Gb2 port must only receive a connection from a Gb1 port further from the chain.


Create separate chains for the CMCs in the primary CMC slot and the second CMC slot.

Redundancy

When both CMC slots in an M1000e are populated, you have CMC redundancy.

The first CMC booted is the active (primary) CMC, while the other becomes the standby CMC. The

standby CMC mirrors the active CMC. The settings for both CMCs are held in NVRAM on the M1000e

chassis operator panel.

Both CMCs must run the same firmware revision. If the firmware revisions differ, the system reports as

redundancy degraded.

Failovers can occur when:

The active CMC fails.

The network cable is removed from the active CMC.

The active CMC is removed from the chassis.

A CMC firmware flash is initiated on the active CMC.

If a CMC failover occurs:

All external iDRAC sessions and all active CMC sessions disconnect and must be reestablished.

Internal iDRAC sessions move to the new active CMC.

Initial Configuration

There are four ways to configure the CMC:

LCD Configuration Wizard: This wizard is accessed via the LCD control panel on the front of the

chassis.

CLI using a null modem cable: This is a direct serial connection to the CMC via a null modem

cable.

Syntax: setniccfg ‐s <IP address><network mask><gateway>

Example: setniccfg ‐s192.168.1.60 255.255.255.0 192.168.1.1

Web GUI: Can be used if you set your management station's IP address to be on the same

subnet as the default CMC IP address.

The default IP address is 192.168.0.120.

CLI using entry 17 on the OSCAR interface for the iKVM.


Integrated KVM

The integrated keyboard, video, and mouse (iKVM)

switch for the M1000e is an optional, hot‐pluggable

module. It provides local keyboard, mouse, and video

access to any of the 16 blades in the enclosure. It also

provides access to the additional Dell CMC command‐

line console.


List the features and capabilities of the iKVM.

Access the OSCAR Setup dialog box.

Features

The optional Avocent® Integrated KVM Switch Module (iKVM):

Provides local keyboard, mouse, and video access to any of the 16 blades in an M1000e

enclosure

Provides access to the additional Dell CMC command line console

Maintains all blade connections as input is switched from each blade

Occupies a single slot accessible through the rear of the enclosure

Redirects local blade video, keyboard, and mouse electrical interfaces to either:

o The iKVM local ports

o The M1000e front panel ports

Local iKVM access can be remotely disabled on a per blade basis, using the blade's iDRAC interface

(access is enabled by default).

The iKVM also has an Analog Console Interface (ACI) compatible RJ45 port that allows the iKVM to tie

the interface to a KVM appliance upstream of the iKVM via CAT5 cabling. Although the ACI port is an RJ‐

45 connector and uses CAT5 (or better) cabling, it is not an Ethernet network interface port. It is only

used for connection to external KVM switches with Analog Rack Interface (ARI) ports. The ACI port

reduces cost and complexity by giving access for sixteen simultaneous servers using only one port on an

external KVM switch.

The iKVM available for the M1000E chassis is analog. The iDRAC handles functions, such as virtual media

or remote desktop, that digital KVMs provided in the previous blade models.


Capabilities

iKVM user interface: On‐Screen Configuration and Activity Reporting (OSCAR). The OSCAR

display allows you to select, configure and manage the blades in the enclosure.

Security: The OSCAR user interface allows you to protect the system with a screen‐saver

password.

Video: The iKVM video connections support video display resolutions ranging from 640 x 480 at

60 Hz up to 1280 x 1024 at 60 Hz.

Flash upgrades: You can update the iKVM firmware using:

o CMC Web‐based interface (requires the firmware file)

o racadm fwupdate command (requires a TFTP server to upload the file using the OSCAR

interface)

CMC access through entry 17: The CMC command‐line interface can be accessed through entry

17 on the OSCAR interface.

Enabling and disabling access: You can enable or disable access through the front panel or CMC

(through entry 17) with the CMC.


OSCAR Configuration

The OSCAR menu has a common feel found throughout the Dell

range of KVMs.

To access the Setup menu:

1. Press <Print Screen> to launch OSCAR.

2. Click Setup to access the Setup dialog box.

You can soft switch the iKVM between blades, using a hotkey

sequence, by pressing <Print Screen> and then typing the first

few characters of its name or number. If you previously set a

delay time (the number of seconds before the Main dialog box is

displayed after <Print Screen> is pressed) and you press the key

sequences before that time has elapsed, the OSCAR interface

does not display.

Integrated DRAC

The Integrated Dell Remote Access Controller (iDRAC) is a systems management hardware and software

solution that provides:

Remote management capabilities

Crashed system recovery


Power control functions


Describe the features of the iDRAC.

List the configuration utilities available on the iDRAC.

Embedded but Separate

iDRACs are embedded in the M‐series family of blades. The iDRAC combines the functionality of the

baseboard management controller (BMC) and Dell Remote Access Controller (DRAC) used on traditional

servers into a single device on the blade’s system board.

The iDRAC has both software and hardware components. The hardware component provides:

Virtual media (vMedia) and virtual KVM (vKVM) functions

Out‐of‐band graphical user interface (GUI)

Out‐of‐band status and inventory information

The iDRAC is powered by standby power. With a blade in the chassis, the iDRAC is always on and

provides management and monitoring capabilities.

Management Features

The iDRAC provides the following management features:

Console Redirection: Provides remote system keyboard, video, and mouse functions.

Dell OpenManage software integration: Enables you to launch the iDRAC Web interface from

Dell OpenManage Server Administrator or IT Assistant.

Dynamic Domain Name System (DDNS) registration.

iDRAC alerts: Alerts you to potential managed node issues through an e‐mail message or SNMP

trap.

IPMI support.

Microsoft Active Directory authentication: Centralizes iDRAC user IDs and passwords in Active

Directory using the standard schema or an extended schema.

Monitoring: Provides access to system information and status of components.

Password‐level security management: Prevents unauthorized access to a remote system.

Remote power management: Provides remote power management functions, such as

shutdown and reset, from a management console.

Remote system management and monitoring: Using:

o Web interface.

o Local racadm command‐line interface.

o SM‐CLP command line over a Telnet/SSH connection.

Role‐based authority: Provides assignable permissions for different systems management tasks.

Secure Sockets Layer (SSL) encryption: Provides secure remote system management through

the Web interface.


System log access: Provides access to the system event log, the iDRAC log, and the last crash

screen of the crashed or unresponsive system. Access is independent of the operating system

state.

Virtual Media: Enables a managed server to access a local media drive on the management

station or CD/DVD ISO images on a network share.

Management Interface

The iDRAC network interface is disabled by default. You must configure it before you can access the

iDRAC. After accessing the iDRAC on the network, you can access it at its assigned IP address with:

The iDRAC Web interface

Telnet

SSH

Supported network management protocols, such as:

o Simple Network Management Protocol (SNMP)

o Intelligent Platform Management Interface (IPMI)

You can configure the iDRAC to send you an e‐mail message or Simple Network Management Protocol

(SNMP) trap alerts for warnings or errors. To help you diagnose the probable cause of a system crash,

iDRAC can log event data and capture an image of the screen when it detects that the system has

crashed.

The CMC monitors and manages all components in the enclosure, including iDRACs. The LCD that is part

of the enclosure displays the iDRAC's integration information and the health status of the blade.

All network connections to the iDRAC are through the network interface (Gb1) on the active CMC. The

CMC routes traffic to the iDRACs on its servers through a private, internal network.


Traditional IPMI‐based baseboard management controller (BMC) features such as hardware monitoring

and power control are supported. With up to two simultaneous sessions per blade, remote

management satisfies virtually any usage model.

Virtual media over IP is standard. The iDRAC emulates:

DVD‐ROM

CD‐ROM

USB Flash Drive

ISO image

USB/legacy floppy

Floppy image

Full USB access is available through the blade server module front panel. Connection to console

redirection and virtual media is through the CMC, with an option available to encrypt those sessions.

Highlights of the iDRAC solution include:

Dedicated management interface

Virtual Media

Console redirection via Virtual KVM

IPMI 2.0 Out‐Of‐Band management


Serial over LAN

SMASH CLP

Blade status and inventory

Active power management

Local Security and Integration with Active Directory

Security Features

The iDRAC provides the following security features:

User authentication through Microsoft Active Directory (optional) hardware‐stored user IDs and

passwords

Role‐based authority, which enables an administrator to configure privileges for each user

User ID and password configuration through the Web interface or SM‐CLP

SM‐CLP and Web interfaces, which support 128‐bit SSL encryption and 40‐bit SSL encryption (for

countries where 128‐bit is not acceptable)

Session time‐out configuration (in seconds) through the Web interface or SM‐CLP

Configurable IP ports (where applicable)

Telnet does not support SSL encryption.

Secure Shell (SSH), which uses an encrypted transport layer for higher security

Login failure limits per IP address, with login blocking from the IP when the limit is exceeded

Limited IP address range for clients connecting to the iDRAC

Out‐of‐band Communication

All network connections to the iDRAC are through the CMC network interface (Gb1). The CMC routes

traffic to the iDRACs on its servers through a private, internal network. Features of the out‐of‐band

network include:

Dedicated internal 100Mb Ethernet connection to each CMC

o The iDRAC connects to the CMC via dedicated, fully redundant 100 Mbps Ethernet

connections.

Passes through the midplane to a dedicated 24‐port Ethernet switch on the

CMC

Exposed to the outside world through the CMC’s external Management

Ethernet interface (Gb1)

o This connection is distinct from the three redundant data Fabrics A, B and C.

Unlike previous generations of Dell blade servers, the iDRAC’s connectivity is

independent of, and in addition to, the onboard GbE LOMs on the blade.

Configurable Web or command‐line interface (CLI) using Virtual Media


Configuration Interfaces

You have several options for configuring the iDRAC. However, using more than one configuration

method at a time can generate unpredictable results.


iDRAC Configuration Utility

Access this at boot time to set up the network and basic security features and for enabling other

features. Press <Ctrl>+<E> during POST to access the utility. If the Web interface to the CMC is not being

used, this is the best opportunity to turn on the Ethernet interface to the iDRAC and set the IP address

for remote access.

iDRAC Web interface

The iDRAC Web interface is a browser‐based management application that you can use to interactively

manage the iDRAC and monitor the managed server. It is the primary interface for day‐to‐day tasks such

as:

Monitoring system health

Viewing the system event log

Managing local iDRAC users

Launching the CMC web interface and console redirection sessions

CMC Web Interface

In addition to monitoring and managing the chassis, you can use the CMC web interface to:

View the status of a managed server

Configure iDRAC network settings

Start, stop, or reset the managed server

The current firmware does not properly perform a graceful operating system shut down.

Chassis LCD Panel

The LCD panel on the chassis containing the iDRAC can be used to view the high‐level status of the

servers in the chassis. During initial configuration of the CMC, the configuration wizard allows you to

enable configuration of iDRAC networking.

Local racadm

The local racadm command‐line interface runs on the managed server. The server is accessed from

either the iKVM or a console redirection session initiated from the iDRAC Web interface.

Racadm is installed on the managed server when you install Dell OpenManage Server Administrator.

Racadm commands provide access to nearly all iDRAC features. You can:

Inspect sensor data, system event logs records, and the current status and configuration values

maintained in the iDRAC.

Alter iDRAC configuration values.

Manage local users.

Enable and disable features.

Perform power functions such as shutting down or rebooting the managed server.


The CMC also has a racadm interface available through slot 17 of the iKVM. It can be accessed remotely

via Telnet, SSH, or from a management station.

OpenManage Integration

M‐series blade servers fully integrate Dell OpenManage, OpenManage Server Administrator, and IT

Assistant. After completing this module, you will be able to:

Describe how OpenManage Server Administrator and IT Assistant integrate with the M1000e.

OpenManage Server Administrator

OpenManage Server Administrator (OMSA) supports the following M1000e management functions:

Power monitoring: This list shows some of the features that you can monitor:

o Amps per power supply

o Aggregate watts consumed

o Aggregate cumulative power drawn

o Resetting of peak power values

o Maximum peak power (watts and amps)

o Power supply input voltage

Dell OpenManage software integration: You can launch the iDRAC web interface from OMSA.


OMSA is covered in greater depth in the OpenManage Server Administrator module of this course.

IT Assistant Integration ‐ now referred to as DMC ‐ provides Power Monitoring for servers supporting

PMBus reports with the following types of data:

Aggregation of power and energy consumed by various groups of servers

Amperage per power supply

Energy and power consumption per blade

Peak power and peak amperage per server

Power consumption and total energy consumed

IT Assistant is covered in greater depth in the IT Assistant module of this course.


ITAssistant

Summary


topic.

Management basic concepts include:

o Management information from each blade and module in the enclosure to the CMC

over redundant 100BaseT paths.

o Management features of M‐series blade servers include:

CMC

iDRAC

iKVM

LCD operator panel

o The retractable operator panel contains the LCD panel, power button, and diagnostic

LEDs.

o The LED on the operator panel is blue when all devices are functioning properly. It blinks

orange to indicate that one of the devices in the enclosure has a problem.

The M‐series operator panel has two menus:

o Health — displays alerts, status information, and health information for the devices in

the enclosure.

o Information — displays module information, service tag, and IP address.

The Chassis Management Controller (CMC) is a hot‐pluggable systems management hardware

and software solution.


o Status, Inventory and Alerting for blade server modules, chassis infrastructure and I/O

Modules

o The CMC can be configured to send e‐mail alerts or SNMP trap alerts for warnings or

errors related to:

Temperatures

Hardware configuration errors

Power outages

Fan speeds

o The CMC has two Ethernet ports

Gb1 is used to connect to the external management network.

Gb2 will allow CMCs in adjacent enclosures to be daisy‐chained.

o Daisy‐chaining CMCs in the rack significantly reduces the number of cables required.

o CMC redundancy is attained when both CMC slots in an M1000e are populated.

o Failovers can occur when:

The network cable is removed from the active CMC.

The active CMC is removed from the chassis.

A CMC firmware flash is initiated on the active CMC.

o The four ways to configure the CMC are:

LCD Configuration Wizard

CLI using a null modem cable

Web GUI

CLI using port 17 on the iKVM

The integrated keyboard, video, and mouse (iKVM) switch for the M1000e is an optional, hot‐

pluggable module.

o The M‐series iKVM is:

Optional

Hot‐swappable

Analog

o The iKVM provides:

An on‐screen utility to select, configure, and manage blades in the enclosure

A screen‐saver password

Flash upgrades

Access to the CMC CLI

o You can configure OSCAR by pressing <Print Screen> and clicking Setup in the resulting

menu.

The Integrated Dell Remote Access Controller (iDRAC) is a systems management hardware and

software solution.

o The iDRAC resides on the blade’s system board and provides

Virtual media (vMedia) and virtual KVM (vKVM) functions

An out‐of‐band graphical user interface (GUI)

Out‐of‐band status and inventory information

o M‐series iDRAC management features include:


Console Redirection

Dell OpenManage software integration

Dynamic Domain Name System (DDNS) registration

iDRAC alerts

IPMI support

Microsoft Active Directory authentication

Monitoring

Password‐level security management

Remote power management

Remote system management and monitoring

Role‐based authority

Secure Sockets Layer (SSL) encryption

System log access

o The iDRAC communicates through the Gb1 connection on the CMC.

o All communication with the iDRAC passes through the CMC.

o The following configuration interfaces are available on the iDRAC:

iDRAC Configuration Utility

iDRAC Web interface

CMC Web interface

Chassis LCD Panel

Local racadm

iVM‐CLI

iDRAC SM‐CLP

IPMI

M‐series blade servers fully integrate Dell OpenManage.

o You can use OpenManage Server Administrator with an M1000e enclosure to:

Monitor power usage

Launch the iDRAC Web interface

o Power management features available through IT Assistant include:

Aggregation of power and energy consumed by various groups of servers

Amperage per power supply

Energy and Power consumption per blade

Peak power and Peak amperage per server

Power consumption and total energy consumed


Module 5: Dell OpenManage Software

Module Objectives


Explain how to use Dell OpenManage Server

Administrator (OMSA) to manage

interactions between blade servers and a

storage area network (SAN).

System Management Basics

This topic introduces the basic concepts of system management. After completing this topic, you will be

able to:

Define systems management.

Describe Dell OpenManage.

List the systems management protocols supported by OpenManage.


What Is Management?

Systems Management uses tools (hardware and software) to perform mundane, simple, or complex

tasks. Systems management can call on multiple components:

Hardware

Remote management hardware, in addition to the base hardware that makes the server work

Software

Agents to monitor status and generate alerts

Software to manage same or different systems

Dell OpenManage System Overview

Dell OpenManage systems management software is a suite of application programs for Dell systems that

allows you to manage your system with proactive monitoring, diagnosis, notification, and remote

access. Each of your managed systems will use applications that include Server Administrator, IT

Administrator, and remote access controller (RAC) software. A management station can be used to

remotely manage one or more managed systems from a central location. By installing IT Assistant on a

management station, you can effectively manage from one to thousands of remote managed systems.


This graphic illustrates the relationship between a management station and its managed systems. It also

shows the operating systems and the Dell OpenManage software products that may be installed on the

managed systems.

The Dell OpenManage Model

Review the OpenManage Model on

this page. Note that OpenManage

consists of Deployment Tools,

Monitoring Tools, and Maintenance

Tools. Review the names of all the

individual tools in the OpenManage

suite, and then focus on the tools in

the Deployment category.


Systems Build and Update Utility

The Dell Systems Build and Update

Utility is a bootable CD that provides:

Step‐by‐step configuration for

an unattended install of the

operating system.

Installation support for

Windows and Linux operating

systems.

Dell‐optimized device drivers,

firmware, and utilities for Dell

PowerEdgeTM servers.


Deployment toolkit

Dell OpenManage Deployment Toolkit

(DTK) provides some of the

components that facilitate unattended

server deployment. DTK includes a set

of utilities that supports both CLI and

file‐based parameters, as well as

documentation that describes its

components. DTK components are:

Unattended server hardware

configuration

Utility partition (WP)

Basic Input/Output System

(BIOS)

Redundant Array of

Independent Disks (RAID)

Network Interface Cards (NICs)

Unattended installation of Dell drivers and management agents per operating system. This feature

contains a set of tools and scripts that can be used by most third‐party deployment software that

has an Automated Deployment Service (ADS).

Server Management: Remote Management

Remote Management Hardware Remote management hardware is often optional. It generally enables you to perform extra

management tasks that would not otherwise be possible. For Dell systems, this additional hardware is

part of a Dell Remote Access Controller (DRAC), which bundles the hardware and software needed for

remote management.

Remote management hardware:

Often works out‐of‐band

Does not require operating system interaction

Functions even when the system is powered down

Systems management software uses management protocols (such as SNMP or CIM) to IMPI components

(such as BMC) and to change settings. These protocols are embedded within the operating system.

Therefore, they are in‐band, available only when the operating system is running.


OutofBand Management An additional adapter card is required to perform out‐of‐band management. Dell recommends the DRAC

family of adapters. These cards enable you to remotely access the information supplied by IMPI and

control the manageable hardware.

DRAC hardware supplied

interfaces include:

Serial

Telnet

Ethernet

InBand Management In‐band management is

accomplished via instrumentation

in the operating system. In‐band

management means that a

management system is dealing

with a managed node through the

operating system and through

agents and drivers.

The Dell Remote Access hardware

supports “in‐band” management,

and allows you to perform three

major types of tasks:

View system related

management information

such as temperature, voltages via new BMC

Perform power management

Take control of the server GUI remotely

Server Management: In‐band Interfaces

OpenManage Server Administrator Server Administrator provides a comprehensive, one‐to‐one systems management solution from an

integrated, Web browser‐based GUI (the Server Administrator home page), and from a command line

interface (CLI) through the operating system. Server Administrator is designed for system administrators

to both locally and remotely manage systems on a network.


IT Assistant IT Assistant provides a one‐to‐many management solution when OpenManage Server Administrator

(OMSA) is also installed on the managed systems. Server Administrator enables you to configure the

RAC.

Systems Management Protocols

Protocols can be described as:

Rules of communication between agents and the system management application

Industry‐standard technologies for cost‐effective management

Protocols gather and transmit management information. Systems management protocols include:

Simple Network Management Protocol (SNMP)

Common Information Method (CIM)

Intelligent Platform Management Interface (IPMI)

Review

Systems management protocols supported by Dell OpenManage software include:




Dell OpenManage Server Administrator

Bundled with Dell OpenManage software, Dell OpenManage Server Administrator (OMSA) provides

configuration and monitoring, and replacing functionality. This topic covers the basics of OMSA.


Describe the functions of the OMSA management service.

Describe the OMSA storage management service.

Describe the OMSA remote access controller.

Describe IT Assistant.

Describe the Dell Online Diagnostics.

If you need more in‐depth information, you may want to take the Dell OpenManage course.


OMSA: Instrumentation Service

The Server Administrator Instrumentation Service monitors the health of a system and provides rapid

access to detailed fault and performance information. The reporting and viewing features allow retrieval

of overall health status for all chassis in the system. At the subsystem level, users can view information

about the voltages, temperatures, fan speed, and memory in the system. A detailed account of every

relevant cost of ownership (COO) detail about your system can be seen in summary view. Version

information for BIOS, firmware, operating system, and all installed software is easy to retrieve.

Additionally, systems administrators can

use the Instrumentation Service to

perform the following essential tasks:

Specify minimum and maximum

values for certain critical

components. The values, called

thresholds, determine the range

in which a warning event for that

component occurs.

Specify how the system responds

when a warning or failure event

occurs. Users can configure the

actions that a system takes in

response to notifications of

warning and failure events.

Populate all of the user‐specific

values for the system, such as the

name of the system, the phone

number of the system’s primary

user, the depreciation method,

whether the system is leased or

owned, and so on.

OMSA Storage Management Service

The Dell OpenManage applications used to

control and monitor the storage subsystem

are OpenManage Storage Management

(OMSM). The application provides storage

management information in an integrated

graphical view. Features of the Storage

Management Service include:

The Storage Management Service

enables you to view the status of local and remote storage attached to a monitored system


Supports SCSE, SATA, and SAS

Does not support Fibre Channel

Enables you to perform controller and enclosure functions for all supported controllers and

enclosures from a single graphical or command line interface without using controller BIOS

utilities

Protects your data by helping you:

o Configure data redundancy

o Assign hot spares

o Rebuild failed drives

Provides a graphical interface that is wizard‐driven with features for novice and advanced users

and detailed online help

Provides a command line interface that is fully featured and scriptable


OMSA: Remote Access Controller

The Server Administrator Remote

Controller (RAC) provides a complete

remote system management solution for

systems equipped with any of the

following instrumentation:

Integrated Dell Remote Access

Controller (iDRAC)

Dell Remote Access Card (DRAC) 5

Embedded Remote Access (ERA)

controller

ERA Option (ERA/O) card

ERA/MC controller

Remote access requires instrumentation

that supports either:

Simple Network Management

Protocol (SNMP)

Common Information Mode (CIM)

These hardware and software solutions are collectively known as Remote Access Controllers (RACs).

Users may choose to perform some of these functions through the BMC, but working through a RAC is

more efficient.

The RAC provides:

Remote access to an inoperable system, enabling you to get the system up and running as

quickly as possible.

Alert notification when a system is down.

Remote restart of a system.

Logs of the probable causes of system crashes and the most recent crash screen.

Remote access through the Server Administrator home page or by directly accessing the

controller’s IP address using a supported browser.


Monitoring Tools: IT Assistant

Use e‐mail, paging, or console

alerting to monitor faults with:

o Disks

o Fans

o Memory

o Thermal conditions

o Voltage

Inventory and asset reporting

that includes:

o Service tag

o Cost of ownership

information

o Specifics of BIOS,

processor(s), and memory

In‐context launch of device‐

specific tools, such as:

o Storage Manager

o Server Administrator

o Remote Access Controller

o PowerConnect Switch Manager

Monitoring Tools: Baseboard Management Controller

The baseboard management controller

(BMC) is responsible for monitoring the

health of the server and components

associated with the server’s system board. It

does not usually monitor devices on the I/O

buses, and it operates independently of the

system board.

The monitoring technology is based on the

Intelligent Platform Management Interface

(IPMI) standard. This standard is defined in

conjunction with Intel and other Original

Equipment Manufacturers, including Dell.

IMPI is being used in the systems

management working group at the

Distributed Management Task Force

(DTMF).


The Dell BMC supports IPMI 1.5 and later. It delivers:

Monitoring of server hardware.

Alerts when potential or actual faults are encountered.

Network and serial access to control server power and reset the server regardless of the server’s

power status.

Maintenance Tools: Dell Server Update Utility

The Dell Server Update Utility (SUU)

simplifies the task of updating BIOS,

firmware, and drivers on servers. Insert

the Dell Server Updates DVD into a

server’s DVD drive to automatically:

Inventory the Dell firmware and

drivers on that server.

Compare the installed

configuration with the content on

the Dell Server Updates DVD.

Report on discrepancies and

recommend updates, including

any prerequisites that impact the update sequence.

Update and (if needed) reboot the server.

The Dell Server Update Utility is designed to simplify single‐server updates and is available on the Dell

Server Updates DVD included as part of the Dell OpenManage Subscription Service.

Maintenance Tools: Dell Online Diagnostics

The Dell Online Diagnostics:

Provides testing of the system

without bringing it down:

The system administrator can run

the diagnostics as a preventative

measure, or after a non‐fatal

failure, to verify the status of the

system hardware.

Helps determine if the cause of the

failure is the hardware:

The online diagnostics enable the administrator to determine if the problem is hardware and to

make a more accurate assessment of the problem.


Provides hardware Field Replacement Unit (FRU) isolation.

Dell Online Diagnostics are now available on the following sources:

Dell Systems Service and Diagnostics Tools CD

Dell Systems Management Tools and Documentation DVD

The OpenManage Subscription Service

Summary


topic.

Basic systems management concepts include:

o Systems management consists of:

Using a variety of tools to remotely manage computer hardware.

o Dell OpenManage consists of:

Deployment tools

Monitoring tools

Maintenance tools

o The Systems Build and Update Utility provides:

Configuration for an unattended operating system installation

Installation support for Windows and Red Hat Linux

Dell‐optimized device drivers, firmware, and utilities

o The Deployment toolkit includes components that facilitate unattended server

deployment.

o Systems management protocols supported by Dell OpenManage software include:




OpenManage Server Administrator (OMSA) provides configuration, monitoring, and reporting

functions.

o The OMSA Instrumentation service monitors the health of a system and provides rapid

access.

o The Dell OpenManage Storage Management application provides storage management

information.

o The OMSA remote access controller provides a complete remote system management

solution instrumentation:

Dell Remote Access Card (DRAC) 5

Embedded Remote Access (ERA) controller


ERA Option

(ERA/O)

card

ERA/MC

controller

o Using IT Assistant

you can:

Be alerted

of specific

system

faults

Generate

inventory

and asset

reports

Launch

device‐

specific tools

o The Dell Online Diagnostics:

Enables you to test a system without downing it

Helps distinguish between hardware and software issues

Provides hardware isolation

Module 6: iDRAC The Integrated Dell Remote Access Controller (iDRAC) is a fully functional DRAC that is embedded on the

system board of M‐series blade servers. This module assumes you are already familiar with DRAC

functionality. If not, you should review the DRAC chapter of the PowerEdge server course.


Describe the basic features of the iDRAC

Explain how to configure the iDRAC

iDRAC Management and Security Features

The Integrated Dell Remote Access Controller (iDRAC) is a systems management hardware and software

solution that provides remote management capabilities, crashed system recovery, and power control

functions for Dell PowerEdge systems.



Describe the iDRAC’s management features

Describe the iDRAC’s security features

iDRAC Management Features

The iDRAC provides the following management features:

Dynamic Domain Name System (DDNS) Registration: Provides the ability to register the iDRAC

name to the Domain Name Servers (DNS).

Remote system management and monitoring: Provides management and monitoring

capabilities using:

o A Web interface

o The local racadm command‐line interface (CU)

o The SM‐CLP command line over a Telnet/SSH connection Support for Microsoft Active Directory authentication: Centralizes iDRAC user IDs and

passwords in Active Directory using the standard schema or Dell extended schema.

Console Redirection: Provides remote keyboard, video, and mouse functions.

Virtual Media: Enables a managed server to access a local media drive on the management

station or CD, DVD, floppy, CD/DVD ISO images or floppy images on a network share. Monitoring: Provides access to system information and status of components.

Access to system logs: Provides access to the system event log, the iDRAC log, and the last crash

screen.

Dell OpenManage software integration: Enables you to launch the iDRAC web interface from

Dell OpenManage Server Administrator or IT Assistant.

iDRAC alert: Alerts you to potential managed node issues through an e‐mail, IPMI, alerts or

SNMP trap.

Remote power management: Provides remote power management functions, such as

shutdown and power up, from a management console.

Intelligent Platform Management Interface (IPMI) support: Provides support for IPMI features

as defined in the IPMI Specification version 1.5 and 2.0.

Secure Sockets Layer (SSL) encryption: Provides secure remote system management through

the Web interface or SSH.

Password‐level security management: Prevents unauthorized access to a remote system by

using DRAC defined users and password authentication.

Role‐based authority: Provides assignable permissions for different systems management tasks.

iDRAC Security Features

The iDRAC provides the following security features:

User authentication through Microsoft Active Directory (optional) or hardware‐stored user IDs

and passwords


Role‐based authority, which enables an administrator to configure specific privileges for each

user

User ID and password configuration through the Web interface or SM‐CLP

SM‐CLP and Web interfaces, which support 128‐bit SSL encryption and 40‐bit SSL encryption (for

countries where 128‐bit is not acceptable)

Session time‐out configuration (in seconds) through the Web interface, SM‐CLP, or racadm

Configurable IP ports (where applicable)

Secure Shell (SSH), which uses an encrypted transport layer for higher security

Telnet does not support SSL encryption.

Login failure limits per IP address, with login blocking from the IP address when the limit is

exceeded

Limited IP address range for clients connecting to the iDRAC

iDRAC Security Features

When opening firewalls for remote access to an iDRAC, you need to make sure that all of the necessary

ports are available. The iDRAC client and listening ports are:

iDRAC client ports iDRAC server listening ports

Port Function Port Function

25 SMTP 22 SSH

53 DNS 23 Telnet

68 DHCP‐assigned IP address 80 HTTP

69 TFTP 443 HTTPS

162 SNMP traps 623 RMCP/RMCP+

636 LDAPS 3668 Virtual Media Service

3269 LDAPS 3669 Virtual Media Service

3770 Virtual Media Secure Service

3771 Virtual Media Secure Service

5900 Console redirection keyboard and mouse

5901 Console redirection video

All ports except 623 are configurable.


Configuration Tasks

Since the iDRAC is part of the blade’s system board, no installation is required. But you still need to do

several configuration tasks to make it useful. This topic discusses these tasks in detail.

After completing this topic, you will be able to explain how to:

Configure the iDRAC

Configure the management station

Configure the managed systems

Use the iDRAC Web interface

Configuring the Management Station

A management station is a computer used to monitor and manage the PowerEdge servers and other

modules in the chassis.

The following are the software installation and configuration tasks that set up a management station to

work with the iDRAC. Before you begin configuring the iDRAC, follow the procedures in this section to

ensure that you have installed and configured the tools you will need.

Step Procedure

1. Set up the management station network

To access the iDRAC, the management station must be on the same network as the CMC RJ45 connection port labeled “GB1.” You can isolate the CMC network from the network the managed server is on, so that your management station may have LAN access to the iDRAC but not to the managed server. Using the iDRAC console redirection feature (see Using GUI Console Redirection), you can access the managed server’s console even if you do not have network access to the server's ports. You can also use iDRAC facilities to perform several management functions on the managed server, such as rebooting the computer.

2. Install and configure a supported Web browser.

For a list of supported Web browsers, see Supported Web Browsers. Note : The iDRAC Web interface is not supported on 64‐bit Web browsers. If you open a 64‐bit browser, access the Console Redirection page, and attempt to install the plug‐in, the installation procedure fails. If this error was not acknowledged and you repeat this procedure, the Console Redirect Page loads even though the plug‐in installation fails during your first attempt. This issue occurs because the Web browser stores the plug‐in information in the profile directory even though the plug‐in installation procedure failed. To fix this issue, install and run a supported 32‐bit Web browser and log in to the iDRAC.

If you use a proxy server, make sure the browser bypasses the proxy for local addresses.


Step Procedure

Make sure the iDRAC is in the Trusted Domains list.

If you use Firefox, disable the Whitelist feature. Otherwise you must install a console redirection viewer for each iDRAC you access—even though all viewers are identical.

3. Install a Java Runtime Environment (JRE).

This step is optional for stations running Internet Explorer. If you use Internet Explorer, an ActiveX control is provided for the console viewer. You can also use the Java console viewer with Internet Explorer if you install a JRE and configure the console viewer in iDRAC Web interface before you launch the viewer. See Configuring Console Redirection in the iDRAC web Interface for more information. If you use Firefox you must install an IRE (or a Java Development Kit [JDK]) to use the console redirection feature. The console viewer is a Java application that is downloaded to the management station from the iDRAC web interface and then launched with Java web Start on the management station. Go to java.sun.com to install a JRE or JDK. Version 1.6 (Java 6.0) or higher is recommended. Note: In the Java Console, set the Runtime Settings parameter to a value of ‐xmx256M.

4. Install Secure Shell clients, if required.

By default, the iDRAC Telnet service is disabled and the SSH service is enabled. Since telnet is an insecure protocol, you should use it only if you cannot install an SSH client or your network connection is otherwise secured. Note: The iDRAC supports only one active SSH connection. If an SSH session is already active, the iDRAC denies all other SSH connection attempts.

5. Install a TFTP server, if required.

Trivial File Transfer Protocol (TFTP) is a simplified form of the File Transfer Protocol (FTP). It is used to update iDRAC firmware using SMCLP. Certificates can be transferred to the iDRAC using racadam or SMCLP. If you choose to use SM‐CLP or racadmracadm when you perform these tasks, an active TFTP server must be running on the computer that can be accessed by the iDRAC. Note: No TFTP server is required if you only use the iDRAC Web interface to upload new iDRAC firmware or install SSL certificates.

6. Install Dell OpenManage IT Assistant.

Your system includes the Dell OpenManage System Management Software Kit. This kit includes, but is not limited to, the following components:

Dell Systems Service and Diagnostics Tools CD

Dell Systems Management Tools and Documentation DVD

The OpenManage Subscription Service


Step Procedure

Use the Dell System Management Consoles CD to install the management console software, including Dell OpenManage IT Assistant, on the management station. For instructions on installing this software, see your Quick Installation Guide.

Configuring the iDRAC

Before you can configure the iDRAC, the blade must be installed in an enclosure with at least one

functioning CMC and IOM. The following list describes the configuration tasks you may need to perform.

It also points to resources you can use to accomplish these tasks.

Step Procedure

1. Configure the management station.

Set up a management station by installing Dell OpenManage software, a browser, and other software utilities as described in Configuring the Management Station.

2. Configure iDRAC networking. Enable the iDRAC network and configure IP, netmask, gateway, and DNs addresses. Note: Changing the iDRAC network settings terminates all current network connections to the iDRAC.

The option to configure the server using the LCD panel is available only during the CMC initial configuration. Once the chassis is deployed, the LCD panel cannot be used to reconfigure the iDRAC.

The LCD panel can be used to enable DHCP to configure the iDRAC network. If you want to assign static addresses, you must use the iDRAC Configuration Utility or the CMC Web interface.

Chassis LCD Panel‐see the Dell Chassis Management Controller User’s Guide.

DRAC Configuration Utility — see LAN.

CMC Web Interface — see Configuring Networking Using the CMC Web Interface.

racadmracadm — see cfgLanNetworking.

3. Configure iDRAC users. Set up the local iDRAC users and permissions. The iDRAC holds a table of sixteen local users in firmware. You can set usernames, passwords, and roles for these users.

iDRAC Configuration Utility (configures administrative user only): see LAN User Configuration

iDRAC Web interface: see Adding and Configuring iDRAC Users

racadmracadm: see Adding an iDRAC User 4. Configure Active Directory. In addition to the local iDRAC users, you can use Microsoft Active

Directory to authenticate iDRAC user logins. See Using the iDRAC with Microsoft Active Directory.


Step Procedure

5. Configure IP filtering and blocking.

In addition to user authentication, you can prevent unauthorized access by rejecting connection attempts from IP addresses outside of a defined range and by temporarily blocking connections from IP addresses where authentication has failed multiple times within a configurable time span.

iDRAC Web interface: see Configuring IP Filtering and IP Blocking

racadmracadm: see Configuring IP Filtering (IpRange), Configuring IP Blocking

6. Configure Platform Events. Platform events occur when the iDRAC detects a warning or critical condition from one of the managed server’s sensors. Configure platform Event Filters (PEFs) to choose the events you want to detect, such as rebooting the managed server.

iDRAC web interface — see Configuring Platform Event Filters (PEF)

racadmracadm — see Configuring PEF Configure Platform Event Traps (PETs) to send alert notifications to an IP address, such as a management station with IPMI software, or to send an e‐mail to a specified e‐mail address.

iDRAC Web interface — see Configuring Platform Event Traps (PET)

racadmracadm — Configuring PET 7. Configure serial over LAN (SOL).

Serial Over LAN (SOL) is an IPMI feature that allows you to redirect the managed server's serial port I/O over the network. SOL enables the iDRAC console redirection feature.

iDRAC Web interface — see Configuring Serial Over LAN

See also Using GUI Console Redirection. 8. Configure iDRAC services Enable or disable the iDRAC network services, such as Telnet, SSH,

and the Web server interface, and reconfigure ports and other service parameters.

iDRAC Web interface: see Configuring iDRAC Services

racadm: see Configuring iDRAC Telnet and SSH Services Using Local racadm

9. Configure Secure Sockets Layer (SSL).

Configure SSL for the iDRAC Web server.

iDRAC Web interface: see secure sockets Layer (SSL)

racadm: see cfgRacSecurity, sslcsrgen, sslcertupload, sslcertdownload, sslcertview

10. Configure virtual media for loading an operating system.

Configure the virtual media feature so that you can install the operating system on the PowerEdge server. Virtual media allows the managed server to access media devices on the management station

or CD/DVD, ISO images, and floppy images as if they were devices

on the managed server, among other functions.


Step Procedure

iDRAC Web interface — see Configuring and Using Virtual Media

iDRAC configuration utility — see Virtual Media Note: Using the iDRAC configuration utility, you can only attach or detach virtual media devices.

11. Install the managed server software.

Install the Microsoft Windows or Linux operating system on the PowerEdge server using virtual media and then install the Dell Open Manage software on the managed PowerEdge server and set up the last crash screen feature.

Console redirection — see Installing the Software on the Managed Server

iVM‐CLI: see Using the Virtual Media Command Line Interface Utility

12. Configure the managed server’s Last Crash Screen feature.

Set up the managed server so that the iDRAC can capture the screen image after an operating system crash or freeze. See Configuring the Managed Server to Capture the Last Crash Screen, Disabling the Windows Automatic Reboot Option.

13. Configure networking using the CMC Web interface.

Note: You must have Chassis Configuration Administrator privilege to set up iDRAC network settings from the CMC.

The default CMC user is root and the default password is calvin.

The CMC IP address can be found in the iDRAC Web interface by clicking System Remote Access CMC. You can also launch the CMC Web interface from this page.

1. Use your Web browser to log in to the CMC Web user interface using a Web address of the form https://czCMC‐IP‐eddress> or https://<CMC‐DNS‐name>. 2. Enter the CMC username and password and click OK. 3. Click the plus (+) symbol next to Chassis in the left column, then click Servers. 4. Deploy iDRAC. 5. Enable the LAN for the server by checking the checkbox next to the server beneath the Enable LAN heading. 6. Enable or disable IPMI over LAN by checking or unchecking the checkbox next to the server beneath the Enable IPMI over LAN heading. Note: By default, DHCP is disabled and a default IP address, netmask and gateway are provided. 7. Check the box to enable DHCP for the blade server. 8. If DHCP is disabled, enter the static IP address, netmask, and default gateway for the server. 9. Click Apply at the bottom of the page.


Step Procedure

14. Update the iDRAC firmware as needed.

Updating the iDRAC firmware installs a new firmware image in the iDRAC flash memory. You can update the firmware using any of the following methods:

SM‐CLP load command

iDRAC Web interface

Dell Update Package (for Linux or Microsoft Windows)

DOs iDRAC Firmware update utility

CMC Web interface (only if iDRAC firmware is corrupted)

Configure the Managed Server

The following are the tasks to set up the managed server to enhance your remote management

capabilities.

Task Details

Install the management software on the managed server

The Dell management software includes the following features:

Local racadm CLI: allows you to configure and administer the iDRAC from the managed system. It is a powerful tool for scripting configuration and management tasks.

Server Administrator: a Web interface that allows you to administer the remote system from a remote host on the network.

Server Administrator Instrumentation Service o Provides access to detailed fault and performance

information gathered by industry‐standard systems management agents and allows remote administration of monitored systems.

o Displays logs of commands issued to or by the system, monitored hardware events, POST events, and system alerts. You can view logs on the home page, print or save them as reports, and send them by e‐mail to a designated service contact.

Server Administration Storage Management Service: provides storage management information in an integrated graphical view.

Use the Dell Systems Management Tools and Documentation DVD to install Server Administrator. For instructions on installing this software, see your Quick Installation Guide.


Task Details

Configure the managed server to capture the last crash screen.

The iDRAC can capture the last crash screen so that you can view it in the Web interface to help troubleshoot the cause of the managed system crash. Follow these steps to enable the last crash screen feature. 1. Install the managed server software. For more information about

installing the managed server software, see the Server Administrator User's Guide. 2. If you are running a Microsoft windows operating system, ensure that the Automatically Reboot feature is deselected in the Windows Startup and Recovery Settings. See Disabling the windows Automatic Reboot Option. 3. Enable the Last Crash screen (disabled by default) in the iDRAC web interface. To enable the Last Crash Screen in the iDRAC web interface, click System Remote Access ‐ iDRAC Network/Security Services, then check the Enable checkbox under the Automatic System Recovery Agent Settings heading. To enable the Last Crash Screen using local racadm, open a command prompt on the managed system and type the following command: racadm config ‐g clgRacTuning ‐o cfgRacTuneAsrEnable 1 4. In the Server Administrator Web‐based interface, enable the Auto Recovery timer and set the Auto Recovery action to Reset, Power Off, or Power Cycle. For information about how to configure the Auto Recovery timer, see the Server Administrator User’s Guide. To ensure that the last crash screen can be captured, the Auto Recovery timer should be set to 60 seconds. The default setting is 480 seconds. The Last Crash screen is not available when the Auto Recovery action is set to Shutdown or Power Cycle if the managed server is powered off.

Disable the Windows Automatic Reboot option.

To ensure that the iDRAC can capture the Last Crash Screen, disable the Automatic Reboot option on managed servers running Microsoft Windows Server. 1. Open the Windows Control Panel and double‐click the System icon. 2. Click the Advanced tab. 3. Under Startup and Recovery, click Settings. 4. Deselect the Automatically Reboot check box. 5. Click OK twice.


Configuring the iDRAC Using the Web Interface

Use the iDRAC Web interface for everyday systems management tasks. You can:

Configure the iDRAC properties and users

Perform remote management tasks

Troubleshoot a remote (managed) system for problems

Most Web interface configuration tasks can also be performed with local racadm commands or with SM‐

CLP commands.

Local racadm commands are executed from the managed server. For more information about

local racadm, see Using the Local racadm Command Line Interface.

SM‐CLP commands are executed in a shell that can be accessed remotely with a Telnet or SSH

connection. For more information about SM‐CLP, see Using the iDRAC SM‐CLP Command Line

Interface.

Configuring the iDRAC Using the Web Interface

The following list explains the tasks you can accomplish with the iDRAC Web interface.

Task Details

Access the iDRAC Web interface To access the DRAC Web interface: 1. Open a supported web browser window. 2. Type the following text string in the Address field:

If the default port has not been changed, Type: https://cziDsQ4c‐IP‐address> and press <Enter>.

The iDRAC Login window appears. You must log in to the DRAC with valid credentials.

Configure the iDRAC NIC. Assuming that the iDRAC has already been configured and is accessible on the network, you can:

Configure the network and IPMI LAN settings

Configure IP Filtering and IP Blocking

Configure platform events. Platform event configuration provides a mechanism for configuring the iDRAC to perform selected actions on certain event messages. The actions include:

No action

Reboot system

Power cycle system

Power off system

Generate an alert


Task Details

Configure platform event filters (PEFs).

When a platform event occurs (for example, a battery warning assert), a system event is generated and recorded in the System Event Log (SEL). If this event matches a platform event filter (PEF) that is enabled and you have configured the filter to generate an alert, a PET or e‐mail alert is sent to up to four configured destinations. If the same platform event filter is also configured to perform an action (such as rebooting the system), the action is performed. Note: Configure platform event filters before you configure the platform event traps or e‐mail alert settings. To configure PEFs: 1. Log in to the iDRAC Web interface. 2. Click System. 3. Click the Alert Management tab. 4. On the Platform Events page, do either of the following:

Place a check next to the Generate Alert heading to group select all events or to check each individual event

5. Click the radio button below the action you would like to enable for each event. Only one action can be set for each event. 6. Click Apply. Note: Generate Alert must be enabled for an alert to be sent to any valid, configured destination (PET or e‐mail).

Configure platform event traps (PETs).

1. Log in to the remote system using a supported web browser. 2. Ensure that you followed the procedures in Configuring Platform Event Filters (PEFs). 3. Configure your PET destination IP address:

a. Click the Enable checkbox next to the Destination Number you would like to activate. b. Enter an IP address in the Destination IP Address box. Note: The destination community string must be the same as the iDR4C community string. c. Click Apply.

Note: To successfully send a trap, configure the Community String value on the Network Configuration page. The Community String value indicates the community string to use in a Simple Network Management Protocol (SNMP) alert trap sent from the iDRAC. SNMP alert traps are transmitted by the iDRAC when a platform event occurs. The default setting for the Community String is Public.

d. Click Send to test the configured alert (if desired). e. Repeat for any remaining destination numbers.


Task Details

Configure e‐mail alerts 1. Log in to the remote system using a supported Web browser. 2. Ensure that you followed the procedures in Configuring Platform Event Filters (PEF). 3. Configure your e‐mail alert settings.

a. On the Alert Management tab, click E‐mail Alert Settings. 4. Configure your e‐mail alert destination.

a. In the E‐mail Alert Number column, click a destination number. There are four possible destinations to receive alerts. b. Ensure that the Enabled checkbox is selected. c. In the Destination Email Address field, type a valid e‐mail address. d. Click Apply.

Note: To successfully send a test e‐mail, the SMTP Server Address must be configured on the Network Configuration page. The IP address of the SMTP Server communicates with the DRAC to send e‐mail alerts when a platform event occurs.

e. Click Send to test the configured e‐mail alert (if desired). f. Repeat step a through step e for any remaining e‐mail alert settings.

Using Console Redirection

The iDRAC console redirection feature enables you to access the local console remotely in either graphic

or text mode. You do not have to sit in front of each server to perform all the routine maintenance. You

can manage servers from your desktop or laptop computer wherever you are.

Console Redirection Considerations

The Console Redirection page enables you to manage the remote system by using the keyboard, video,

and mouse on your local management station to control the corresponding devices on a remote

managed server. This feature can be used in conjunction with the Virtual Media feature to perform

remote software installations.

The following conditions apply:

A maximum of two simultaneous console redirection sessions are supported. Both sessions view

the same managed server console simultaneously.

The first logged‐in user will be notified if a second user connects.


A console redirection session should not be launched from a Web browser on the managed

system for its own console redirection.

The managed system should be set to a screen resolution and refresh rate listed in the table.

The screen resolution on the management station should be set higher than the resolution of

the managed system.

A supported Web browser must be installed on the management station.

If you are using Firefox or want to use the Java Viewer when using Internet Explorer, you must

install a Java Runtime Environment (JRE) on the management station.

Screen Resolution Refresh Rate (Hz)

640 x 480 60, 72, 75, 85

720 x 400 70

800 x 600 60, 70, 72, 75, 85

1024 x 768 60, 70, 72, 75, 85

1280 x 1024 60

Configuring Console Redirection in the iDRAC Web Interface

To configure console redirection in the iDRAC Web interface, perform the following steps:

Access the iDRAC’s Web interface from the management station.

Click System and then click the Console tab.

Click Configuration to open the Console Redirection Configuration page.

Using the Video Viewer

The Video Viewer provides a user interface between the management station and the managed system.

It enables you to see the managed server's desktop and control its mouse and keyboard functions from

your management station. The Video Viewer provides various control adjustments. The Help menu

provides for more information on these functions.

When you connect to the remote system, the Video Viewer starts in a separate window. You may need

to adjust the color mode and synchronize the mouse pointers. The bar across the top of the viewer

contains a menu where you can adjust the viewer settings.

Using the Video Viewer

The following table describes the menu options to adjust the viewer settings.

View

Pause Temporarily pauses console redirection.

Resume Resumes console redirection.

Refresh Redraws the viewer screen image.

Capture Current Screen Captures the current remote system screen to a .bmp file on Windows or a .png file on Linux. A dialog box is displayed that allows you to save the file to a specified location.

Full Screen To make the Video Viewer expand into full screen mode, select Full Screen from the Video menu.


Exit When you have finished using the Console and have logged out (using the remote system’s logout procedure), select Exit from the Video menu to close the Video Viewer window.

Mouse

Synchronize Cursor The Mouse menu enables you to synchronize the cursor so that the mouse on the client is redirected to the mouse on the server.

Options

Color Mode Allows you to select a color depth to improve performance over the network. For example, if you are installing software from virtual media, you can choose the lowest color depth (3‐bit gray), so that less network bandwidth is used by the console viewer, leaving more bandwidth for transferring data from the media. The color mode can be set to:

15‐bit color

7‐bit color

4‐bit color

4‐bit gray

3‐bit gray Media

Virtual Media Wizard The Media menu provides access to the Virtual Media Wizard, which allows you to redirect to a device or image such as a:

Floppy drive

Floppy image

CD

DVD

Image in ISO format

USB flash drive For information about the Virtual Media feature, see Configuring and Using Virtual Media. You must keep the Console Viewer window active when using Virtual Media.

Help

Activates the Help menu.

Summary


topic.

The iDRAC implements several management and security features.


o The iDRAC provides both in‐band and out‐of‐band management solutions such as blade

power management, logging critical events, and remote management.

o M‐series iDRACs provide security features such as:

Active Directory authentication and authentication using local users

40‐bit and 128‐bit encryption

IP address restriction when login fails

Since the iDRAC is part of the blade’s system board, no installation is required, but some

configuration is.

o To set up the management station, the administrator should set up Web browsers,

TFTP, and SSH clients.

o When configuring the iDRAC, configure:

Network settings

Users

Active Directory

Event filters

The vKVM

vMedia

o You can also enable services such as Web, SSH, and Telnet.

o The iDRAC network can also be configured using the CMC Web interface.

o When configuring the managed server, the System administrator must:

Install OpenManage System software

Configure the Auto Recovery feature

Enable the Last Crash Screen capture

o System administrators can configure the settings of iDRAC NIC, Platform Event Trap, E‐

mail alerts, and Platform Event Filters through iDRAC Web interface. Most Web

interface configuration tasks can also be performed with local racadm commands or

with SM‐CLP commands.


Module 7: Network Fabrics and IOMs This module provides information on Network

Fabrics A, B and C.


Describe the features of Fabric A and the TOE

key

Describe the features and use restrictions of

mezzanine Fabrics B and C

Compare the features of the three types of

mezzanine cards compatible with Dell

M1000e blade servers

Explain how InfiniBand technology functions and list its advantages

Describe the features and functionality of InfiniBand mezzanine cards and IOM

Chassis Overview

The M1000e chassis can house 6 x I/O

modules. This allows for a greater diversity

of roles for all of the enclosed blade servers.

The 6 x I/O slots are classified as three

separate fabrics. Each fabric contains two

slots, 1 and 2, which relate to the two ports

found on the server side I/O cards. The

fabrics can be used independently of each

other, though each fabric must contain the

same technology. For example, Fabric A is

hardwired to the two LANs On Motherboard

(LOMs) on the blade server mainboards, so

the I/O modules in Fabric A must be

Ethernet. Fabrics B and C can be used for

Ethernet, Fibre Channel and InfiniBand.


Fabric A Broadcom Ethernet Controllers

Fabric A consists of two embedded Broadcom Ethernet controllers. The controller was designed as a

Convergence Network Interface Card (CNIC) that combines network functionality into a single chip.

Fabric A includes:

LAN On Motherboard (LOM)

Fabric A Features

The bidirectional data for a single Broadcom Gb LOM is carried over a PCI Express x4 bus.

SerDes (serial to parallel and parallel to serial) transceivers transmit data across the midplane on

a single lane to the I/O module bays A1 and A2

Support for 10/100/1000Gb Ethernet speeds

iSCSI boot

Wake‐On‐LAN (WOL)

PXE

TOE (TCP Offload Engine)

o Requires optional TOE license key (all systems ship with the TOE key populated)

Dell LOM TOE License Key

Optional LOM features are activated with

the population of the TOE License Key.

TOE License Key


Expansion Cards Overview

The M‐series blade servers support optional Fabric expansion Mezzanine cards designed to support high

speed network communication technologies including:

InfiniBand

10Gb Ethernet

Fibre Channel

Mezzanine Cards

Mezzanine cards supported on the PowerEdge 1000e

Blade Servers include:

Emulex LPe1105‐M4 FC HBA

QLogic QME2472 FC HBA

Dell BCM57086S Ethernet Mezzanine

Mezzanine cards installed in the Fabric B or Fabric C card slot require a matching I/O module installed in

the corresponding I/O bay to support data flow for that fabric.

The AMD Chipset connection to Mezzanine card slot C supports a single PCIe x8 link, which cannot be

split into two PCIe x4 links.

Dell BCM57086S

The Dell BCM57086S Ethernet Mezzanine card requires two PCIe x4 links.

As a result, the M605, M805, and M905 blade servers cannot support the Dell BCM57086S Ethernet

Mezzanine card in Mezzanine card slot C (slot restriction).

The Dell BCM57086S Ethernet Mezzanine card is the only Mezzanine card restricted from slot C;

all other existing Mezzanine cards are supported.

The Dell BCM57086S Ethernet Mezzanine card is supported in Mezzanine card slot B.

This limitation only applies to the M605; the PowerEdge M600 is not affected.

Emulex LPe1105‐M4 FC HBA

The following list details the functionality of the Emulex LPe1105‐M4 FC HBA.

Provides 4, 2, or 1Gb FC connectivity among up to 16 Dell blade servers and a SAN fabric

Comprehensive virtualization capabilities with support for N_Port ID Virtualization (NPIV) and

Virtual Fabric

Administration via HBAnyware® for Emulex HBAs anywhere in the SAN

Common driver with all generations and types of Emulex HBAs

Firmware releases deployed independently from the driver


Support for Industry Standard Fibre Channel Security Protocol (FC‐SP)

QLogic QME2472 FC HBA

The QME2472 HBAs is an "Enterprise Class HBA" specifically designed for the Dell 4 Gbps M1000e Blade

Server Chassis blades M‐series blades.

Provides 4, 2, or 1Gb FC connectivity among up to 16 Dell blade servers and a SAN fabric

Robust interoperability simplifies deployment and upgrades of SAN hardware and software

Administration via SANSurfer

Capability of booting from SAN


Qlogic SANBlade QNE247M Fibre Channel HBA Technical Specifications

Host Bus Specifications Feature Description

Bus Interface X4 PCI Express

Memory 1 MB SRAM per port

1 MB FLASH (SPI)

2 KB NVRAM (SPI)

Compliance Conforms to PCI Express Base Specification rev. 1.0a

PCI Express Card Electromechanical Specification rev. 1.0a

PCI Bus Power Management Interface Specification revision 1.1

Fibre Channel Specifications Feature Description

Data Rate 4/2/1 Gbps auto‐negotiation (4.25/2.125/1.0625 Gbps)

Performance 150,000 IOPS per port

Throughput 800 Mbps per port

Topology Point‐to‐Point (N_Port)

Arbitrated Loop (NL_Port)

Switched Fabric (N_Port)

Logins Support for F_Port and FL_Port login. 2,048 concurrent logins and 2,048 active exchanges

Class of Service Class 2 and 3

Protocols FCP (SCSI‐FCP)

IP (FC‐IP)

FICON (FC‐SB‐2)

FC‐TAPE (FCP‐2)

Buffer Credits 3 credits available (2,112 Byte frame payload)

Compliance SCSI‐3 Fibre Channel Protocol (SCSI‐FCP)

Fibre Channel Physical and Signaling Interface (FC‐PH)

Fibre Channel 2nd Generation (FC‐PH‐2)

Third Generation Fibre Channel Physical and Signaling Interface (FC‐PH‐3)

Fibre Channel Arbitrated Loop (FC‐AL‐2)

Fibre Channel Fabric Loop Attachment Technical Report (FC‐FLA)

Fibre Channel‐Private Loop Direct Attach Technical Report (FC‐PLDA)

Fibre Channel Tape (PF‐TAPE) profile

SCSI Fibre Channel Protocol‐2 (FCP‐2)

Second Generation FC Generic Services (FC‐GS‐3)

Third Generation FC Generic Services (FC‐GS‐3)

Fibre Channel Framing and Signaling (FC‐FS)


IOMs

Information on the IOMs available for Dell

PowerEdge M1000e blade servers follows.

Dell/Emulex PT‐1016 Fibre Channel Pass‐Through

The Dell PT‐1016 by Emulex is designed for use in

situations where a direct connection between each

blade server and external SAN switches is required.

Dell/Emulex PT‐1016

The Dell PT‐1016 by Emulex provides investment

protection for data centers with pre‐existing networking infrastructures by providing a direct connection

between the servers and the SAN. The PTM provides up to 16 4Gb Fibre Channel connections with 1:1,

non‐switched connections between the server blade and the SAN. The PTM design is hot pluggable and

provides for high‐availability failover when deployed in pairs within the server chassis.


Dell/Emulex PT‐1016 Fibre Channel Pass‐Through

Capabilities Provides 4, 2, or 1Gb connectivity between up to 16 Dell blade servers and a SAN fabric

Each port provides a dedicated connection for guaranteed bandwidth to each server port

With 16 external ports in a single blade, the PTM provides the highest port density available in

Dell’s connectivity options

Provides uncompromising scalability to the fullest capability of the Dell chassis

Hot‐plug capability and redundancy provide true high‐availability for 24x7 data center operation

Investment protection: uses existing SAN infrastructure with no interoperability issues

Key Features

16 ports of 4Gb Fibre Channel connectivity between servers and SAN

Transparent connection between SAN and Server

Simplicity of unmanaged design speeds deployment

Simplicity of unmanaged design eliminates configuration hassles

Eliminates interoperability issues with transparent, pass‐through protocol

Compact, ultra‐dense form factor

4, 2 or 1 Gb Fibre Channel connectivity on each port to match existing server and SAN

installation

Hot‐pluggable design eliminates downtime during service actions

Intended to be installed in pairs for redundancy and high availability

Firmware can be upgraded in the field, eliminating downtime (future, not in initial release)

Completely dedicated connection for each server blade

Up to 256Gb full‐duplex bandwidth

Technical Specifications

The Dell PT‐1016 by Emulex Pass‐Through Module has the following technical specifications:

Product Specification Summary Item Description

Physical interface Hot‐pluggable industry‐standard copper and optical SFP transceivers at all ports

Blade type Fibre Channel Pass‐Through

Performance 4Gb Fibre Channel per port, non‐blocking

Port Configuration Sixteen 4Gb Fibre Channel internal connections to Server Blades

Sixteen 4Gb Fibre Channel external uplinks

Port Type 4Gb Fibre Channel optical SFPs, shortwave


Brocade 4424 Fibre Channel Switch

Switch Overview The Brocade 4424 is a 24‐port Fibre Channel module with 8 external ports and 16 internal ports

designed for installation in a blade server chassis. The product ships with 12 ports enabled and is

upgradable to 24 ports through Ports on Demand (POD) licensing. These ports support link speeds up to

4Gbit/sec.

Technical Specifications The following table describes the technical specifications for the Brocade 4424 Fibre‐Channel I/O

modular switch.

Brocade 4424 Features Item Description

Scalability Full fabric architecture with maximum of 239 switches

Certified maximum 32 switches, 7 hops

Performance 1.063 Gbit/sec line speed, full duplex 2.125 Gbit/sec line speed, full duplex

Fabric latency (Fabric Switch Mode only)

<2.1 sec with no contention

Maximum frame size 2112‐byte payload

Class of Service Class 2 Class 3 Class F (interswitch frames)

Port Types FL_port

F_port

E_port

N_port

Fabric services (Fabric Switch Mode only)

Simple Name Server, Registered State Change Notification

PowerPC 440GP processor 333MHz

SDRAM memory 256 MB – for system memory at 64 bits wide with 8 ECC bits


Brocade 4424 Features Item Description

Boot flash memory 4 MB

Compact flash memory 256 MB

InfiniBand Technology

The InfiniBand standard is a key element in the speed and efficiency of Dell

blade server I/O systems.

About InfiniBand

In 1999, two competing input/output (I/O) standards called Future I/O

(developed by Compaq, IBM and Hewlett‐Packard) and Next Generation I/O (developed by Intel,

Microsoft and Sun) merged into a unified I/O standard called InfiniBand. InfiniBand is an industry‐

standard specification that defines an input/output architecture used to interconnect servers,

communications infrastructure equipment, storage and embedded systems. InfiniBand is a true fabric

architecture that leverages switched, point‐to‐point channels with data transfers up to 120 gigabits per

second, both in chassis backplane applications as well as through external copper and optical fibre

connections.

How InfiniBand Works

InfiniBand is a pervasive, low‐latency, high‐bandwidth interconnect that requires low processing

overhead and is ideal to carry multiple traffic types (clustering, communications, storage, management)

over a single connection. As a mature and field‐proven technology, InfiniBand is used in thousands of

data centers, high‐performance compute clusters and embedded applications that scale from two nodes

up to a single cluster that interconnect thousands of nodes.


Advantages of InfiniBand

Powerful multi‐core CPUs, server virtualization, blade architectures, increasing storage demands, and

the need to reduce power usage are all placing increased demands on I/O solutions. InfiniBand can

address these demands while delivering low‐latency in a high‐bandwidth interconnect. Advantages of

InfiniBand include:

Superior performance ‐ InfiniBand is the only shipping solution that supports 20Gb host

connectivity and 60Gb switch to switch links. InfiniBand also provides world‐class application

latency with measured delays of 1µs end‐to‐end.

High efficiency ‐ InfiniBand offloads all data transport mechanisms to the adapter cards to

relieve CPUs so they can be fully used on customer workloads. In addition, InfiniBand uses

Remote Direct Memory Access (RDMA) to optimize data transfer and reduce internal memory

usage to further enable CPUs to focus on application processing.

Fabric consolidation and low energy usage ‐ With less than 5W per 20Gb port InfiniBand can

consolidate networking, clustering, and storage data over a single fabric, significantly lowering

the overall power, real estate, and management overhead required for servers and storage.

Reliable and stable connections ‐ By enabling fully redundant I/O fabrics, with automatic path

failover and link layer multi‐pathing abilities to meet the highest levels of availability, InfiniBand

is perfectly suited to meet the mission critical needs of today’s enterprise data center.


Data integrity ‐ Mission‐critical applications require that data has not been corrupted as it

travels across the fabric. InfiniBand enables the highest levels of data integrity by performing

cyclic redundancy checks (CRCs) at each fabric hop and end‐to‐end across the fabric to ensure

the data is correctly transferred.

These advantages combine to make InfiniBand the industry’s most powerful interconnect.

Cabling

InfiniBand uses a base 2.5‐Gbps clock rate that necessitates that InfiniBand ports and cables are thicker

and bulkier than Unshielded Twisted Pair or UTP, commonly found in Ethernet applications. Generally,

the faster a link is clocked, the more crosstalk is generated. This is because of the rapid rise and fall of

the carrier voltage as bits are transmitted. Additionally, because of the signal attenuation, the receiver

must be able to interpret the signal correctly when it is received. At higher InfiniBand speeds, the effects

of crosstalk and data skew limit cable lengths and consequently limit the number of nodes that can be

supported within a cluster. See the table below.

For InfiniBand 4X installations, although the copper cabling lengths are relatively limited when

compared to Gigabit Ethernet, very large clusters can be constructed by carefully planning cabling runs

and equipment placement. As an example, the world’s largest InfiniBand‐based HPC cluster, built using

Cisco SFS 7000 Series InfiniBand Server Switches, consists of more than 4,500 dual‐processor servers. By

contrast, using copper cabling only, the largest InfiniBand 12X HPC cluster that can be built is 256 nodes

using standard “side‐by‐side” rack configuration. By arranging the racks in a “U” shape, it is possible to

build a 512‐node cluster, although this complicates the cabling and maintenance aspects of the cluster.

Link Speeds

These variables determine at what speed the InfiniBand fabric will run and its capable bandwidth:

Link Width: The number of bonded channels (send/receive pairs) used for an InfiniBand link.

Defined IB link widths are 1X, 4X and 12x.

Signaling Rate: The simplex, raw, bit‐rate of an InfiniBand channel. Defined IB signaling rates are:

o SDR (Single Data Rate ‐ 2.5 Gbps/channel)

o DDR (Double Data Rate ‐ 5 Gbps/channel)

o QDR (Quad Data Rate ‐ 10 Gbps/channel)

Link Speed: The total simplex capacity of an InfiniBand link calculated as the signaling rate of

each channel multiplied by the number of channels present in the link. For example, a 4X DDR IB

Link has a simplex link capacity of 5.0 Gbps X four channels = 20 Gbps in each direction.

QDR signaling rate standard has been defined, but silicon that is able to drive a signal at this rate is not

yet in production by any vendor.


Defined IB Link Speeds

SDR

1X SDR 2.5 Gbps

4X SDR 10 Gbps

12X SDR 30 Gbps

DDR

1X DDR 5 Gbps

4X DDR 20 Gbps

12X DDR 60 Gbps

QDR

1X QDR 10 Gbps

4X QDR 40 Gbps

12X QDR 120 Gbps

InfiniBand Physical‐Layer Characteristics

The InfiniBand physical‐layer specification supports three length widths, designated 1X, 4X, and 12X,

over both copper and fiber optic media, as stated earlier in this module. The base data rate, 1X single

data rate (SDR), is clocked at 2.5 Gbps and is transmitted over two pairs of wires—transmit and

receive—and yields an effective data rate of 2 Gbps full duplex (2 Gbps transmit, 2 Gbps receive). The 25

percent difference between data rate and clock rate is due to 8B/10B line encoding that dictates that for

every 8 bits of data transmitted, an additional 2 bits of transmission overhead is incurred.

InfiniBand 4X and 12X SDR interfaces use the same base clock rate that uses multiple pairs, referred to

as lanes, to increase bandwidth capacity. Therefore, an InfiniBand 4X interface realizes a data rate of 8

Gbps using 8 pairs of wires and a 12X interface realizes 24 Gbps using 24 pairs. Although it is possible for

InfiniBand switches to forward frames between an InfiniBand 1X and 4X, or 4X and 12X link, there is a

performance penalty because of the way that InfiniBand transmits packets.

InfiniBand switches use cut‐through switching to provide low‐latency transport between devices, which

does not require the entire packet to be buffered before being forwarded. This enables InfiniBand to

provide low‐latency network transmission, but at the same time requires that the ingress and egress link

speeds are the same and that the packet is received and transmitted in the same format.

However, when different link speeds are mixed in InfiniBand networks, InfiniBand transmits portions of

the packet in parallel. Therefore, for an InfiniBand 1X link, a packet is transmitted serially; for an

InfiniBand 4X link, the packet is divided into four chunks and transmitted in parallel across the four 2.5‐

Gbps lanes. For a 12X link, a packet is divided into 12 chunks and transmitted in parallel across twelve

2.5‐Gbps lanes.

Parallel Transmission in InfiniBand Links

Because the packet is transmitted in parallel, if a packet is switched between a 12X and a 4X interface, a

store‐and‐forward penalty is incurred to change transmission formats. To maintain the performance of


cut‐through switching, most InfiniBand fabrics consist of a single link speed. For example, all links are 4X

SDR with multi‐path forwarding used to provide additional bandwidth within the InfiniBand network.

However, if an application can tolerate the small store‐and‐forward delay, mixing 12X and 4X in the

same InfiniBand network may be an acceptable solution.

Mellanox ConnectX IB MDI InfiniBand Host Channel

Adapter (HCA) Mezzanine Card

The Mellanox ConnectX IB MDI InfiniBand Host Channel

Adapter (HCA) mezzanine card for the Dell PowerEdge M‐

series blade servers delivers low‐latency and high‐bandwidth

for performance‐driven server and storage clustering

applications in Enterprise Data Center and High‐Performance

Computing environments. Clustered databases, parallelized

applications and transactional services applications will

achieve significant performance improvements resulting in

reduced completion time and lower cost per operation.

ConnectX IB adapters simplify network deployment by

consolidating clustering, communications, storage, and

management I/O and by providing enhanced performance in

virtualized server environments.


Benefits These are the main benefits of the Mellanox ConnectX Dual‐Port InfiniBand Mezzanine Card:

World‐class cluster performance

High‐performance networking and storage access

Guaranteed bandwidth and low‐latency services

Reliable transport

End‐to‐end storage integrity

I/O consolidation

Virtualization acceleration

Scales to tens‐of‐thousands of nodes

Key Features Key features of the Mellanox ConnectX Dual‐Port InfiniBand Mezzanine Card include:

1.2us MPI ping latency

20Gb InfiniBand ports

CPU offload of transport operations

End‐to‐end QoS and congestion control

Hardware‐based I/O virtualization

TCP/UDP/IP stateless offload

Additional Features Additional features of the Mellanox ConnectX Dual‐Port InfiniBand Mezzanine Card include:

World Class Performance and Scalability: Clustered applications running on multi‐socket servers using

multi‐core processors benefit from the reliable transport connections and advanced multicast support

offered by ConnectX lB. End‐to‐end Quality of Service (QoS) enables partitioning and guaranteed service

levels while hardware‐based congestion control prevents hot spots from degrading the effective

throughput. ConnectX lB is capable of scaling to tens‐of thousands of server and storage nodes.

Hardware Offload Architecture: Clustered and client/server applications achieve maximum

performance over ConnectX IB, because CPU cycles are available to focus on critical application

processing instead of networking functions. Network protocol processing and data movement overhead

such as RDMA and Send/Receive semantics are completed in the adapter without CPU intervention.

Applications using TCP/UDP/IP transport can achieve industry‐leading throughput when run over

ConnectX lB and its hardware‐based stateless offload engines.

I/O virtualization: ConnectX lB support for hardware‐based I/O virtualization is complementary to Intel

and AMD virtualization technologies. Virtual machines (VM) within the server are enabled with

dedicated I/O adapter resources and guaranteed isolation and protection. Hypervisor offload features

remove software‐based virtualization overheads and free up CPU cycles enabling native operating

system performance for VMs and higher server utilization by supporting more VMs per physical server.


Storage Accelerated: A unified InfiniBand cluster for computing and storage achieves significant cost‐

performance advantages over multi‐fabric networks. Standard block and file access protocols leveraging

InfiniBand RDMA result in high‐performance storage access. Data reliability is improved through the use

of Tb0‐compliant Data Integrity Field (DIF). Fibre Channel (FC) over InfiniBand (FCoIB) features enable

the use of cost‐effective bridges for connecting to FC SANs.

Software Support: All Mellanox adapter cards are compatible with legacy TCP/IP and OpenFabrics‐based

RDMA protocols and software. They are also compatible with InfiniBand and cluster management

software available from OEMs. The adapter cards are supported with major operating system

distributions.

Technical Specifications The following table shows the technical specifications for the Mellanox ConnectX Dual‐Port InfiniBand

Mezzanine Card.

PowerEdge M1000e (InfiniBand Interconnect) Specifications Feature Description

InfiniBand IBTA Specification 1.2 compliant

10 or 20Gb per port

RDMA, Send/Receive semantics

Hardware‐based congestion control

Atomic operations

16 million I/O channels

256 to 4Kbyte MTU

2GB messages

9 virtual lanes: 8 data + 1 management

Enhanced InfiniBand Hardware‐based reliable transport

Hardware‐based reliable multicast

Scalable Reliable Connected transport

Enhanced Atomic operations

Service oriented I/O

Fine‐grained end‐to‐end QoS

Hardware‐Based I/O Virtualization Address translation and protection

Multiple queues per virtual machine

Native OS performance

Complimentary to Intel and AMD I/OMMU

Additional CPU Offloads TCP/UDP/IP stateless offload

Intelligent interrupt coalescence

Full support for Intel I/OAT

Compliant to Microsoft RSS and NetDMA

Storage Support T10‐compliant Data Integrity Field support

Fibre Channel over InfiniBand (FCoIB)

CPU Compatibility AMD X86, X86_64

Intel X86, EM64T, IA‐32, IA‐64


PowerEdge M1000e (InfiniBand Interconnect) Specifications Feature Description

PCI Express Interface PCIe Base 2.0 compliant, 1.1 compatible

2.5GT/s link rate x8 (20+20Gb bidrectional bandwidth)

Fits Dell PowerEdge M1000e server blades

Support for MSI/MSI‐X mechanisms

Connectivity Works with Dell PowerEdge M600 and PowerEdge M605 blades

Management and Tools OpenSM

Interoperable with third‐party subnet managers

Firmware and debug tools (MFT, IBADM)

Operating Systems/Distributions Novell SLES, Red Hat Enterprise Linux (RHEL), Fedora, and other Linux distributions

OpenFabrics Enterprise Distribution (OFED)

Protocol Support Open MPI, OSU MVAPICH, HP MPI, Intel MPI, MSMPI, Scali MPI

IPoIB, SDP, RDS

SRP, iSER, FCoIB and NFS RDMA

uDAPL

InfiniBand IOM: Cisco SFS M7000E IB

Switch

Cisco SFS7000E Series InfiniBand Switches

provide an ideal server interconnect for

distributed application environments,

such as high‐performance computing. The

Cisco SFS7000E Series meets the

performance demands of distributed

applications by offering:

The unprecedented bandwidth of InfiniBand

Superior latency characteristics

Enterprise‐class high availability and management features

The latest evolution of the Cisco SFS 7000 Series supports dual‐speed InfiniBand 4X double data rate

(DDR) and single data rate (SDR) interfaces that deliver 20 Gbps and 10 Gbps bandwidth respectively,

per port. The low latency and high bandwidth offered by Cisco SFS 7000D InfiniBand Server Switches

enable a new class of distributed applications and systems that deliver greater business agility and

competitive advantage.

Overview The Cisco SFS M7000E Switch delivers an ideal combination of price and performance with the Dell

M1000e Blade platform. The switch provides DDR InfiniBand connectivity to up to 16 blade server


modules and provides 8 DDR InfiniBand uplink ports. Each DDR InfiniBand port can support 20 Gbps of

fully non‐blocking bandwidth to each port. The port‐to‐port latency through the switch is 140 ns.

When fully populated, the Dell M1000e can have four Cisco SFS M7000E blade modules to drive 1.28

terabits of nonblocking traffic for I/O‐intensive applications. The blade switch is also a hot‐pluggable

expansion module, allowing on‐the‐fly capacity expansion without application disruption. The Cisco SFS

M7000E is designed to be a compelling high performance computing (HPC) solution in the blade market.

The Cisco SFS M7000E has been certified with the Cisco SFS 7000 Series InfiniBand Server Switches and

Cisco SFS 3000/3500 Multi‐fabric Server Switches. Cisco provides a complete DDR InfiniBand switching

family to complement the Cisco SFS M7000E switch to deliver HPC solutions. For example, by using the

Cisco SFS‐7024D Switches in the core (288‐port, 4x DDR InfiniBand switches) to aggregate multiple blade

chassis, a customer can build clusters with tens of thousands of processors using Dell M1000 blade

systems.

Because the Cisco SFS M7000E is an unmanaged switch, a subnet manager is required for the operation

of the InfiniBand fabric. Cisco recommends running the Cisco High Performance Subnet Manager on a

blade server module in the InfiniBand fabric. The Cisco SFS M7000E is also qualified with other SFS

management applications such as the Fabric Analysis and Correlation Toolkit.


Technical Specifications

The following tables show the technical specifications for the Cisco SFS M7000E InfiniBand Dual‐Port

Switch.

Cisco SFS M7000E InfiniBand DualPort Switch Features Feature Description

20‐Gbps (DDR), 4x InfiniBand Ports The Cisco SFS M7000E offers 24 4x DDR IB ports of 20‐6bps links for the most demanding high‐performance computing (HPC) applications. The switch has full bisectional bandwidth of 960 Gbps. Each port is autosensing in both single‐data rate (SDR) and double‐data rate (DDR) modes. 16‐ports provide connectivity to server modules and 8 uplink ports are available for external connectivity.

Low Latency The Cisco SFS M7000E is a cut‐through switch with less than 140 nanoseconds of port‐to‐port latency. The switch is ideal for applications that have large inter‐process communication (IPC) or simply depend on quick movement of data for competitive advantage in the market.

Powered Ports All ports are powered to enable connectivity with passive and active copper cables as well as optical cables.

Hot‐Plug Swappable The Cisco SFS M7000E Switches are hot‐plug swappable into any of the four I/O slots in the rear of the Dell M1000E blade chassis. The administrator can reconfigure and maintain the fabric without powering down the server nodes.

Scale‐out Fabric Topologies Each server blade may contain up to 2 HCA cards. When the blade chassis is fully populated with the Cisco SFS M7000E Switches, a customer can build multiple, independent fabrics with the same server nodes. This type of configuration offers extremely high throughput.

Unmanaged Switch The Cisco SFs M7000E is an unmanaged switch to provide a power‐efficient, cost‐effective solution for HPC customers.

Complete Ecosystem Cisco offers an InfiniBand platform that has been certified and extensively tested to provide optimal productivity. The Cisco SFS M7000E is part of a greater InfiniBand offering from Cisco.

Cisco SFS M7000E InfiniBand DualPort Switch Technical Specifications Feature Description

Size (H x W x D) 1.15 x 12.10 x 10.75 in.

System Weight 3.75 lb.

Temperature Operating: 40 to 120°F (5 to 50°C)

Nonoperating: ‐40 to 140°F (‐40 to 60°C)

Power Consumption 56W

Environmental Standards ROHS‐5 compliant, WEEE compliant

Mean time between Failure (MTBF)

>100,000 hours


Diagnostic Indicators

Diagnostic LEDs The following table shows the diagnostic indicators to troubleshoot the Cisco SFS M7000E InfiniBand

switch.

LED Type Indicator Description

Port Status LEDs Diagnostic Status (DIAG) Blinking Green: Traffic Active

Solid Green: Logical link exists

Off: Link error or Subnet Manager is not running

System Status LEDs Diagnostic Status (DIAG) Solid Blue: System is operating normally

Blinking Blue: POST is in progress

Amber: System is receiving power but is not functioning properly

Off: System power is off

Power Status LEDs Diagnostic Status (DIAG) Solid Green: Power is applied to the switch

Off: Power is not applied to the switch

Power‐On Indicators

The following LED sequence occurs after installation into the chassis I/O bay and the switch is functional.

1. Switch Status Power LED turns GREEN after a few seconds. CMC module in Blade system verifies

that the Switch is correctly connected to an InfiniBand HCA.

2. Port Status LEDs turn GREEN after approximately 30 seconds.


3. Switch Status LED turns BLUE.

After installation into the chassis I/O bay, if the switch is not operational, the following are true:

1. The Switch Status Power LED does not turn GREEN.

2. The Switch Status LED blinks AMBER.

Summary

Fabric A consists of two embedded Broadcom Ethernet controllers, which are designed as a

(CNIC) that combines network functionality into a single chip.

o The bidirectional data for a single Broadcom Gb LOM is carried over a PCI Express x4 bus

connected to the ESB2‐D Southbridge on the M600 or the MCP55 on the M605.

o Optional LOM features are activated with the population of the TOE License Key.

Optional Fabric expansion Mezzanine cards designed to support high‐speed network

communication technologies in Fabrics B and C include:

o 10Gb Ethernet

o Fibre Channel

o InfiniBand

The data for each mezzanine fabric is carried over a PCI Express x8 interface connected to the

Northbridge.

Mezzanine cards supported include:

o Emulex LPe1105‐M4 FC HBA

o QLogic QME2472 FC HBA

o Dell BCM57086S Ethernet Mezzanine

Mezzanine cards (which require a matching I/O module installed in the corresponding I/O bay)

installed in the Fabric B or Fabric C card slot require a matching I/O module installed in the

corresponding I/O bay to support data flow for that fabric.

o The PowerEdge M605 cannot support the Dell BCM57086S Ethernet Mezzanine card.

IOMs in Dell PowerEdge blade servers include:

o Dell/Emulex PT‐1016 Fibre Channel Pass‐Through, designed for use in situations where a

direct connection between each blade server and external SAN switches is required.

o Brocade 4424 24‐port Fibre Channel switch

InfiniBand is an industry‐standard specification that defines an input/output architecture used

to interconnect servers, communications infrastructure equipment, storage and embedded

systems.

o InfiniBand addresses the needs of today’s complex and powerful servers by providing:

Superior performance

High efficiency

Fabric consolidation and low energy usage

Reliable and stable

Data integrity


o The Mellanox ConnectX IB MDI InfiniBand Host Channel Adapter (HCA) mezzanine card

delivers low‐latency and high‐bandwidth for performance‐driven server and storage

clustering applications.

o Cisco SFS7000E Series InfiniBand Switches offer:

The unprecedented bandwidth of InfiniBand

Superior latency characteristics

Enterprise‐class high availability and management features


Module 8: Troubleshooting

Common Configuration Issues for Blades

Objectives

In this section you will learn about common configuration issues that can avoid the need for parts

replacement.

Receive Message stating that the CMC’s are not redundant

Solutions

Plug both CMC’s into management network via Ethernet cable. Primary CMC will have a blue status indicator led.

CMC firmware does not match. Make sure both CMC’s are running the same firmware version.

Update the Firmware

Firmware Updates are critical for M1000e and blades. Update all components to current versions at

the same time. If this is not done, unexpected errors or behavior may occur from the CMC, iDRAC,

and/or blade. Plan for maintenance windows to perform updates.


CMC

iKVM

iDRAC

Bios

M6220 firmware

LOM BIOS/firmware

HBA Mezzanine BIOS/firmware

FLEXADDRESS INFORMATION: See http://www.delltechcenter.com/page/FlexAddress for information

on FlexAddress.

An example of why it is important to update all firmware/bios to current versions across all components

in the enclosure and blades as mentioned in item 2. Even if FlexAddressing is not enabled, the CMC

requires all pertinent component firmware be at a version that supports FlexAddressing. See items 3

and 4.

Message received in the iDRAC sel log

When the message “Link tuning sensor failed” is received:

Do not replace hardware. Update the firmware for the appropriate device listed in the error message. This could be the iDRAC, LOM, or HBA Mezzanine card.

Example of error: LinkT/FlexAddr: Link tuning sensor failed to get link tuning or flex address data from BMC/iDRAC

FC (QLGC or EMLX) Mezzanine would experience "F1/F2" during post.

Do not replace hardware. Update the QLGC or EMLX mezzanine BIOS.

QLogic QME2472 – BIOS 2.04 or later

Emulex LPe1105‐M4 – BIOS 3.03a3 and firmware 2.72A2 or later

Server non responsive at power up

A new installation or configuration change, the most common problem is fabric mismatch. The blade status/identification indicator will blink amber if there is an installed mezzanine card that does not match the I/O module installed in the M1000e enclosure.


Explanation of fabrics: See CMC get dc info and get io info output below. Notice that the font color matches the fabric types.

Fabric A will always be Ethernet – Switch 1 and switch 2 correspond to the LOM’s in each of the blades. Note that the LOM’s are not listed under the server Daughtercard (DC1 type, DC 2 type) information below. (the term daughtercard is used instead of mezzanine card)

Fabric B can be either Ethernet or fibre – switch 3 and switch 4 correspond to daughtercard (DC1 type). Note that both switch 3 and switch 4 fabric must match. Example of unsupported configuration: switch 3 = Ethernet and switch 4 = fibre. Also note that the daughtercard fabric must match the switch fabric. In the example below, fabric B is Ethernet.

Fabric C can be either Ethernet or fibre – switch 5 and switch 6 correspond to the daughtercard (DC2 type) – Note that both switch 5 and switch 6 fabric must match. The daughtercard (DC2 type) must match the switch fabric. In the example below, fabric C is Fibre.

It is possible to install one switch for a fabric and leave the second switch blank. This is a valid configuration.

‐ If a switch is not installed, no mezzanine cards can be installed for that fabric. ‐ If a switch is installed, not all servers must have the corresponding mezzanine card installed.

Troubleshooting Resource

Dell Enterprise Technology Center ‐ http://www.delltechcenter.com

1. Use menu on left side of page to choose Blades:


2. At the bottom of the page choose from several information resources:

Support.dell.com

Firmware can manually be downloaded from this site. Information is located under servers.

1. Choose Drivers and Downloads from the main page:


2. Choose method for finding information as appropriate.

Ftp.dell.com

FTP site where updates are available. This site is automatically accessed by the servers.


Module 9: Navigating Dell Information and Tools Dell Enterprise is committed to offering customers information and support through many channels. Our

engineers and subject matter experts are dedicated to bringing you experience and guidance, technical

“how to” tips, tools, best practices and solutions.

Upon completion of this module, you will be able to:

Navigate through many of the support sites and tools designed and offered by Dell Enterprise.

Recognize the benefits of these tools.

Relate the relevancy of this information to your job.

Dell System E‐Support Tool (DSET)

The Dell System E‐Support Tool (DSET) provides the ability to collect hardware, storage and operating

system information from a Dell PowerEdge server. This information is consolidated into a single System

Configuration Report that can be used for troubleshooting or inventory collection of a system. The

browser user interface provides a convenient means to view specific data through hierarchical menu

trees.

Navigate and explore the Dell System E‐Support Tool (DSET) at this link:

http://support.dell.com/support/topics/global.aspx/support/en/dell_system_tool?c=us&l=en&s=gen

Support.dell.com

This comprehensive support site offers a wide‐variety of technical support options. You may research

your issues using Dell manuals and forums as well as check on order status and download drivers you

need. The site offers comprehensive information on all Dell products and features links to many tech

support, customer care, and chat options.

Navigate to the Dell’s Support page at this link: http://support.dell.com/

1. Find the Troubleshooting and FAQs section and click the link. Follow these steps to use the troubleshooting wizard:

a. Choose Select a Model.

b. With Select your Product Model highlighted, select Servers, Storage, Networking.

c. With Servers, Storage, Networking highlighted, select PowerEdge Server.

d. With PowerEdge Server highlighted, select the model of your choice, then confirm your choice.


e. Spend 10 minutes using the wizard.

2. Support.dell.com offers all customers options for signing up and managing Dell Technical Update information. The options include Dell Technical Updates and Management of Your Updates. The site requires that you sign up initially for an account after which updates concerning your equipment will automatically be e‐mailed to you.

a. Click this link, then review the web page: http://support.dell.com/support/topics/global.aspx/support/ notifications /en/index?c=us&l=en&s=gen&~ck=ln&lnki=0

b. Sign up for this service now or save the link for future reference.

Learndell.com

Dell offers comprehensive training and certification programs for Dell hardware and popular software

applications. Training can be customized to meet your unique requirements.

Navigate to the LearnDell site at this link:

http://www.learndell.com/Dell/

Spend five minutes reviewing the learning opportunities offered through Dell Learning.

Enterprise Support Services

Each level of Dell’s Enterprise Support incorporates different options, allowing Dell and you to create the

right match for your unique needs. Dell also offers help to ensure on‐going performance of your EMC

technology.

Navigate and explore the Enterprise Support site at this link:

http://www.dell.com/content/topics/global.aspx/services/ent_support/ent_support?c=us&cs=555&l=e

n&s=biz

Spend five minutes reviewing the different support options offered by Dell Enterprise.

Thank You!

This concludes the Dell Blade Server Configuration Course. Dell invites you to consider the following

related courses:

Dell SAN Data Protection.

2 day course

Dell SAN Management.

3 day course

Dell OpenManage and System Management.

2 day course


PowerEdge Configuration and Troubleshooting.

2 day course

Dell Desktop and Laptop Configuration and Troubleshooting.

2 days

Go to www.learndell.com to register!

dellcert - blades training

Documents