ern-ipm devkit: user guide...cern-ipmc devkit – user guide ep-ese-be 5 2.1 power the ern-ipm...

CERN-IPMC DevKit: User guide

This document introduces the CERN-IPMC Development kit. This setup is a start-up platform that

can be used as a module tester or a debugging platform. This paper lists the features available

and how to use them.

CERN-IPMC DevKit – User guide v.4.0b

Sources: CERN-IPMC – https://cern-ipmc.web.cern.ch Support e-mail – [email protected]

https://cern-ipmc.web.cern.ch/

mailto:[email protected]

CERN-IPMC DevKit – User guide EP-ESE-BE

2

Table of Contents 1. Introduction ......................................................................................................................................... 3

2. Hardware ............................................................................................................................................. 4

2.1 Power .................................................................................................................................................. 5

2.2 Management interface ........................................................................................................................ 5

2.3 UART interfaces ................................................................................................................................. 6

2.4 Ethernet .............................................................................................................................................. 6

2.5 I2Cs ..................................................................................................................................................... 6

2.6 IPMB Bus ........................................................................................................................................... 7

2.7 IPMC JTAG interface ......................................................................................................................... 7

2.8 JTAG Master Interface ....................................................................................................................... 8

2.9 Management controller reset .............................................................................................................. 8

3. Software ............................................................................................................................................... 9

3.1 Connection to Raspberry PI ................................................................................................................ 9

3.2 JTAG Player ....................................................................................................................................... 9

3.3 IPMB Analyser ................................................................................................................................. 10

3.4 DevKit Control tool .......................................................................................................................... 12

3.5 IPMC Test tool.................................................................................................................................. 13


1. Introduction

AdvancedTCA standard defines Intelligent Platform Management Controllers (IPMC) to

control the user blade. It is responsible for hardware management and monitoring with the hot

swap control and temperature/voltage checking. As the module is mandatory for all of the blades

and ATCA was selected for the electronics upgrade of the CERN experiments. The EP-ESE group

has designed an IPMC solution based on the Pigeon Point’s product. It was adapted to be

configurable and run on a common hardware providing all of the interfaces required to control

a blade. Therefore, the pinout provides connectivity to: 9 AMC ports, 51 User I/Os, 3 UART

interfaces, 2 I2C buses, one JTAG master interface as well as all of the specified pins required to

detect the blade position and status. Additional details can be found in the technical overview

document1.

In this context, a system was designed to test all of the interfaces after production. This

product was extended as a development kit for the user as it offers a set of tool that helps

debugging and learning how the module works.

1 CERN-IPMC Tecnical overview: https://cern-ipmc.web.cern.ch/doc

https://cern-ipmc.web.cern.ch/doc


4

2. Hardware

The CERN-IPMC development kit is made of an electronic cards that can be used out or in

an AdvancedTCA crate. The block diagram below details the system.

According to scheme, all of the interfaces are connected. The system is supervised via the

uMGT controller, which receives UART commands on its dedicated USB port – named

Management. The “Management interface commands” annexe lists all of the instructions that

can be used. Because of the limited number of ports on the 8bit controller, a CPLD is used as a

multiplexer for the user I/Os. The same idea has been used to avoid duplicating the AMC

emulation bloc 9 times. Therefore, only one I/O or one AMC port can be tested at the time. The

master JTAG port is connected to a 10 pin header – named Master JTAG –, allowing programming

an external FPGA, while the slave JTAG to program the IPMC is either connected to a second 10

pin header – named IPMC JTAG – and the raspberry pi. All of the I2C ports are connected to

multiplexing mechanism, which can be used to redirect a port to a connector and GPIOs pins of

the raspberry pi. Additional components, which are a sensor temperature and an EEPROM

memory, are also connected to the sensor and management buses coming from the IPMC.

Finally, the IPMB-A/B buses are also connected to a 10 pin connector compatible with the shelf

manager development kit from Pigeon Point (KIT BTC 7200R ATCA).


5

2.1 Power

The CERN-IPMC devkit is designed to work in both standalone and in-crate mode.

Therefore, a power connector is located on the front side of the blade and can be used as

described below:

Figure 1:Power connector

In order to protect the board of input conflicts, a switch allows selecting the power source

from external or backplane. The 3 position switch selects:

Top position: external power

Middle position: not connected

Bottom position: backplane source

The picture bellows show the switch, which is located next to the external input connector.

Finally, the additional jumper SW1 connects the 5 volts to the input of the 3.3 volts

regulator, which supply the IPMC module and management controllers. Removing the jumper

turn-off the module under test, allowing the operator to change the IPMC without rebooting the

raspberry pi.

2.2 Management interface

The CERN-IPMC development kit control is made via the USB to Serial interface available

on the “management” connector located on the front of the card. This input is linked to the


6

ATMEGA128 controller through an FTDI chip, which takes care of the USB to Serial conversion.

Once the FTDI drivers are installed on the master computer, a new serial port is detected and

can be used using the following configuration:

9600 Bauds,

8bit,

No parity,

No control.

The implemented commands are listed into the “Management interface commands”

annexe, located at the end of the document.

2.3 UART interfaces

All of the UART interfaces available on the CERN-IPMC modules are routed to USB ports

using FTDI chip for translation. In this context, UART[0] is connected to pins 57/60, UART[1] to

pins 58/61 and UART[2] to pins 75/76 or 92/93 depending on the SW4/SW5 configuration. The

latest UART is the optional one from the CERN-IPMC. It is based on user I/Os pins and can be

routed to USER_IO[0:1] (pin 75/76) or USER_IO[24:26] depending on the SW4/SW5

configuration. The pictures below shows the options:

2.4 Ethernet

An Ethernet port is available on the IPMC mezzanine card. It can be used to send IPMI

commands (RMCP/RMCP+) directly to the controller, to open a Serial Over LAN (SOL) session

and/or to configure a Xilinx FPGA using the Xilinx Virtual Cable server. This port is directly

connected to the Ethernet connector located on the bottom of the front side. It can be directly

connected to the Raspberry pi.

2.5I2Cs

The CERN-IPMC implements 3 types of I2C buses: IPMB (A, B and L, which are connected

to the MMC or the IPMB connector), the sensor I2C and the Mgt I2C. In order to test these two

last buses, a temperature sensor (LM75) is connected to the Sensor I2C bus and an EEPROM

(AT24C64D) to the Mgt bus. Additionally, all of the I2C buses can be redirected to the I2C OUT


7

connector through I2C buffers, which are enabled depending the configuration set by the

management controller. The picture below shows the connector pinout:

Additionally, the IPMB interfaces, as well as the output selected on I2C OUT, are routed to the

Raspberry pi connector, allowing to spy the buses and analyse IPMI traces using Wireshark.

Additional details are introduced in the Software section of the document.

2.6 IPMB Bus

In order to allow using the CERN-IPMC on benches, the IPMB-A and IPMB-B buses are

routed to a 10 pins connector on the top of the IPMC development kit. The pin assignment was

designed in a way to be compatible with BTC 7200R ATCA kit from Pigeon Point. However, when

the cars is used within a crate, the IPMB buses are routed to the backplane connector and the

IPMB-A/B one should be left unconnected. The picture below shows the pinout of the 10 pins

connector:

2.7 IPMC JTAG interface

Even if the HPM.1 is recommended to configure the module with the latest version of the

software. The mezzanine card implements a JTAG connector allowing to flash the controller.

Therefore, this connectivity is reported to a 10 pin connector located on the top of the card –

IPMB JTAG. It connects the pins to be compliant with the Flashpro tool from Microsemi as

described below:


8

Additionally, the JTAG slave interface is also connected to the raspberry pi, allowing to

program the card via a JTAG player running on GPIOs. In case of Raspberry Pi use, the SW6 jumper

has to be set to VCC (RPI configuration is not supported yet). How to configure the controller via

a Raspberry pi is described in the Software section.

2.8 JTAG Master Interface

The CERN-IPMC JTAG master interface is directly routed from the DIMM-DDR3 VLP

connector to a 10 pin one, located on the top of the card – MASTER JTAG. It allows connecting

the interface to external devices, like Xilinx development kit. The module was tested, at CERN,

with a Basys 3 development kit from Diligent. The picture below describes the 10 pin connector:

2.9 Management controller reset

The management controller of the development kit can be set in reset by connecting the SW3 jumper.


9

3. Software

The CERN-IPMC development kit software runs on a Raspberry pi version 3B+, included in

the setup. The operating system, based on Raspbian – Linux –, is pre-installed on the SD card and

support a set of tools allowing to play with the setup.

In order to use the tools described below, one USB port of the Raspberry has to be

connected to the Management interface, as well as the Ethernet port.

3.1 Connection to Raspberry PI

On the first boot, the WLAN interface is not configured. Therefore, it is recommended to

connect the HDMI output to a monitor and directly control the raspberry from a local keyboard.

The graphical user interface can be used to configure the wireless network in order to get the

system ready. Once the module is connected to the network, it is possible to access via SSH with

the following credentials:

Username: pi

Password: raspberry

Note: It is strongly to keep the Ethernet interface configuration as they are, except for expert

person.

3.2 JTAG Player

The JTAG player tool allows reconfiguring the CERN-IPMC with the default binaries.

Therefore, the user customization will be loose. It is recommended to use the interface when

new firmware is available or to reset a module to manufactory default. The program can be

simply ran using the “jtag_player -reset” command.


10

When the configuration finishes, the program print a dictionary providing details of the

different programming stages. On success, the result shall be similar to the one presented in the

picture below:

The JTAG player tool can also be used to flash a custom firmware by replacing the -reset

instruction by a filename. However, this action is not recommended except in case of a specific

message from the support team.

3.3 IPMB Analyser

The IPMB analyser tool allows sniffing one of the IPMB bus connected to the CERN-IPMC:

IPMB-A, IPMB-B or IPMB-L. It captures all of the bytes exchanged on the I2C lines and store it into

PCAP files that can be read with Wireshark2. It can be executed using the following command:

> ipmb_sniffer -t <timeout> -f <output file> -p <port>

2 A specific version of wireshare, patched to analyse IPMB packet, can be downloaded here.

https://espace.cern.ch/ph-dep-ESE-BE-ATCAEvaluationProject/PP_IPMC/Public%20documents/wireshark-1.10.6-IPMB.zip


11

Where the parameters are:

Timeout: Acquisition time in second. Wait for Enter to be pressed when not specified.

Output file: Path to the output file where the data are saved.

Port: Select IPMB port to spy. The value can be ipmba, ipmbb or ipmbl.

The picture below shows an example of data capturing with a timeout set to 60 seconds,

on the IPMB-L bus.

When the timeout value is not set, the program wait for the touch Enter to be pressed to

stop. In this case, the acquisition delay is not based on time but on a user input.

Finally, the files can be extracted from the Raspberry using an SCP program or via

MobaXterm for example. Then, it can be opened in Wireshark, which can analyse the transaction

and make them verbose. The picture below shows the result of the IPMB-L acquisition.


12

3.4 DevKit Control tool

The DevKit Control tool allow playing with the IPMC interfaces connected to the uMGT

controller. It sends the instruction detailed in the “Management interface commands” annexe

depending on the action passed in parameter. The list of available commands can be get using

the -h argument:

> devkit_ctrl -h

Depending the command, additional parameter might be requested. Using the “-a <action>

-h” argument explains the usage of the specific command. Then, calling the program with the

right action and arguments display the result in the terminal. The picture below shows a call of

the get_power instruction.

Because it might be interesting to script a list of instruction, the tool also support JSON

output. It shall be specified using the “--json" argument when calling the program, as illustrated

below.


13

3.5IPMC Test tool

The IPMC Test tool perform a check of the CERN-IPMC hardware. It is based on the same

test that is performed on module reception after production. The tester print information during

the execution and can store all of the result in a pickle file when the output argument is defined

(optional). The tester can be executed using the following command:

> ipmc_test [-o <output.p>]


14

Annexe 1: Connectors description

The table below lists all of the connectors:

REF. DESCRIPTION

J1 Do not used (uMGT JTAG connector)

J2 Do not used (AMC JTAG connector)

J3 Do not used (CPLD debug connector)

MASTER JTAG Master JTAG from IPMC:

Pin 1: TCK

Pin 3: TDO

Pin 4: VCC (3.3V)

Pin 5: TMS

Pin 6: TRST

Pin 9: TDI

Pin 10: GND

IPMC JTAG Slave JTAG to CERN-IPMC:

Pin 1: TCK

Pin 2: GND

Pin 3: TDO

Pin 6: VCC (3.3V)

Pin 5: TRST (Payload_nrst)

Pin 10: TDI

IPMB-A/B IPMB-0 connector:

Pin 1: IPMB-B SCL

Pin 2: GND

Pin 3: IPMB-B SDA

Pin 4: GND

Pin 6: GND

Pin 7: IPMB-A SCL

Pin 8: GND

Pin 9: IPMB-A SDA

Pin 10: GND

I2C OUT I2C output from mux. selection:

Pin 1: SCL

Pin 2: GND

Pin 3: SDA

Pin 10: GND


15

Annexe 2: Jumpers description

The table below describes all of the jumpers:

SWITCH DESCRIPTION

SW1 IPMC and uMGT enable: connect IPMC and uMGT power to VCC.

SW2 -48V to 5V DC/DC enable: turn ON backplane power DC/DC

SW3 uMGT reset: reset uMGT controller when set.

SW4/SW5 UART[2] routing:

No jumpers: not connected

[0:1]: connected to user I/O[0:1]

[24:26]: connected to user I/O[24:26]

SW6 IPMC JTAG TRST signal (used as payload_nrst during runtime):

No jumper: recommended except for programming. Pin

used as payload_nrst.

VCC: Force TRST signal for module programming

RPI: Connected to a raspberry pi GPIO (automatic

control)


16

Annexe 3: Management interface commands

COMMAND DESCRIPTION

DEVKIT RELATED COMMANDS

GETDEVICE Returns a description string “IPMCTP_CTRLER_V1.0”

GETMGTVOLTAGE Get MGT voltage (must be around 3.3V)

GETIPMCVOLTAGE Get IPMC voltage (must be around 3.3V)

GETIPMCCURRENT Get IPMC drawn current

SELI2C 0x<id> Select I2C port to redirect on “beagle_ipmb_l1” connector:

<id> values:

00: IPMB-A

01: IPMB-B

02: IPMB-L

03: IPMC Sensor

04: IPMC Mgt

ATCA RELATED COMMANDS

SETHA 0x<addr> Set the IPMC Hardware address (addr must be an hex. value)

SETHSW Set the IPMC’s handle switch pin to VCC

CLRHSW Set the IPMC’s handle switch pin to GND

SETIO 0x<id> Set the selected IO to VCC (id must be an hex. value)

CLRIO 0x<id> Set the selected IO to GND (id must be an hex. value) *1

GETIO 0x<id> Get value of the selected IO (id must be an hex. value) *1

GETBLUELED Get blue led status *2

GET12VEN Get 12V Enable status

ASSERTPGOODA Assert PowerGood A signal

ASSERTPGOODB Assert PowerGood B signal

DEASSERTPGOODA De-assert PowerGood A signal

DEASSERTPGOODB De-assert PowerGood B signal

AMC RELATED COMMANDS

SETAMCGA 0x<addr> Set AMC’s geographical address

SELAMC 0x<port> Select the IPMC port to be connected to the AMC

SETPS1 Emulate AMC insertion

CLRPS1 Emulate AMC extraction

CLOSEAMCHS Close AMC handle switch

OPENAMCHS Open AMC handle switch

GETAMCMPEN Get MP Enable signal from AMC emulator

GETAMCPPEN Get PP Enable signal from AMC emulator

ASSERTMPGOOD Assert MP good signal on AMC emulator

DEASSERTMPGOOD De-assert MP good signal on AMC emulator

ASSERTPPGOOD Assert PP good signal on AMC emulator

DEASSERTPPGOOD De-assert PP good signal on AMC emulator

ern-ipm devkit: user guide...cern-ipmc devkit – user guide ep-ese-be 5 2.1 power the ern-ipm...

Documents