lhcb’s experiment control system

Clara Gaspar, March 2006

LHCb’s Experiment Control System

Step by Step

2Clara Gaspar, March 2006

Overview❚ LHCb’s Experiment Control System

❙ What do we (JCOP/LHCb) provide❙ What sub-detectors/sub-systems need to implement

❚ PVSS & Framework reminder❚ Interfacing Electronics Boards

❙ SPECS & CC-PC Tools❙ The Configuration DB

❚ Hierarchical Control❙ The FSM Toolkit

❚ Note: This tutorial is meant as an overview❙ The PVSS & Framework and the FSM courses are still

required in order to use the tools!


ECS Scope

Detector Channels

Front End Electronics

Readout Network

High Level Trigger

Storage

L0

Expe

rimen

t Con

trol S

yste

m

DAQ

DCS Devices (HV, LV, GAS, Temperatures, etc.)

External Systems (LHC, Technical Services, Safety, etc)

TFC


ECS Generic Architecture

...

To Devices (HW or SW)

Com

man

dsSt

atus

&

Alar

ms

ECS

DCS DAQ

DetDcs1

DetDcsN

SubSys1

SubSys2

Dev1

Dev2

Dev3

DetDaq1

SubSysN

DevN

LHCT.S.

...GAS

DSS

Abst

ract

leve

ls


What do we provide?❚JCOP + LHCb Online provide:

❙Not complete applications, but:❙A Framework, i.e. a set of tools to help sub-

systems create their control systems:❘Complete, configurable components (ex. CAEN HV)❘Tools for defining User Components:

〡Electronics boards (SPECS/ CC-PC)〡Specific equipment/software tasks (DIM protocol)

❘Other Tools, for example:〡FSM for Building Hierarchies〡Configuration DB 〡Archiving, Alarm handling, etc.


We also provide:❚ Integration of Infrastructure Services:

❘Power Distribution and Rack/Crate Control❘Cooling and Ventilation Control❘Magnet Control (Monitoring)❘Gas Control❘Detector Safety System

❚And interface to:❘LHC machine❘Access Control System❘CERN Safety System

❚Sub-detectors can use these components:❙For defining logic rules (using their states)❙For high-level operation (when applicable)

❘Switch ON, Switch Off, Set parameters


And also Database Tools

PVSSPVSS

PVSSPVSS

. To Offline

...

...PVSS

PVSSPVSS

PVSS

Cond..DB

Conf.DB

To Offline

...

...

PVSSArch.

❚ Interfaces to the three Logical Databases in the Online System

Expe

rimen

tal E

quip

men

t


Online Database Contents

❙Configuration DB contains:❘All data needed to configure the HW (or SW) for the

various running modes〡Ex.: HV V0 Settings, Pedestal settings, trigger settings, etc.

❙PVSS Archive contains:❘All monitoring data read from HW for monitoring and

debugging of the Online System〡Ex.: HV Vmon Readings, temperatures, pedestal readings, etc.

❙Conditions DB contains:❘A subset of the monitoring data read from HW if it is

needed for Event processing (prob. packaged differently)〡Ex.: HV Vmon Readings if changed by more than n Volts

❘Some configuration data once it has been used〡Ex.: Trigger settings used by a particular run


The Configuration DB❚The Configuration DB will contain:

❙All "static" information about the devices❘Connectivity, addresses, etc. (also inventory and history)➨ Developed within LHCb (supports queries)

❙All "dynamic" data needed by the devices (for different running modes and different versions):

❘Settings (voltages, alarm limits, etc.), Calibration constants, Pedestals, FPGA code (probably a pointer to it), etc.

➨The settings for a particular running mode are called a “Recipe” (partial recipes available)

➨The JCOP FW component implements a cache:〡Can be used without Oracle for tests〡Can pre-load several recipes before “Start of Run”


What needs to be done:

❚Start bottom up❙Integrate each device into PVSS❙Define configuration recipes

❘for the various running modes❙Build a hierarchy for each sub-system

❘According to the guidelines❙Integrate the devices in the hierarchy


Device Integration❚Device Types

❙HV & LV channels❘CAEN, ISEG, WIENNER -> JCOP Framework

❙Analog inputs❘ELMB -> JCOP Framework

❙Electronics boards❘SPECS & CC-PC -> Tools to describe boards❘TELL1 -> FW component (for common part)

❙Other Components❘HW or SW -> FwDIM component

➨ Needs: PVSS, Framework, DIM,…


PVSS


PVSS Distribution


Datapoint Concept❚DP type -> DP

Configs


Graphical Objects❚Reference Panels

❙Can be “inherited” dynamically❙“$parameters” get replaced by instance

value


Building User Interfaces

❚Static Part -> Drag & Drop❚Dynamic part -> Control Scripts ("C" like)

❙A few usefull calls for accessing DPs:❘dpGet (string dpName, <data_type> value)❘dpSet (string dpName, <data_type> value)❘dpConnect (string callback, string dpName)

❙A few usefull calls for accessing Widgets:❘getValue (string widgetName, string

widgetProperty, <widget dependent data>)❘setValue (string widgetName, string

widgetProperty, <widget dependent data>)


PVSS Features❚Open Architecture

❙We can write our own managers➨It can be interfaced to anything (FSM, DIM)

❚Highly Distributed❙130 Systems (PCs) tested➨No major problem found

❚Standard Interface❙All data of all sub-systems defined as

DataPoints!


Demo-1❚Start PVSS console❚Create a project

(add installation tool)❚PVSS basic functionality

❙PVSS Managers❙Parameterization Module

❘Datapoint structures❙Graphic editor


Demo-2❚Install Framework

❘fwCore❘fwAnalogDigital❘fwCaen❘fwConfigurationDB❘fwDIM❘fwSpecs❘fwHw

❚CAEN component:❙Create Crates/Boards/Channels

❘“Crate0” will be used by FSM later❙Show Operation panels


DIM Distributed Information Management System

❙Publish/Subscribe mechanism❘Servers publish Services.❘Clients subscribe to Services:

〡On change or at regular intervals❘Clients can send commands to

Servers❙Services

❘A set of data 〡any type or size〡Identified by a name

❙A Name Server❘Keeps a list of available Services


DIM Some Characteristics

❙Transparency❘DIM clients do not know where their interlocutors are.❘DIM components can move from one machine to another, all

connections are transparently re-established.❙Available on mixed environments:

❘UNIX (HP-UX, Sun-OS, Sun-Solaris, IBM-AIX, DEC-OSF, Linux), Windows, VMS, Real-time OSs (OS9, LynxOS, VxWorks)

〡API available in “C”, C++ and Java❙Easy to Use

❘One “call” and a process can become a server or a client.❘Monitoring and Visualization Tools Available.❘Documentation and examples at: http://www.cern.ch/dim

http://www.cern.ch/dim




PVSS<->DIM❚FwDIM component:

❙Server is a DIM Server❙Client is a PVSS Manager (PVSS00dim)❙Correspondence: PVSS DPs <-> DIM Services

❘Can be setup graphically via fwDIM panel❘Or via a script library

❙When setup❘When Server updates Service data goes into DP❘Writing to DP will send a DIM Command

❙Documentation at: ❘http://www.cern.ch/lhcb-online/ecs/fw/FwDim.html

http://www.cern.ch/lhcb-online/ecs/fw/FwDim.html


Non-standard components

❚Integrating user components:❙Create a DIM server (C or C++)

❘Publishes device status & data❘Receives Commands

❙ Create a PVSS Datapoint❘That matches the structure of DIM services

❙Connect the DP to the DIM services❘Using the FwDIM tools

❙Make a PVSS panel to control the device❚ Used for: farm monitoring, trigger algorithms, etc.


Demo-3❚FwDIM

❙Configure DIM_DNS_NODE❙Start a DIM server (ex.: pvss_dim_server)❙Start DIM visualization tool

❘DIMTree on Windows❘DID on Linux

❙Start fwDIM.pnl❘Connect services to DPs❘Visualize from PVSS


Electronics Interface❚CC-PC & SPECS tools:

❙Low-level Software❘A “C” library for accessing board components

〡Via I2C, JTAG or parallel bus (and FPGA programming)❘A Generic DIM server for PVSS Communication

❙PVSS Tools (FW components: fwCcpc/fwSpecs)❘A library (PVSS scripting) for accessing board

components on any board with a CC-PC/Specs (equivalent to the low-level library)

❘A graphical user interface providing the functionality available in the library


Electronics Integration❚Electronics Boards:

❙Can use the CCPC/SPECS FW Tools for tests, but accessing the “chips” is not enough

❙Boards have to be modeled in PVSS according to guidelines (ex. registers have to correspond to datapoints) in order to:❘Provide access to the Conf. DB

〡Select a device/group of devices and say:Save as “Physics” recipe.

❘Be able to archive the data❘Be able to send the data to the Cond. DB❘Integrate into the FSM, Generate alarms, etc.


Electronics Integration❚We provide a tool for modeling boards

and their components (FWcomponent: FwHw)

❘Declaring boards (access via SPECS or CC-PC)Containing:

〡Groups of Chips (recursive) Containing:

❘Chips (TTCrx, Beetle, etc.)Containing:❘Registers

(access via I2C/JTAG/Parallel Bus)❙Contacts:

❘Ricardo Fernandes: SPECS❘Stefan Koestner: CC-PC


Electronics boards❚Demo Setup

SPECSMaster SPECS

Mezzanine

I2Cwidget

Croquette

ServerPC: pclbcecs03

SPECS

I2C

ClientPC: Portable

SpecsSrv

PVSS

PVSS00dimEthernet

DNS

SupportPC: pclhcb155

PVSS

PVSS00dim

Note: The DNS should run on a stable machine (same as PVSS), not on a portable…


Demo-4❚FwSpecs:

❙Server PC: pclbcecs03❘Configure DIM_DNS_NODE❘Start SpecsServer remotely

❙Client PC: portable❘Configure DIM_DNS_NODE❘Start SpecsClient “direct access” panel

〡Exercise I2C, JTAG, DCU❘Explain the “Monitoring” feature❘Show Advanced Panel (User Scripts)

❙Documentation at (not this version yet):❘ http://www.cern.ch/lhcb-online/ecs/PVSS_SPECS

❚ FwCcpc: very similar❙ Tools will be presented at Online meeting

http://www.cern.ch/lhcb-online/ecs/PVSS_SPECS


Custom Electronics❚Demo Example

SPECSMaster SPECS

Mezzanine

Velo Board

Server PC

SPECS

SpecsSrv

TTCrxBeetle1

Beetle2

I2C


Demo-5❚FwHw

❙Create HW types:❘ TTCrx, Beetle and VeloBoard

❙Configure Default Settings❙Create veloBoards

❘“Operate” the board❙Interface to Configuration Database

(cache)❘Save recipes (“PHYSICS”, “TEST”, etc.)❘Download recipes


Electronics guidelines❚FwHw: Some Guidelines

❙If a “chip” has many registers❘If they can be written in one single operation

〡Declare them as 1 register of size N〡This will optimize configuration time

❘Some (a few) can also be declared separately〡If they are often accessed individually

❚After using FwHw to define the boards:❙Design a user interface to operate each board type

❘The library fwSpecs or fwCcpc will give you access to the data to be visualized or sent to the board ex.: fwSpecs_read(“board1.ttcrx1.reg2”,…)


Control Hierarchy

...

ECS

DCS DAQ

DetDcs1

DetDcsN

SubSys1

SubSys2

Dev1

Dev2

Dev3

DetDaq1

SubSysN

DevN

LHCT.S.

...GAS

DSS

❚Building a Control Hierarchy❙And integrating Devices

❚Needs: FwFSM, LHCb guidelines


Control Units❚Each node is able to:

❙Summarize information (for the above levels)❙“Expand” actions (to the lower levels)❙Implement specific behaviour

& Take local decisions❘Sequence & Automate operations❘Recover errors

❙Include/Exclude children (i.e. partitioning)❘Excluded nodes can run is stand-alone

❙User Interfacing❘Present information and receive commands

DCS

Temp

Tracker Muon

HV

GAS

HV


Device Units❚Device Units

❙Provide the interface to real devices:(Electronics Boards, HV channels, trigger algorithms, etc.)❘Can be enabled/disabled❘In order to integrate a device within FSM

〡Deduce a STATE from device readings (in DPs)〡Implement COMMANDS as device settings

❘ Commands can apply the recipes previously defined

DevN


❚The FwFSM Component is based on:❙PVSS for:

❘Device Description (Run-time Database)❘Device Access (OPC, Profibus, drivers)❘Alarm Handling (Generation, Filtering, Masking, etc)❘Archiving, Logging, Scripting, Trending❘User Interface Builder❘Alarm Display, Access Control, etc.

❙SMI++ providing:❘Abstract behavior modeling (Finite State Machines)❘Automation & Error Recovery (Rule based system)

The Control FrameworkDe

vice

Uni

ts

Cont

rol U

nits


SMI++❚Method

❙Classes and Objects❘Allow the decomposition of a complex system

into smaller manageable entities❙Finite State Machines

❘Allow the modeling of the behavior of each entity and of the interaction between entities in terms of STATES and ACTIONS

❙Rule-based reasoning❘Allow Automation and Error Recovery


SMI++❚Method (Cont.)

❙SMI++ Objects can be:❘Abstract (e.g. a Run or the DCS)❘Concrete (e.g. a power supply or a temp. sensor)

❙Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts)

❙Logically related objects can be grouped inside "SMI domains" representing a given sub-system (Framework: Control Unit)


SMI++ Run-time Environment

ProxyProxyProxy

Hardware Devices

Obj

Obj

Obj

SMI Domain

ObjObjObj

Obj

Obj SMI Domain

❙Device Level: Proxies❘ drive the hardware:

〡deduceState〡handleCommands

❘ C, C++, PVSS ctrl scripts❙Abstract Levels: Domains

❘ Implement the logical model❘ Dedicated language - SML❘ A C++ engine: smiSM

❙User Interfaces❘ For User Interaction

❙All Tools available on: ❘ Windows, Unix (Linux) ❘ All communications are transparent

and dynamically (re)established


SMI++❚SMI++ - The Language

❙SML –State Management Language❘Finite State Logic

〡Objects are described as FSMstheir main attribute is a STATE

❘Parallelism〡Actions can be sent in parallel to several objects.

Tests on the state of objects can block if the objects are still “transiting”

❘Asynchronous Rules〡Actions can be triggered by logical conditions on the

state of other objects


SML – The languageclass: HighVoltage state: NOT_READY /initial_state action: GOTO_READY do SWITCH_ON all_in PSS if (all_in PSS in_state ON) then move_to READY endif move_to ERROR state: READY when ( any_in PSS in_state TRIP ) do RECOVER action: RECOVER do CLEAR all_in PSS do SWITCH_ON all_in PSS … action: GOTO_NOT_READY … state: ERROR … object: SubDetHV is_of_class HighVoltage

class: PowerSupply /associated state: UNKNOWN /dead_state state: OFF action : SWITCH_ON state: ON action : SWITCH_OFF state: TRIP action : CLEAR … object: PS1 is_of_class PowerSupply object: PS2 is_of_class PowerSupply object: PS3 is_of_class PowerSupply … objectset: PSS {PS1, PS2, PS3, …}

❚ Devices: ❚ Sub System:

❚ Objects can be dynamically included/excluded in a Set


SML example (automation)

object: RUN_CONTROL state: TEST_MODE when (LHC::STATE in_state PHYSICS) do PHYSICS action: PHYSICS do GOTO_READY all_in SubDetHVS if (all_in SubDetHVs in_state READY) do START_RUN DAQ … move_to PHYSICS_MODE state: PHYSICS_MODE …

object: LHC::STATE /associated state: UNKNOWN /dead_state state: PHYSICS state: SETUP state: OFF …

❚ External Device:

❚ Sub System:


PVSS/SMI++ Integration

❚ Graphical Configurationof SMI++ Using PVSS


Building Hierarchies❚Hierarchy of CUs

❙Distributed over several machines❘"&" means reference to

a CU in another system❙Editor Mode:

❘Add / Remove / Change Settings

❙Navigator Mode❘Start / Stop / View


Control Unit Run-Time❚Dynamically generated operation

panels(Uniform look and feel)

❚ Configurable User Panels


Features of PVSS/SMI++

❚Task Separation:❙SMI Proxies/PVSS Scripts execute only

basic actions – No intelligence❙SMI Objects implement the logic behaviour❙Advantages:

❘Change the HW -> change only PVSS

❘Change logic behavioursequencing and dependency of actions, etc -> change only SMI rules


Features of PVSS/SMI++

❚Error Recovery Mechanism❙Bottom Up

❘SMI Objects react to changes of their children〡In an event-driven, asynchronous, fashion

❙Distributed❘Each Sub-System recovers its errors

〡Each team knows how to recover local errors❙Hierarchical/Parallel recovery❙Can provide complete automation even

for very large systems


Demo-6❚Show a simple Hierarchy

❙Install fwLHCb_FsmDomains❙In installs standard LHCb FSM Domain Types❙And it creates:

❙Show Include/Exclude and Enable/Disable❙Show “Temp” FSM and Alarm Handling

VELODCS

VELOMotors

VELOTemp


Sub-detector FSM Guidelines

❚Started defining naming conventions.❚Defined standard “domains” per sub-detector:

❙ DCS❘ DCS Infrastructure (Cooling, Gas, Temperatures, pressures, etc) that is

normally stable throughout a running period❙ HV

❘ High Voltages or in general components that depend on the status of the LHC machine (fill related)

❙ DAQ❘ All Electronics and components necessary to take data (run related)

❙ DAQI❘ Infrastructure necessary for the DAQ to work (computers, networks,

electrical power, etc.) in general also stable throughout a running period. ❚And standard states & transitions per domain.❚ Doc available in EDMS:

❘ https://edms.cern.ch/document/655828/1

https://edms.cern.ch/document/655828/1


FSM Guidelines❚State Diagram for Trigger and DAQ Domains:

❙Possible intermediate “CONFIGURING” and “STARTING” states if operations slow…


MUONDCS

MUONHV

MUONDAQI

MUONDAQ

Hierarchy

VELODCS

Infrast. DCS HV DAQI DAQ L0 TFC HLT LHC

VELOHV

VELODAQI

VELODAQ

VELODCS_1

VELODCS_2

VELODAQ_1

VELODAQ_2

ECS

VELODev1VELO

Dev1VELODev1VELO

DevN

SubFarm1SubFarm

N


MUONDCS

MUONHV

MUONDAQI

MUONDAQ

Hierarchy & Conf. DB

VELODCS


VELOHV

VELODAQI

VELODAQ

VELODCS_1

VELODCS_2

VELODAQ_1

VELODAQ_2

ECS

VELODev1VELO

Dev1VELODev1VELO

DevN

Conf.DB

3

2

1

1 Configure/mode=“PHYSICS”(Get “PHYSICS” Settings)Apply Settings

2

3

1

1


Demo-7❚Using type: DAQ_Domain❚Create:

❚Create type VeloBoard❚Integrate veloBoard1, veloBoard2,…

❙Apply recipes on “Configure” command❙Note: There will be a “configurator” object per CU

which gets recipes from DB to cache

VELODAQ

VELODev1VELO

Dev1VELOBoard1

VELOFEE

VELOTELL1


Demo-8❚Using type: ECS_Domain❚Create:

VELODAQ

VELODev1VELO

Dev1VELOBoard1

VELOFEE

VELOTELL1

VELODCS

VELOMotors

VELOTemp

VELOHV

VELO


The End❚Questions?


MUONDCS

MUONHV

MUONDAQI

MUONDAQ

Hierarchy & Partitioning

VELODCS


VELOHV

VELODAQI

VELODAQ

VELODCS_1

VELODCS_2

VELODAQ_1

VELODAQ_2

ECS

VELO

lhcb’s experiment control system

Documents