EPICS Experience at Fermilab

Geoff SavageAugust 2005

Controls and Monitoring Group

31 Aug 2005 MICE DAQ and Controls Workshop 2

Outline Fermilab and DØ What is EPICS? EPICS at DØ - RunIIa EPICS at DØ – RunIIb EPICS and the CFT EPICS at Fermilab EPICS lessons

31 Aug 2005 MICE DAQ and Controls Workshop 3

Main Injector & Recycler

Tevatron

Chicago

p source

Booster

p

p

p p

1.96 TeV

CDFDØ

Fermilab

31 Aug 2005 MICE DAQ and Controls Workshop 4

DØ Detector

Scintillating Fibers

Liquid Argon/Uranium

Calorimeter

Central MuonLayer A

Silicon Tracker

Superconducting Solenoid

Central MuonLayer B

Central MuonLayer C

Forward MuonLayer C

Muon Toroid

31 Aug 2005 MICE DAQ and Controls Workshop 5

What is EPICS? Experimental Physics and Industrial Control

System• Collaboration• Control System Architecture• Software Toolkit

Integrated set of software building blocks for implementing a distributed control system

www.aps.anl.gov/epics

31 Aug 2005 MICE DAQ and Controls Workshop 6

Why EPICS? Availability of device interfaces that match or

are similar to our hardware Ease with which the system can be extended to

include our experiment-specific devices Existence of a large and enthusiastic user

community that understands our problems and are willing to offer advice and guidance.

31 Aug 2005 MICE DAQ and Controls Workshop 7

EPICS Architecture

Input OutputController

(IOC)

OperatorInterface

(OPI)

Local Area Network (LAN)

OPI – a workstation running EPICS tools.Operating Systems: Linux and Windows XP.

IOC – platform supporting EPICS run-time database.Example: VME based PPC processor running vxWorks, Linux workstation.

LAN – communication path for Channel Access (CA), the EPICS communication protocol.

31 Aug 2005 MICE DAQ and Controls Workshop 8

IOC Architecture

31 Aug 2005 MICE DAQ and Controls Workshop 9

IOC Database Collection of records Each record represents a system parameter

(process variable, PV)• Unique name• Set of attributes• Attributes and value can be modified

Records must process to do something• An input record can read a value every 10 seconds• A CA write to an output record causes the record to

process• Either input or output, not both

31 Aug 2005 MICE DAQ and Controls Workshop 10

EPICS Records Input

• Analog In (AI)• Binary In (BI)• String In (SI)

Algorithm/control• Calculation (CALC)• Subroutine (SUB)

Output• Analog Out (AO)• Binary Out (BO)

Custom – only needed when existing record types or a collection of existing record types are inadequate

31 Aug 2005 MICE DAQ and Controls Workshop 11

Channel Access

IOC(CA Server)

LAN

EPICS Tool(CA Client)

• One client can connect to many servers.• A channel connects to the value or an attribute of a PV.• Each channel has a unique name = record name + attribute.• CA Services:

• Search – find a channel by name• Get – retrieve a channel value• Put – modify a channel value• Add monitor – notification of state change

31 Aug 2005 MICE DAQ and Controls Workshop 12

EPICS at DØ - RunIIa Support for new devices and additional field busses Centralized database with supporting tools for defining

the control system configuration Report EPICS alarms to our global event reporting

system Interface to the Python programming language used to

develop host tools for downloading and monitoring Detector configuration management - COMICS Archiving Gateways

31 Aug 2005 MICE DAQ and Controls Workshop 13

MIL-STD-1553B Bus Restricted detector access while running Provides a robust and highly reliable

connection to electronics in the remote collision hall

Developed a queuing driver, device support, and a generic record

12 IOCs with ~70 1553 busses from the counting house to the detector and ~10 busses within the counting house

31 Aug 2005 MICE DAQ and Controls Workshop 14

DØ Specific Records SVX sequencer - configures and controls the

ADCs in the tracking detectors High voltage

• Implemented using a state machine model• Linear ramping with retries• Trip condition recovery• Limits control

Rack monitor - Generic I/O module• 64 A/D inputs, 4 D/A output, 4 16-bit I/O words

MIL-1553 – debugging/testing on a 1553 bus

31 Aug 2005 MICE DAQ and Controls Workshop 15

DØ Specific Device Support Access hardware registers across the MIL-1553

bus and VME backplane using standard EPICS records – long input, long output, …

Support for DZERO specific records• Bira 4877 power supplies• SVX sequencers• Rack monitor

Complex MIL-1553 bus transactions• Dual port memory on the AFE boards• Write the address to access followed by a read or

write of the data

31 Aug 2005 MICE DAQ and Controls Workshop 16

Describe a Device with Records

TemplateFile

GeneratorFile

EPICSDatabase

EPICStool

Multiple record definitions with substitutionparameters This defines a device.

Define instances of a template. Assign valuesto the substitution parameters in the template file.

DatabaseLoad

ASCII fileInstances of records read by IOC to create record database.

31 Aug 2005 MICE DAQ and Controls Workshop 17

Centralized Database System

OracleHardwareDatabase

TemplateFile

DBCreation

EPICSDatabase

RecordExtract

GeneratorFile

EPICSDatabase

IOC Id

TemplateExtract

GeneratorFile

InstanceCreation

PythonScripts

WebAccess

31 Aug 2005 MICE DAQ and Controls Workshop 18

PV Naming Convention <det> [subdet]_ <devtype>_ <loc>/ <attr> [:io][. field]

• <det>  - Required detector system CTL, SMT, CFT, CPS, CAL, FPS, ICD, MUO

• [subdet] - Optional subdetector North, South, East, West, Forward, Test

• <devtype> - Required Device Type ADC, BLS, HV, PLS, CETEC, ...

• <loc>  - Required Location indicator For example a crate number: M201, PC03, F313

• <attr> - Required device attribute PEDS, MODE, SFM, CTEC, HEAD, VER, ...

• [:io] - Optional Input/Output designation         Direction is from the viewpoint of the front end crate

• [.field] - Optional Epics field name, default is .VAL Example: CTL_LV_M201/P5V

31 Aug 2005 MICE DAQ and Controls Workshop 19

Global Event Reporting Collect and distribute significant events from

detector and DAQ• Run control• Alarms - data corruption

Added a connection to EPICS• Allows for notification of all EPICS alarms• Otherwise a user must register an event

Significant Event System (SES)

31 Aug 2005 MICE DAQ and Controls Workshop 20

SESServer

FrontEnds

HostApps

SELogger

AlarmWatcher

AlarmDisplay

COOR

L1L2L3

Collector/Router

DataLogger

Disk

SE LogFile

GuidanceFiles

Message Key:SE = Significant EventFIL = Filter configurationRC = Run ControlFM = Filtered SE Message = Filter

FIL

FM

FMRC

SE

SE SE

SE

SE

SESE

31 Aug 2005 MICE DAQ and Controls Workshop 21

Alarm Display

31 Aug 2005 MICE DAQ and Controls Workshop 22

Python Interface “Python is an interpreted, interactive, object-

oriented programming language.” A Python interface to Channel Access (C

library) has been developed and extended to be almost object oriented

Tkinter and Python Mega Widgets used to develop graphical user interfaces (GUI)

Developed a GUI framework to speed development time and provide a consistent look and feel

31 Aug 2005 MICE DAQ and Controls Workshop 23

Left click each tab tomove between displays.

Look for statusinformation.

IOC Resource Monitor Display

Processor name.

Each box is a PV

31 Aug 2005 MICE DAQ and Controls Workshop 24

31 Aug 2005 MICE DAQ and Controls Workshop 25

Detector Configuration Management

The COMICS server manages the configuration of the detector

Receives load requests from:• Run control• Expert GUIs• Program APIs

Load map is a directed graph (tree)• Tree node (intermediate)

Establishes a layered hierarchy Establishes an execution order

• Action node (leaf) Issues EPICS CA requests

31 Aug 2005 MICE DAQ and Controls Workshop 26

Configuration Management Tree

IntermediateTree Node

CFTSMTCAL MUO

DEV2 DEV3

S0 S2S1

DEV0 DEV1 T0

Root Node

Configuration Data

Action NodeUses EPICS

Channel Access

Links determine the

order of execution

DØ

31 Aug 2005 MICE DAQ and Controls Workshop 27

Comics Expert GUI

31 Aug 2005 MICE DAQ and Controls Workshop 28

Archiving Data Maintain a history of PV values Channel Archiver – EPICS tool

• Fast sampling rates• Immediate analysis• Infrequent access to old historical data• Large files• Stored to tape once a week

EPICS/Oracle Archiver - DØ• Long-term studies• Slow sampling rates• Easy access to data at any moment• Minimal maintenance• Data stored directly in an Oracle database

31 Aug 2005 MICE DAQ and Controls Workshop 29

Gateways Accelerator

• Gateway link to ACNET system• Bidirectional• Data access only (no control)

Cryogenics and Gas systems• Gateway link to DMACS system• Portable Channel Access Server – EPICS tool• Read-only• Data access only (no control)

31 Aug 2005 MICE DAQ and Controls Workshop 30

ACNET Gateway

xmlrpc Server

ToACNET

FromACNET

ToFromITC

ToEPICS

FromEPICS

CACHE

REMOTE

EXPORT

xmlrpc

xmlrpc

ITCCLIENTS

CA

CA

EPICSRECORDS

CLIENTS xmlrpcCLIENTS

D0 BD

31 Aug 2005 MICE DAQ and Controls Workshop 31

EPICS at DØ - RunIIb CanBus driver for EPICS

• Weiner power supplies Ethernet driver for EPICS

• Gigabit Ethernet communication• Raw Ethernet protocol

Both use support from asynDriver – EPICS tool• Asynchronous Driver Support• “asynDriver is a general purpose facility for interfacing

device specific code to low level communication drivers.”• “A primary target for asynDriver is EPICS IOC device

support but, other than using libCom, it is independent of EPICS.”

Framework reduces development time.

31 Aug 2005 MICE DAQ and Controls Workshop 32

Support for CFT Device support for the AFE board using the

MIL-1553 driver AFE and AFE power supply templates Record and device support for the SVX

sequencers Expert GUI for downloading

• Does not use COMICS Calibration software

• Runs on processor in VRB crate• Communicates with AFE

31 Aug 2005 MICE DAQ and Controls Workshop 33

Analog Front End

Fiber

Wave guide

Cryostat

VLPC

Analog Front End

Stereo Board

Cassette

MIL-1553Communication

31 Aug 2005 MICE DAQ and Controls Workshop 34

SVX Sequencer

AFE

VBD

VRB

VRBC

Processor

Sequencer

Sequencer

Sequencer

Sequencer

Level 3VME

MIL-1553Communication

CalibrationSoftware

31 Aug 2005 MICE DAQ and Controls Workshop 35

EPICS at Fermilab SMTF

• Superconducting module test facility• Cavity test in October• DOOCS for LLRF – speaks CA• EPICS for everything else• No ACNET – beams division controls system

ILC• Control system to be decided• Management structure now forming

31 Aug 2005 MICE DAQ and Controls Workshop 36

EPICS Lessons Three Layers

• Tools EPICS tools OPI development - CA library

• Applications Build base Build application - combine existing pieces Develop device templates

• Development Records, device, and driver support

EPICS is not simple to use• Need expertise in each layer• Support is an email away

IOC operating system selection

31 Aug 2005 MICE DAQ and Controls Workshop 37

The End