epics - experimental physics and industrial control system ......science advisory committee...

This material is based upon work supported by the U.S. Department of Energy Office of Science under Cooperative Agreement DE-SC0000661.

Michigan State University designs and establishes FRIB as a DOE Office of Science National User Facility in support of the mission of the Office of Nuclear Physics.

Sheng Peng Controls & Computing Department Manager

FRIB Introduction and FRIB Controls

FRIB – a DOE-SC National User Facility Enabling Scientists to Make Discoveries Properties of nucleonic matter

• Classical domain of nuclear science

• Many-body quantum problem: intellectual overlap to mesoscopic science – how to understand the world from simple building blocks

Nuclear processes in the universe • Energy generation in stars, (explosive) nucleo-synthesis

• Properties of neutron stars, EOS of asymmetric nuclear matter

Tests of fundamental symmetries • Effects of symmetry violations are

amplified in certain nuclei

Societal applications and benefits • Bio-medicine, energy, material

sciences, national security

, Slide 2 S. Peng, October 2012 EPICS Meeting @ Pohang

Users are organized as part of the independent FRIB Users Organization • FRIBUO has 1268 members (92 US Colleges and Universities, 10 National

Laboratories, 58 countries) as of 2 October 2012

• Chartered organization with an elected executive committee (Chair is Michael Smith, ORNL)

• FRIBUO has 20 working groups on experimental equipment

• 41 members from 21 states visited Capital Hill on 5 March 2012

Science Advisory Committee • Review of equipment initiatives (Feb. 2011)

• Review of FRIB Integrated Design (March 2012)

FRIB Users Organization Over 1250 Users Ready for Science

August 2011

Joint Users Meeting

284 participants

fribusers.org


FRIB Project at MSU Project of $680M ($585.5M DOE, $94.5M MSU)

Dec. 2008: DOE selects MSU to establish FRIB

June 2009: DOE and MSU sign corresponding cooperative agreement

Sept. 2010: CD-1 granted; conceptual design complete & preferred alternatives decided

April 2012: Lehman Review, readiness to baseline


Michigan State University 57,000 people; 36 sq mi; $1.8B annual revenue; 552 buildings


Site Layout: FRIB addition connects to existing NSCL building


Building Configuration


Final Design of Conventional Facilities Complete

New Construction View from

Southeast


Ready for Civil Construction to Begin

Site preparation activities complete; pilings for earth retention system being installed as long-lead procurement

Web cams at www.frib.msu.edu


http://www.frib.msu.edu/

Facility Layout


Accelerator Design & Requirements

Delivers FRIB accelerator as part of a DOE-SC national user facility with high reliability & availability

Accelerate ion species up to 238U with energies of no less than 200 MeV/u

Provide beam power up to 400 kW

Satisfy beam-on-target requirements

Energy upgrade by filling vacant slots with 12 SRF cryomodules

Maintain ISOL option

Upgradable to multiuser simultaneous operation of light/heavy ions with addition of a light-ion injector


Facility performance expectations • Rare isotope production with primary beams up to 400 kW, 200 MeV/u uranium

• Fast, stopped and reaccelerated beam capability

• Experimental areas and scientific instrumentation for fast, stopped, and reaccelerated beams

• World-class science on day one

Experimental Systems project scope • Production target facility

• Fragment separator

Non-TPC contributions to Experimental Systems • Beam stopping systems,

reaccelerator, experimental areas, experimental equipment

Experimental Systems Scope Defined and Unchanged since April

Experimental areas

for fast, stopped, and

reaccelerated beams


Facility Layout


FRIB Driver Accelerator Layout

Delivers FRIB accelerator as part of a DOE-SC national user facility with high reliability & availability

Accelerate ion species up to 238U with energies of no less than 200 MeV/u

Provide beam power up to 400kW

Satisfy beam-on-target requirements

Energy upgrade by filling vacant slots with 12 SRF cryomodules

Maintain ISOL option

Upgradable to multiuser simultaneous operation of light/heavy ions with addition of a light-ion injector


FRIB Resonators and Cryomodules: Beam Dynamics Specifications


Conventional Facility

Racks, cable tray, conduit arrangement

How does tunnel look like


Target Building


Controls Technical Scope

Global systems •Global timing system •Machine protection system

•Network and computers

High level applications •Database •High level applications

»Console application »Web application

•Physics applications

Low level controls »Vacuum/Power Supply/RF »Cryomodule & Cryoplant »Front End/Ion Source/RFQ »Stripper

Personnel protection system

Diagnostics support

Conventional facility integration


An integrated control system • Technically Integrated • No fence between ASD and ESD

A large distributed control system based on EPICS framework

User interfaces/tools with consistent look and feel • Remote supervisory control and data acquisition of devices/detectors • Integrated control room toolkit (e.g. snapshot/restore, alarm handler) • Integrated physics applications and model-based beam control

Low level device control and process control

• PLC, IOC and instrumentation

35 μs machine protection mitigation (10μs for MPS electronics)

Global timing system distributes trigger, timestamp and real-time information facility-wide

High-bandwidth/reliability network and computing platform to support control system operation

Personnel Protection System to ensure safe operation

Diagnostics high speed data acquisition

High availability and long lifetime as a user facility

Controls Scope Requirements Defined


Control System Scale

It is a large scale distributed control system • ~200m * 200m physical distribution

• ~150 Computers/EPICS Input/Output Controllers (IOC)

• ~100 Programmable Logic Controllers (PLC)

• Thousands of network attached intelligent devices » RF controller

» Power supply controller

» Vacuum gauge/pump controller

» Programmable logic controller

• ~ 3000 network ports

• >500 timing drop points

• ~2000 MPS fast protection inputs

• ~750 racks and more than 100 with controls devices


EVG/EVR + PTP

PLC HIPROM module works

Deploy with 900W project

Looking for a PMC module for EVG to simplify the effort


EVG/EVR test platform works


Testing result :

Convergence process, PTP looks great

0 1 2 3 4 5 6 7 8-250

-200

-150

-100

-50

0

50

100

150

200Process of PTP synchronizing

Time(minutes)

off

set(

us)

2

4

6

8

Toffset : master and slave time offset


http://www.youtube.com/watch?v=1JfZbqT1OA8&feature=youtu.be

6 nodes & 8 nodes Histogram

-20 -15 -10 -5 0 5 10 15 200

500

1000

1500

2000

25006 Distribution: std=1.9749

Time(us)

Num

ber

-20 -15 -10 -5 0 5 10 15 200

500

1000

1500

2000

25008 Distribution: std=1.8542

Time(us)

Num

ber


EVR core requirement

GTP RXFrom SFP

Inte

rnal

Bu

s

Heartbeat

EventCode[7:0]

DistBus[7:0]

CharacterIsK[1:0]

EventClock

EventCode[7:0]

EventClock

LosHeartbeat

LosSynch

TimeofdayEventCode[7:0]

EventClock

DistBusReceiver

DistBus[7:0]

EventClockCharacterIsK[1:0]

RamreadyCheckSumErr

EventMappingRam

EventCode[7:0]

EventClock

ReadRam

Read/WriteRam

Trig

ger

Bu

s

Latchtimeofday

EventFIFOEventCode[15:0]

EventClock

Timestamp[63:0]

TimestampLacth[63:0]

ReadFIFO

FIFO_full

FIFO_empty

ConfigureRegRead/WriteReg

TriggerOutTriggerOut[7:0]

Loca

l Bu

s

Bus Bridge

Local Bus

Local Interface

Stan

dar

d B

us

EVR Receiver Top

EventClock


Function of FGPDB

AdvancedMC™ connector for ATCAﾮ, type B+HARTING:16 04 170 5104 000

FPGA

SPARTAN-6XC6SLX150T3FGG900C

J30AFC

J31AFC

1xGTP1xGTP

32 pairs LVDS+2 Clock

J1

PCI-E 1x

PCI-E

8 ports M-LVDS+2 clocks64 I/Os J2 FMC

34 pairs LA +2 clocks

6 Management

1xGTP

J3SFP

J4SFP

GTP

GTP

ROM Winbond

W25Q128FVSIG

MCU

ATmega1281-16MU

7 I/Os

6 I/

Os

CLOCK

UA

RT

4/I

Os

Power Supply

4 management

32 pairs LVDS+2 ClockMale connectorERNI:973101

AdvancedMC™ Plug ConnectorHARTING: 16 23 170 1032 000

Female ConnectorERNI:973032

DDR3

50

x2 I/

Os

ROM Winbond

W25Q128FVSIG

6 I/

Os

5 I/Os

5 I/Os

1xGTP

DDR3 4Gbx2

MT41K256M16

50

x2 I/

Os

LEDs 16bit

2x17 header

Switch 16bit

2x15 header

J5 RJ45

J6 PCI-E

GP

IO

J7 CLK-/+

MTCA.4

Two SFPs

Ethernet

FMC(LPC)

8Gb DDR3

GPIOs

Update Online

Support Tree & Multi-circle topology


FGPDB


FPS Topology

MASTER

Concentrator

Concentrator Concentrator

EndNode1 EndNode2 EndNode64

...

...

RPS

GTS

SFP

SFP

SFP

MASTER

.

.

.

End

Node

End

Node

End

Node...

End

Node

End

Node

End

Node...

RPS

GTS

Tree topology • Easy to implement

• Almost same delay for all end nodes

• Need 120.75MHz reference clock which is higher than system clock

• Poor message capacity

• Hard to extend

Multi-Circle topology • 80.5MHz Reference clock which is

as same as system clock

• Easy to extend

• Rich message capacity

• Different delay for different end node • Need a big master

VS


Controls Network


Green-Core Switch, Red-Distribution Switch


E-Traveler production version in field

Naming convention in RDB

E-log in field

Lattice RDB

Stockbook FRIB version beta-release

RDB/Apps go well


https://controls.frib.msu.edu/traveler/



https://controls.frib.msu.edu/names/

https://controls.frib.msu.edu/names/

https://controls.frib.msu.edu/logbook/



http://controls.frib.msu.edu/conf/

Physics applications deliverables are identified and reviewed

XAL is adopted and built

The hardware representation and online model for electrostatic quads are added

XAL configuration files for FRIB Linac and ReA3 are generated

Benchmarking is going on against off line modeling tool

Technical Status for Physics Applications

FRIB Segment1 to Stripper Energy Gain

LB Source to Diagnostics Box2


Allen Bradley PLC • Vacuum

• Cryo

UDP over Ethernet • LLRF

• PS (To Be Designed)

Virtualized IOC .vs. Hot-swappable IOC • Under evaluation

Debian/Ubuntu

What is our RTOS?

Low Level Control


RTOS: vxWorks or Linux-RT variants


In general everything from AD/DA,FPGA towards user will be handled by Controls, everything towards the detector will be handled by Diagnostics.

Treaty point

High Level Application

and computing platform

EPICS IOC and Data

Acquisition software

A/D, D/A and FPGA

Analog Front End

Detector and

Mechanical Design

Controls…Diagnostics


Test Set-Up

Box for shielding

BPM2ReA3 Beam Pipe (balcony)BPM1

Pre-Amp Stage 140 dB gain

4x Heliax ¼”

Length: ~ 46' (23 m)

4x Pick-up cable

DC Power Supply

5 V, 1 Amp

Connector SMA male

Connector SMA male,

Adapter SMA to BNC

Distance between BPM1 and BPM2

is about 2.08 m (82").

Beam

Pre-Amp Stage 2, 40 dB gain

Balcony

Ground Level

Band Pass Filter, 160 MHz

Trigger signal

(Beam Pulse

Generator)

4x

4x

4x

4x

BPM card

(has SMB female

inputs)

VME crate

DC Power Supply

15 V

Rack (ground floor)

Patch Panel with SMA

fem bulkhead adapters

8x Jumper Cables

RG142, same length

Provided by Fermilab

VME CPU card

Provided by Fermilab

Laptop

Controls

network


PPS

Access Control • AB Guardlogix

• SIL3 redundant with Compact Guardlogix

ODH • Compact Guardlogix

• Oxigrad .vs. PureAire

Requirement tracking and process • Enterprise Architect

Integrate with Conventional Facility • BACnet is agreed interface

Controls role • Full Support: LCW

• Interface with all signals

• Interface with some signals

• No interface


Wants to know more about FRIB

[email protected]

Welcome to Michigan!

Q&A


mailto:[email protected]

epics - experimental physics and industrial control system ......science advisory committee...

Documents