high level software structure for the european xfel...
TRANSCRIPT
HIGH LEVEL SOFTWARE STRUCTURE FOR THE EUROPEAN XFELLLRF SYSTEM
Christian Schmidt∗, Valeri Ayvazyan, Julien Branlard, Łukasz Butkowski, Olaf Hensler,Martin Killenberg, Mathieu Omet, Sven Pfeiffer, Konrad Przygoda,
Holger Schlarb, DESY, Hamburg, GermanyAdam Piotrowski, FastLogic sp. z o.o., Łodz, Poland
Wojciech Cichalewski, Filip Makowski TUL-DMCS, Lodz, Poland
AbstractThe Low Level RF system for the European XFEL is
controlling the accelerating RF fields in order to meet thespecifications of the electron bunch parameters. A hardwareplatform based on the MicroTCA.4 standard has been chosento realize a reliable, remotely maintainable and high perform-ing integrated system. Fast data transfer and processing isdone by field programmable gate arrays (FPGA) within thecrate, controlled by a CPU via PCIe communication. Inaddition to the MicroTCA.4 system, the LLRF comprisesexternal supporting modules also requiring control and mon-itoring software. In this paper the LLRF system high levelsoftware used in E-XFEL is presented. It is implemented asa semi-distributed architecture of front end server instancesin combination with direct FPGA communication using fastoptical links. Miscellaneous server tasks have to be exe-cuted, e.g. fast data acquisition and distribution, adaptationalgorithms and updating controller parameters. Furthermorethe inter-server data communication and integration withinthe control system environment as well as the interface toother subsystems is described.
INTRODUCTIONThe Deutsches Elektronen-Synchrotron (DESY) in Ham-
burg is currently building the European X-ray Free ElectronLaser (E-XFEL) [1]. This hard X-ray light source generatesup to 27000 coherent laser pulses per second with a durationof less then 100 fs and a wavelength down to 0.05 nm. Forthis, electrons have to be accelerated to 17.5 GeV using a2 km particle accelerator based on superconducting radiofrequency technology. Precision regulation of the RF fieldsinside the accelerating cavities is essential to provide a highlyreproducible and stable electron beam. RF field regulationis done by measuring the stored electromagnetic field insidethe cavities. This information is further processed by thefeedback controller to modulate the driving RF source, usinga low level RF system. Detection and real-time processingare performed using most recent FPGA techniques. Perfor-mance increase demands a powerful and fast digital system,which was found with the Micro Telecommunications Com-puting Architecture (MicroTCA.4) [2]. Depending on theapplication either parallel processing for latency efficiencyor pipelined processing of a large number of similar RF sig-nals is aspired. In-tunnel installation with limited access
requires a compact system with implied redundancy. Fur-ther remote access to status information, software upgradesand maintenance is ensured. Finally the modularity andscalability of this system guarantees to have later upgradesin order to meet demands within a lifetime of the machineand beyond. DESY currently is operating the Free ElectronLaser (FLASH), which is a user facility of the same typeas E-XFEL but at a significantly lower maximum electronenergy of 1.2 GeV. The LLRF system for FLASH is equalto the one of E-XFEL, which allows for testing, developingand performance benchmarking in advance of the E-XFELcommissioning [3]. A picture of the FLASH installation canbe found in Fig. 1.
Figure 1: Installation of MicroTCA crates inside the FLASHtunnel underneath the accelerator module. The radiationshielding (yellow cabinet) protects the electronic rack inside.The insert shows the MicroTCA crate installed in this rack.
LLRF SYSTEM LAYOUTThe present LLRF system for a single E-XFEL RF station
controlling 32 cavities is given in Fig. 2. LLRF systems witha different quantity of cavities are build as a subset of thisconfiguration. Machine operation is performed in a pulsedmode. The duty cycle, i.e. the repetition rate (10 Hz) ofthe RF pulse to the RF pulse length (about 1 ms) is about1 %. The impact to the RF regulation is such that within in aregular mode of unchanged operating conditions, i.e. stableworking point consecutive pulses are similar, however theinitial conditions vary as well as long term changes occur. In
Proceedings of ICALEPCS2015, Melbourne, Australia - Pre-Press Release 23-Oct-2015 11:00WEPGF029
Control System InfrastructureISBN 978-3-95450-148-9
1 Copy
right
©20
15CC
-BY-
3.0
and
byth
eres
pect
ivea
utho
rsPr
e-Pr
essR
eleas
e23-
Oct
-201
511
:00
CIC CIC
DAC
VM
Ctr
l
Ctr
l
DCM TMCBLOGMREFMPSMREFM LOGM
LLRF Front end server -Slave
ReflectedForward
ReflectedForward
ReflectedForward
waveguide
Klystron
...8xdfdf
ReflectedForward
df
ReflectedForward
8
df df df df
Pzt. Drive
8
ReflectedForward
ReflectedForward
8888
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
888888
ReflectedForward
Setpoint
MIMO FB
Master
BLC
ToroidBCM
BAM
Pzt Ctrl.
BBF
ToroidBAMBCM
Pzt Ctrl.
Pzt. Drive
ORC
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
...IQCal
ADC
RTM
IF
Forw./Refl. signal correction
Lin
LFFDetuning Detuning
PVS
LLRF Front end server Master
Slave
Feedforward FF Correction
DCM TMCB
Forw./Refl. signal correction
PSM
Figure 2: Basic layout of the LLRF station for the master and slave control used in E-XFEL. Signal processing path as wellas communication links are presented. Communication between the master and slave system is done using optical links, aswell as for connecting further subsystems to the LLRF (marked as BAM, BCM and Toroid).
order to optimize the field regulation a temporal separationof feedback is applied in the following way:
• µs scale: intra pulse feedback and digital signal pro-cessing on the FPGA
• ms scale: pulse to pulse adaptation, removing repetitivecontrol errors, processed in the CPU
• s scale: long term drift compensation and slow feed-back mechanism by external modules or feedbacks
Intra Pulse Data ProcessingFast regulation steps are processed within the FPGA. Fur-
thermore the fast regulation loop is separated into a prepro-cessing step, running on a different FPGA than the maincontroller application, which collects data from several pre-processing boards using low latency communication linksin the backplane of the MicroTCA.4 crate. The 16 cavitydata acquisition and preprocessing section is identical forthe master and the slave subsystem. The main controllersums up the two partial vector sums (PVS), resulting fromthe contribution of cryomodule (CM) 1 and 2 on the masterLLRF system and CM 3 and 4 on the slave LLRF system.This total vector sum is processed within a multiple input,multiple output (MIMO) feedback controller [4] and addedto the feed-forward signal in order to to generate the klystrondrive. Further beam based information transferred throughoptical links are processed in order to improve the regulation
performance (BBF) or compensate effects like beam loading(BLC).
Pulse to Pulse AdaptionRepetitive control errors such as generated by cavity detun-
ing, are rather compensated by some additional feed-forwardinput instead of the feedback. An model based iterativelearning feed-forward algorithm (LFF) processing data frompulse to pulse on the CPU is used to play some correctionsignals in order to relief the action required from the feed-back. Changes in the actuator chain are visible in offsetsof the correction signals and are compensated by automaticscaling and rotating the output vector (ORC).
Long Term Drift SuppressionLong term drifts, especially in the measurement chain are
currently the most critical part in the regulation loop. Thereexist different approaches to compensate for this, e.g. refer-ence signal tracking or passive methods like ensuring stableenvironmental conditions. Here a special drift compensa-tion module (DCM) is used which measures a priori the RFpulse a reference pulse and corrects the RF signals for eachchannel accordingly. Further beam based measurements areused to quantify the control error with respect to the electronbeam and slowly adapt the operating conditions by changingLLRF setpoints.The combination of these feedback mechanisms is the
basis to meet the given RF field regulation requirements of
WEPGF029Proceedings of ICALEPCS2015, Melbourne, Australia - Pre-Press Release 23-Oct-2015 11:00
ISBN 978-3-95450-148-92Co
pyrig
ht©
2015
CC-B
Y-3.
0an
dby
ther
espe
ctiv
eaut
hors
Pre-
Pres
sRele
ase2
3-O
ct-2
015
11:0
0
Control System Infrastructure
MTCA crate
MC
H
CP
U
TM
G
AM
C 1
2
AM
C 1
1
AM
C 1
0
AM
C 0
9
AM
C 0
8
AM
C 0
7
AM
C 0
6
AM
C 0
5
MP
S
CT
RL
External modules
User interfaces
Pzt DCMLOGMREFM
PCIe
Sw
itch
Low latency link
Tmg/Intlck
Optical link
Eth
.
PSM
Sw
itch
Ethernet
jddd = main operaional GUIMatlabPython....
Data storage
Front End CPU
LLRF Control serverTiming server MPS server Radmon server
Notations / legend
TMG
Machine ProtectionMPS
DCM
LOGM
REFM
RPC / TINE
0MQ
Multicast
Timing
Drift compensation module
Reference sub-distribution module
Local osciallator generation module
long term data archievershort term multi dataset distribution
DAQ sender
Data preprocessingData reductionData filtering
Watchdog
Supporting servers
Quench detectionStatistics and PerfomancePiezo AutomationPiezo relaxationVS calibration...
External modules
Monitor and control of: - DCM - PSM - LOGM - REFM - Piezo
PSM
PZT
AMC
CTRL
Power supply module
Piezo control module
Controller
Advanced Mezzanine card
MCH MicroTCA carrier hub
Figure 3: Front end server structure and communication links within the MicroTCA.4 LLRF system.
0.01 % and 0.01 deg in amplitude and phase. Control andcommunication with the firmware is based on the LLRFfront end server.
ARCHITECTURE OF THE LLRF FRONTEND SERVERS
Having a pulsed mode of operation requires to synchro-nize all subsystems to a frequent trigger. This is realized by atiming distribution system which triggers all subcomponentswithin the crate at a given time. Further the servers runningon the front end CPU are triggered by the same mechanism.This synchronization puts the subtasks within one RF pulsein line which is: starting the RF pulse, processing data withinthe FPGA, transferring data to the CPU, processing data andfurther sub-distribute, update control variable withsin theFPGA. Communication between the FPGA and the CPU isdone by direct memory access over a PCIe link [5]. Fixedpoint representation from the FPGA is converted to physi-cal meaningful data before further distributing the data tographical presentation, other processing tasks or data storage.Inter process communication on the same CPU is realizedusing the ZeroMQ communication protocol [6]. Java basedDOOCS is in charge of the graphical user interface [7]. Datacommunication to the GUI is done by RPC or TINE calls,which is further extended to use third party display and pro-cessing programs like Matlab. A sketch of the inter-servercommunication topology, inter-crate communication linksand user interfaces can be found in Fig. 3.The LLRF control server itself can be seen as the basic
interface to the MicroTCA.4 hardware such that commu-nication to front end registers is being controlled by thisparticular device. Data acquisition and pre-processing is ahigh priority task and should be treated as quasi real time ca-pable. Therefore other tasks displayed as supporting servers
will have a lower priority since the system is still operableeven without having them running. The DAQ sending deviceis collecting data from the LLRF control server before trans-mitting them further to a data storage. Here the data fromall subsystems is synchronized and stored for later analysisor directly used in higher level automation routines. Thisdata storage is extremely helpful for post mortem analysisof system malfunctions.
Decoupling of the basic LLRF control server from othersupporting servers turned out to be rather helpful in terms ofmaintenance and further developments. Having the front endserver optimized at a given state it usually requires only mi-nor adaptations during the running process. Further changesin the functionality of the server should be ideally transpar-ent to other clients, which can be done except of interfacechanges.
Extensions and additional status and processing informa-tion is likely often adapted to new requirements. Maintainingsuch systems does not directly influence the main controllerand also bugs usually induced by new developments aretransparent to third party clients. Having defined clear inter-faces allows to have a modular mechanism. For example thesame LLRF front end server can be used for different subsys-tems even if additional supporting servers are not requiredor wanted. This reduces the overall load on the processingunit by reducing the number of subtasks to the required min-imum. The goal of having a highly modular system for theMicroTCA.4 hardware and its corresponding firmware alsoreflects on the software architecture.
Basic Front End ServersThe basic idea for a front end server is to have one instance
capable to communicate with the hardware device installedin the system. For example comparing Fig. 3, the devices
Proceedings of ICALEPCS2015, Melbourne, Australia - Pre-Press Release 23-Oct-2015 11:00WEPGF029
Control System InfrastructureISBN 978-3-95450-148-9
3 Copy
right
©20
15CC
-BY-
3.0
and
byth
eres
pect
ivea
utho
rsPr
e-Pr
essR
eleas
e23-
Oct
-201
511
:00
of timing control (TMG), machine protection (MPS) aresingle server - single device pairs. For the LLRF controlserver there are in total 7 devices controlled by one serverwhich can be seen as one combined system. This comparisonreflects the complexity and large amount of data processingtaking place. Reducing the functionality of this server tobasic functionalities is a consequence to distribute tasks inan effective way.
Supporting ServersThe subcategory of supporting servers contains several
functionalities separated into frequently running instancesand user triggered applications. As an example the routineof automatic piezo tuning should be described here. Inorder to minimize the cavity detuning, piezo actuators areused to play a mechanical stimulus to compensate to RFfield induced deformations of the cavity. This is done bycomputation of the actual detuning within the statistics andperformance server. Necessary data for this computationis send from the LLRF control server as described above.After the detuning parameters are computed, they are furthersend to the piezo automation server which computes thewaveform to be played. This is transferred back to the LLRFfront end server which sends this waveform to the firmwareregisters as a control command sequence for the next pulse.A different example can be found with the VS calibrationinstance, which is triggered unfrequent by user requests inorder to recalibrate the measurement chain of the LLRFsystem. Permanent operation of this server is not possiblesince the calibration method is invasive to machine operationand usually requires a special setup.
External Device ServersWithin the LLRF system there exist next to the Mi-
croTCA.4 crate supporting components which also requireto be controlled and monitored. The setup varies with thedifferent locations the system is implemented. Each of thissystem requires a server, however there are many functional-ities within which can be transferred in between. All theseservers share special classes which can be plugged togetherin a modular way. Communication is done over ethernetwhich would basically allow to run the instances also ona different CPU if high load on the front end CPU wouldrequire this.
CONCLUSION AND OUTLOOKIn this paper the MicroTCA.4 based LLRF control system
in view of the LLRF software structure for the E-XFEL hasbeen presented. Basic layout ideas and structuring as wellas communication paths are discussed. This architecture iscurrently operating in the FLASH facility, meeting the re-quired goals in terms of RF field stability. However the basic
functionalities are successfully running, continues develop-ments are ongoing which further improve the overall systemperformance, as well as the maintainability and reliability.Currently all acceleration modules being installed in the
E-XFEL are tested and characterized in advance. Part ofthis program is done by the LLRF system since gained infor-mation will be used for later automation routines. Storageof this data is done by a database which requires automaticacquisition and load to the systems being installed in the E-XFEL. Furthermore special automation routines developedfor the tests are directly transferable to E-XFEL and also toFLASH.
Fast error tracking helps to minimize machine downtime.Therefore status and health information is monitored for allsubsystems, however automatic processing this informationis essential to support operators with the key informationinstead of a large amount of error notifications. Systematicfailure analysis methods are currently under development.Finally the LLRF system containing hardware, firmware
and software are going to be used in other facilities whichmight use different control systems. To integrate this in adifferent environment, a control system independent serveris developed, which decouples functionalities of the serverfrom requirements of the control system [8]. All these modi-fications demand restructuring work, however themodularityof the systems as described in this paper helps to minimizethe effort and reduces the risk of system malfunctions.
REFERENCES[1] The European X-Ray Free Electron Laser Technical Design
Report, http://xfel.desy.de
[2] MicroTCA® is a trademark of PICMG, MicroTCA.4 specifi-cations: http://www.picmg.org
[3] J. Branlard et al.,The European XFEL LLRF System, Proceed-ings of the 3th International Particle Accelerator Conference2012, New-Orleans, USA, 2012.
[4] C. Schmidt et al., Precision Regulation of RF Fields withMIMO Controllers and Cavity-based Notch Filters, Proceed-ings of the XXVI International Linear Accelerator Confer-ence 2012, Tel-Aviv, Isreal, 2012.
[5] M. Killenberg et al., “Drivers and Software for MicroTCA.4”,WEPGF015, These Proceedings, ICALEPCS’15, Melbourne,Australia (2015).
[6] http://zeromq.org
[7] E. Sombrowski, R. Kammering, K.R. Rehlich , A HTML5Web Interface for JAVA DOOCS Data Display,These Pro-ceedings, ICALEPCS’15, Melbourne, Australia (2015).
[8] M. Killenberg et al., “Integrating control applications intodifferent control systems”, TUD3O05, These Proceedings,ICALEPCS’15, Melbourne, Australia (2015).
WEPGF029Proceedings of ICALEPCS2015, Melbourne, Australia - Pre-Press Release 23-Oct-2015 11:00
ISBN 978-3-95450-148-94Co
pyrig
ht©
2015
CC-B
Y-3.
0an
dby
ther
espe
ctiv
eaut
hors
Pre-
Pres
sRele
ase2
3-O
ct-2
015
11:0
0
Control System Infrastructure