Availability Performance of LCLS X-Ray FEL at SLAC
William Colocho for the LCLS team.
2
Outline
• Quick LCLS Overview
• Availability and downtime vs. time
• Downtime per system and planned upgrades
• Tuning and Availability
• Operational Availability Goals
• Questions
ARW 2011/LCLS AvailabilityCape Town, April 2011
3
• Operated by Stanford University for the US Department of Energy
• Study the molecular world in ways never before possible
• 3 of 6 LCLS scientific instruments fully commissioned
• By February 2011 1,162 unique scientists from 28 countries have submitted 427 proposals to use LCLS • Typically two experiments are scheduled for a five day user run.
LCLS FEL at SLAC
ARW 2011/LCLS AvailabilityCape Town, April 2011
4
A Tool for the Global Science Community
ARW 2011/LCLS AvailabilityCape Town, April 2011
“By 2013 six instruments will give researchers unprecedented tools for a broad range of research in material science, medicine, chemistry, energy science, physics, biology and environmental science” (LCLS fact sheet:http://www-group.slac.stanford.edu/com/docs/lcls_fact_sheets.pdf )
5
LCLS Free Electron LASER
Parameter Hard x-rays Soft x-rays Short pulse soft
Short pulse hard
unit
Final e- energy 6.7 – 14.7 3.3 – 6.7 3.3 – 6.7 6.7-14.7 GeV
Bunch Charge 150,250 150,250 20,40,80 20,40,80 pC
Pulse Duration 70-250 70-400 <10 <10 fs (fwhm)
Repetition Rate 1,10,60,120 1,10,60,120 1,10,60,120 1,10,60,120 Hz
Photon Bandwidth
~0.2-0.5 ~0.2-0.5 ? ? %
ARW 2011/LCLS AvailabilityCape Town, April 2011
For full list see: https://slacportal.slac.stanford.edu/sites/lclscore_public/Accelerator_Physics_Published_Documents/LCLS-parameters.pdf
• Each experiment selects a set of parameters to optimize their science goal.• Each configuration has different hardware and software requirements.
6
LCLS Major Components
• 197 Quadrupoles• 158 X, 159 Y Steering
Correctors• 175 Beam Position
Monitors• Klystrons/Modulators
– 4 Injector Stations– 31 “Linac 2” Stations– 48 “Linac 3” Stations
• 33 Undulators• 33x5 CAM Movers• ~1.8 Km of Vacuum• ~124 Vacuum Pumps• ~ 45 CAMAC crates• 2 Injector LASER
systems• 33 Beam Shut Off Ion
Chambers
ARW 2011/LCLS AvailabilityCape Town, April 2011
7ARW 2011/LCLS Availability
Cape Town, April 2011
LCLS Availability: 94.8%
Program Delivered User OffConfig. Change
Tuning DownScheduled
OffPhotons Hardware
USER 87.6 3.1 4.2 1.9 3.3 0.6 94.8 96.7
COMM 85.8 6.1 3.2 1.9 2.8 0.3 95.2 97.2
ALL 87.1 3.8 4.0 1.9 3.2 0.5 94.9 96.8
Percent of Scheduled Hours (%)
Plots and tables with the latest LCLS Availability information are automatically updated every shift during user runs.
COMM: Hutch Commissioning Programs.
Run I: 92.8% Photon, 97.0% HardwareRun II: 92.5% Photon, 97.2% Hardware
Runs I and II Photon Availability combined“Tuning” and “Configuration Change” as one category.
8
LCLS Availability Terms
• Photon Availability: (Delivered + User Off + Configuration Change) / Scheduled On
• Hardware Availability: (Delivered + User Off + Configuration Change + Tuning)/ Scheduled On
• Scheduled On: Delivered + User Off + Configuration Change + Tuning + Down
ARW 2011/LCLS AvailabilityCape Town, April 2011
9
Photon or Operational Availability
Short term performance as important as long term.
Users scheduled for machine time for five day intervals.
ARW 2011/LCLS AvailabilityCape Town, April 2011
10
Trends• Tuning, Downtime and Configuration Change time improved during the user run.
• Tuning and Configuration Change time reduction likely due to increased operational efficiencies.
• Downtime reduction not fully explained. Further analysis (Weibull analysis) needed.
•Notice correlation between tuning time and down time. (Down time leads to tuning time).
ARW 2011/LCLS AvailabilityCape Town, April 2011
11ARW 2011/LCLS Availability
Cape Town, April 2011
Downtime Statistics and future upgrades
12
MTBM and MDT• Mean Time Between Maintenance Includes Corrective Maintenance and it does not include Preventive Maintenance.
• We accept that there will be preventive maintenance scheduled.
• This MTBM number should better reflect “Maintainability” when it most matters: During beam delivery to users.
•Only Selected Systems shown. See conference paper for full list.
• Data is 6 month snapshot (69.7 operating days).
ARW 2011/LCLS AvailabilityCape Town, April 2011
SystemMTBM (Days)
MDT (Hours)
1.3 Power supplies 17.4 0.41.6 Linac Modulators 10 0.31.7 Subbooster supplies 23.2 0.93.1 S-band Klystrons 34.9 0.83.3 Hi-power Subboosters 34.8 1.13.6 sol. Stat subboosters, Low-Level RF 6.3 0.65.1 Electrical (on site) 23.2 0.35.3 Cooling water (mechanical) 69.7 1.46.2 LCLS gun laser system 7.7 0.97.1.1 Micros & Camac crates 11.6 1.47.1.3 Timing 23.2 0.57.1.4 Vacuum I&C 34.9 0.27.1.15 MPS 34.8 1.27.2.1 MCC:: (VMS System) 69.7 1.57.3.2 PPS 7.7 17.4.1 Controls Software 8.7 0.510.1 Procedural Error 23.2 0.410.2 Physicists Software 34.9 0.912.1 PCDS Hardware 13.9 0.2
13
Tuning Reduction Task Force
• Members from Physics, Operations and Software groups.
• Working on reducing tuning time during Machine Development and Repair Days recovery.
• Expect this effort to also reduce tuning time during User runs.
• Tuning reduced operational availability by ~2%
ARW 2011/LCLS AvailabilityCape Town, April 2011
14
Tuning Task Force
• Identify how tuning time is spent; Mine data from electronic logbook. Talk to Operators.
• Measure execution time of software procedures.
• Understand Machine configuration reproducibility.
• Ongoing development of software to simplify and automate tuning procedures.
ARW 2011/LCLS AvailabilityCape Town, April 2011
15
Tuning: Need Metrics Pie!
• Injector Emittance, LI21 Emittance X, LI28 Emittance X, LTU Emittance X, Injector Matching, LI28 Matching, LTU Matching, Injector Phase Scans, Charge Change, Energy Change, Mirror reconfiguration, DL2 Dispersion with Quads, DL2 Dispersion with CORs, Undulator Re-pointing, Cathode Alignment, BSY Steering, LTU Steering, Laser Heater Alignment, Laser Heater Timing, C-Iris steering, MPS Trips at some settings, MPS Trips with Slotted Foil, Taper reconfiguration, Gas Detector Calibration, E-loss scan, Klystron Phasing, RF Amplitude Calibration, Zeroing e+ correctors, BBA, LI21 Emittance Y, LI28 Emittance Y, LTU Emittance Y, XCAV Set-points for FEL, L3 Steering…
ARW 2011/LCLS AvailabilityCape Town, April 2011
16
LCLS Availability Goal is 95%
• That is 95% Operational Availability to Users.
• Before run III, goals did not take Tuning into account.
• A tuning budget (~2%) will be included in the availability plan. This sets the hardware availability goal at 97%.
ARW 2011/LCLS AvailabilityCape Town, April 2011
17
Realizable Goals
ARW 2011/LCLS AvailabilityCape Town, April 2011
Sub-System
RUN III
Previous Goal New Goal
Power Supplies (See Session 9, Wednesday) 99.6 99.42 99.77 Magnets (See Session 14, Thursday) 100 99.96 99.95 RF
99.37 98.98 99.44 Vacuum
99.93 99.86 99.95 Utilities
99.84 99.38 99.84 Guns & lasers
99.52 99.42 99.64 Controls
98.36 98.23 98.66 Non-Radiation Safety
100 99.95 99.95 Alignment
100 99.82 99.95 Other
99.82 99.92 99.93 Unassigned
99.99 99.98 99.95 Photon Controls
99.93 99.98 99.95 Total
96.7 95.00 97.00
• Goals based on past performance
• Remove worst two weeks or events from historical data (Life of the project ~2.5 years).
• System owners in charge and accountable for given system’s availability performance.
18
Conclusion
• LCLS science program is ongoing.
• Two year hardware availability ~ 97%
• Photon availability of 95% realizable.
• Multiple upgrade projects underway.
• Further analysis needed to better understand long term time dependence of individual subsystem and component failure rate.
ARW 2011/LCLS AvailabilityCape Town, April 2011
19
Thank You!
• Questions.
ARW 2011/LCLS AvailabilityCape Town, April 2011
If we use failure rates as a metric for determining which systems (and therefore groups) will receive attention and/or resources; are we setting up a system where failure is rewarded?
In a climate of decreasing resources and budget constrains: Is there a model that correlates Availability and Reliability with Budget Allowance?
20ARW 2011/LCLS Availability
Cape Town, April 2011