welcome to the mpp workshop 2013 in annecy!
Post on 02-Jan-2016
24 Views
Preview:
DESCRIPTION
TRANSCRIPT
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013 1
WELCOME to the MPP Workshop 2013 in Annecy!
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Past reviews
●MP related reviews that were done in the past• External Machine Protection review (June 05 - overall system)• BIC audit summary (Sept 06)• BIC audit sequel summary (June 09)• LBDS audit summary (Feb 07)• LBDS audit sequel summary (June 09)• BLM audit summary (July 08)• BLM audit sequel summary (June 09)• MPS internal review (June 2010)• Collimation system (several reviews)• External LHC Machine Protection Review (September 2010 - are we ready
to go beyond 30 MJ?)• Machine Protection Workshop for linear accelerators (June 2012)• Machine Protection Workshop (March 2013)
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Mandate
●Discuss mid-and longer-term improvements of the MP systems for the LHC + injector complex:
• review of the current operational experience with MP systems during the first running period (2009-2013).
• understanding the planned changes of MP equipment during LS1 and the consequences/potential limitations for operation after LS1.
• identify areas where improvements are required.
• ensuring coherence between the different MP systems.
• identify misses.
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Program
● Indico Page●6 sessions, 2 ½ days, ~ 85 participants
• MPS experience (2008-2012) and outlook (D.Wollmann)
• Injection and LHC beam dumping system (J.Uythoven)
• Beam Diagnostics (B.Dehning)
• Collimation and movable devices (S.Redaelli)
• Electrical circuit related protection (M.Zerlauth)
• Operation after LS1 (J.Wenninger)
• Summary and discussion (R.Schmidt)
• Executive summary @ CERN in special LMC (date tbc)
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
A bit of general info first….
●No (more) coffee break on MON morning!
●Free WiFi in the whole hotel, consume with moderation
●Lunch scheduled for 12.30 in the restaurant (seated)
●Rooms will be ready around 16.00
● In case of any problems, please contact Mariane
(+41 76 487 87 53)
5
LHC Machine Protection WorkshopM. Zerlauth March 2013
Thanks to : B.Dehning, K.Fuchsberger, B.Goddard, S.Redaelli, J.Uythoven, R.Schmidt, J.Wenninger, D.Wollmann, MPP and rMPP members et al
1v0
MPS issues and MP approach concerning operation and MDs
• 3 years of MPS in short • Organization and issues• MP protection during commissioning and production• Non-standard operation• Conclusions
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
3 years of LHC Machine Protection in short ● Some 3499 clean beam dumps performed since Oct 2009:
1582 beam dumps above 450GeV.In 2012 a majority of dumps with beam energies > 100MJ (reaching max 146MJ).
● No beam induced magnet quenches @3.5/4.0TeV.● No serious equipment damaged due to beam (excl. heating, corrector coils of
RQX.3L2, ALICE detector).● Reasons and MPS response reasonably well documented, analyzed and validated
for all beam dumps > 450GeV. More details see talk from B. Todd
7
2012
2011
2010
Courtesy B. ToddCourtsey of B.Todd – Evian 2012
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
In the end there was reason to be content…
>23 fb-1
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Driving along the edge throughout…● LBDS 12V● Quench detection issues on IPQ, 600A EE● TL collimators● Injecting H9● False collimator settings (2 x TCTV IR2, 2x IR3)● Roman Pot Controls● BLM High Voltage Cable ● OFSU reference problems● BSRT Mirror● MDs and other non-standard machine
operation● MKI flashovers ● QFB not usable in squeeze due to poor signal● Instrumentation problem in triplet L8 after TS2● Loss of redundant protection (60A power
permits, LHCb dipole , CMS solenoid,…)● ….
9
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Mandate and Organization of MP Work● Assure (potentially) dangerous situations are recognized, documented and
distributed and actions followed up throughout commissioning and production
● MPP (in present form since Sept 2008) for scientific ground work and preparing decisions – ties and certain overlap with COLL WG, LIBD, LBOC, AWG,..• Will need to increasingly look into HL-LHC and injectors as well
● rMPP added during production run (Sep 2010) to follow up and decide on ‘urgent’ issues and non-standard operation• Very good for ad-hoc discussions (outside large bodies), sometimes had to decide in
absence of few key members -> Still need a common line from rMPP
• Need for (more) regular meetings? Update of rMPP membership?
• Follow-up & responsibility of recommendations/ procedures less obvious (MD documents, checklists,….)
● Interplay of MPP and MP3 for performance of magnet powering system to be defined/improved for post LS1?
10
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Examples of issues and how we dealt with them
● Safety critical issues in MPS systems● Inversion of HTS instrumentation cable on
RB.A45 found following CL quench on 7th April 2011 due to cooling problem on DFBAI.L5 ● Stopped powering for verification of
all >6kA circuits for 1.5 days
● Common 12V dependency in LBDS system: Found 8th of June 2012 during Lab test (follow up of LBDS review on redundant powering) ● LHC coordinator and rMPP informed,
ongoing fill dumped + operation stopped for some 5 hours for implementation of watch-dog
11
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Examples of issues and how we dealt with them
● Wrong TL-TCDI settings for SPS Q20 optics found on 19th of November 2012 (up to 1.3 σ)● CCC informed and operation stopped for 12 hours to recalculate + revalidate
● Missing BLM HV noticed by sanity check on 12th of April 2011● System reconfigured and validated (using supply and spare cable from LSS),
later SIS interlock added
12
Cell 12-34
First indication in Pre-op Check
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Examples of issues and how we dealt with them
● Issues in MPS systems resulting in known (and protected) failure cases• E.g. non redundant LBDS powering, MKI flash-over, injection of H9,… • Operation in general continued, further minimizing risk through longer term
mitigations● Issues with non-MPS systems with certain potential to become unsafe
• E.g. OFSU without reference, BSRA, RP controls• Operation in general continued straight away, short-term mitigations put in
place to further minimize risk, e.g. postponed next intensity step, new SIS interlocks and additional sequencer and operator checks
13
Missing Orbit reference
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Would a MP ‘responsible’ be beneficial?
● Major issues were mostly ‘dormant’ failures and appropriate action was taken immediately• Lessons should be learned and commissioning procedures tightened up
● Some issues persisted for some time & reaction might vary as function of individuals knowledgeable about it
● Double role of EIC/Machine Coordinator to optimize performance and assure protection not always easy to combine
● To increase and facilitate support to operation crews: Role of an ‘MP responsible / week’ for post LS1?• Contact (representing rMPP) for operation crews in case of doubts and
non-standard operation• Assure necessary documentation, follow-up of operational issues, e.g.
analysis of dumps,...• Follow-up of MPS commissioning, operational changes + necessary
revalidations
14
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Documentation of MPS tests and issues - Commissioning● Very rigorous and thorough approach to preparation, execution & documentation● Required time was in general allocated /available ● Commissioning procedures need to be updated and revised during LS1
• Profit to organize execute as well ‘difficult’ tests (redundant powering,…)● MPS site worked fine – driven by few individuals in parallel to coordination, progress &
status of advancement not obvious● Use ACCTEST for more dependable + automatic sequencing, dependency tracking + follow-
up
15
See talk of K.Fuchsberger
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
From commissioning towards production● Commissioning mindset should be maintained throughout full running period, suffered
towards end increasingly due to scheduling pressure, routine (we have done this before without problem), relatively smooth running, fatigue….
● Initial (commissioned) safety level certainly degrades under such circumstances● Possibilities to counter-act?
16
• Tracking of (relevant) system changes• More coherent approach for follow-up of magnet &
beam related MPS issues• Define & enforce minimum (non-negotiable)
validation costs of changes, e.g. new setting validation with loss-maps & asynch dumps, critical interventions,… -> Takes ‘responsibility’ off the piquet teams, operation crews, coordinators,…
• Automatic tools/dependency models could help to enforce this for well known cases
• MP responsible for remaining cases
CERN
markus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Change Management
17
● MPS HW and SW changes tracked during past 2 years in MPS commissioning SharePoint Site, however…
● We have to clearly define what is mandatory to be tracked
● Requires corresponding culture and level of responsibility of experts (many changes not even known/advertised!)
● Even more so for SW deliverables -> Culture of best practice (Good design, Unit Testing, Integration Testing , Functional Testing)
● Could a commissioning tool like ACCTEST (in conjunction with a dependency management) be re-used to enforce re/validation process after changes?● Concentrate on few most critical &
reoccurring items first, like powering system, settings validations, …
CERN
benjamin.todd@cern.ch ITER – Machine Protectionmarkus.zerlauth@cern.ch MPP Workshop – Annecy 2013
During production running● Post Mortem DB with operator and MPS expert comments (works well for dumps above
injection energy, how to improve for dumps @ injection?!)• Can and need to do more automated validation checks (pre and post-op, IQC,
XPOC…), difficulty is vast failure space and too frequent false positives● Regular distribution of MPS check lists during intensity ramp-ups and operation to MP
system experts with all dumps above injection energy for comments
18
• Considered very useful by MP experts, should this be distributed to anyone else?
• Regular reflection on past issues, comparison with own fault tracking and reminder for follow-up
• Problem of ‘self-validation’ could be improved by true approval circuit
• Should be enforced on more regular basis (e.g. did none during Ion running)
CERN
markus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Machine Developments 1/2
19
● Machine Developments per definition explore new machine and machine protection territory and hence risks!
● MP documents did allow to address and mitigate risks – did preparation phase increase as well the efficiency of the MDs?
MD 1 MD 2 MD 3 MD 4 QT FTMD 1 FTMD 20
1
2
3
4
5
6
7
8
# MDs with MP Doc
Unwanted Dumps
MD 1 MD 2 MD 3 MD 4 QT FTMD 1 FTMD 20
1
2
3
4
5
6
7
8
9
# MDs without MP Doc
Unwanted Dumps No Docs
● Un-intentional losses of beams as (one possible) indicator for ‘efficiency’, e.g. hidden interlocks during change of SBF, (not counting machine unavailability, equipment failures,…)
● With detailed MP MD doc ~ 0.38 unintentional dumps/MD
● Without doc ~ 0.67 unintentional dumps/MD
Courtesy of D.Wollmann
CERN
markus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Machine Developments 2/2
20
● Additional factors influencing ’efficiency’● Increasingly flexible plannings towards the end, many documents done ‘last minute’,
little time for comments and iterations● Operations not sufficiently aware of MD details and requested machine preparations
Proposal for post LS1:MD – 4 weeks:● For all MDs up-to-date MD request
document (parameter space, interlocks, program..) precondition for being scheduled
MD -2 weeks:● Draft procedure for MD Class C ● True EDMS approval circuit, all rMPP
members to signMD:● EDMS approval precondition to start● Simple additional SIS interlocks to
enforce defined parameter space ?!● OP shifters responsible to assure correct
execution of MD and eventual checklists
CERN
markus.zerlauth@cern.ch MPP Workshop – Annecy 2013
Summary
21
LHC Machine Protection Systems have been working well during run 1
No evidence of major loopholes or uncovered risks in MPS architecture, however a few near-misses have revealed short-comings in designs, commissioning
procedures and operation of MPS systems
Organization and response of coordinators, OP, (r)MPP, equipment experts to these problems was adequate
As a result of many different factors some of the initial rigor appeared to be lost at all levels towards to end of the run
Could an MP responsible be an additional asset for operations and help counteracting some of the issues observed?
It might become increasingly challenging to maintain systems at their needed level of dependability (additional operational challenges post LS1, allocated resources
and time,..)
top related