1 review of recent run l. coney – ucr mice cm34 – oct 2012

Coney - CM34 - Oct 2012 1

Review of Recent Review of Recent RunRun

L. Coney – UCRL. Coney – UCR

MICE CM34 – Oct 2012MICE CM34 – Oct 2012

2Coney - CM34 - Oct 2012

OutlineOutline Organization & PreparationOrganization & Preparation ShiftersShifters Goals of RunGoals of Run IssuesIssues Next..Next..


Organization & Organization & PreparationPreparation

Micemine Run Plan worked wellMicemine Run Plan worked well http://micewww.pp.rl.ac.uk/projects/http://micewww.pp.rl.ac.uk/projects/

operations/wiki/RunPlan20121012operations/wiki/RunPlan20121012


Organization & Organization & PreparationPreparation

Run plan continued…Run plan continued… Used to schedule shifters and Used to schedule shifters and

software expertssoftware experts Used to lay out daily planUsed to lay out daily plan


Organization & Organization & PreparationPreparation Run plan wiki page used to define Run plan wiki page used to define

beamline configurations for data beamline configurations for data requested (although not primary goal requested (although not primary goal for this run period)for this run period)


Shifters…Shifters… As of CM33, we have a shifter training regimeAs of CM33, we have a shifter training regime And have trained some people to be qualified And have trained some people to be qualified

shifters (intro role) and Lead Shifters (more shifters (intro role) and Lead Shifters (more advanced)advanced)

However…However… Recent running has made the need for more Recent running has made the need for more

qualified/trained individuals very clearqualified/trained individuals very clear


Need New/More ShiftersNeed New/More Shifters Recognized the need for training – now need Recognized the need for training – now need

to implement trainingto implement training Difficult to do while running – difficult to Difficult to do while running – difficult to

properly train w/o runningproperly train w/o running Need some training w/o beam but with Need some training w/o beam but with

equipmentequipment Develop training scheduleDevelop training schedule

Corollary – develop training updates when new Corollary – develop training updates when new equipment arriveequipment arrive

Need all MOMs able to lead/arrange trainingNeed all MOMs able to lead/arrange training Sign up more MICE – THIS MEANS YOU!Sign up more MICE – THIS MEANS YOU! Cannot exploit tiny group of (primarily UK-Cannot exploit tiny group of (primarily UK-

based) dependable individualsbased) dependable individuals This becomes even more true for Step IV This becomes even more true for Step IV

runningrunning


Goals of RunningGoals of Running Test significant upgrades and changes to both the Test significant upgrades and changes to both the

Online Systems and MAUS software packages – NOT to Online Systems and MAUS software packages – NOT to take datatake data

Overall OnlineOverall Online New OS installed on all MLCR computersNew OS installed on all MLCR computers New master server, new account setups, new UPSsNew master server, new account setups, new UPSs

DAQDAQ Final live data test of new DAQ system – upgraded DATE & Final live data test of new DAQ system – upgraded DATE &

OSOS New interface with C&MNew interface with C&M New DAQ hardwareNew DAQ hardware

C&MC&M Full read/write from CDBFull read/write from CDB Overall system test of Step I Run ControlOverall system test of Step I Run Control Expert testing/feedback on Run ControlExpert testing/feedback on Run Control New HV user docsNew HV user docs

SoftwareSoftware Online test server setup functionalOnline test server setup functional New control room branch of MAUSNew control room branch of MAUS Improved TOF/CKOV online plotsImproved TOF/CKOV online plots


Goals of RunningGoals of Running DAQ – Yordan’s talkDAQ – Yordan’s talk C&M – Pierrick’s talkC&M – Pierrick’s talk Software – Chris Rogers’ talk this Software – Chris Rogers’ talk this

morningmorning

Mixed results – confirm that we DO need Mixed results – confirm that we DO need regular runningregular running Partially successfulPartially successful Unfortunately – not smooth run – while this Unfortunately – not smooth run – while this

was expected due to the many changes, many was expected due to the many changes, many of the problems were not anticipatedof the problems were not anticipated

No real useful data until Sunday….although No real useful data until Sunday….although this was not the goal of the runthis was not the goal of the run


Recent Run – IssuesRecent Run – Issues TargetTarget

Fatal error in operation – related to laser gainFatal error in operation – related to laser gain Discovered Discovered beforebefore frame lowered – required remote expert frame lowered – required remote expert

interventionintervention Target DAQ display (RATS) did not start correctly – Target DAQ display (RATS) did not start correctly –

would not update – required remote expert interventionwould not update – required remote expert intervention ISIS beam loss did not correlate – tripped ISIS (3V+) ISIS beam loss did not correlate – tripped ISIS (3V+)

when target DAQ showed below 2Vwhen target DAQ showed below 2V Needed expert every day Needed expert every day

HV controlsHV controls SY527 remote control not working – required C&M SY527 remote control not working – required C&M

expert on site to fix (related to ongoing work for expert on site to fix (related to ongoing work for newnew HV HV system – interfered)system – interfered)

Luminosity monitorLuminosity monitor Power supply turned OFF incorrectly (left at 10.7 V Power supply turned OFF incorrectly (left at 10.7 V

instead of ramped down)instead of ramped down) DAQ crate not turned ONDAQ crate not turned ON Suspect result of UPS installation – systems not Suspect result of UPS installation – systems not

returned to original statereturned to original state


Recent Run – IssuesRecent Run – Issues DAQ – crashed/needed expert interventionDAQ – crashed/needed expert intervention C&MC&M

Some new applications not workingSome new applications not working Application launcher start issuesApplication launcher start issues Neutron monitor readout not workingNeutron monitor readout not working All required on-site expert intervention – in addition to All required on-site expert intervention – in addition to

new systems being testednew systems being tested Online MonitoringOnline Monitoring

Gone – result of changeover to new DAQ – will probably Gone – result of changeover to new DAQ – will probably notnot be easy to resurrect be easy to resurrect

Online Reco/MAUSOnline Reco/MAUS Did not work – new plot upgrades could not be tested – Did not work – new plot upgrades could not be tested –

bug, plots not filledbug, plots not filled Software group is assessing prep process and will debug Software group is assessing prep process and will debug

offlineoffline Will need to test at later dateWill need to test at later date

Datamover Datamover Crashed first time runCrashed first time run Crashed again when attempted to fixCrashed again when attempted to fix Need expert phone or debugging documentation Need expert phone or debugging documentation


Way Forward..Way Forward.. We are making progressWe are making progress

Previously we were focused on getting the basics going Previously we were focused on getting the basics going – Implementation of necessary functionality– Implementation of necessary functionality

Now – move focus to Now – move focus to reliabilityreliability Need to Need to KEEP THINGS WORKING!KEEP THINGS WORKING! Schedule regular 3-day weekend runsSchedule regular 3-day weekend runs MAUS – already using “release” system of MAUS – already using “release” system of

changeschanges C&M C&M

need do same with the controls and monitoring codeneed do same with the controls and monitoring code Stabilize applicationsStabilize applications Development computer/system separate from Development computer/system separate from

iocpc1/iocpc2iocpc1/iocpc2 Coordination between C&M developers Coordination between C&M developers mustmust happen happen

DAQ – staged upgrades – especially with new DAQ – staged upgrades – especially with new equipmentequipment


Regular RunningRegular Running Why?Why?

Implement changesImplement changes Operate equipment long idleOperate equipment long idle PracticePractice Train shift personnelTrain shift personnel LearnLearn

NEED coordination of vital personnelNEED coordination of vital personnel Worked (almost) this timeWorked (almost) this time CDB expert on leaveCDB expert on leave MAUS expert in US in advance of runMAUS expert in US in advance of run

Software/on call experts/remote availabilitySoftware/on call experts/remote availability If you are responsible for a system you must be If you are responsible for a system you must be

contact-ablecontact-able


SummarySummary While not exactly smooth…run While not exactly smooth…run

last weekend accomplished much last weekend accomplished much of what needed to doof what needed to do

Clear need for more trained shift Clear need for more trained shift personnelpersonnel

Clear need for regularly Clear need for regularly scheduled runningscheduled running


MICE Step IVMICE Step IV Includes both Spectrometer Includes both Spectrometer

SolenoidsSolenoids Two trackers installed within the Two trackers installed within the

SS magnetsSS magnets One AFC (Focus Coil magnet & One AFC (Focus Coil magnet &

Absorber)Absorber)


New Equipment…New Equipment… We will have much more in the We will have much more in the

Hall!Hall!


Step IV: Operations & Step IV: Operations & OnlineOnline

Moving toward Step IV runningMoving toward Step IV running Many configurations to test/runMany configurations to test/run Long term running/every opportunity takenLong term running/every opportunity taken All ISIS cycles – every day – Differs from past All ISIS cycles – every day – Differs from past

OperationsOperations Develop idea of not-so-remote control roomDevelop idea of not-so-remote control room Develop plan for staffing & commissioning of cooling Develop plan for staffing & commissioning of cooling

channelchannel

Develop robust system of OperationsDevelop robust system of Operations Includes all facets of Online Group (DAQ, C&M, Online Includes all facets of Online Group (DAQ, C&M, Online

Reconstruction/Analysis, Infrastructure)Reconstruction/Analysis, Infrastructure)

Goal – next phase of OperationsGoal – next phase of Operations FunctionalFunctional Safe/SecureSafe/Secure ReliableReliable Easy to useEasy to use


Step IV – OperationsStep IV – Operations Continue:Continue:

Pre-run prepPre-run prep Further simplify runningFurther simplify running

Improve:Improve: Automate procedures Automate procedures fewer errors fewer errors Move from spreadsheet Move from spreadsheet only ConfigDB only ConfigDB Increase Online Reconstruction capabilitiesIncrease Online Reconstruction capabilities

Complete suite of detectors, Online analysis, Global recoComplete suite of detectors, Online analysis, Global reco Reduce required number of shifters…Reduce required number of shifters…

New equipment arriving – plan for support & New equipment arriving – plan for support & expertiseexpertise

Modify procedures to include SC magnets and Modify procedures to include SC magnets and LH2 systemLH2 system

Train shift personnelTrain shift personnel Develop plan to staff multitude of shiftsDevelop plan to staff multitude of shifts


Step IV – OperationsStep IV – Operations Safety Safety

Need understand implications under STFC rules of Need understand implications under STFC rules of new equipmentnew equipment

Made arrangements to meet with Jane Vickers (ISIS Made arrangements to meet with Jane Vickers (ISIS safety) to begin preparation/planningsafety) to begin preparation/planning

Need understand from experts what each system Need understand from experts what each system entails/requires for operationentails/requires for operation

PPSPPS Need understand implementation of system with SC Need understand implementation of system with SC

magnets and LH2 systemmagnets and LH2 system Hall AccessHall Access

Limited – not every day – scheduled access onlyLimited – not every day – scheduled access only Increase efficiency of data-takingIncrease efficiency of data-taking Hall search dropped Hall search dropped twicetwice in 3 days last weekend. in 3 days last weekend.

Not acceptable.Not acceptable. Experts must be availableExperts must be available


General OperationsGeneral Operations SS1, SS2, Tracker1, Tracker2, FC, solid SS1, SS2, Tracker1, Tracker2, FC, solid

absorbers, LH2 absorber & systemabsorbers, LH2 absorber & system

SS commissioningSS commissioning How will this work? How long?How will this work? How long? Access needed to magnet? How often?Access needed to magnet? How often?

Tracker commissioningTracker commissioning How done? How long? Plan for alignment?How done? How long? Plan for alignment? Analysis immediate? Or delayed?Analysis immediate? Or delayed? What access required?What access required?

LH2 + FC system commissioningLH2 + FC system commissioning Needs 1 monthNeeds 1 month Once LH2 system in place – not coming out againOnce LH2 system in place – not coming out again


Operations OrganizationOperations Organization Overall Operations ManagerOverall Operations Manager

Local, familiar with STFC safety and operational Local, familiar with STFC safety and operational systemssystems

Provide oversight, link between MOMS, safety Provide oversight, link between MOMS, safety responsibility from project managerresponsibility from project manager

Monthly MOMsMonthly MOMs On call for 1 month – Daily running duties as nowOn call for 1 month – Daily running duties as now

MICE Shift personnelMICE Shift personnel Currently 2 shifters for ~8 hour shiftCurrently 2 shifters for ~8 hour shift Prefer to move to 1 MICE shifterPrefer to move to 1 MICE shifter

Current on-call experts – BLOC, SOC, TROCCurrent on-call experts – BLOC, SOC, TROC New system-specific experts – 3 per system – not New system-specific experts – 3 per system – not

necessarily current MICEnecessarily current MICE Magnet & cryo experts – maintenance of magnet Magnet & cryo experts – maintenance of magnet

systems, manage cryogens and vac pumping rigs & systems, manage cryogens and vac pumping rigs & instrumentation, cooling and powering expert, quench instrumentation, cooling and powering expert, quench behavior expertbehavior expert

LH2 expert – control engineering knowledge, safety LH2 expert – control engineering knowledge, safety proceduresprocedures


Step IV OperationsStep IV Operations Given current MICE schedule for Step Given current MICE schedule for Step

IV equipment and installation…IV equipment and installation… We We mustmust be prepared for 24/7 running be prepared for 24/7 running

in order to get Step IV data before ISIS in order to get Step IV data before ISIS shutdown in August 2014shutdown in August 2014

Can our systems handle this?Can our systems handle this? Target ok? Magnets ok?Target ok? Magnets ok? DAQ ok? Controls ok? Datamover ok?DAQ ok? Controls ok? Datamover ok? Shift coverage ok?Shift coverage ok?

Another reason to limit Hall access.Another reason to limit Hall access.


Floor Open..Floor Open.. Questions?Questions?

Suggestions?Suggestions?

Volunteers?Volunteers?



FunctionalFunctional DAQ – incorporate all detectors – improve rateDAQ – incorporate all detectors – improve rate DAQ upgradesDAQ upgrades All C&M for each elementAll C&M for each element Alarm Handler values set appropriatelyAlarm Handler values set appropriately Add capability to Online Reco/AnalysisAdd capability to Online Reco/Analysis Automate run infrastructure – Run ControlAutomate run infrastructure – Run Control Automatic use – read/write to Configuration DatabaseAutomatic use – read/write to Configuration Database

Safe/SecureSafe/Secure Incorporated fully commissioned PPSIncorporated fully commissioned PPS Implemented formal shifter trainingImplemented formal shifter training Updated Operations/Online documentation & instructionsUpdated Operations/Online documentation & instructions Developing comprehensive list of safety-critical maintenanceDeveloping comprehensive list of safety-critical maintenance Updating safety paperwork – developing overall system for Updating safety paperwork – developing overall system for

MICEMICE Using new target controller with all necessary interlocks & Using new target controller with all necessary interlocks &

BPSBPS Online system security reviewed – improvements madeOnline system security reviewed – improvements made Access limited to micenetAccess limited to micenet



Easy to useEasy to use Simplified Ops procedures – single key exchange ISISSimplified Ops procedures – single key exchange ISIS MICE control access to HallMICE control access to Hall Improving Online Reconstruction/Data Quality plotsImproving Online Reconstruction/Data Quality plots Need add Online Reco for KL/EMR/Accel AnalysisNeed add Online Reco for KL/EMR/Accel Analysis Run ControlRun Control Mature Alarm HandlerMature Alarm Handler

ReliableReliable New OS implemented – automatic updatesNew OS implemented – automatic updates Automatic/systematic backups in placeAutomatic/systematic backups in place New DAQ and C&M computers installedNew DAQ and C&M computers installed UPS coverage for critical systemsUPS coverage for critical systems Developing “stockroom” of spares – Developing “stockroom” of spares –

crates/computers/etccrates/computers/etc Automatic write to EPICS ArchiverAutomatic write to EPICS Archiver Extensive remote monitoring of Extensive remote monitoring of

equipment/computers/hardware/environment in EPICSequipment/computers/hardware/environment in EPICS Archive of all software – Launchpad repositoryArchive of all software – Launchpad repository

1 review of recent run l. coney – ucr mice cm34 – oct 2012

Documents

run periodconey cm34

clearconey cm34

runningconey cm34

runconey cm34

new systems

coney ucrmice cm34

new equipment

original stateconey