1 operations summary l. coney – ucr mice cm34 – oct 2012

Coney - CM34 - Oct 2012 1

Operations Operations SummarySummary

L. Coney – UCRL. Coney – UCR

MICE CM34 – Oct 2012MICE CM34 – Oct 2012

2Coney - CM34 - Oct 2012

Online Updates – New Online Updates – New DAQDAQ


DAQDAQ DAQDAQ

New DAQ in placeNew DAQ in place Implemented new acquisition library independent of Implemented new acquisition library independent of

DATE and EPICS – leads to flexibility and is easy to DATE and EPICS – leads to flexibility and is easy to useuse

Written simple applications for use in the control Written simple applications for use in the control room room

Have thin C++ wrapper of EPICS – flexible/easy – Have thin C++ wrapper of EPICS – flexible/easy – also independent of DATE and DAQ hardwarealso independent of DATE and DAQ hardware

Communication with CDB done by Controls & Communication with CDB done by Controls & Monitoring (Run Control)Monitoring (Run Control)

At end – interface with DATEAt end – interface with DATE New zero suppression of fADCsNew zero suppression of fADCs Very high DAQ rate now possibleVery high DAQ rate now possible New pulser trigger availableNew pulser trigger available


DAQDAQ DAQ can now handle rates above the DAQ can now handle rates above the

initial 600 muons/spill goalinitial 600 muons/spill goal

New software/hardware system in New software/hardware system in place, debugged, tested, and place, debugged, tested, and functionalfunctional

All that remains on Yordan’s All that remains on Yordan’s Christmas list for the “PID” DAQ is Christmas list for the “PID” DAQ is the programmable FPGA logic for the programmable FPGA logic for the particle triggerthe particle trigger And the EMR and Trackers…And the EMR and Trackers…


Online Systems Updates Online Systems Updates – C&M– C&M

Been very busy since GlasgowBeen very busy since Glasgow TargetTarget Daresbury C&MDaresbury C&M Spectrometer SolenoidSpectrometer Solenoid LH2 MonitoringLH2 Monitoring Run ControlRun Control RF TunersRF Tuners Auto-SMS notificationAuto-SMS notification More system monitoringMore system monitoring State MachinesState Machines


C&M - TargetC&M - Target Rewritten Target C&M systemRewritten Target C&M system

Better EPICS complianceBetter EPICS compliance Web interfaceWeb interface CDB compatibleCDB compatible Online analysisOnline analysis Run priorityRun priority Root file outputRoot file output

Making similar changes to Tracker Making similar changes to Tracker systemsystem


C&M C&M DLDL

SS standalone system being usedSS standalone system being used Being improved for final SS trainingBeing improved for final SS training

FC standalone readyFC standalone ready


C&M C&M LH2 LH2

MonitoringMonitoring ArchivingArchiving Control expert onlyControl expert only

RF TunersRF Tuners Resonant frequency Resonant frequency

depend on depend on geometrygeometry

Change cavity Change cavity shapeshape

Controls system Controls system


C&M – Run ControlC&M – Run Control Interfaces with:

DATE TargetMon State Machine Environment Standard operation Records all running parameters to CDB Records all meta-data at end of run Verifies readiness of subsystems Sets running parameters Tags run parameters


Elog ChangesElog Changes Modifications to improve ease of use Modifications to improve ease of use

and understandability – see Henry and understandability – see Henry for input to new systemfor input to new system


Step IV: Operations & Step IV: Operations & OnlineOnline

Moving toward Step IV runningMoving toward Step IV running Many configurations to test/runMany configurations to test/run Long term running/every opportunity takenLong term running/every opportunity taken All ISIS cycles – every day – Differs from past All ISIS cycles – every day – Differs from past

OperationsOperations Develop idea of not-so-remote control roomDevelop idea of not-so-remote control room Develop plan for staffing & commissioning of cooling Develop plan for staffing & commissioning of cooling

channelchannel

Develop robust system of OperationsDevelop robust system of Operations Includes all facets of Online Group (DAQ, C&M, Online Includes all facets of Online Group (DAQ, C&M, Online

Reconstruction/Analysis, Infrastructure)Reconstruction/Analysis, Infrastructure)

Goals – next phase of OperationsGoals – next phase of Operations FunctionalFunctional Safe/SecureSafe/Secure ReliableReliable Easy to useEasy to use



FunctionalFunctional DAQ – incorporate all detectors – improve rateDAQ – incorporate all detectors – improve rate DAQ upgradesDAQ upgrades All C&M for each elementAll C&M for each element Alarm Handler values set appropriatelyAlarm Handler values set appropriately Add capability to Online Reco/AnalysisAdd capability to Online Reco/Analysis Automate run infrastructure – Run ControlAutomate run infrastructure – Run Control Automatic use – read/write to Configuration DatabaseAutomatic use – read/write to Configuration Database

Safe/SecureSafe/Secure Incorporated fully commissioned PPSIncorporated fully commissioned PPS Implemented formal shifter trainingImplemented formal shifter training Updated Operations/Online documentation & instructionsUpdated Operations/Online documentation & instructions Developing comprehensive list of safety-critical maintenanceDeveloping comprehensive list of safety-critical maintenance Updating safety paperwork – developing overall system for Updating safety paperwork – developing overall system for

MICEMICE Using new target controller with all necessary interlocks & Using new target controller with all necessary interlocks &

BPSBPS Online system security reviewed – improvements madeOnline system security reviewed – improvements made Access limited to micenetAccess limited to micenet



Easy to useEasy to use Simplified Ops procedures – single key exchange ISISSimplified Ops procedures – single key exchange ISIS MICE control access to HallMICE control access to Hall Improving Online Reconstruction/Data Quality plotsImproving Online Reconstruction/Data Quality plots Need add Online Reco for KL/EMR/Accel AnalysisNeed add Online Reco for KL/EMR/Accel Analysis Run ControlRun Control Mature Alarm HandlerMature Alarm Handler

ReliableReliable New OS implemented – automatic updatesNew OS implemented – automatic updates Automatic/systematic backups in placeAutomatic/systematic backups in place New DAQ and C&M computers installedNew DAQ and C&M computers installed UPS coverage for critical systemsUPS coverage for critical systems Developing “stockroom” of spares – Developing “stockroom” of spares –

crates/computers/etccrates/computers/etc Automatic write to EPICS ArchiverAutomatic write to EPICS Archiver Extensive remote monitoring of Extensive remote monitoring of

equipment/computers/hardware/environment in EPICSequipment/computers/hardware/environment in EPICS Archive of all software – Launchpad repositoryArchive of all software – Launchpad repository


Organization & Organization & PreparationPreparation

Micemine Run Plan worked well – Micemine Run Plan worked well – schedule & planschedule & plan

http://micewww.pp.rl.ac.uk/projects/http://micewww.pp.rl.ac.uk/projects/operations/wiki/RunPlan20121012operations/wiki/RunPlan20121012


Need New/More ShiftersNeed New/More Shifters Recognized the need for training – now need Recognized the need for training – now need

to implement trainingto implement training Difficult to do while running – difficult to Difficult to do while running – difficult to

properly train w/o runningproperly train w/o running Need some training w/o beam but with Need some training w/o beam but with

equipmentequipment Develop training scheduleDevelop training schedule

Corollary – develop training updates when new Corollary – develop training updates when new equipment arriveequipment arrive

Need all MOMs able to lead/arrange trainingNeed all MOMs able to lead/arrange training Sign up more MICE – THIS MEANS YOU!Sign up more MICE – THIS MEANS YOU! Cannot exploit tiny group of (primarily UK-Cannot exploit tiny group of (primarily UK-

based) dependable individualsbased) dependable individuals This becomes even more true for Step IV This becomes even more true for Step IV

runningrunning


Goals of RunningGoals of Running Test significant upgrades and changes to both the Test significant upgrades and changes to both the

Online Systems and MAUS software packages – NOT to Online Systems and MAUS software packages – NOT to take datatake data

Overall OnlineOverall Online New OS installed on all MLCR computersNew OS installed on all MLCR computers New master server, new account setups, new UPSsNew master server, new account setups, new UPSs

DAQDAQ Final live data test of new DAQ system – upgraded DATE & Final live data test of new DAQ system – upgraded DATE &

OSOS New interface with C&MNew interface with C&M New DAQ hardwareNew DAQ hardware

C&MC&M Full read/write from CDBFull read/write from CDB Overall system test of Step I Run ControlOverall system test of Step I Run Control Expert testing/feedback on Run ControlExpert testing/feedback on Run Control New HV user docsNew HV user docs

SoftwareSoftware Online test server setup functionalOnline test server setup functional New control room branch of MAUSNew control room branch of MAUS Improved TOF/CKOV online plotsImproved TOF/CKOV online plots


Goals of RunningGoals of Running DAQ – Yordan’s talkDAQ – Yordan’s talk C&M – Pierrick’s talkC&M – Pierrick’s talk Software – Chris Rogers’ talk this Software – Chris Rogers’ talk this

morningmorning

Mixed results – confirm that we DO need Mixed results – confirm that we DO need regular runningregular running Partially successfulPartially successful Unfortunately – not smooth run – while this Unfortunately – not smooth run – while this

was expected due to the many changes, many was expected due to the many changes, many of the problems were not anticipatedof the problems were not anticipated

No real useful data until Sunday….although No real useful data until Sunday….although this was not the goal of the runthis was not the goal of the run


Way Forward..Way Forward.. We are making progressWe are making progress

Previously we were focused on getting the basics going Previously we were focused on getting the basics going – Implementation of necessary functionality– Implementation of necessary functionality

Now – move focus to Now – move focus to reliabilityreliability Need to Need to KEEP THINGS WORKING!KEEP THINGS WORKING! Schedule regular 3-day weekend runsSchedule regular 3-day weekend runs MAUS – already using “release” system of MAUS – already using “release” system of

changeschanges C&M C&M

need do same with the controls and monitoring codeneed do same with the controls and monitoring code Stabilize applicationsStabilize applications Development computer/system separate from Development computer/system separate from

iocpc1/iocpc2iocpc1/iocpc2 Coordination between C&M developers Coordination between C&M developers mustmust happen happen

DAQ – staged upgrades – especially with new DAQ – staged upgrades – especially with new equipmentequipment


MICE Step IVMICE Step IV Includes both Spectrometer Includes both Spectrometer

SolenoidsSolenoids Two trackers installed within the Two trackers installed within the

SS magnetsSS magnets One AFC (Focus Coil magnet & One AFC (Focus Coil magnet &

Absorber)Absorber)


Step IV Ops – New Step IV Ops – New Equipment…Equipment…

We will have much more in the We will have much more in the Hall!Hall!


Step IV – OperationsStep IV – Operations Continue:Continue:

Pre-run prepPre-run prep Further simplify runningFurther simplify running

Improve:Improve: Automate procedures Automate procedures fewer errors fewer errors Move from spreadsheet Move from spreadsheet only ConfigDB only ConfigDB Increase Online Reconstruction capabilitiesIncrease Online Reconstruction capabilities

Complete suite of detectors, Online analysis, Global Complete suite of detectors, Online analysis, Global recoreco

Reduce required number of shifters…Reduce required number of shifters…

New equipment arriving – plan for New equipment arriving – plan for commissioning, support & expertisecommissioning, support & expertise

Modify procedures to include SC magnets Modify procedures to include SC magnets and LH2 systemand LH2 system

Train shift personnelTrain shift personnel Develop plan to staff multitude of shiftsDevelop plan to staff multitude of shifts


Step IV – OperationsStep IV – Operations Safety Safety

Need understand implications under STFC rules of Need understand implications under STFC rules of new equipmentnew equipment

Made arrangements to meet with Jane Vickers (ISIS Made arrangements to meet with Jane Vickers (ISIS safety) to begin preparation/planningsafety) to begin preparation/planning

Need understand from experts what each system Need understand from experts what each system entails/requires for operationentails/requires for operation

PPSPPS Need understand implementation of system with SC Need understand implementation of system with SC

magnets and LH2 systemmagnets and LH2 system Hall AccessHall Access

Limited – not every day – scheduled access onlyLimited – not every day – scheduled access only Increase efficiency of data-takingIncrease efficiency of data-taking Hall search dropped Hall search dropped twicetwice in 3 days last weekend. in 3 days last weekend.

Not acceptable.Not acceptable. Experts must be availableExperts must be available


Operations OrganizationOperations Organization Overall Operations ManagerOverall Operations Manager

Local, familiar with STFC safety and operational Local, familiar with STFC safety and operational systemssystems

Provide oversight, link between MOMS, safety Provide oversight, link between MOMS, safety responsibility from project managerresponsibility from project manager

Monthly MOMsMonthly MOMs On call for 1 month – Daily running duties as nowOn call for 1 month – Daily running duties as now

MICE Shift personnelMICE Shift personnel Currently 2 shifters for ~8 hour shiftCurrently 2 shifters for ~8 hour shift Prefer to move to 1 MICE shifterPrefer to move to 1 MICE shifter

Current on-call experts – BLOC, SOC, TROCCurrent on-call experts – BLOC, SOC, TROC New system-specific experts – 3 per system – not New system-specific experts – 3 per system – not

necessarily current MICEnecessarily current MICE Magnet & cryo experts – maintenance of magnet Magnet & cryo experts – maintenance of magnet

systems, manage cryogens and vac pumping rigs & systems, manage cryogens and vac pumping rigs & instrumentation, cooling and powering expert, quench instrumentation, cooling and powering expert, quench behavior expertbehavior expert

LH2 expert – control engineering knowledge, safety LH2 expert – control engineering knowledge, safety proceduresprocedures


Step IV OperationsStep IV Operations Given current MICE schedule for Step Given current MICE schedule for Step

IV equipment and installation…IV equipment and installation… We We mustmust be prepared for 24/7 running be prepared for 24/7 running

in order to get Step IV data before ISIS in order to get Step IV data before ISIS shutdown in August 2014shutdown in August 2014

Can our systems handle this?Can our systems handle this? Target ok? Magnets ok?Target ok? Magnets ok? DAQ ok? Controls ok? Datamover ok?DAQ ok? Controls ok? Datamover ok? Shift coverage ok?Shift coverage ok?

Another reason to limit Hall access.Another reason to limit Hall access.

1 operations summary l. coney – ucr mice cm34 – oct 2012

Documents

daq daq

micenetconey cm34

parametersconey cm34

useconey cm34

trackersconey cm34

new systemconey cm34

tracker systemconey

coney ucrmice cm34