part i – shifter duties part ii – acr environment part iii – run control & daq part iv –...
TRANSCRIPT
Shifter Instructions
Instructions for the SCT ACR Shifter
Version 0.4 – February 2010
ContentsPart I – Shifter Duties
Part II – ACR environment
Part III – Run Control & DAQ
Part IV – Beam
Part V – DCS
Part VI – Data Quality Monitoring
Part VII – Problems
2SCT Shifter Manual
Shifter Duties
Part I - Instructions for the SCT ACR Shifter
Before your shift
1. Make sure your training is up to date : courses Level 1, Level 2, Level 3 and Level 4A and SCT Shifter Qualification
2. Your access rights should be valid (ATL_CR) and SCT:shifter rights.
3. Read documentation on Twiki page : http://atlasop.cern.ch/twiki/bin/view/Main/AtlasOperation and go to shifter manual.
4SCT Shifter Manual
Beginning of your shift
1. Come to the ACR at least 15 minutes before your shift and consult with the previous shifter and check:
1. Current ATLAS conditions
2. Particular item under watch?
3. Where we stand respect to run plan?
4. LHC status and plan for the coming shift
2. Your access privileges will be granted automatically 2 hours before the beginning your shift
3. Check the run plan• http://atlasop.
cern.ch/atlas-point1/wmi/current/Run%20Status_wmi/AtlasRunCom.html
4. Read previous shifter e-log and start your new entry
5. Log in to the DCS, FSM Alarms and Run Com (Check-List Utility) 5SCT Shifter Manual
During your shift1. Do the checks requested in the Check-List
2. Monitor DCS alarms on the DCS Alarm Panel
3. Look for ERROR and FATAL messages at the DAQ coming from the SCT
4. Check Monitoring Histograms and DQM status flags
5. Check beam conditions are stable
6. Monitor Cooling status and alarms
7. Document in the e-log (SCT and IDGen) problems and happenings
8. In case of problems contact relevant experts and/or Run Coordinator
6SCT Shifter Manual
End of shift1. Submit e-log entries SCT and IDGen (log out of the e-log)
2. Inform the next shifter on the current situation of the running and about any special conditions or plans.
3. Log out of the Check-List, DCS and FSM Alarms
4. Do not leave if following shifter did not arrive (after 30 minutes call Run Coordinator if shifter has not arrived).
7SCT Shifter Manual
ACR environmen
t
Part II - Instructions for the SCT ACR Shifter
ACR Layout
SCT Shift DeskACR, CERN, Meyrin
Shift Leader DeskACR, CERN, Meyrin
9SCT Shifter Manual
ACR Desktop
TDAQ Screen DQM Screen DCS and Alarm Screens
1. Desks have 4 Screens to be used by the shifter
2. Get familiar with the environment
3. Some useful Utilities can be found in the General tab
4. Example of ScreenShot of the ACR Desktop
10SCT Shifter Manual
SCT Shifter Manual
RunCom1. In the General tab find
RunCom
2. Look for the SCT tab and log yourself in
3. Log out of the Check-List and FSM
4. Read the checks and comments, some of the requirements have an information page
5. DQ checks will appear every hour
Relevant for ATLAS runs but NOT for SCT Standalone 11
ATLAS e-log1. e-log can be found under the tab General
2. To be able to submit entries you have to login with your P1 login/password
3. Make entries if problems occur do not wait until the end of your shift
4. Do a Shift Summary (and select ID-Gen and SCT) for every shift
Remember to select all options and attach plots (if any) before submitting
12SCT Shifter Manual
ATLAS e-log Sample
13
Short summary…..
*******************************************15.00: start of shift
15.04: Recovered module Crate 22 Ch 4 which showed a trip error.
17.03: end of run 139303
17:05 Expert takes over DAQ to investigate XXX18:30 DAQ tests concluded. The result is that blah blah blah... and more blah, blah, blah… Changes have been applied, blah, blah, blah…
18.40: SCT on stand-by19.17: Start of Physics run 13933419.30: SCT on during LB 6.
20.00: Monitoring the number of errors, stable and around 20-30,see separate report from MON shifter.…21.12: Run 139336. Try a short cosmics run again(8 bins, 1000 triggers, timeout 60s)Result status -> no fail, but quite a few problematic modules?
… SCT Shifter Manual
SCT Shifter Manual
Atlas Operation1. Information for shifters will be posted in the Operations page.
• From outside P1 : http://atlasop.cern.ch/twiki/bin/view/Main/AtlasOperation
SCT Links
General Information
14
SCT Shifter Manual
SCT WhiteBoard1. Special announces, SCT DAQ Panel Configuration and SCT Phones
• From outside P1 : http://atlasop.cern.ch/twiki/bin/view/Main/SCTDetectorWhiteBoard
• From P1 : ID Menu SCT WhiteBoard
Important Announcements
Phones
15
SCT Shifter Manual
SCT Shifter Manual
1. Information for your shift• From outside P1 : http://atlasop.
cern.ch/twiki/bin/view/Main/SctShifterManual
• From P1 : Link from the Atlas Operation Twiki
16
pplications1. DAQ Panel go to TDAQ tab
2. FSM Alarms and DCS go to ID SCT tab
3. RunCom go to General tab
4. e-log go to General tab
5. konqueror go to General tab
6. ksnapshot go to General tab
17SCT Shifter Manual
Run Control & DAQ
Part III - Instructions for the SCT ACR Shifter
DAQ Panel
Log Window
Options
Applications
19SCT Shifter Manual
DAQ PanelSelect the setup script (/det/tdaq/scripts/setup_TDAQ.sh will always point to the production release)
1
Select the partition db file (i.e., /atlas/oks/tdaq-02-00-03/combined/partitions/ATLAS.data.xml)
2
Get the partition(s) described in db(the Part Name list will be automatically filled)
3
Tell the DAQPanel to acquire the information
4
20SCT Shifter Manual
DAQ Panel : Main1. Monitor Partition: Use it to
Spy on the ATLAS Partition status
2. Log Manager: Browse the log messages produced by the applications
3. MRS: It starts a viewer of the MRS (Message Reporting System) messages flowing through the system
4. Busy: It starts the Busy Presenter. Use it to understand whether/why the system is in a busy state
21SCT Shifter Manual
Monitor Partition GUI
22
RootController
CommandsTab Panels
MRS Log Window
Run Information & Settings
SCT Shifter Manual
MRS
23
Change subscription
Message format
Clone table
Start from DAQPanel – Main tab – MRS button
SCT Shifter Manual
Busy Monitor
24
1. No data flowing through the system? • The system may be in a BUSY state
2. Start the Busy Presenter from the DAQPanel: Main tab, BUSY• RED: the corresponding subsystem is busy• PINK: the corresponding subsystem is busy but out of the
PARTITION• GREY: the corresponding subsystem is not part of the run
If SCT is Busy and do not know what to do CALL DAQ EXPERT (162034) for instructions
SCT Shifter Manual
SCT Gui
25
1. DAQPanel in the SCT Desk has an additional tab called SCT 2. Use the button to launch the SctGui. Use the GUI to:
• Run Mode Physics scans• Alternatively check DCS conditions• Probe and check modules : if OUT of ATLAS running and you
have been instructed to do it
SCT Shifter Manual
SCT Gui : Short Tour
26
Data shows different options depending on the kind of display selected (Module Configuration in this case)Views: Geometry and ROD cratesList of: Barrels/Wheels or Crates in the configuration
Different module groups
Each of the squares is a module
SCT Shifter Manual
SCT Shifter Manual
SCT Gui : Short Tour
27
Example: SCT View
SIDE
Coordinates of the module under the mouse
0 – No Data1 – Bad Data2 – CLK/2a – Corrupt DataA – ABCD Error codeC – Config Readback modeE – Returning eventsJ – Junk DataO – Other
SCT Gui : Short Tour
28
You can access to the different menus using the toolbar or with a right click on the empty part of the screen
- Module Configuration: Change modules in the configuration- Probe Data: Check for modules with errors and recover them. Using this display you can perform actions like Power Cycle, Reset, Configure…- Data Manager: Allows you to access to results and plots from a test- Test Results: Show test results for all the modules- DCS Conditions: Several of the DCS parameters related with the modules are available through the SctGui (Temperature, LV, HV,...)
SCT Shifter Manual
SCT Gui : Short Tour
29
How to check the ROD which is BUSY?
• Right Click at the TIM Module and select TTC Status
- If Right Click on any of the Crates a menu will show three possibilities:- Clear Latch - Mask ROD Busy - Raise ROD error (manual stop-less removal) : Use this one if a
busy appears and is not picked up by the automatic removalOnly do this if DAQ EXPERT (162034) ask you to do it
TTC Status
Physics Mode Scan
30
- In SctGui : Tools Menu Physics Mode Scan - Need to be in Physics mode (standalone) or in ATLAS partition- Uses triggers to acquire events, number of events and crates can
be selected- Results are displayed in the SctGui – Test Results
Physics Mode
Remember…
31
I. Important to be specially vigilant at the beginning of the run. Problem most likely will happen at the point. Look for:a) FATAL / ERRORS from the SCT appearing in the MRSb) Check BUSY status (and TTC Status). If no triggers are coming
through one of the systems could be busyc) Stop-less removal could happen at this stage. Ask to restart
the run if:• 3 Barrel RODs or 7 RODs (Barrel + EndCap) get disabled
d) Always write Information on SctGui Run comment window (pop-ups at the start and end of the run)
e) Run a Physics Mode Scan and check for errors
II. During the run :a) Keep checking on disabled RODsb) Check ROS Monitoring every hour. Look for errors and
anomalous values in the resultsc) Continue to check for errors in MRS d) Document everything
SCT Shifter Manual
Beam
Part IV - Instructions for the SCT ACR Shifter
33
Also available on the WEB @http://atlasop.cern.ch/atlas-point1/operRef.php?subs=dcs/dcs/process.php?page=ATL_LHC
Beam Intensity From LHC (P4)
<#protons> / bunch
Beam IntensityFrom Beam Pick Up <#protons> / bunch
SCT Shifter Manual
ATLAS Stand-by Mode
34
• ATLAS should ALWAYS be in stand-by mode except when:
• STABLE BEAMS have been declared Physics Mode
• Beam is not possible due to LHC conditions LHC NO BEAM Mode
• In ALL OTHER situations ATLAS MUST be in stand-by mode
• "Stand-by Mode" is state of ATLAS hardware• it is decorrelated from the data aquisition state • it is possible to take data (not physics quality) with
ATLAS in stand-by
• SCT in Stand-by : LV ON and HV at 50 V
SCT Shifter Manual
Fill Pattern
35
Beam Injection: I
36
1. ATLAS Shift Leader receive LHC INJECTION WARNING via the LHC handshake panel. Big Sound coming from the DCS Desk!
2. ATLAS shift leader replies ATLAS INJECTION = PREPARE, within few seconds, otherwise phone call from CCC
3. ATLAS shift leader asks the shifters to prepare for for INJECTION:• Make sure that SCT is in Standby and Beam
Protection is ACTIVE• Check the INJECTION PERMIT mask. SCT
should NOT be MASKED
LHC Page
37
How to see if SCT is in STANDBY?
Masks
Link to the LHC Handshake page
SCT Beam Page
38
SCT Status : Injection Permit and
HV
Link to the LHC Handshake page
Beam Protection OFF / ON
SCT Beam Page
39
SCT Status : Injection Permit and
HV
Beam FlagProtection OFF /
ONLHC Status
Beam Injection : II
40
4. ATLAS Shift Leader turns the Injection Key5. Shift Leader uses hand-shake page and declares
ATLAS INJECTION READY6. LHC will inject Pilot bunch...
During Injection
41
• Monitor carefully beam losses (Backgrounds) o ATLAS BCM and BLM readingso LHC BLMs
• Monitor Beam parameters, position, ... o BCM, LUCID, MBTS, L1Calo rates, Tile
integrated currents
• Inform Shift Leader if some problem is found.
Instructions and limits will be set soon
ATLAS in Beam Mode
42
• Only with STABLE BEAM• All systems should have Low Voltages (Front end powered,
system can be read out)• All systems should have High Voltages / Bias Voltages, systems
are providing signals from particles
• SCT ON only with STABLE BEAM and GOOD BEAM Conditions (TBD)
Stable Running
43
• When STABLE BEAM Flag becomes TRUE. Shift Leader will ask systems to go ON
➔ Check STABLE BEAM Flag AND Beam BACKGROUNDS : 1. Inform SCT Run Coordinator and ask for approval2. HV can go ON3. Inform Shift Leader that system is ON
➔ Keep Checking Beam Conditions at ALL TIMES during your shift.
Removal of Permit
44
1. If the Beam Mode= STABLE BEAMS, Injection Permit will be removed AUTOMATICALLY.
2. If a Beam Dump is requested anywhere around the LHC, our BCM receives a post mortem request which removes the ATLAS Injection Permit.
• Without beam: If the LHC Page 1 indicates that the beam will be absent from LHC for a period of a few hours “Quite Time”.
SCT will be informed that some “Quiet Time” is available. SCT Run Coordinator should be informed by shifter and/or ATLAS Run Coordinator.
Beam Dump
45
1. Planned Dump :• LHC send an ADJUST/ DUMP warning via the hand-
shake• SCT will go to Standby automatically when BEAM
DUMP state goes to INMINENT• Shift leader ask shifters PIX, SCT, MUO, LArg to put
their systems in safe state. Send SCT to Standby if not already
2. Non planned Dump :• Send system to Standby if not already• Ask Shift Leader for an explanation and post in the e-
log as much information as you can gather. Do not wait up to the end of your shift to post it!
• Inform Run Coordinator and DAQ On-call expert (162034)
• If qualified perform the calibrations for a Non planned dump, if not wait for the On-call expert
Operation Flow-Chart
46
NO BEAMShort Interruption
INJECTION PERMIT=FALSE
PROTECTION ACTIVESTABLE BEAM=False
NO BEAMLong Interruption
INJECTION PERMIT=FALSE
PROTECTION OFFSTABLE BEAM=False
INJECTION
INJECTION PERMIT=True
PROTECTION ACTIVESTABLE BEAM=False
STABLE BEAM
INJECTION PERMIT=False
PROTECTION ACTIVESTABLE BEAM=True
RAMP/SQUEEZE/..
INJECTION PERMIT=True
PROTECTION ACTIVESTABLE BEAM=False
Injection handshake
warningInject and dump
cycleor lose the beam
(no dump)
LHC Page 1 announces NO BEAM
for > 8hours
NO
YES
1) SL notifies ARC and shifters in the ACR2) Shifters inform their SRC if instructed to3) SRC requests to ARC to override STABLE BEAM flag4) ARC checks LHC plan, gives permission for STABLE BEAM flag override5) SRC can override STABLE BEAM flag and notifies SL
SL Removes Injection Permit
Injection handshakewarning
1) SL notifies ARC2) Remove STABLE BEAM override
LHC Page 1: Beam <2hours
1) SL notifies ARC2) Remove STABLE BEAM override3) SL goes through Injection handshake
Beam
dum
p
ARC= ATLAS Run Coordinator.SRC= Sub system Run CoordinatorSL = Shift Leader
SL goes through Injection Handshake & brings ATLAS to STDBY, gives Injection Permit
Beam
dum
p
Acronyms
ATLAS in Beam Mode
47
1
2
3
4 5
6
7
8
LBDSBeam-1
LBDSBeam-2
MKIBeam-2CIBC
CIBC.R3
CIBC.L3
CIBC.L4
CIBC.R4 CIBC
.L5CIBC.R5
CIBC
CIBC.L6
CIBC.R6
CIBC.L7
CIBC.R7
CIBC.L8
CIBC.R8 CIBC
CIBC.R1
CIBC.L1
CIBC.R2
CIBC.L2
CIBC
MKIBeam-1
INJECTION PERMIT= is Injection into LHC Allowed?
If INJECTION PERMIT is FALSE:• Beam cannot be injected• Ongoing injection is STOPPED or PAUSED
But Beam is NOT dumped
Remember...
48
I. In NON SAFE situations SCT must be in Standby :o Single or Double beam operation during the commissioning
phaseo During beam injection and while STABLE BEAMS = False o During fills even if there is no beam in the machineo Standby : LV ON and HV 20 V
II. Special Configuration applies for Beam Commissioning :o Splash Configuration : LV ON, EndCaps at 20 V and Barrel
at 5 V (OUT of the data taking)
This is the SAFE configuration up to further notice
III. Only SCT Run Coordinator or someone designated by the Run Coordinator is allowed to give you permission to switch ON or change running conditions (Commissioning Configuration Standby Configuration)o ON Configuration : LV ON and HV 150 V
DCS
Part V - Instructions for the SCT ACR Shifter
FSM
50
State machine hierarchyDetector hardware and hierarchical structure represented by FSM entities
State & command propagationStates upwards (using programmed logic with flexible rules)Commands downwards
StatusError handling upwards
PartitioningAsynchronous operation of sub-trees possible
State/Status/Command propagation can be inhibitedAccess control
Alarm Screen
51
1. Active and non-acknowledged alarms2. Acknowledgement:
• Only for some alarms• Operator needs to acknowledge
to remove it3. For each alarm:
• Get details for experts• Mask alarm on UI level until
the alarm condition goes• Insert alarm to Elog with comment• Display trend plot of value• In dev: alarm help on TWiki
4. Summary alert:• Hides several individual alerts in a single alarm entry• Access to individual alerts via “Details”
Log in as yourself !
DCS Screen
52
Log in as yourself !
Beam Flags Summary
SCT Status
Beam Status
Alarms
FSM Modes
53
• Padlocks:No control, Free
No control, Included
Exclusive, Owner
Shared, Owner
Exclusive, NOT Owner
Shared, NOT Owner
No control, Locked Out
• Padlock background:Tree complete
Excluded children(partitionable trees)
Disabled children(non-partitionable )
• Control:No propagation, NO commands
Propagation, NO commands
Propagation (if Included), Commands by you
Propagation (if Included), Commands by all
Propagation (if Included), NO commands
Propagation (if Included), Commands by all
No propagation, NO commands
Tree complete
Excluded children(partitionable sub-trees)
Disabled children(non-partitionable sub-trees or devices)
FSI Monitor in FSM
54
ON/OFF flag shows when FSI laser is ON (light in SCT), in the Barrel and/or both End-caps. Note FSI laser is usually ON, even when FSI partition is not running.
Access under: SCT -> INFRASTRUCTURE -> FSI
History plots show peak and mean power in each SCT region:fibres 0 & 1 are Barrel, fibres 2 & 3 are both End-caps.
FSI Monitor in FSM
55
1. Persistent HV trips on certain SCT end-cap modules.• Very rarely, overpower of the laser may cause
persistent HV trips on certain end-cap modules. The watch list below shows which modules are expected to trip and in which order, if the laser power were to exceed upper safety levels.
• 4114, 5929, 7834, 7937,4231, 3743, 6731, 7926
2. If persistent HV trips are seen on the above modules, first check whether FSI is ON using the laser power monitor in the FSM (previous slide). If so call the FSI expert with a list of which modules have tripped and in which region (barrel or end-cap). The FSI expert will reduce the laser power to this region, so the modules can be recovered.
3. If the ID cooling is about to be stopped (e.g. imminent power intervention) or restarted, please notify the FSI expert who will ensure the alignment system is running to record any interesting thermal motions.
ROD Racks
56
Access to ROD
Crates
Power
Remember...
57
I. Use TOP Level of the Tree to send detector to ON (GOTO_READY) and Standby (GOTO_STANDBY)
II. Inform the Shift Leader when going to ON or STBY. On their side they should ask you to do the transition. BE AWARE of the beam STATUS.
III. Cooling loop 186 is OUT. This loop has a leak and can not be repaired without access to the detector volume.
IV. Do not try to take OUT modules of the configuration without expert advice.
V. Check that the FSM tree is complete. Nothing should be out of the Tree (grey). But be aware that a few modules will be out and one of the cooling loops.
VI. Document problems in the e-log
VII. Call the DCS expert (162035) if you get ALARMS. Investigate the WARNINGS.
Data Quality Monitoring
Part VI - Instructions for the SCT ACR Shifter
Check-List
59
DQ Check-List • Every hour the list will pop-up for you to fill • Read the comments and look to the documentation if have any question
Comment
Help
IF Problem found : • Check in the List of known problems and monitoring documentation• Call Monitoring expert (160037) if : plots stopped updating, monitoring processes crashed,... and you do not have instructions on how to act
Accessing Apps
60
Main Monitoring Applications. From the Main tab in the DAQ Panel :
• DQM Display : this application displays a sub-set of the histograms with tests applied to them.• OHP : a larger set of histograms is available in it. Results of the check of the DQM are also shown there.
OHP Configuration
Buttons to OHP and DQM
Main tab
OHP
61Online Histogram Presenter – OHP
If NULL monitoring could be not working or between updates
Colors show the result of the Checks from DQM
Links to different tabs
DQM Display
62
Data Quality Monitoring Display – DQMD• Colors : GREEN, YELLOW, RED show status of the checks• Grey : Out of the Configuration or Masked (Link Masking +
Stopless removal)
SCT Tree SCT tab
Bad Module - RED
Histogram Display
63
Displays the content of the Information Server (IS) • Histograms will be found under ATLAS partition• Providers are called : SCT-PT-X while SCT-PT will be the gathered information from all of them • SCT-PT is the important one for the DQM. If this is not updated DQM will not update• Contains all histograms created by Monitoring. Useful for further debugging
Mon Advanced tab Last Update (60 s)
Press to get last update
MDA
64
Browse results for previous runs• Directory structure is the same as the one in the Monitoring files and in IS• For a given run files are saved every X (60) Luminosity Blocks (LB). An additional file is saved at the end of the run EOR• Useful to check if things are new or known
Mon Advanced tab
Summary Histogram
65
Displays a general view of the status of the system• Bins show an average over the current run
> 3
5 h
its
> 7
0 %
err
or
rate
< 9
0 %
effi
cient
> 0
.00
1 N
O
OU
T +
Mask
ed O
nlin
e
Large Number of Entries (>100) on ANY of the bins is BAD!
Number of modules OUT of the configuration is around 30, rest is Masked which should be small.
During WARM START, Noise and Inefficient will be high.
Remember…
66
I. Look at the Status of the DQM, if it turns red for Barrel or End-Caps, something really bad happened, if no clear reason – call DAQ expert (162034)
II. Check DQM if several modules (around 50) go RED. Investigate the problem – call DAQ expert (162034)
III. Loop over the histograms in OHP especially:a) Summary histograms – Look at contents of each of the bins,
check for increases. • BAD! If an increase of more than 30
b) Error summaries – Some modules could show some errors but a low rate, check the scale.• BAD! If more than 30 modules with around 100% error rate
c) Noise Occupancy – Check for a lot of modules with high occupancy (red) or white modules. Few white modules are worse than few red ones. • BAD! White modules after enough statistics or lots of red
ones.IV. Document problems and read documentation during or previous to
your shift. If found a problem call expert.
Problems
Part VII - Instructions for the SCT ACR Shifter
Problems – I
68
I. SCT sets a BUSY:Stop-less removal will start and a window will appear in the Run Control Desk. If this does not happen you can use the TTC menu. Ask Shift Leader to restart a run if more than 3 Barrel RODs or 7 RODs (Barrel + EndCap) get disabled.
II. Frequency Veto: A bunch of messages like these or similar appear in MRS:
Something is not working properly in the Trigger. Ask the RunControl shifter to check it. Good practice would be set triggers to HOLD while investigating.
III. Monitoring stops updating:Easiest way is to restart the whole monitoring segment, but probably restarting the Gatherer would work.
Problems – II
69
I. Cooling goes OFF:Call the Cooling expert (162449) and the SCT Run Coordinator. Document the alarms and errors from the FSM. Follow the error documentation procedure described in:
Shifter Manual page Cooling Infrastructure Failures
II. Power Cut:Same as above.
III. ….