geneva 15.06.2010 cms week, rpc operation review anton dimitrov rpc detector review hardware status...

Geneva 15.06.2010 CMS Week, RPC Operation Review

Anton Dimitrov

RPC Detector RPC Detector ReviewReview

Hardware Status and New Hardware Status and New Technical ProceduresTechnical Procedures

Geneva 15.06.2010CMS Week, RPC Operation Review

Anton Dimitrov

CMS RPC EQUIPMENTCMS RPC EQUIPMENT

Detector and Service EQUIPMNETDetector and Service EQUIPMNET Known ProblemsKnown Problems ProceduresProcedures


Anton Dimitrov

DETECTOR AND SERVICE EQUIPMENT

BARREL ENDCAP TRIGGER SPARE

UXC

Spare

USC

Spare

UXC USC

Spare

UXC USC

Trigger

EndcapBarrel

EQUIPMENT ALREADY UPLOADED in EMDb

EQUIPMENT TO BE UPLOADED in EMDb – till end of June 2010


Anton Dimitrov

KNOWN PROBLEMS

Chambersin SG Slow Control Masked Rolls

Threshold Control TPH Sensors

PROBLEMS by TYPE

PROBLEMS by SEVERITY

1) Problems for long shutdown/CMS opening2) Problems for NEXT UXC ACCESS3) Reappearing Problems


Anton Dimitrov

Technical ProceduresTechnical Procedures How to deal with a new equipment How to deal with a new equipment

problems? problems? How to create a work package when How to create a work package when

access is announced?access is announced? How to replace equipment in UXC?How to replace equipment in UXC? How to replace equipment in USC?How to replace equipment in USC?

How to deal with an equipment problem

Operation Manager

P5 SHIFTER RPC Equipment Manager

HARDWAREEXPERT

elog Savannah

Sav

anna

hA

utom

atic

mai

l

Automatic mail

Sav

anna

h

ACCESS CMS RC&TC, GLIMOSCMS RC&TC, GLIMOSSLIMOS, RFM, CSLSLIMOS, RFM, CSL

Subdetector RFM(Operation ManagersOperation Managers)

WP

EMDB System Experts

EMDB Manager

RP Technicalshifter

BUFFERZONE

EMDBEQUIPMENT

UXC

AnnounceAccess

RP

Sw

eep

Access O

pen

Authorize

Req

uest

/Mod

ify

Req

uest

/Mod

ifyO

pen/

Clo

se

Ope

n/C

lose

ControlGive Access/Key

Change D

ata (Full System

Rights)

Supervise (Full R

ights)

Operational SpareOperational

Defective

Che

ck E

quip

men

t

Repair

Still Defective Tested

WasteBroken

Take after RP check

Use WP name to check in equipment


Anton Dimitrov

Work Package ListWork Package List## WP nameWP name DateDate DuratioDuratio

nnTask DescriptionTask Description probprob

lllocatiolocatio

nn11 RB LV board RB LV board

inspectioninspection14.06.114.06.1

001.1h1.1h W+1/S7 board replacedW+1/S7 board replaced 11 X2A11X2A11

22 RB HV testRB HV test 31.05.131.05.100

1.9d1.9d RPC-DT noise testRPC-DT noise test 11 X1 PPX1 PP

33 RE HV checkoutRE HV checkout 31.05.131.05.100

2d2d Investigate RE+1/2/21_22Investigate RE+1/2/21_22 11 X1 PPX1 PP

44 RE LV connect. RE LV connect. replacementreplacement

31.05.131.05.100

2.1h2.1h RE-3/2,3/25 Anderson RE-3/2,3/25 Anderson changedchanged

11 X3S51X3S51

55 RB LV connector RB LV connector checkcheck

31.05.131.05.100

2.2d2.2d W-1/S5/RB3-W-1/S5/RB3- 11 X4S11X4S11

66 RB LV module checkRB LV module check 20.05.120.05.100

1.1h1.1h W-1/S5/RB3-W-1/S5/RB3- 11 X4S11X4S11

77 RE LV module RE LV module replacementreplacement

20.05.120.05.100

1.1h1.1h Due to channel RE-Due to channel RE-3/2,3/253/2,3/25

11 X3S51X3S51

88 RPC LV connector RPC LV connector checkcheck

19.05.119.05.100

39min39min RE-3/2,3/25 & W-1/5/RB3-RE-3/2,3/25 & W-1/5/RB3- 22 X4S11, X4S11, X3S51X3S51

99 RB HV check outRB HV check out 04.05.104.05.100

2.1h2.1h W+2/8/RB2in & W+2/8/RB2in & W+1/7/RB3+W+1/7/RB3+

22 X1PPX1PP

1100

RPC HV system RPC HV system inspectioninspection

26.04.126.04.100

2.1d2.1d 4 spare RB cables for 4 spare RB cables for SVmaxSVmax

3 RE probl (2 3polar 3 RE probl (2 3polar changed)changed)

77 X1 PPX1 PP

1111

RB HV check outRB HV check out 08.04.108.04.100

5.2h5.2h Trip channel at W0 PPTrip channel at W0 PP 11 X2N39X2N39

1122

RB HV chain check RB HV chain check outout

16.03.116.03.100

1.9d1.9d 7 HV prob(W+2,W+1,W-7 HV prob(W+2,W+1,W-2,RE-1)2,RE-1)

77 X1 PPX1 PP


Anton Dimitrov

Work Packages by Work Packages by systemsystem

SystemSystem # WP# WP DurationDuration # Problems# Problems InsideInsideMTAMTA

OutsideOutsideMTAMTA

RB HVRB HV 55 6.2 days6.2 days 1313 1010 33RB LVRB LV 44 2.3 days2.3 days 22 11 11RE HVRE HV 33 6 days6 days 66 66 00RE LVRE LV 33 2.2 days2.2 days 11 11 00

RG RG TriggerTrigger

0 (2)0 (2) sharedshared 22 22 00

TotaTotall

1515 RB 8.5d RB 8.5d RE 8.2dRE 8.2d

2424 2020 44


Anton Dimitrov

BARREL HV BARREL HV INTERVENTIONSINTERVENTIONS

WheelWheel No. ProblemsNo. Problems No. No. Components Components

changedchangedRB+2RB+2 77 66RB+1RB+1 88 33RB0RB0 11 22RB-1RB-1 00 00RB-2RB-2 11 11

TOTALTOTAL 1717 1212


Anton Dimitrov

Barrel HV System: Summary Barrel HV System: Summary by Components II Trimesterby Components II Trimester

WheWheelel

HV HV BoardBoardA3512A3512

NN

Short red Short red cablecable

Brd -> YBrd -> Y

YY Long Long black black

cable US-cable US-UXUX

BeyoBeyond PPnd PP

SVMSVMax < ax < 96009600

i1 i1 settings settings > 10uA> 10uA

Rare Rare ProbleProble

mm

RB+RB+22 -- 11 33 22 -- -- -- --

RB+RB+11 11 -- 11 11 44 11 --

RB0RB0 -- -- 11 11 -- 11 -- --RB-1RB-1 -- -- -- -- -- -- -- --RB-2RB-2 11 -- -- -- -- -- -- --TotTotalal

22 11 44 44 11 55 11 00


Anton Dimitrov

Barrel HV System Status II Barrel HV System Status II trimestertrimester

Broken Y in Distributor Boxes: 6/480; ~ 1.2% Broken Y in Distributor Boxes: 6/480; ~ 1.2% (0.6%*)(0.6%*)

Broken long black cables (USC – UXC): 11 Broken long black cables (USC – UXC): 11 (7*)(7*) Spare black cables used 17/480; ~ 3.5% Spare black cables used 17/480; ~ 3.5% (2.7%*)(2.7%*) 473** RPC in DG mode 473/480; ~ 98.54% 473** RPC in DG mode 473/480; ~ 98.54%

(98.75%*)(98.75%*) 5 RPC in SG mode 10/1020; ~ 1% 5 RPC in SG mode 10/1020; ~ 1% (1%*)(1%*) 1 RPC totally dead 1/480; ~ 0.21% 1 RPC totally dead 1/480; ~ 0.21% (0.21%*)(0.21%*)N.B. N.B. ** => => Barrel HV system status for I trimester Barrel HV system status for I trimester

20102010**** => => 1 RPC OFF1 RPC OFF due to LV problem (next slide)due to LV problem (next slide)


Anton Dimitrov

Barrel Low Voltage/Slow Barrel Low Voltage/Slow Control Status II TrimesterControl Status II Trimester1 RPC LV OFF 479/480 ~ 1 RPC LV OFF 479/480 ~ 0.21%0.21% (0%*)(0%*)18 RPC I2C lines (in 17 chambers) not 18 RPC I2C lines (in 17 chambers) not working in the Barrel 18/800 ~ working in the Barrel 18/800 ~ 2.25% 2.25% (2.13%*)(2.13%*)Redundant DT lines working for 8/18 ~ Redundant DT lines working for 8/18 ~ 44%44% (9/17~53%*)(9/17~53%*)10/800 I2C lines in the Barrel not 10/800 I2C lines in the Barrel not controlled at all ~ controlled at all ~ 1.25% 1.25% (8/800 ~ 1%*)(8/800 ~ 1%*) Vth or Vmon values not settable on chip Vth or Vmon values not settable on chip basis in 4 cases in the Barrel basis in 4 cases in the Barrel (no change*)(no change*)


Anton Dimitrov

Barrel Threshold Control Barrel Threshold Control HistoryHistory

DateDate ProblematiProblematic FEB/chipc FEB/chip

No THR No THR ControlControl

DT line DT line WorkingWorking

No No redundantredundant

ControlControl22.01.20022.01.200

88 66 1212 77 5521.05.20021.05.200

99 88 1313 88 5523.11.20023.11.200

99 66 1414 88 6616.03.20116.03.201

00 44 1717 99 8815.06.20115.06.201

00 44 1818 88 1010


Anton Dimitrov

RE+2/2, 3/11- TrippingRE+2/2, 3/11- Tripping: Problem localized in HV distributor so fixed it by using : Problem localized in HV distributor so fixed it by using

the spare channel. (On April08)the spare channel. (On April08) RE-1/3/31, 32- SpikesRE-1/3/31, 32- Spikes: Problem localized in RE-1/3/31 (TOP) so disconnect this : Problem localized in RE-1/3/31 (TOP) so disconnect this

gap and run rest of three gaps. (On April08)gap and run rest of three gaps. (On April08) RE-1/3/05, 06-TrippingRE-1/3/05, 06-Tripping: Localized ~ at PP so fixed it by changing the tripolar : Localized ~ at PP so fixed it by changing the tripolar

connector in UXC. (Reported on April10)connector in UXC. (Reported on April10) RE-3/2, 3/26-TrippingRE-3/2, 3/26-Tripping - Fixed by changing connector in UXC at PP. (On April 28)- Fixed by changing connector in UXC at PP. (On April 28) RE-1/3/35, 36-TrippingRE-1/3/35, 36-Tripping – Actually problem was in RE-1/2/01,02 because channels – Actually problem was in RE-1/2/01,02 because channels

were swapped. Put back the channels at original position. Problem do not were swapped. Put back the channels at original position. Problem do not reproduce. (On April 28 reproduce. (On April 28

RE+1/2/21, 22- TrippingRE+1/2/21, 22- Tripping – During TA31052010 switch on back this channel so it – During TA31052010 switch on back this channel so it start working without any intervention and it is working up to now.(Reported on start working without any intervention and it is working up to now.(Reported on May06)May06)

RE-2/2,3/11-TrippingRE-2/2,3/11-Tripping- During TA31052010 swapped the channel with RE-- During TA31052010 swapped the channel with RE-2/2,3/12 both are working without any problem. (Reported on June 02).2/2,3/12 both are working without any problem. (Reported on June 02).

During last 3 months total 7 (6 trpping+1spikes) HV problems were reported

Saleh Muhammad NCP Islamabad

Endcap HV Interventions and StatusEndcap HV Interventions and Status


Anton Dimitrov

Endcap LV Interventions and StatusEndcap LV Interventions and Status

RE+2/2, 3/31- Tripped (LV FEB)RE+2/2, 3/31- Tripped (LV FEB) Fixed the problem by changing the problematic Fixed the problem by changing the problematic module with good spare onemodule with good spare one. (. (On March 11).On March 11).

RE-3/2, 3/25 –Tripped (LV FEB)RE-3/2, 3/25 –Tripped (LV FEB) Localized the problem in the module channels. Localized the problem in the module channels. Fixed the problem by changing the module also by repairing the connector which Fixed the problem by changing the module also by repairing the connector which break during changing the module . (Reported on May19)break during changing the module . (Reported on May19)

LV modules serial no.47LV modules serial no.47 which was replaced form P5 test it in bld.904 testing setup. which was replaced form P5 test it in bld.904 testing setup. The board did not give the problem again so put it back into system at P5 on May31. The board did not give the problem again so put it back into system at P5 on May31.

On 1On 1stst of April the X2S32 rack (YE-1 far side, X2 level) all electronics become off due of April the X2S32 rack (YE-1 far side, X2 level) all electronics become off due to rising the temperature of MAO at 29C and LV boards at 40C. This happen due to to rising the temperature of MAO at 29C and LV boards at 40C. This happen due to power failure of the turbine. But after 4 hours power came back and just switching on power failure of the turbine. But after 4 hours power came back and just switching on the MAO and boards everything start working. the MAO and boards everything start working.



Anton Dimitrov

Endcap Summary (RPC TCM)Endcap Summary (RPC TCM) SystemsSystems HVHV LVLV Slow controlSlow control Gas Gas

RPC TCM RPC TCM New New pending pending Problem Problem

Spare Spare cables cables in usein use

DB spare DB spare ch. in use ch. in use

Single Single gap gap

RPCsRPCs

Full Full RPC RPC outout

New New pending pending problemproblem

Full Full RPCs RPCs

outout

TH TH setting setting

ProblemProblempendingpending

T and RH T and RH sensor sensor

problem problem pendingpending

Known big leaks Known big leaks channelschannels

April26, 2010April26, 2010 33 00 11 1818 11 00 11 77 00 22

May 03, 2010May 03, 2010 00 11 11 1717 11 00 11 77 00 22May 10, 2010May 10, 2010 11 00 11 1717 11 00 11 77 00 22May 17, 2010May 17, 2010 11 00 11 1717 11 00 11 77 00 22May 23,2010May 23,2010 11 00 11 1717 11 11 11 77 00 22June 02,2010June 02,2010 00 00 11 1717 11 00 11 77 00 22June 07, 2010June 07, 2010 00 00 11 1717 11 00 11 77 00 22June 14, 2010June 14, 2010 00 00 11 1717 11 00 11 77 00 22



Anton Dimitrov

Trigger USC electronicsTrigger USC electronics CAEN VME controller failedCAEN VME controller failed in one Trigger in one Trigger

Crate on night 18/19 April 2010. Crate on night 18/19 April 2010. Replaced by a spare on 19 April. It was Replaced by a spare on 19 April. It was checked that only mezzanine board with optics checked that only mezzanine board with optics was damaged (most probably optical was damaged (most probably optical transmitter).transmitter).

At the moment no other At the moment no other problems - 100% of the system problems - 100% of the system functional!!!functional!!!

Karol Buńkowski, CERN, UW


Anton Dimitrov

Trigger Trigger Link System Link System ~10 March: one ~10 March: one Half Box tripped during normal operationHalf Box tripped during normal operation (not (not

during power on), first we have replaced the LV module, but it did during power on), first we have replaced the LV module, but it did not helped, so then we have replaced all boards from that Half Box not helped, so then we have replaced all boards from that Half Box ((CBCB, , 22x x MLBMLB,,44xx SLB SLB), and this helped. Test showed that:), and this helped. Test showed that: CB Actel damagedCB Actel damaged, the CB itself is OK,, the CB itself is OK, GOLs on both MLB damagedGOLs on both MLB damaged,, SLBs are OK,SLBs are OK,

RB+1 far (24 April 2010)RB+1 far (24 April 2010) - - pproblems with the CCU ringroblems with the CCU ring:: FEC mezzanine was not well attached to the CCU board.FEC mezzanine was not well attached to the CCU board. When it was corrected CCU errors appeared again. So the FEC When it was corrected CCU errors appeared again. So the FEC

mezzanine was swapped with other one. After that both FECs were mezzanine was swapped with other one. After that both FECs were working correctly. Possible reason: working correctly. Possible reason: FEC and the ribbon fiber FEC and the ribbon fiber connectors “not compatible”???connectors “not compatible”???

RB+2RB+2__near near ((31 May 31 May 2010) - 2010) - the CCU errors become very the CCU errors become very frequentfrequent,, ~every second (before that ring had a few CCU errors ~every second (before that ring had a few CCU errors per day)per day), , The ring B for that tower was disconnected, after that no CCU errors.The ring B for that tower was disconnected, after that no CCU errors.

CCU errors appear only on one-two rings, with low rate. In most of CCU errors appear only on one-two rings, with low rate. In most of the cases are recovered by the reset_fec automatically executed by the cases are recovered by the reset_fec automatically executed by the online software. the online software.

The system is working stable - 100% of the system functionalThe system is working stable - 100% of the system functional



Anton Dimitrov

Trigger FirmwareTrigger Firmware 22 April - CB firmware updated - 22 April - CB firmware updated -

problem with the LB firmware problem with the LB firmware loading from FLASH correctedloading from FLASH corrected

Karol Buńkowski, CERN, UW 14 June 2010


Anton Dimitrov

Trigger Online Software Trigger Online Software StatusStatusThe Trigger Supervisor control over the full system (Link The Trigger Supervisor control over the full system (Link

System, Trigger System, RBC/TTU) is working fine.System, Trigger System, RBC/TTU) is working fine. Latest improvements:Latest improvements:

Monitoring panels in TSMonitoring panels in TS(the data from the XDAQs to the TS are send with use of the flash lists)(the data from the XDAQs to the TS are send with use of the flash lists)

Panel in the TS LBox Worker:Panel in the TS LBox Worker: Disabling the LBox XDAQs from the TS controlDisabling the LBox XDAQs from the TS control Loading the LB firmwareLoading the LB firmware Presenting the LBox XDAQs statusPresenting the LBox XDAQs status

Panel in the TS Cell:Panel in the TS Cell: disabling the TC XDAQs from the TS controldisabling the TC XDAQs from the TS control Presenting the TC XDAQs statusPresenting the TC XDAQs status

Alarms in the HOTSOPTAlarms in the HOTSOPT GUI for disabling the Sorter Crate inputs (in case of the Trigger Crate GUI for disabling the Sorter Crate inputs (in case of the Trigger Crate

malfunction)malfunction) To do next:To do next:

Better management of the configuration data (different local config keys Better management of the configuration data (different local config keys for different parts of the system),for different parts of the system),

FEB configuration from the DBFEB configuration from the DB Storing monitoring data in the condition databaseStoring monitoring data in the condition database


Ian Crotty 14 June 2010

RPC CoolingRPC Cooling

Two main changes have occurred this year• Regulation Valve change on the “Endcap cooling circuit

Week 17 2010

• The changes of set Point and regulation Parameters sinceWeek 20 2010

Continuing ConcernsLow contingency

Temps still high in EndcapsTemp gradients along Z in RE


RB Temperature Log since April 2010RB Temperature Log since April 2010


RE Temperature Log since April 2010RE Temperature Log since April 2010


Endcap Circuit Set-point Changed to Endcap Circuit Set-point Changed to 16.5°C16.5°C


X2 Power Cut & New Set-point Change X2 Power Cut & New Set-point Change


RPC Temperature Distributions RPC Temperature Distributions

+Z -ZRB020

.33C

19.9

0C

19.7

9C

20.2

5C

20.2

5C

18.6

7C

19.4

1C

20.8

3C

21.3

3C

20.3

2C

21.9

2C

0.5

C 0.8

C 0.6

C0.7

C0.8

C0.75 C

Gains since April 2010


Anton Dimitrov

Gas StatusGas Status New gas leak in WP2_S05_RB2 notified to CERN New gas leak in WP2_S05_RB2 notified to CERN

gas group. gas group. W0 readout board replaced (middle of March). No W0 readout board replaced (middle of March). No

effect on RPC operationeffect on RPC operation Warning due to purifiers fixed.Warning due to purifiers fixed. New Flow cell threshold settings for warnings and New Flow cell threshold settings for warnings and

alarms uploaded into DB. This did not solve the alarms uploaded into DB. This did not solve the sporadic warnings due to spikes on cells readout. sporadic warnings due to spikes on cells readout. For this is needed a new approach in the warning For this is needed a new approach in the warning treatment to be implemented into RPC-DCS. treatment to be implemented into RPC-DCS. Discussion on this is ongoing with G.Polese. Discussion on this is ongoing with G.Polese.

L. Benussi


Anton Dimitrov

Gas Flow cell Reading Gas Flow cell Reading fluctuationsfluctuations

L. Benussi


Anton Dimitrov

ConclusionsConclusions Partitions that cannot be threshold controlled are Partitions that cannot be threshold controlled are

increasing in time, leading to the dilemma increasing in time, leading to the dilemma “noise “noise vs. masking”!vs. masking”!

Apart from that Detector Performance is stable!Apart from that Detector Performance is stable! For better Detector performance threshold For better Detector performance threshold

refinement is required => DB is highly needed!refinement is required => DB is highly needed! Temperatures in Endcap still higher than 20°C, but Temperatures in Endcap still higher than 20°C, but

decreasing after the change of coolant set-point. decreasing after the change of coolant set-point. Signals of after pulsing raised again the question of Signals of after pulsing raised again the question of

individual chamber HV working point definition.individual chamber HV working point definition. Still lot of effort to be put in order to update all the Still lot of effort to be put in order to update all the

instructions! instructions!

geneva 15.06.2010 cms week, rpc operation review anton dimitrov rpc detector review hardware status...

Documents