Online System Status

LHCb WeekBeat Jost / Cern

9 June 2015

Beat Jost, Cern

Hardware and infrastructure work

❏ LS1 is over❏ A lot of work has been performed during the last

~30 months➢ All control PCs have been replaced/virtualized➢ All PCs upgraded to new OS (W2008S/W7/SLC6)➢ Configuration Management changed from Quattor to

Puppet➢ Controls infrastructure upgraded to new version of PVSS

❏ Much hardware replaced➢ ~600 farm nodes extracted➢ ~800 new farm nodes installed (90% cabled and in

production)➢ ~25 SPECS masters replaced by new hardware➢ ~30 CAN control PCs replaced by Credit-card PCs

2LHCb Week, 9 June 2015

Beat Jost, Cern

Software

❏ Main challenge was to support the Split HLT➢ Whole control of the farm had to be adapted/redone

➥ Concurrent running of HLT1 and HLT2➥ New task architecture in the farm nodes➥ New management of conditions

❏ Completely new is the online Alignment and Calibration to provide the HLT with the best Alignment and Calibration constants➢ HLT1 selected/enriched data for alignment tasks on local disks➢ Alignment runs on all farm nodes (~10 minutes)➢ Calibration for Rich and OT done online using either output of online

reconstruction or monitoring data coming from HLT1.➢ Calibration constants available typically within seconds after change

of run

❏ To synchronize and sequence all these all these activities automatically a new Scheduler has been developed➢ For the moment the activities are performed manually, though

3LHCb Week, 9 June 2015

Beat Jost, Cern

Software II

❏ Completely new system for HLT2 and Data Quality monitoring➢ As HLT2 runs asynchronously and on different runs in

different nodes, the normal Data monitoring cannot be used

➥ New system, based on HLT2 output files, has been devised and implemented

➥ Same for the final Data Quality monitoring, running Offline Brunel

➢ Dry tested, but not yet in production

4LHCb Week, 9 June 2015

Beat Jost, Cern

Where are we?

❏ All collision data have been taken without major problems

❏ Most of the data are still spinning around on the local disks

❏ Velo and Tracker alignment works❏ So far, all actions are performed manually❏ For a few runs, HLT2 has been run and files

transferred to offline for test processing➢ Some minor problems transferring the conditions to the

offline database, will be fixed today or tomorrow

5LHCb Week, 9 June 2015

Beat Jost, Cern

Summary

❏ After all the changes and new development the system is in very good shape➢ Thanks to extensive testing on the way, especially on

data saved end of 2012

❏ Operationally we still have to learn a lot to run the system in a mainly automatic fashion

Many thanks to all involved

6LHCb Week, 9 June 2015