Cardiff University’s Condor Pool

Industrial PerspectivesJames Osborne

Contents

IntroductionApplication DiversityEnglish HeritageVelindre Cancer CentreFull Economic CostingCharging ModelsQuestions

Introduction

1 Central Manager RHEL 3800 Windows XP SP230 Submit Nodes Max (IP Address)200 Execute Nodes (Execute Always)600 Execute Nodes (Execute Idle)Condor Installed Via Application ObjectCondor 6.6.11 ~ 60% + 6.6.2 6.6.9 6.6.10

Application Diversity

Architecture Energy Plus

Biosciences DB, BLAST PAUP, Structure

Chemistry Dmarel

Computer Science Image Processing Radiotherapy

Engineering Travelling Salesman

History and Archaeology Artefacts Longbarrows

Application Diversity

Mathematics CFD of Polymers

Optometry Macromolecules

English Heritage

English Heritage

Collaboration with HISAR Alex Bayliss from English Heritage Very difficult to track her down

Artefacts ~ 300BC Prof John Hines

Longbarrows ~ 3700BC Prof Alasdair Whittle

Condorised Application

Oxcalc Developed by Christoper Ramsey Developed at University of Oxford We use version 3.10

Oxcalc interprets radiocarbon data using Bayesian statistics to date archaeological artefacts based on their carbon isotope ratios

Data Collection

Scientist takes a sample of the artefactThe sample is heated until it vaporisesThe vapour is then analysedThe ratio of C12 to C14 is measured

Artefacts

Artefacts – Sensitivity

English Heritage still collecting the dataData will be available in September 2006300 – 400 artefacts (jewellery, pottery, tools)1 hour per job on a P412 – 16 days on a single workstation (E)1 hour using the entire pool (E)

Longbarrows

Longbarrows - Stability

English Heritage have collected the data6 Longbarrows in Southern England80 – 100 models per Longbarrow1 hour per job on a P420 – 25 days on a single workstation (E)1 hour using the entire pool (E)~1.5 hours total (A)

The Future

English Heritage have already paid us EH + HISAR share an internal budget code Via TRF into our internal condor account Paid £500 for 25,000 CPU Hours Currently used < 10% of those CPU Hours EH work with other institutions EH intend to use us in the future with others

Velindre Cancer Centre

Velindre Cancer Centre

Collaboration with COMSC Mary Chin and Geraint Lewis from Velindre

Radiotherapy Prof David Walker Jon Giddy

Condorised Applications

BeamNRC Simulates x-rays emitted from radiotherapy

machines based on plan filesDosXYZNRC

Splits radiotherapy plan files (pre Condor) Joins BeamNRC results files (post Condor)

Radiotherapy Machines

Condorised Application

Radiotherapist provides plan file Radiotherapist runs Perl script Perl script calls DosXYZNRC to split plan file Perl script writes 18 submit files Radiotherapist runs condor_submit 18 beams, 90 pieces, 500,000 events 1620 jobs per patient per plan 1.5 hours per job on a P4 (A)

Condorised Application

3.5 months on a single workstation (E) 3 hours using the entire pool (E) ~12 hours total (A) ~10 Gigabytes of results Radiotherapist runs another Perl script Perl script calls DosXYZNRC to join results Radiotherapist evaluates the results

The Future

Velindre have agreed to paid us V + COMSC share an internal budget code Via TRF into our internal condor account Will pay £500 for 25,000 CPU Hours Currently used ~ 30% of those CPU Hours Future potential for 1400 patients / year 3.4 million hours using the entire pool (E) 6 months using the entire pool (E)

Full Economic Costing

Background

Full economic costing is a hot topic Some guidelines available from

EPSRC, HEFCE, HEFCW JISC…Some guidelines from other universitiesNo guidelines for HPC or HTCFunding bodies require FEC in new grantsFEC of Condor was a pre-emptive strike

Categories

BuildingsEquipmentMaintenancePowerStaff

FEC for OAWS

BuildingsEquipment

Execute and submit nodes and networkingMaintenance

Equipment warrantiesPowerStaff

FEC for CondorEquipment

Central managerMaintenance

Equipment warrantyPower

Additional power consumedStaff

Additional staff required

Equipment

Dell 1750Equipment cost £2240.37Racking cost £1000.00 pa

Rack space, UPS connection, 100mbit network connection, power, and air conditioning

Total cost £1560.10 pa Assuming a 4 year update cycle

Additional Power

Measure power consumption When Idle When running a job using max CPU When running a job using max DISK

Calculate additional power consumed Max Condor = (Max CPU – Idle) + (Max DISK – Idle)

Watts Up Pro

Measures Watts, Volts, Amps,

WattHrs, Cost, Avg Kwh, Mo Cost, Max Wts, Max Vlt, Max Amp, Min Wts, Min Vlt, Min Amp, Pwr Fct, Dty Cyc, Pwr Cyc

Freq – 1 second Duration – 15 minutes

Power ConsumptionWatt Hours Consumed - P3 1.0GHz 512MB

0

5

10

15

20

Idle Cpu Disk

Machine State

Wat

t Hou

rs

Watt Hours Consumed - P3 2.8GHz 512MB

0

5

10

15

20

25

Idle Cpu Disk

Machine State

Wat

t Hou

rs

Watt Hours Consumed - P4 1.8GHz 512MB

0

5

10

15

20

25

Idle Cpu Disk

Machine State

Wat

t Hou

rs

Watt Hours Consumed - P4 3.0GHz 512MB

0

10

20

30

40

Idle Cpu Disk

Machine State

Wat

t Hou

rs

Pool Power Consumption

Machines in inventory TotalMachines in class

Watt Hours consumed after 15 minutes IDLE

Watt Hours consumed after 15 minutes CPU

Watt Hours consumed after 15 minutes DISK

Watt Hours consumed after 15 minutes CONDOR

Watt Hours consumed after 60 minutes

Kilowatt Hours consumed after 60 minutes

Cost of electricity

Cost of machines in class per hour

WH WH WH WH kWH GBP/kWH GBPP3 500MHz 64MB 2P3 500MHz 128MB 11P3 733MHz 128MB 86P3 766MHz 128MB 22P3 800MHz 128MB 1P3 866MHz 128MB 7P3 933MHz 256MB 9P3 1.0GHz 256MB 40P4 1.7GHz 512MB 15P4 1.8GHz 512MB 4P4 2.2GHz 512MB 162P4 2.4GHz 512MB 2P4 2.66GHz 512MB 44P4 2.8GHz 512MB 178P4 3.0GHz 1024MB 319

TOTAL 902

COST OF CONDOR POOL PER CPU HOUR 4.82084

COST OF CONDOR PER CPU HOUR 0.00534461

178 9.8 14.3 11.6 6.3 25.2 0.0252 0.1 0.44856

19 12.4 21.6 14.5 11.3 45.2 0.0452 0.1 0.08588

0.1705 25.3 37.1 28.7 15.2 60.8 0.0608 4.2864

Additional Staff

Condor support engineer £ 24.26 phAdministrative assistant £ 21.08 ph

Charging Models

Internal Charging

Equipment £1560.10 pa Equipment cost £560.10 pa assuming a 4 year update cycle Racking cost £1000.00 pa Power Additional power consumption of Condor costs £0.005 per CPU per hour Staff

Condor Support Engineer +20% costs £29.11 per hour Administrative Assistant +20% costs £25.30 per hour

+5% for pay increase +15% for internal mark up

External ChargingEquipment £2260.10 pa Equipment cost £560.10 pa assuming a 4 year update cycle Racking cost £1000.00 pa + 70% costs £1700.00 pa Power

Additional power consumption of Condor costs £0.01 per CPU per hour Double that for the internal charging policy

Staff Condor Support Engineer costs £125.00 per hour + VAT + EXPENSES Standard consultancy fees Administrative Assistant +40% costs £29.51 per hour + VAT

+10% for pay increase +30% for external mark up Double that for the internal charging policy

Conclusions

Still early daysOnly two industrial partnersOnly one EH has actually paid usFull economic costing completeCharging policy 90% complete

Questions ?