20121017 openstack cern accelerating science

Accelerating Sciencewith OpenStack

Tim [email protected]

@noggin143

OpenStack Summit San Diego17th October 2012

mailto:[email protected]

What is CERN ?

OpenStack Summit October 2012 Tim Bell, CERN 2

• Conseil Européen pour la Recherche Nucléaire – aka European Laboratory for Particle Physics

• Between Geneva and the Jura mountains, straddling the Swiss-French border

• Founded in 1954 with an international treaty

• Our business is fundamental physics , what is the universe made of and how does it work


Answering fundamental questions…• How to explain particles have mass?

We have theories and accumulating experimental evidence.. Getting close…

• What is 96% of the universe made of ?We can only see 4% of its estimated mass!

• Why isn’t there anti-matterin the universe?

Nature should be symmetric…

• What was the state of matter justafter the « Big Bang » ?

Travelling back to the earliest instants ofthe universe would help…

Community collaboration on an international scale

Tim Bell, CERN 4OpenStack Summit October 2012

Tim Bell, CERN 5

The Large Hadron Collider

OpenStack Summit October 2012

6

The Large Hadron Collider (LHC) tunnel

OpenStack Summit October 2012 Tim Bell, CERN

Accumulating events in 2009-2011


Heavy Ion Collisions



Tier-1 (11 centres):•Permanent storage•Re-processing•Analysis

Tier-0 (CERN):•Data recording•Initial data reconstruction•Data distribution

Tier-2 (~200 centres):• Simulation• End-user analysis

• Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid• In a normal day, the grid provides 100,000 CPU days executing over 2 million jobs


• Data Centre by Numbers– Hardware installation & retirement

• ~7,000 hardware movements/year; ~1,800 disk failures/year

Xeon 51502%

Xeon 516010%

Xeon E5335

7%Xeon

E534514%

Xeon E5405

6%

Xeon E541016%

Xeon L5420

8%

Xeon L552033%

Xeon 3GHz4%

Fujitsu3%

Hitachi23% HP

0% Maxtor

0% Seagate15%

Western Digital

59%

Other0%

High Speed Routers(640 Mbps → 2.4 Tbps) 24

Ethernet Switches 350

10 Gbps ports 2,000

Switching Capacity 4.8 Tbps

1 Gbps ports 16,939

10 Gbps ports 558

Racks 828

Servers 11,728

Processors 15,694

Cores 64,238

HEPSpec06 482,507

Disks 64,109

Raw disk capacity (TiB) 63,289

Memory modules 56,014

Memory capacity (TiB) 158

RAID controllers 3,749

Tape Drives 160

Tape Cartridges 45,000

Tape slots 56,000

Tape Capacity (TiB) 73,000

IT Power Consumption 2,456 KW

Total Power Consumption 3,890 KW

Our Challenges - Data storage


• >20 years retention• 6GB/s average• 25GB/s peaks• 30PB/year to record


45,000 tapes holding 73PB of physics data

New data centre to expand capacity


• Data centre in Geneva at the limit of electrical capacity at 3.5MW

• New centre chosen in Budapest, Hungary

• Additional 2.7MW of usable power

• Hands off facility• Deploying from 2013

with 200Gbit/s network to CERN

Time to change strategy• Rationale

– Need to manage twice the servers as today– No increase in staff numbers– Tools becoming increasingly brittle and will not scale as-is

• Approach– CERN is no longer a special case for compute– Adopt an open source tool chain model– Our engineers rapidly iterate

• Evaluate solutions in the problem domain• Identify functional gaps and challenge them• Select first choice but be prepared to change in future

– Contribute new function back to the community


Building Blocks


Bamboo

Koji, Mock

AIMS/PXEForeman

Yum repoPulp

Puppet-DB

mcollective, yum

JIRA

Lemon /Hadoop

git

OpenStack Nova

Hardware database

Puppet

Active Directory /LDAP

Training and Support

• Buy the book rather than guru mentoring• Follow the mailing lists to learn• Newcomers are rapidly productive (and often know more than us)• Community and Enterprise support means we’re not on our own


Staff Motivation

• Skills valuable outside of CERN when an engineer’s contracts end


Prepare the move to the clouds• Improve operational efficiency

– Machine ordering, reception and testing– Hardware interventions with long running programs– Multiple operating system demand

• Improve resource efficiency– Exploit idle resources, especially waiting for disk and tape I/O– Highly variable load such as interactive or build machines

• Enable cloud architectures– Gradual migration to cloud interfaces and workflows

• Improve responsiveness– Self-Service with coffee break response time


Public Procurement Purchase ModelStep Time (Days) Elapsed (Days)

User expresses requirement 0

Market Survey prepared 15 15

Market Survey for possible vendors 30 45

Specifications prepared 15 60

Vendor responses 30 90

Test systems evaluated 30 120

Offers adjudicated 10 130

Finance committee 30 160

Hardware delivered 90 250

Burn in and acceptance 30 days typical 380 worst case

280

Total 280+ Days


Service Model


• Pets are given names like pussinboots.cern.ch

• They are unique, lovingly hand raised and cared for

• When they get ill, you nurse them back to health

• Cattle are given numbers like vm0042.cern.ch

• They are almost identical to other cattle• When they get ill, you get another one

• Future application architectures should use Cattle but Pets with strong configuration management are viable and still needed

Supporting the Pets with OpenStack

• Network– Interfacing with legacy site DNS and IP management– Ensuring Kerberos identity before VM start

• Puppet– Ease use of configuration management tools with our users– Exploit mcollective for orchestration/delegation

• External Block Storage– Currently using nova-volume with Gluster backing store

• Live migration to maximise availability– KVM live migration using Gluster– KVM and Hyper-V block migration


Current Status of OpenStack at CERN

• Working on an Essex code base from the EPEL repository– Excellent experience with the Fedora cloud-sig team– Cloud-init for contextualisation, oz for images with RHEL/Fedora

• Components– Current focus is on Nova with KVM and Hyper-V– Tests with Swift are ongoing but require significant experiment code

changes

• Pre-production facility with around 150 Hypervisors, with 2000 VMs integrated with CERN infrastructure, Puppet deployed and used for simulation of magnet placement using LHC@Home and batch


When communities combine…• OpenStack’s many components and options make

configuration complex out of the box• Puppet forge module from PuppetLabs does our configuration• The Foreman adds OpenStack provisioning for user kiosk to a

configured machine in 15 minutes


http://forge.puppetlabs.com/

Foreman to manage Puppetized VM


Active Directory Integration• CERN’s Active Directory

– Unified identity management across the site– 44,000 users– 29,000 groups– 200 arrivals/departures per month

• Full integration with Active Directory via LDAP– Uses the OpenLDAP backend with some particular configuration

settings– Aim for minimal changes to Active Directory– 7 patches submitted around hard coded values and additional filtering

• Now in use in our pre-production instance– Map project roles (admins, members) to groups– Documentation in the OpenStack wiki


Welcome Back Hyper-V!

• We currently use Hyper-V/System Centre for our server consolidation activities

– But need to scale to 100x current installation size

• Choice of hypervisors should be tactical– Performance– Compatibility/Support with integration components– Image migration from legacy environments

• CERN is working closely with the Hyper-V OpenStack team– Puppet to configure hypervisors on Windows– Most functions work well but further work on Console, Ceilometer, …


Opportunistic Clouds in online experiment farms

• The CERN experiments have farms of 1000s of Linux servers close to the detectors to filter the 1PByte/s down to 6GByte/s to be recorded to tape

• When the accelerator is not running, these machines are currently idle

– Accelerator has regular maintenance slots of several days– Long Shutdown due from March 2013-November 2014

• One of the experiments are deploying OpenStack on their farm– Simulation (low I/O, high CPU)– Analysis (high I/O, high CPU, high network)


Federated European Clouds

• Two significant European projects around Federated Clouds– European Grid Initiative Federated Cloud as a federation of grid sites providing

IaaS– HELiX Nebula European Union funded project to create a scientific cloud based

on commercial providers


EGI Federated Cloud Sites

CESGA CESNET INFN SARA

Cyfronet FZ Jülich SZTAKI IPHC

GRIF GRNET KTH Oxford

GWDG IGI TCD IN2P3

STFC

Federated Cloud Commonalities

• Basic building blocks– Each site gives an IaaS endpoint with an API

• OCCI? CDMI ? EC2 ? Libcloud ? Jclouds ?– Image stores available across the sites – Federated identity management based on X.509 certificates– Consolidation of accounting information to validate pledges and usage– Common security policies and computing rules

• Multiple cloud technologies in use– OpenStack– OpenNebula– Proprietary


Next Steps• Deploy into production at the start of 2013 with Folsom running the Grid

software on top of OpenStack IaaS• Support multi-site operations with 2nd data centre in Hungary• Exploit new functionality

– Ceilometer for metering– Bare metal for non-virtualised use cases such as high I/O servers– X.509 user certificate authentication– Load balancing as a service

Ramping to 15,000 hypervisors with 100,000 to 300,000 VMs by 2015


What are we missing (or haven’t found yet) ?

• Best practice documentation for– Monitoring and KPIs as part of core functionality– Guest disaster recovery solutions– Migration between versions of OpenStack

• Roles within multi-user projects– VM owner allowed to manage their own resources (start/stop/delete)– Project admins allowed to manage all resources– Other members should not have high rights over other members VMs

• Global quota management for non-elastic private cloud– Manage resource prioritisation and allocation centrally– Capacity management / utilisation for planning


Conclusions• Production at CERN in next few months on Folsom

– Our emphasis will shift to focus on stability– Integrate CERN legacy integrations via formal user exits– Work together with others on scaling improvements

• Community is key to shared success– Our problems are often resolved before we raise them– Packaging teams are producing reliable builds promptly

• CERN contributes and benefits– Thanks to everyone for their efforts and enthusiasm– Not just code but documentation, tests, blogs, …


References


CERN http://public.web.cern.ch/public/Scientific Linux http://www.scientificlinux.org/Worldwide LHC Computing Grid http://lcg.web.cern.ch/lcg/

http://rtm.hep.ph.ic.ac.uk/Jobs http://cern.ch/jobsDetailed Report on Agile Infrastructure http://cern.ch/go/N8wpHELiX Nebula http://helix-nebula.eu/EGI Cloud Taskforce https://wiki.egi.eu/wiki/Fedcloud-tf

http://public.web.cern.ch/public/

http://www.scientificlinux.org/

http://lcg.web.cern.ch/lcg/

http://rtm.hep.ph.ic.ac.uk/

http://cern.ch/jobs

http://cern.ch/go/N8wp

http://helix-nebula.eu/

https://wiki.egi.eu/wiki/Fedcloud-tf

Backup Slides


CERN’s tools

• The world’s most powerful accelerator: LHC– A 27 km long tunnel filled with high-tech instruments– Equipped with thousands of superconducting magnets– Accelerates particles to energies never before obtained– Produces particle collisions creating microscopic “big bangs”

• Very large sophisticated detectors– Four experiments each the size of a cathedral– Hundred million measurement channels each– Data acquisition systems treating Petabytes per second

• Top level computing to distribute and analyse the data– A Computing Grid linking ~200 computer centres around the globe– Sufficient computing power and storage to handle 25 Petabytes per

year, making them available to thousands of physicists for analysis


Our Infrastructure

• Hardware is generally based on commodity, white-box servers– Open tendering process based on SpecInt/CHF, CHF/Watt and GB/CHF– Compute nodes typically dual processor, 2GB per core– Bulk storage on 24x2TB disk storage-in-a-box with a RAID card

• Vast majority of servers run Scientific Linux, developed by Fermilab and CERN, based on Redhat Enterprise

– Focus is on stability in view of the number of centres on the WLCG


New architecture data flows


Virtualisation on SCVMM/Hyper-V

OpenStack Summit October 2012 Tim Bell, CERN 45Mar-

10

Apr-10

May-10

Jun-10Jul-1

0

Aug-10

Sep-10

Oct-10

Nov-10

Dec-10

Jan-11

Feb-11

Mar-11

Apr-11

May-11

Jun-11Jul-1

1

Aug-11

Sep-11

Oct-11

Nov-11

Dec-11

Jan-12

Feb-12

Mar-12

Apr-12

May-12

Jun-12Jul-1

2

Aug-12

Sep-12

Oct-12

0

500

1000

1500

2000

2500

3000

3500

WindowsLinux

Scaling up with Puppet and OpenStack

• Use LHC@Home based on BOINC for simulating magnetics guiding particles around the LHC

• Naturally, there is a puppet module puppet-boinc• 1000 VMs spun up to stress test the hypervisors with Puppet,

Foreman and OpenStack


http://lhcathome.web.cern.ch/LHCathome/

https://github.com/arusso23/puppet-boinc

https://github.com/arusso23/puppet-boinc

20121017 openstack cern accelerating science

Technology