IPv6 pre-GDB (10 Jun 2014)

David Kelsey (STFC-RAL)WLCG GDB, CERN

11 Jun 2014

Aims of pre-GDB IPv6

• Background– CERN is running out of IPv4 network addresses– Other resources/sites likely to follow soon(ish)– WLCG needs all outward facing services to be run on dual-stack

IPv4/IPv6 systems to support IPv6-only clients• When and which services not yet decided

• Aims of IPv6 workshop:– Present the recent work HEPiX IPv6 working group/WLCG task force– Provide guidance on the configuration and management of systems,

sites and services– Encourage others to deploy IPv6 to join the testing activities– Future plans (experiments and working group)

10/06/2014 WLCG IPv6 2

Pre-GDB IPv6 agenda

• https://indico.cern.ch/event/313194/• ~30 people in person & ~20 on Vidyo• IPv6 technical background

– Edoardo Martelli & Francesco Prelz

• IPv6 at CERN– Edoardo Martelli

• File transfer testing – the IPv6 testbed activities– Tony Wildish

• Status and configuration of some services– Many speakers looking at different storage and batch solutions

10/06/2014 WLCG IPv6 3

Agenda (2)

• Experiment testing and plans– Alastair Dewhurst (ATLAS, CMS and LHCb) & Costin Grigoras (ALICE)

• Monitoring– Duncan Rand (PerfSONAR) & Marek Elias (NAGIOS)

• An IPv6 quiz!• Site status and experiences

– The WLCG IPv6 site-readiness survey (DPK)– Chris Walker (UK), Marek Elias (FZU) & Oksana Shadura (BITP, ALICE))

• Next steps – discussion• All slides available on Indico agenda page

11/06/2014 WLCG IPv6 4

Services & IPv6

• WLCG Software IPv6 database– http://hepix-ipv6.web.cern.ch/wlcg-applications

• Storage and data transfer (lots of good news re IPv6)– DPM (Sam Skipsey)– dCache (Ulf Tigerstedt)– StoRM (Chris Walker)– FTS3 (Michail Salichos)– XRootD (Lukasz Janyst)

• Batch systems (Francesco Prelz)– Still lots of IPv6 problems but some good news

11/06/2014 WLCG IPv6 5

Survey of all WLCG Tier 0/1/2• Announced on 28 May 2014• We asked all sites to respond before 6th June• Questions

– Is your site already offering connectivity, routing and naming services for IPv6 end systems?

– If so, have you already enabled IPv6 on some of the services you manage?– If not, are there plans for this? If so, what are the timelines?– Does your site currently have problems with allocating a sufficient number of

IPv4 addresses? Or foreseen in the near future? – Other work, other comments

• Complete wiki table at

https://www.gridpp.ac.uk/wiki/2014_IPv6_WLCG_Site_Survey

10/06/2014 WLCG IPv6 6

Results to date

• Many thanks to those replying quickly• VERY much an initial analysis (errors are mine)• The table is live– Please add new lines or modify your answer when the

situation changes (just change the “date” field)

• We had responses from CERN (Tier 0) and– 11 Tier 1 sites (2 missing)– 85 Tier 2 sites (~70 missing)

• We will send another reminder– Then start to issue tickets against sites

10/06/2014 WLCG IPv6 7

Tier 0 and 11 Tier 1 sites

• IPv6 connectivity?– 1 yes (CERN), 7 partial, 4 not yet

• Enabled services?– Limited, mainly DNS, web, email etc.

• When will you be ready?– 1 now (CERN), 6 within 1 year, 5 not defined

• Lack of IPv4 addresses?– Just 1 (CERN)

10/06/2014 WLCG IPv6 8

Tier 2 sites (85 of them)

• IPv6 connectivity?– 16 yes, 9 partial, 60 not yet

• Enabled services?– Some DNS, web, email etc, but ~13 sites have deployed more widely

• Some test-bed, some production

• When will you be ready? (for those with no current IPv6)– 8 within 1 year, 1 within 2 years, 6 are planning (no timetable)– ~ 45 have no plans or timetable is unknwon

• Lack of IPv4 addresses?– 5 sites now, many say OK now but problems in a few years– Many sites note current use of private IPv4 for worker nodes

10/06/2014 WLCG IPv6 9

11/06/2014 WLCG IPv6 10

11/06/2014 WLCG IPv6 11

11/06/2014 WLCG IPv6 12

ALICE GRID SERVICES IPV6 READINESScostin.grigoras@cern.ch

CENTRAL SERVICES STATUS

All servers were assigned IPv6 addresses 1y ago ~4 months ago the DNS was switched to the

production name and the firewall was opened for all (>50) servers

DNS load balancing publishes aliases with both the IPv4 and the IPv6 of the respective servers

No problems detected, but most services are not contacted via IPv6

Requests from 86 IPv6 networks to alimonitor.cern.ch since then (3% of requests) 14

WLC

G p

re-G

DB

- IPv6

Work

shop, 2

01

4-0

6-

10

$ host alice-ldap.cern.chalice-ldap.cern.ch has address 137.138.99.165alice-ldap.cern.ch has address 137.138.99.166alice-ldap.cern.ch has IPv6 address 2001:1458:201:b49f::100:falice-ldap.cern.ch has IPv6 address 2001:1458:201:b49f::100:10

SITE DEPLOYMENT

Still only 4 sites have deployed IPv6 on the VoBoxes BITP, CSC, DCSC_KU, NECTEC BITP has fully enabled it site-wide More sites seem to have it deployed for users

Not interfering with the production Minor issues ironed out, usually related to

firewall configuration Sites are encouraged to deploy it as soon as

possible

15

WLC

G p

re-G

DB

- IPv6

Work

shop, 2

01

4-0

6-

10

IPV6-ONLY WN AT CERN DEPENDS ON

Upgrade Perl in AliEn Switch to Xrootd 4.0.0+ for client and central

services Either switch EOS as well to Xrootd 4.0.0+ or

access data via IPv4 local addresses Hopefully more sites will offer SE access on

IPv6 by then

16

WLC

G p

re-G

DB

- IPv6

Work

shop, 2

01

4-0

6-

10

11/06/2014 WLCG IPv6 17

11/06/2014 WLCG IPv6 18

11/06/2014 WLCG IPv6 19

11/06/2014 WLCG IPv6 20

Future plans

11/06/2014 WLCG IPv6 21

Next testing

• Continue the file-transfer testbed/mesh– Useful way of gaining experience– GridFTP: Would like all Tier 1s involved– GridFTP: Tier 2s (Open invitation to join)

• FTS3 pilot and IPv6– Taken forward by FTS3 pilot, experiments, Ops

• IPv6/PhEDEx: More storage options to be tested (at a few sites)

11/06/2014 WLCG IPv6 22

Experiments

• ALICE moving to dual-stack and testing all– XrootD V4 required

• ATLAS– Imperial to test BigPanda instance– Panda Dev instance at CERN to be dual-stack

• CMS– Test dual-stack glideinWMS– Dashboard monitoring– Test CRAB3

11/06/2014 WLCG IPv6 23

Experiments (2)

• LHCb job submission– All DIRAC internal communication is authenticated– All communication needs to be tested• on separate VMs, starting June 2014

– Tests started on lxplus-ipv6 (dual-stack)• Works for IPv4• Next – tests on IPv6-only

11/06/2014 WLCG IPv6 24

Data Management tests

• ATLAS, CMS and LHCb will all use FTS3– GridFTP, XrootD and http

• ATLAS already doing IPv4 stress tests of Rucio– Could then do IPv6 stress testing

• CMS– Test a few endpoints of each storage technology– Next – test AAA with IPv6• Nebraska will enable IPv6 access to their XrootD• Requires update to new XrootD client

11/06/2014 WLCG IPv6 25

IPV6 Network Monitoring

• PerfSONAR monitoring• As we turn on production dual-stack– We need to collect monitoring data

• A good way for a new site to get started– Ask Imperial to add to HEPiX IPv6 dashboard

• Encourage this for sites not in the IPv6 group• NAGIOS and other monitoring– Deploy new plugins to test IPv6 services

11/06/2014 WLCG IPv6 26

Dual-stack on production

• Several sites successfully moved to dual stack• Drive according to Experiment use-cases– Agreed sites and services

• Over coming months– More volunteer sites will do this– As part of working group– In a controlled manor with experiment agreement

• Other sites welcome to join WG to do this

11/06/2014 WLCG IPv6 27

Other issues

• We cannot wait for all sites to be IPv6 ready– We must plan for a subset being dual-stack

• Interesting to test IPv6-only WNs behind http Squid dual-stack proxies

• Documentation/help/guidance required– RAL Tier 1: not clear what a site needs to do– BITP/ALICE: how do I move to dual stack?– We must document their experiences

11/06/2014 WLCG IPv6 28

To do list

• Complete the Site readiness survey• Encourage deployment of dual-stack PerfSONAR as good way

of getting started• Remaining Tier 1s to join GridFTP mesh

– Tier 2s welcome to join too

• PhEDEx tests of more storage technologies• Track experiment use cases and testing• Move more sites/services to dual stack

– In a controlled and agreed way

• Improve documentation and guidance

11/06/2014 WLCG IPv6 29

Links

• HEPiX IPv6 webhttp://hepix-ipv6.web.cern.ch

• HEPiX IPv6 wikihttps://w3.hepix.org/ipv6-bis/

• Working group meetingshttp://indico.cern.ch/categoryDisplay.py?categId=3538

• WLCG Operations IPv6 Task Forcehttp://hepix-ipv6.web.cern.ch/content/wlcg-ipv6-task-force-0

• Paper accepted for publication in proceedings of CHEP2013

10/06/2014 30WLCG IPv6

11/06/2014 WLCG IPv6 31

Questions?