dcache, a distributedhigh performance storagesystemfor hpcdcache storage for the large hadron...

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 1

ISC 2013

Patrick Fuhrmann

dCache, a distributed high performance storage system for HPC

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 2

Content •  Who are the dCache people ? •  Where are we coming from ?

–  Some words about WLCG, and others –  Some dCache deployments

•  Where do we want to go and why ? –  HTC to HPC –  HPC for our current customers

•  But there is more than just performance. –  Multi Tier storage model –  Multi Protocol Support –  Consistent Authentication and Authorization

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 3

•  German Project with Helmholtz Alliance Data Centers

•  Large Scale DM and Analysis

•  WP1: Federated Identities •  WP2: Federated Storage

Who are we ? dCache is an international

collaboration.

Nordic e-Infrastructure for Computing

Chicago

Hamburg Kopenhagen

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 4

What do we do ?

We – design –  implement – and deploy

Data Storage and Management software for data intensive communities.

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 5

Where are we coming from ?

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 6

From High Energy Physics

HERA and the Tevatron in the past and now

The Large Hadron Collider in Geneva

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 7

High Energy Physics (the sensors)

The Atlas detector, 12 m long, 6 in diameter

and 12.000 tonnes

The ring: 27 Km long, -271 degrees cold, some billion Euros and looking for the Higgs and for Dark Matter. Collisions every 25 nsec, filled with 13.000 bunches running with nearly speed of

light. The ring needs 120 MW and 50 MW for cooling.

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 8

And its computer:

The LHC Computing Grid

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 9

The Grid never sleeps: this image shows the activity on 1 January 2013, just after midnight, with almost 250,000 jobs running and 14 GiB/sec data transfers. Image courtesy Data SIO, NOAA, US Navy, NGA, GEBCO, Google, US Dept. of State Geographer, GeoBasis, DE/BKG and April 2013 issue of the CERN Courier

High Energy Physics (the computer, GRID)

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 10

Now, where is dCache ?

We do ½ of their storage So we have 50% of their famous Higgs

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 11

dCache storage for the Large Hadron Collider around the world

Germany 25.5 PB

Europe 66 PB

US 34 PB

Canada 9.6 PB

Spain 13.7 PB

France 8.8 PB

Netherlands 4.5 PB

Nordics 4.5 PB

Russia 5 PB

DESY LMU

Aachen Wuppertal

0 0.5

1 1.5

2 2.5

3 3.5

4 4.5

Increase over 2.5 years

About 115 PBytes only for WLCG in 8(+2) out of 11(+3) Tier 1 centers

and about 60 Tier 2’s, which is about ½ of the entire WLCG data.

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 12

But there are more …

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 13

Other customers

THE FRENCH NATIONAL INSTITUTE FOR AGRICULTURAL RESEARCH

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 14

And how do we do this ?

With joy … and

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 15

LHC Computing Storage Element medium single stream performance

DISK

DISK

DISK

Tape

Tape

Tape

Other sites

DISK Local Compute Farm

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 16

This is quite nice but getting a bit boring ….

So, where do we want to go ?

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 17

HPC Computing Possibly high single stream performance

DISK

DISK

DISK

Tape

Tape

Tape

DISK

Local Supercomputer IO Nodes CORES 512 Titan 18.000 4096 Tianhe-2 16.000

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 18

Having a look into real HPC performance numbers.

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 19

Real Data Rates for HPC Three file system partitions in front of the Titan

Courtesy: Oak Ridge National Lab, Spider FS Project

Equivalent to 5000 and 10000

Cores per

File system in GRID Terms

Each GRID core

consumes about

6Mbytes/s

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 20

So the question arises …

Can dCache do this in a single instance ?

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 21

Core Count of FERMIlab

Courtesy: Vicky White, Institutional Review 2011

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 22

40 PBytes Tape

Information provided by Catalin Dumitrescu and Dmitry Litvintsev

US-CMS Tier I 14 PBytes on

Disk 770 Write

Pools

420 Read Pools

26 Stage Pools

***

260 Front

Nodes Total:

6 Head 280 Pool/Door

Physical Hosts

US CMS dCache Setup to serve the farm and

the wide area connection to CERN and the US Tier II’s

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 23

As network and spinning disks are becoming the bottleneck,

we can even do better …

Or

Using Multi-Tier Storage

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 24

Multi Tier Storage

DISK

SSD

SSD

SSD

Streaming

DISK

DISK

Streaming

Chaotic

Tape

Tape

Tape

Parallel NFS

to compute nodes

Wide area resp. cloud

protocols DISK

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 25

Why do we want to go HPC ?

•  The LHC experiment (e.g. ATLAS) are seriously looking into HPC. They would like to utilize free resources in HPC worldwide. Feasibility evaluations are ongoing. If they decide to go for it, they need Grid Storage Elements to ensure access to their worldwide data federation.

•  The HPC community begins to share data. Right now this is still all manual. But they could learn from the LHC Grid. We share and transfer data automatically for about a decade, including proper authentication and authorization at the source and endpoints.

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 26

Just performance is not sufficient for BIG DATA

in the future

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 27

NASA Evaluation of scientific data flow

Courtesy: Goddard Tech Transfer News | volume 10, number 3 | summer 2012

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 28

Scientific Storage Cloud

LOFAR antenna Huge amounts of data

X-FEL (Free Electron Lasers) Fast Ingest

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 29

Scientific Storage Cloud

•  The same dCache instance can serve –  Globus-online transfers via gridFTP –  FTS Transfers for WLCG via gridFTP or WebDAV –  Private upload and download via WebDAV –  Public anonymous access via plain http(s) –  Direct fast access from worker-nodes via NFS4.1/pNFS (just a

mount like GPFS or Lustre but with standards)

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 30

Now, performance seems to be ok…

how about automated worldwide data transfers ?

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 31

How can you do worldwide automated transfers I

•  Use ‘globus online’ a worldwide transfer services. •  dCache provides the necessary interfaces,

including authentication.

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 32

•  Run your own “File Transfer Service, FTS(3)”. •  The Software is provided by EMI/CERN DM. •  FTS uses gridFTP or http(s). •  FTS can do network shaping. •  FTS does the necessary tries. •  FTS is free and “Open Source” •  dCache provides the necessary interfaces to FTS,

including authentication.

How can you do worldwide automated transfers II

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 33

The Dynamic http/WebDAV federation

How can you do worldwide automated transfers III

•  Still prototype status •  Collaboration between dCache.org and CERN DM,

started with EMI

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 34

Dynamic Federation

Federation Service Best

Match Engine

GEO IP

LFC Catalogue

Other http enabled SE’s

dCache

Any cloud provider

Candidate Collection

Engine

ROOT

WGET CURL

Nautilus Dolphin

Konqueror

Portal One or more

candidates

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 35

•  A user (individual) usually holds a set of credentials to identify him/herself against services. –  Passport, Driver license, credit card –  Google account, Twitter, –  X509 Certificates (GRID, Country Certificate Authority)

•  Federated Data Services should –  Understand as many as possible of those credentials –  Be able to map different ones to the same individual

•  dCache does all this with : –  User/password –  Kerberos –  X509 Certificates and Proxies –  SAML assertions (in development within LSDMA)

Some remarks on authentication and authorization

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 36

In summary •  dCache has a long history in serving Big Data communities with

PetaBytes of local and remote storage and Gbytes/sec of transfer performance.

•  dCache is successfully moving into the “Scientific Cloud” direction, incorporating HTC and HPC.

•  Focusing on High Individual Throughput as well as scaling out. •  Moreover, making sharing of scientific data easy and secure.

–  Making all data available via a set of industry access protocols: •  NFS 4.1/pNFS for local high performance access (like local mount) •  WebDAV and http for Web Sharing •  CDMI and S3 (in preparation) for cloud acces. •  GridFTP for fast wide area transfers.

–  Mapping various different credentials to a single ‘user’ •  X509 Certificates •  User/Password •  Kerberos •  SAML/OpenID

dCache for HPC | ISC 2013 Leipzig | Patrick Fuhrmann | 17 June 2013 | 37

The End

further reading www.dCache.org