“The Pacific Research Platform:

Leading Up to the National Research Platform”

Opening Keynote

The National Research Platform Workshop

Montana State University

Bozeman, MT

August 7, 2017

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technology

Harry E. Gruber Professor,

Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

http://lsmarr.calit2.net1

30 Years Ago NSF Brought to University Researchers

a DOE HPC Center Model

NCSA Was Modeled on LLNL SDSC Was Modeled on MFEnet

1985/6

Launching the Nation’s Information Infrastructure:

NSFnet Supernetwork and the Six NSF Supercomputers

NCSA

NSFNET 56 Kb/s Backbone (1986-8)

PSCNCAR

CTC

JVNC

SDSC

PRP’s Backbone is One Million Times Faster!

I-WAY: Information Wide Area YearSupercomputing ‘95

• The First National 155 Mbps Research Network

– 65 Science Projects

– Into the San Diego Convention Center

• I-Way Featured:

– Networked Visualization Applications

– Large-Scale Immersive Displays

– I-Soft Programming Environment

– Led to the Globus Project

UIChttp://archive.ncsa.uiuc.edu/General/Training/SC95/GII.HPCC.html

See Session 1

Talk by

Ian Foster

NSF’s PACI Program was Built on the vBNS

to Prototype America’s 21st Century Information Infrastructure

The PACI Grid Testbed

National Computational Science

1997

vBNS

led to

NSF’s OptIPuter Project: Using Supernetworks

to Meet the Needs of Data-Intensive Researchers

OptIPortal–

Termination

Device

for the

OptIPuter

Global

Backplane

Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PIUniv. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST

Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent

2003-2009

$13,500,000

In August 2003,

Jason Leigh and his

students used

RBUDP to blast

data from NCSA to

SDSC over the

TeraGrid DTFnet,

achieving18Gbps

file transfer out of

the available

20Gbps

LS Slide 2005

We Have Been Working Toward PRP for 15 Years:

NSF OptIPuter, Quartzite, Prism Awards

PI Papadopoulos,

2013-2015

PI Smarr,

2002-2009PI Papadopoulos,

2004-2007

Thirty Years After NSF Adopts DOE Supercomputer Center Model

NSF Adopts DOE ESnet’s Science DMZ for High Performance Applications

• A Science DMZ integrates 4 key concepts into a unified whole:

– A network architecture designed for high-performance applications,

with the science network distinct from the general-purpose network

– The use of dedicated systems as data transfer nodes (DTNs)

– Performance measurement and network testing systems that are

regularly used to characterize and troubleshoot the network

– Security policies and enforcement mechanisms that are tailored for

high performance science environments

http://fasterdata.es.net/science-dmz/

Science DMZ

Coined 2010

The DOE ESnet Science DMZ and the NSF “Campus Bridging” Taskforce Report Formed the Basis

for the NSF Campus Cyberinfrastructure Network Infrastructure and Engineering (CC-NIE) Program

See Deep Dive 6

On High-Perf

Networking

Based on Community Input and on ESnet’s Science DMZ Concept,

NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways

Red 2012 CC-NIE Awardees

Yellow 2013 CC-NIE Awardees

Green 2014 CC*IIE Awardees

Blue 2015 CC*DNI Awardees

Purple Multiple Time Awardees

Source: NSF

(GDC)

Logical Next Step: The Pacific Research Platform Creates

a Regional End-to-End Science-Driven “Big Data Superhighway” System

NSF CC*DNI Grant

$5M 10/2015-10/2020

PI: Larry Smarr, UC San Diego Calit2

Co-Pis:

• Camille Crittenden, UC Berkeley CITRIS,

• Tom DeFanti, UC San Diego Calit2,

• Philip Papadopoulos, UCSD SDSC,

• Frank Wuerthwein, UCSD Physics and SDSC

Letters of Commitment from:

• 50 Researchers from 15 Campuses

• 32 IT/Network Organization Leaders

See Deep Dive 3

on NSF Research CI

Developing Regional DMZs

Require a Lot of Consultation-Sociotechnilogical Problem

• May 2014 LS Gives Invited Presentation to UC IT Leadership Council

– Strong Support from UC and UCOP CIOs

• July 2014 LS Gives Invited Talk to CENIC Annual Retreat

– CENIC/PW Agrees to Act as Backplane

– CIO Support Extends to CA Private Research Universities

• December 2014 UCOP CIO and VPR’s Provide PRP “Momentum Money”

• January 2015 Kickoff of PRPv0 by Network Engineers

– Begins Every Two Week Conference Calls, Now Weekly

• March 2015 LS Invited “Blue Sky” Presentation to UC VCR/CIO Summit

– NSF PRP Proposal Submitted With Letters of Commitment From:

– 50 Researchers from 15 Campuses

– 32 IT/Network Organization Leaders

• July 2015 NSF Announces Funding for PRP

• October 2015 PRP Grant Begins

Big Data Science Data Transfer Nodes (DTNs)-

Flash I/O Network Appliances (FIONAs)

UCSD Designed FIONAs

To Solve the Disk-to-Disk

Data Transfer Problem

at Full Speed

on 10G, 40G and 100G Networks

FIONAS—10/40G, $8,000FIONette—1G, $1,000

Phil Papadopoulos, SDSC &

Tom DeFanti, Joe Keefe & John Graham, Calit2

John Graham, Calit2

See Deep Dive 4

How Does PRPv1 Work?

We Measure Disk-to-Disk Throughput with 10GB File Transfer

4 Times Per Day in Both Directions for All PRP Sites

January 29, 2016

From Start of Monitoring 12 DTNs

to 24 DTNs Connected at 10-40G

in 1 ½ Years

July 21, 2017

Source: John Graham, Calit2

Increasing Participation Through

PRP Science Engagement Workshops

Source: Camille Crittenden, UC Berkeley

UC San DiegoUC Merced

UC Davis UC Berkeley

See Session 3

on Engagement

The First National Research Platform Workshop

on August 7-8, 2017

Announced in I2 Closing Keynote:

Larry Smarr “Toward a National Big Data Superhighway”

on Wednesday, April 26.

Co-Chairs:

Larry Smarr, Calit2

& Jim Bottum, Internet2

150 Attendees

PRP’s First 1.5 Years:

Connecting Multi-Campus Application Teams and Devices

Particle Physics: Using the PRP to Fix

the “Last Mile Problem” of the LHC in California

ATLASCMS

U.S. Institutions

Participating in LHC

LHC Data

Generated by

CMS &

ATLAS

Detectors

Analyzed

on OSG

Maps from www.uslhc.us

See Talks by

Harvey Newman,

Frank Wuerthwein,

& Rob Gardner

Global Scientific Instruments Will Produce Ultralarge Datasets Continuously

Requiring Dedicated Optic Fiber and Supercomputers

Square Kilometer Array Large Synoptic Survey Telescope

https://tnc15.terena.org/getfile/1939

3.2 Gpixel Camera

Tracks ~40B Objects,

Creates 10M Alerts/Night

Within 1 Minute of Observing

2x100Gb/s

See Session 5 Talk

By Heidi Morgan

“First Light”

In 2019

100 Gbps FIONA at UCSC Allows for Downloads to the UCSC Hyades Cluster

from the LBNL NERSC Supercomputer for DESI Science Analysis

300 images per night.

100MB per raw image

30GB per night

120GB per night

250 images per night.

530MB per raw image

150 GB per night

800GB per night

Source: Peter Nugent, LBNL

Professor of Astronomy, UC Berkeley

Precursors to

LSST and NCSA

NSF-Funded Cyberengineer

Shaw Dong @UCSC

Receiving FIONA

Feb 7, 2017

Cancer Genomics Hub (UCSC) Was Housed in SDSC, But NIH Moved Dataset

From SDSC to Uchicago - So the PRP Deployed a FIONA to Chicago’s MREN

1G

8G

Data Source: David Haussler,

Brad Smith, UCSC

15GJan 2016

See Deep Dive 5

on Community

Data Resources

PRP Is Linking the Laboratories of

the Pacific Earthquake Engineering Research Center

http://peer.berkeley.edu/

PEER Labs: UC Berkeley, Caltech, Stanford,

UC Davis, UC San Diego, and UC Los Angeles

John Graham Installing FIONette at PEER Feb 10, 2017

40G FIONAs

20x40G PRP-connected

WAVE@UC San Diego

PRP Now Enables

Distributed Virtual Reality (and Distributed Machine Learning)

PRP

WAVE @UC Merced

Transferring 5 CAVEcam Images from UCSD to UC Merced:

2 Gigabytes now takes 2 Seconds (8 Gb/sec)

Visualizing Enormous Datasets Resulting from Supercomputer Simulations

Using EVL’s DTN GPU Services (PRP Provided EVL DTN)

Cosmological Volume Simulation

• Computed on the ALCF Mira – 29 Billion Particles

– 40 TB per Snapshot and 500 Snapshots

• Then Sent to NCSA Blue Waters – 1st -Level Data Analytics & Visualization

– Using “yt”

• Web Services on EVL’s DTN – Used to Convert a Snapshot

– To a DZI Multi-Resolution Image

– Displayed Using SAGE2

– In the SC16 NCSA Booth

Cosmologist Katrin Heitmann of ANL

Standing in Front of the SAGE2 Display at SC16

Slide from Maxine Brown, EVL Director

See Deep Dive 8

on SC17 Demos

The Prototype PRP Has Attracted

New Application Drivers

Scott Sellars, Marty Ralph

Center for Western Weather and Water Extremes

Frank Vernon - Expansion of HPWREN

Tom Levy, Cultural Heritage

Cryo EM

GPU JupyterHub:

2 x 14-core CPUs

256GB RAM

1.2TB FLASH

3.8TB SSD

Nvidia K80 GPU

Dual 40GbE NICs

And a Trusted Platform

Module

GPU JupyterHub:

1 x 18-core CPUs

128GB RAM

3.8TB SSD

Nvidia K80 GPU

Dual 40GbE NICs

And a Trusted Platform

Module

PRP UC-JupyterHub Backbone

Next Step: Deploy Across PRP

Source: John Graham, Calit2

UCB UCSD

Cryo-electron Microscopy (cryo-EM)

Has Driven a “Resolution Revolution” in the Last Five Years

Exposure (every 60 seconds):

X & Y dimensions: 7420 x 7676 Pixels

Frames per Movie: 10 - 50

Size: 3 - 10 GB per Movie

Every 24 hours:

Number of Movies: ~1400

Data Size: ~5 TB

Typical Datasets:

Length of Time: 2 - 6 Days

Total size: 10 - 30 TB

Each Cryo-EM ‘Image’ is Actually a Movie

Source: Michael A. Cianfrocco,

Elizabeth Villa, & Andres Leschziner, UCSD

Using PRP to Connect Cryo-EM across California

With End Users and Computational Facilities

Long term:

‣Partner with Cryo-EM Facilities to Stream Data

Straight from Microscopes (over PRP) to SDSC

‣Perform All Cryo-EM Analysis (from Micrographs

to 3D Models) via Web Browser on SDSC

‣Expand Computing to Other XSEDE Resources

(e.g. Xstream) and DOE’s NERSC

Short term:

‣Provide 2D and 3D Analysis on Particle Stacks on

Comet at SDSC

Source: Michael A. Cianfrocco, UCSD

**

SDSC

NERSC

Xstream

3 Supercomputer Centers

cosmic-cryoem.org

~20 Microscopes in CA

UCLA

UC Davis

UC Santa Cruz

SF Bay

UC Berkeley, LBNL,

UCSF, Stanford

San Diego

UCSD, TSRI, Salk*

Extending

to MSU

HPWREN Real-Time Network Cameras on Mountains

for Environmental Observations and Fires

San Diego County Red Mountain Fire Cameras

• Southeast (left) “Highway” Fire

• Southwest (center rear) “Poinsettia” Fire

• West (right) “Tomahawk” Fire

Source: Frank Vernon,

Hans Werner Braun HPWREN

May 14, 2014

PRP Backbone Sets Stage for 2017 Expansion

of HPWREN, Connected to CENIC, into Orange and Riverside Counties

• PRP CENIC 100G Link

UCSD to SDSU

– DTN FIONAs Endpoints

– Data Redundancy

– Disaster Recovery

– High Availability

– Network Redundancy

• CENIC Enables PRP

10G Links Between

UCSD, SDSU, & UCI

HPWREN Servers

• Potential Future UCR

CENIC Anchor

UCR

UCI

UCSD

SDSU

Source: Frank Vernon,

Hans Werner Braun HPWREN UCI Antenna Dedicated

June 27, 2017

UC Catalyst Project: Linking Cultural Heritage and Archaeology Datasets

at UCB, UCLA, UCM and UCSD with CAVEkiosks

48 Megapixel CAVEkiosk

UCSD Library

48 Megapixel CAVEkiosk

UCB Library24 Megapixel CAVEkiosk

UCM Library

See Session 2 Talk by

Chris Hoffman

Director: F. Martin Ralph Website: cw3e.ucsd.edu

Big Data Collaboration with:

Source: Scott Sellers, CW3E

Collaboration on Atmospheric Water in the West

Between UC San Diego and UC Irvine

Director, Soroosh Sorooshian, UCSD Website http://chrs.web.uci.edu

Calit2’s FIONA

SDSC’s COMET

Calit2’s FIONA

Pacific Research Platform (10-100 Gb/s)

GPUsGPUs

Complete workflow time: 20 days20 hrs20 Minutes!

UC, Irvine UC, San Diego

Major Speedup in Scientific Work Flow

Using the PRP

Source: Scott Sellers, CW3E

New NSF CHASE-CI Grant Creates a Community Cyberinfrastructure

Adding a Machine Learning Layer Built on Top of the Pacific Research Platform

Caltech

UCB

UCI UCR

UCSD

UCSC

Stanford

MSU

UCM

SDSU

NSF Grant for High Speed “Cloud” of 256 GPUs

For 30 ML Faculty & Their Students at 10 Campuses

for Training AI Algorithms on Big Data

See Session 3 Talk by

Tom DeFanti

We are Now Investigating

How the PRP Prototype Might Be Extended to National-Scale

From the text of the PRP cooperative agreement:

After approximately 18 (or TBD) months, a site visit and comprehensive review of

progress towards meeting project milestones and goals and overall performance and

management processes will take place, including user community relationships,

scientific impacts, and the status of the project as a model for potential future

national-scale, network-aware, data-focused cyberinfrastructure attributes,

approaches, and capabilities.

PRP is Partnering with the Advanced CyberInfrastructure –

Research and Education Facilitators (ACI-REF) NSF Grant to Explore Extension

PRP Connected

ACI-REF has also spawned the 28-member Campus Research Computing consortium (CaRC), funded by the NSF as a Research Coordination Network (RCN).

CaRC is dedicated to sharing best practices, expertise, and resources, enabling the advancement of campus- based research computing activities around the nation.

Jim Bottum, Principal Investigator

ACI-REF

CaRC

See Session 1 Talk by

Tom Cheatham

Expanding to the Global Research Platform

Via CENIC/Pacific Wave, Internet2, and International Links

See

Session 5

on GRP

PRP

PRP’s Current

International

Partners

Korea Shows Distance is Not the Barrier

to Above 5Gb/s Disk-to-Disk Performance

Netherlands

Guam

Australia

Korea

Japan

Our Support:

• US National Science Foundation (NSF) awards

CNS 0821155, CNS-1338192, CNS-1456638, CNS-1730158,

ACI-1540112, & ACI-1541349

• University of California Office of the President CIO

• UCSD Chancellor’s Integrated Digital Infrastructure Program

• UCSD Next Generation Networking initiative

• Calit2 and Calit2 Qualcomm Institute

• CENIC, PacificWave and StarLight

• DOE ESnet