kantor lsst networks and sitestical2019.redclara.net/media/attachments/2019/08/26/kantor-lsst... ·...

1TICAL2019- September3,2019

LSSTDistributedComputingandNetworks

JeffreyKantorSr.ManagerTICAL2019

September3,2019

2TICAL2019- September3,2019 Photo:GianlucaLombardi

Isan8.4metertelescopebeingbuiltinChile

Willconducta10yearsurveyofthesky

Beginsoperationsinlate2022

Photo:Gianluca Lombardi

LargeSynopticSurveyTelescope(LSST)


Eachimageis:• 9.6squaredegrees• 3.2gigapixels• 6.4GBofdata

Observingis:• 2x15sec

exposure/pointing• ~2000images/night• 15TB/night• 7PB/yr

LSSTObservingCadenceandDataRate


SummitandBaseSites

TelescopeandCameraDataAcquisition

CrosstalkCorrectionLong-termstorage(copy1)ChileanDataAccessCenter

DedicatedLongHaulNetworks

Tworedundant100Gbit linksfromSantiagotoFlorida(existing

fiber)Additional100Gbit link

(spectrumonnewfiber)fromSantiago– Florida

(ChileandUSnationallinksnotshown)

SatelliteProcessingCenter

CC-IN2P3,Lyon,France

DataReleaseProduction(50%)FrenchDAC

ArchiveSiteArchiveCenter

AlertProductionDataReleaseProduction(50%)CalibrationProductsProduction

EPOInfrastructureLong-termStorage(copy2)

DataAccessCenterDataAccessandUserServices

HQSiteScienceOperationsObservatoryManagementEducationandPublicOutreach

CameraSiteCameraSupport

LSSTSitesandLong-HaulNetworks


NightlyDataFlows

Archive Site

Base Site

VOEventBroker Serverin La Serena

(Non-DM)

10M Alertsper night(average)

60secondrequirementstopshere

VOEvent Broker Server(Non-DM)

OCS/TCSData(upto1Gbps)

EngineeringFacilityDBDataQualityMetricsAtmosphericTelescopeImagesandSpectra

DMData(upto39Gbps)Northbound SouthboundCrosstalkImages CalibrationImagesRawImages DIAObject CatalogWavefront Images DIASource CatalogRawCalibrationImages

RawandCrosstalkCorrectedImages24- 30TB/night6.4GBperimage

18-bituncompressedcompressedatBaseSite


Non-NightlyDataFlowsandDistributedComputing

Archive Site

Base Site

US DAC

CalibrationproductsL2DataProducts

Physical ShipmentsArchive Site to Base SiteAnnually

• Hardware• Data Release Products

(secondary method)

Chilean DAC

EPO Center

CalibrationproductsL2DataProducts

L2DataProducts

DataRelease:CoaddsObjectcatalogSourcecatalogForcedSourcecatalog

Eastbound WestboundRawimages 1/2ofL2DataProductsCalibrationproductsEngineeringFacilityDatabase1/2ofL2DataProducts(coadds,catalogs,SDQA)

French Site


LSSTDataProductsandDistributedComputing

− Nightlydataproducts– Alerts– Differenceimagesandcatalogs– 60slatencyfromtimeofreadoutfromcamera

− Annualdataproducts– Processallaccumulateddatafromstartofsurvey(distributedUS,France)– Producesallnightlydataproductsplus– Catalogsofdeep,faintobjects– “Forced”photometricmeasurements

− Supportingcommunity-developeddataproducts• “Nearby”computingandstorageatDataAccessCenters• Software(middleware,pipelines,algorithms,tools)


CurrentLSSTSchedule– 3.5MonthsContingency

Camera

FullIntegration&Verification

OperationalReadinessReview

EarlyIntegration&Testing

DataManagement

Telescope&Site

+3.5

ConstructionStart

FullSystemIntegration“Start”

ScienceOperationsStart

FinalDRPRelease

PreCommissioningPreparations

Facility Support

FullOperations

CD-4

ComCamonSummit

Pre-

NSFMREFC

DOEMIE

Commissioning

Operations

ScheduleContingency

17July2019

MIEtoOps

CriticalPath

EngineeringFirstLight

SystemFirstLight

CriticalPathisDome/TMA/AIV

Now


LSSTLongHaulNetworkLinks(BaselineFY21)

Tucson

LaSerena

CerroPachon

Santiago

SaoPaolo

Chicago40x10GbAURA(ScienceData)

20x10GbAURA(OCS/TCSControl)

Panama

1x100Gb1x40– 100GbREUNA

2x100GbAmlight/ANSP/RedCLARA

CC-IN2P3Lyon

Miami

100Gb(spectrum)Amlight/AngolaCable

Champaign

2*100GbNCSA/ICCN

Santiago

BocaRaton

10GbAmlight/internet2

10GbIN2P3

10x10GbNon-LSSTAURA(Fail-over) 100Gb

(spectrum)AmLight/RNP/RedCLARA(viaBuenosAires)

SecondaryPath

PrimaryPath

SLAC,OtherDOE

10GbESnet(institutional)

2x100GbEsnet/FLR/i2


LSSTInternationalNetworks(Chile– US)

Dark Fiber

LSSTBorderRouter

LSSTCerro Pachon/

La Serena

Spectrum

Spectrum

SaoPaolo

Miami FL Boca RatonFL

LSST Dedicated Primary Path

LSST Dedicated Backup Path

Central LinkSantiago

Panama

REUNASantiago

Dark Fiber


LSSTUSNetworks(Florida– Chicago)

Internet2AL2S

AtlanticWave100G

LSST(NCSA)

LSST

Spectrum

Spectrum

Spectrum

ESNET

ESN

ET

ChicagoIL Site A

AtlantaGE

JAXFL

Dark Fiber

Dark FiberBoca RatonFL

MiamiFL

FLRWaves

LSST Dedicated Primary Path

AmLight shared

FLR SharedLSST + AmLight

LSST Dedicated Backup Path

I2 AL2S If needed

ChicagoIL Site B

LSST Network – US Networks

I2 AL2

S

10/1/19 2x20G10/1/20 2x100G

Plan A: 10/1/20 1x100GPlan B: 10/1/21 1x100G

AmLig

ht P

rote

cted

FLRnet

10/1/2019

*

* Backhaul being addressed


LSSTNorthernNetworks(Champaign– Chicago– Lyon)

NCSA

600 W.Chicago

710 NLSDChicago

Internet2

100G Shared

MREN

100G Shared

IN2P3

ESnet

ESnet

100G Shared

100G Shared

GEANT

AL2S

MANLAN

RENATER

2x10G

Atlantic WaveAtlanta

ESnet

ESnet


ESNet forLSST

WASH

PHILNEWY32AOANEWY1118TH

BOSTALBA

BUFF

CLEV

JACK

CHAR

RALE

CHAT

ASHB

PITT

ATLA

DETR

INDI

CINC

LOUI

NASH

MEMP

JCSN

BATO

DALL

KANS

STLO

TULS

SANA

ALBQ

ELPA

DENV

PUEB

PHOE

TUCS

SALT

LASV

SAND

LOSA

SANL

RENO

BOIS

SEATOLYM

PORT

EUGE

SACR

SUNN

STAR

CHIC EQCH

HOUS

HOUH

BALT

Ciena_Map v12pld 9/30/2016

ESnet6 Optical Footprint: Add/DropsWave options Atla-Chic

atla-cr5rtr

LSSTDWDM

ESnetIP/MPLS core

star-cr5rtr

chic-cr5rtr

to Florida

MREN 8700

StarLight

NCSArtr1

ATLA HUB180 PeachTree(Site A)

Pre-FY21

100G100G

100G

100G

100G

100G

Before FY21 but after 2018 there will be a switch between the DWDM and alta-cr5 rtr. The handoff will be 1x100G.

atla-cr5rtr

LSSTDWDM

star-cr5rtr

chic-cr5rtr

to Florida

MREN 8700

Starlight

NCSArtr1

Post-FY21

100G 100G

100G

100G

100G

ESnetDWDM

ESnetDWDM

ESnetDWDM

LSST SW

LSST SW

LSST SW

100G

100G

100G

100G

100G

100G

PS

PS

PS

Last update 1/18/2018

600 W. Chicago(Site B)

710 N. Lake Shore Drive(Site C)

LSST SW

PS

NCSArtr2

NCSArtr2

710 N. Lake S

hore Drive

(Site C

)

600 W. C

hicago

(Site B)

100G

ATLA HUB180 PeachTree(Site A)

100G

FY20(2x20Gbps) FY21+(2x100Gbps)


End-to-EndNetworkBandwidthEvolution

Note:LSSTinternet,web,voice,videogooverAURAcircuits,whichareshared,andarecurrentlylimitedby1GFirewallsinLSand10Gbps internet2linksintheUS.ThiswillbeimprovedbytheendofFY19bythemoveto10GfirewallsinLS.


FirstOpticLightDemonstration– December2017

− TheLSSTNetworkEngineeringTeam(NET)completedthefirstsuccessfultransferofdigitaldataoverLSST/AURAfiberopticnetworksfromtheSummitSiteonCerroPachon,ChiletotheBaseSiteinLaSerena,ChileandontotheArchiveSiteatNCSAinChampaign.Asetof6x10Gbps NetworkInterfacecardsonDataTransferNodes(DTN)configuredwithiPerf3generatedasustaineddatarateofapproximately44gigabitspersecond,overaperiodof24hours.Thisexceededthetesttargetof40gigabitspersecond.

DTN 1 NIC1NIC2

NIC3

DTN 2 NIC1NIC2

NIC3

DTN 3 NIC1NIC2

NIC3

LSST

DWDM

Transponder

10x10G

12

34

56

La Serena

ManagementRouter

REUNA

LSST

DWDM

Transponder

10x10G

12

34

56

AmLight

AndesLight1

Brocade

MLXe

1/11/2

1/31/4

1/51/6

3/2

3/1

10G SR+

1G UTP

DWDM

AmLight

Pacific

Atlantic

100G Leased

AmLight

AMPATH01

Brocade

MLXe

7/2

AmLight

AMPATH02

Brocade

MLXe

7/2

6/2

FLR +

Internet2

Internet2

Rtsw-chic

0/0/5

100G

VID 3890

VID 3891

VID 3892

VID 3893

VID 3894

VID 3895

VID 3890

VID 3891

VID 3892

VID 3893

VID 3894

VID 3895

VID 3890

VID 3891

VID 3892

VID 3893

VID 3894

VID 3895

Level3 SCL AMPATH6000W

ChicagoICCN NCSA/NPCF

VID 3890

VID 3891

VID 3892

VID 3893

VID 3894

VID 3895

7/1

7/1

DTN 3

NCSAEX9214

NIC

NCSAMX960

DTN 1NIC

DTN 2NIC

DTN 4NIC

DTN 5NIC

DTN 6NIC

NCSAnet

NIC

9/1/0

VID 3890

VID 3891

VID 3892

VID 3893

VID 3894

VID 3895

VID 3890

VID 3891

VID 3892

VID 3893

VID 3894

VID 3895

NIC

NIC

NIC

NIC

NIC


Base– LDF100Gbps NetworkDemonstration– November2018

− FollowinguponthefirstsuccessfultransferofdigitaldataoverLSST/AURAfiberopticnetworksfromtheSummitSiteonCerroPachon,ChiletotheBaseSiteinLaSerena,ChileandontotheArchiveSiteatNCSAinChampaign inDecember2017,thatachieved44Gbps sustaineddatarate.

− 100Gbps demonstrationfromLaSerena– NCSANovember14,duringSupercomputing2018.Datasetis10TBofDECam publicdata.UsingSCInet connectionsfromMiami– Dallas– Chicago.DM-provided“watch”JupyterHubapplicationwillbeusedtomonitortransferfromDataTransferNode(DTN)toGPFSfile-system/disk.

Via Atlantic

DWDM

LSSTDWDM10x10G

1

2

3

4

5

6

La Serena REUNA

LSSTDWDM10x10G

1

2

3

4

5

AmLightAndes1

1/1

1/2

1/3

1/4

1/5

3/2

3/110G SR

1G UTP

DWDM

AmLight

Via Pacific

100G Leased AmLightAMPATH01

7/2

AmLightAMPATH02

7/2

8/1

Cent

ury

Link

100G LR4

CLARA/SCL AMPATH ICCN NCSA/NPCF

5x100G

7/1

7/1

NCSAEX9214

NCSAMX960

NIC

9/1/0

100G DTNeth0

6/1

Dell Z9100

7

8

9

10

en

p6

s0

100G DAC

6

7

8

9

10

1/6

1/7

1/8

2/6

2/7

AmLightAndes2

3/2

3/1

6/1

SC18SCINet

StarLight & MREN

100GDTN

Version: SC18.0929.01

Date: 09.29.2018

Author: LSST-NET

Ready

Pending

OOB

NVMeNVMeNVMe

NVMe

100G

DTN

NVMeNVMe

NVMeNVMeNVMe

NVMeNVMe

NVMe

Co

rsa

De

ll Z

91

00

-ON

AMIS

Sens

or

8700HD

SC18

AMIS

Sens

or

8/2

AMISSensor

Mirror

Mirror

Mirror

1

14

3321

OOB

SC15

SC14

SL-10

100G DTNs


VirtualNetworkOperationsCenter(VNOC)

− TheLSSTLong-HaulNetworkisasingleoperationalnetworkimplementedwithdiverseprovidersandlinksthatmustworkseamlesslytogether

− TheVNOCprovidesasingle,integratedoperationalcapabilityforend-to-endengineering,performancemonitoring,security,maintenance,andallotheroperations

− Thegoaliscompletevisibilityinalllinksandsitestoallparticipants,withasingleentrypointforinformationandassistance


SummitSite(RemoteDisplayonly)

VirtualNetworkOperationsCenter(VNOC)Sites

HQSite(RemoteDisplayOnly)

FrenchSite(???)

SantiagoSite(FullVNOCSite)

ArchiveSite(FullVNOCsite)

BaseSite(FullVNOCsite)

MiamiSite(FullVNOCsite)

kantor lsst networks and sitestical2019.redclara.net/media/attachments/2019/08/26/kantor-lsst... ·...

Documents