intelligent optical systems for a 5g world · vehicle a’s view [1] qiu, hang, et al....

Dan Kilper

June 16, 2020

Intelligent Optical Systems for a 5G World

Overview

• A new network dichotomy

• Optics in Edge Clouds• Disaggregating Optical

Systems• A New Physical Layer for

6G

2

Metro-Edge is the New Long Haul

3

Long Haul shifts to DCI: All about Data Center to Data Center

Massive Enterprise Networks

4

Equinix GXI 2019

Two very different networks

DCI Network

Metro Networks

5

Enterprise Hyperscale DC, IXP

Edge DC, Micro DC, CORD, Access Nodes

In the 5G World

Heterogeneous, pay-as-you-grow, mesh networks

Metro-Regional Edge Cloud Networks

6

Willing to give up capacity for something more

Hyperscale Public Clouds

• Tiered access-metro-long haul network

• All wireless processing at cell site

• Deterministic < 1 ms in building or home

• > 10-20 ms for cloud

7

Telco Mesh Network

Core Cloud

Metro/Core Network

User Eqmt

< 1 ms edge > 20 ms access and core

Bldg

Cell

Access/Backhaul Network

Edge Clouds• Put data center in central office or access-metro boundary

• Cloud RAN: send digitized RF to data center for processing

• Sub-millisecond round trip to edge cloud data center

Edge Cloud Core Cloud

Access Pt

User Eqmt

< 1 ms edge > 1 ms core

BldgCell

8

DCI Network

> 20 ms core

Why Latency Matters: Cloud Assisted Autonomous Vehicles

9

Edge Cloud

Vehicle A’s view

[1] Qiu, Hang, et al. "Augmented Vehicular Reality: Enabling Extended Vision for Future Vehicles." Proceedings of the 18th International Workshop on Mobile Computing Systems and Applications. ACM, 2017.

Vehicle B’s view

Vehicle A’s augmented view

Self-Driving Cars

Vehicle A Vehicle B

E2E Latency ~10 ms required for user QoE

Nokia

Courtesy of D. Raychoudhuri

Other Latency & Bandwidth Sensitive Edge Cloud Applications

10

Line of Sight mmWave Handovers

Phased array antenna synchronization

Coordinated Multi-Point (CoMP) cluster formation

Optics moving data in and between edge data centers

11

64x10G MIMO

100x10G Small Cells

200G Large Cells

Local Wireless Provider

Telco 1

Telco 2

Service Provider A

Service Provider A

Service Provider B

10 λ10 λ

12 λ

Replace some ASIC processing with Photonic Switches

Si MEMS Crossbar, M. Wu UC Berkeley

Network Domains: On-Demand Autonomous Systems

64x10G MIMO

100x10G Small Cells

200G Large Cells

Service Provider A

Service Provider A10 λ

10 λ

12 λ

Service Provider B

Telco 2

Local Wireless Provider

Telco 1

12

New Research Frontier for Optical Systems

• Achieving ultra-low latency• Achieving ubiquitous high capacity

• Efficiently adapting capacity to meet demand

• Using optics in cross-domain, multi-tenant environments

• Meet cost & scalability requirements at the edge

13

Technology is Changing in Response….

14

PON vs DWDM ROADMs

15

T. Pfeiffer, Nokia, ONDM 2018

Problems: low capacity, tree topology, protocol delays, not scalable

Advantages: switched, cheap, relatively simple control

Problems: too expensive, too slow/static, amplifiers, complex control, proprietary single vendor

Advantages: mesh networks, high capacity, scalable in size and capacity

Converging!Simplify control, integrated photonics…Add WDM, mesh…

OpticalAmplifiers:

EDFARaman

MUX DEMUX

OpticalFibers:

SSMF, LEAF

Wavelength Selective Switch:

WSS

<200 km

96 ‘Wavelengths’ (50 GHz)

Reconfigurable Optical Add Drop Multiplexer: ROADM

How do we achieve low cost and scalable, software

controllable, optically amplified and switched systems?

16

Market Driven Computer Dis-AggregationEnabled Hyperscale DC Architecture

Computer Disaggregation Spectrum

18

Max Disaggregated Max Integrated

AppleBuild Your Own from scratch/ hyperscale

Single Integrator: Configure Your Own

Single Integrator: Choose Your Model

Single Integrator: Accessorize(Display, Keyboard)

Choice of Operating System

Disaggregated ROADM Optical Systems!?

19

Disaggregated Optical SystemsAg

greg

ated

Fully Aggregated: Entire transport network acts as a single managed system

End to End System

API

Transponder TransponderTerminal ILA ROADM Terminal

Fully Disaggregated: Everything is a separate network element

APIAPIAPI API API API

Partially Disaggregated: Transponder is one element, Open Line System (OLS) is other

Open Line System

APIAPI API

Disa

ggre

gate

d

20

Courtesy of Victor Lopez, Telefonica

Optical System Disaggregation Spectrum

21

Status Quo

Build Your Own from scratch/ hyperscale

Single Integrator Configure Your Own

Single Integrator Choose Model/ Choose C&M

Single Integrator Accessorize:Open Line Systems

Choice of Control & Management Platform

Max Disaggregated Max Integrated

openROADM

Build Your Own?

Will build your own optical system at the component level ever be a reality?

If a control plane is available, then yes

It’s all about the control plane

Still require expert user: most likely carrier or hyperscale

Some History• Late 90’s: MCI/Globecom tried to build their own systems from

components• ~2000: Unified control plane attempt to merge control of optical

systems into L3 control: Software Disagg.• GMPLS/MPLS was result

• Mid 00’s: JDSU/Nortel introduce ‘generic’ ROADM building block systems

• Late 00’s: Coherent transceivers change system engineering (no dispersion maps, PMD)

• Early 10’s: Enterprises/DC operators buy their own optical networks

• 2020: 5G is here!

23

The Disaggregation Lifecycle

24

Time

Perf

orm

ance

Sustained Growth

More research, Growth of Eco-System/ Supply Chain

Integrated System Growth Slows

Transition to Disaggregation

Research

Fact or Fiction?

What’s holding us back?

25

Control is the problem…

26

SDN: Control for Disaggregated Systems

27

Scheduler

Cross-Domain Interface

Policy Manager

Network State/ Abstraction

Ntwk State/ Abstraction

DataCenterSDNos

AmplController

WANSDNos

Network SDN OS

Control Data Translator

PCE

PCE

OPL OS

ROADMOA OA

ROADMOAROADM ROADMROADMOCS

OTN/Eth

Layer 2 Controller

Layer 2 Cntrl

OPL OS

OTN/Eth OTN/Eth

Layer 2 Controller

OPL OS

OTN/Eth

Domain 2 Domain 3Domain 1

OpenFlow, Proprietary, TL1

Proprietary System Control

Flat Network OS Control

28

Scheduler

Cross-Domain Interface

Policy Manager

Network State/ Abstraction

DataCenterSDNos

WANSDNos

Network OS

PCE

ROADMOA OA

ROADMOAROADM ROADMROADMOCS

OTN/Eth OTN/Eth OTN/EthOTN/Eth

Domain 2 Domain 3Domain 1

OCM

Power Levelling

Flat Network OS Control

29

Scheduler

Cross-Domain Interface

Policy Manager

Network State/ Abstraction

DataCenterSDNos

WANSDNos

Network OS

PCE

ROADMOA OA

ROADMOAROADM ROADMROADMOCS

OTN/Eth OTN/Eth OTN/EthOTN/Eth

Domain 2 Domain 3Domain 1

OCM

Power Levelling

Ampl. Control

Provisioning Control Chn Discovery

SDN Optical Physical Layer Control

30

Scheduler

Cross-Domain Interface

Policy Manager

Network State/ Abstraction

Ntwk State/ Abstraction

DataCenterSDNos

AmplController

WANSDNos

Network SDN OS

Control Data Translator

PCE

PCE

OPL OS

ROADMOA OA

ROADMOAROADM ROADMROADMOCS

OTN/Eth

Layer 2 Controller

Layer 2 Cntrl

OPL OS

OTN/Eth OTN/Eth

Layer 2 Controller

OPL OS

OTN/Eth

Domain 2 Domain 3Domain 1

Disaggregated System SDN Research

• P. Castoldi, et. al. “Disaggregated optical network control and orchestration of heterogeneous domains” OECC/PSC 2019

• N. Sambo, et. al. “Enabling locally automated reconfigurations in disaggregated networks” OECC/PSC 2019

• Q. P. Van, et. al. “Container-Based Microservices SDN Control Plane for Open Disaggregated Optical Networks” ICTON 2019

• A. Mayoral, et. al. “Multi-layer service provisioning over resilient Software-Defined partially disaggregated networks” JLT to appear 2019

• A. Sgambelluri, et. al. “Fully Disaggregated ROADM White Box with NETCONF/YANG Control, Telemetry, and Machine Learning-based Monitoring” OFC 2018

• M. Shiraiwa, et. al. “Experimental Demonstration of Disaggregated Emergency Optical System for Quick Disaster Recovery” JLT 2018

• L. Gifre, et. al. “Autonomic Disaggregated Multilayer Networking” JOCN 2018

31

MetroHaul: Sgambelluri, et. al. OFC 2018

Transmission Experiments

• Where software meets physics• Starting to see transmission results on

disaggregated systems• Experiments are hard!

• No re-circulating loops!

32

Open Line System Experiment• 8 different transceiver vendors• Up to 1945 km in lab experiments• Use TIP GNpy performance estimation tool

33M. Filer, et. al. JLT 2018

GN Model Performance Prediction Accuracy

TIP, PoliTo used Microsoft commercial development test lab

Physical Layer Control Algorithms• Power drifts over time and new channels are

provisioned: need periodic power adjustments to stay within margins

• Custom simulation to study control performance

34

[1]

A B

[1,2,3,4] [1,4]

[3]

[2]C

D

[3]

[4]

[ ]

Wait for Chn 1

Ready to Adjust

[i,j,k] = channels adjusting upstream

4

511

12

22

23

3130

29

28

27

2625

24

21

13 15

20

18

19

1614

109

17

87

123

6

Kilper & White OFC 2007

Disaggregated Systems Enable Cross-Domain Transmission

Domain 1 Domain 2

Orchestrator

Local Controller

Local Controller

Data

Pla

neCo

ntro

l Pla

neAp

plic

atio

n Pl

ane

ROADMROADM

AgentAgent

User/App

RyuRyu

ONOS

REST

Openflow

CDCP

IXP 1

IXP 2

Control signaling channelOptical data channel

User interface

35

Y. Li, et. al. OFC 2017 Postdeadline; JLT 2018

New component technology with new system control technology

• Integrated photonic PIC OPM to enable continuous per-channel SLA enforcement

• Coordinate SDN QoT estimators to determine path that meets requested performance

TX1x3 Splitter

WSS WSS WSS WSS70km 70km 55km

20km

20km

OPM

RX

OPMWSS

WSS

Agent Agent

ONOS Orchestrator

Domain 1 Domain 2

WSS

OPM

WSS

OPMROADM3

ROADM4 ROADM6

ROADM5ROADM1 ROADM2

IXP

WSS

WSS WSS

WSS

Control Plane

Data Plane

Ryu Controller Ryu Controller

36

OPM: OSNR Monitor

BPF Powermeter∆t

Power meter

Re-Routing Through Controller Negotiation

• Two Scenarios:• Intra-domain re-routing• Inter-domain re-routing

Inter-domain path setup

OSNR monitoring

Intra-domain rerouting

Inter-domain rerouting

Success?

Below threshold?

Success?No

Yes

Yes

Yes

No

Yes

Success?Yes

Traffic fail

Inter-domain traffic request

NoSuccess?

No

Yes

No

No

Inter-domain path computation

Intra-domain impairment?

λ1

Scenario 1:

Scenario 2: λ1

IXP

IXP 37

OSNR monitoring for working path in Domain 1 (good)

OSNR monitoring in Domain 2 (not good)

Intra-domain rerouting in Domain 2

OSNR monitoring for rerouting path in Domain 1 (good)

Scenario 1

OSNR monitoring for working path in

Domain 1 (not good)

Inter-domain rerouting in Domain 1

Intra-domain rerouting in Domain 1

fail

Inter-domain rerouting in Domain 2

OSNR monitoring for rerouting path in Domain 2 (good)

OSNR monitoring for rerouting path in Domain 2 (good)

Scenario 2

6G

• Edge cloud research frontier• Rethink physical layer networks

• What should the new Internet Physical Layer look like?

• Optics & Wireless truly converged• Cross-technology, multi-tenant, and

multi-domain• Embedded electronics with ‘embedded

photonics’• Intelligence and advanced functionality

38

Densification of Wireless Access

• Tiered, North-South architecture & capacity

• Relatively small number of base stations Metro Core

Distribution Rings

PON

P2P

Microwave BH

Access Link

WDMmm Wave

Core OCS

OLT

Long Haul

WDM-PON

Access OCS

Macro RH

Micro/pico RH

BBU/DC

Today

39

Siloed Aggregation

• Patchwork of single operator domains siloed into trees

• Need to go back to core for CoMP on separate trees

Metro Core

Long Haul

Access Link

WDMmm Wave

Core OCS

OLT

WDM-PON

Access OCS

Macro RH

Micro/pico RH

BBU/DC

CORD/BBU pool

Future 1

40

Mesh

• Dense mesh of fiber and fixed wireless connections at the edge

• Optically amplified switched edge fiber systems

• Multi-tenant, multi-domain at physical layer

Metro Core

Long Haul

Access Link

WDMmm Wave

Core OCS

OLT

WDM-PON

Access OCS

Macro RH

Micro/pico RH

BBU/DC

CORD/BBU pool

Future 2

41

Why Mesh?

• Latency improves by staying in physical layer• mesh = more physical layer paths

• East-West capacity for edge applications• CoMP radio, locality aware networks,

rapid mmWave handovers• Better resilience, disaster response• Easier to plan—network chooses the routes,

not the installer

• Less traffic in metro core

42

NYC Fiber

crowncastle.com

Low cost optical switches: PIC-Based ROADMs

• DuPONT PLC ROADM• Photonics Online 2005: “DuPont Photonics

Announces Most Cost-Effective ROADM Solutions Based On Planar Lightwave Circuits”

• JDSU PLC ROADM

44

Amplified optical switching has never left the ground

• Only PONs use fast (< 1 sec) switching in commercial systems

• 25 Years of research and no working solutions in the field!

• Complex channel interactions• Solutions too expensive• Lab prototypes don’t scale to field• Not just about devices: software

problem as well

45

1

10

100

1000

10000

100000

1000000

-4 -3 -2 -1 0 1 2 3 4 5 6 7

Commercial Optical System Switching

• Commercial optical packet switches

• < 10 nodes• Restricted to ring topo.

• Critical gap for metro-edge cloud networks

• Fast circuit switching• Metro Edge: 100-1000’s

nodes over 100’s km• Robust, stable, low cost

46

Switching Frequency, 1/s

Nod

es x

Dia

met

er (1

00km

)

ROADM

Optical ‘Packet’ SwitchingPONs

Amplified Optical Switching

10-4 10-2 1 100 104 106

Switch speed: Min. Sec. mSec µSec

It’s not about devices, its about how the entire system works in

the larger technology eco-system

47

• Harlem city-scale smart city testbed enabling experimentation in optical, wireless and edge cloud computing networks.

• Application domains include AR, VR, connected car, smart city (with high-bandwidth sensing), industrial IoT, …

• Experimentation platform for ultra high BW and low latency tightly coupled with edge computing

Augmented Reality

Smart City + Connected CarCloud

InfrastructureRoadsideAP

Roadway sensors & lighting

In-car guidance display

Image/Video

Industrial Control

48

COSMOS Smart City Testbed

• GHz mmWave capable SDRs with SDN optical whitebox connectivity

• Edge cloud facilities at large nodes and university data centers

• Smart intersections, gigabit centers, IoT toolkits

49

Mininet Optical Emulator• Emulate multi-layer networks

• Including transmission physics

• Run actual SDN code on emulated networks

• Validate against physical/lab networks and extend to larger scale

50

Servers & IP Routers

ROADM Fiber Network

Team lead R. Lantz

SDN Controllers

R. Lantz, et. al. OFC 2020

Conclusions

• Physical layer networks are transforming• Metro-Regional Edge Cloud & Wireless

Networks are Frontier for Optical Systems Research

• Focus on rich functionality, software control, & integrated photonics

• Need system & network scale research—not a single problem to ‘solve’

• 6G is the wireless-optical physical layer Internet

51

The Team

• Dr. Yao Li, Twitter

• Dr. Weiyang Mo, Juniper

• Jiakai Yu

• Shengxiang Zhu

• Tasha Adams

• Mariya Bhopalwala

• Farida Sari

• Haris Khan, Infinera

• Ian Tillman

• Christian Rios

• Aishik Biswas

• Aamir Quraishy

52

Collaborators

• G. Zussman, K. Bergman, Columbia University

• D. Raychaudhuri, I Seskar, Rutgers University

• M. Ruffini, TCD

• M. Veeraraghavan, R. Foley, R. Williams, UVA

• O. Sylvain, Fordham U.

• S. Foster, Georgetown U.

• N. Peyghambarian, R. Norwood, I. Djordjevic, B. Carter, U. Arizona

• C. Banks, B. Lincoln, Silicon Harlem

• R. Lantz

Thank You

Our Group:wp.optics.arizona.edu/dkilper/

COSMOS:www.cosmos-lab.org

CIAN:cian-erc.uawebhost.arizona.edu/

53

This work was supported by the NSF under grants #CNS-1827923, CNS-1650669, CNS-1650685, PFI-AIR-TT: 1601784, ECCS-1547406, CNS-1737453 and ECC-0812072