An Evo lu t ion o f Network Te lemet ry

Tal Mizrahi

Marvell

Telemetry BoF

3/23/2018 3

Network Telemetry

Are we reinventing the wheel?

Self-driving Networks High reliability

Easy-to-manage

Software-defined Networks

High speed

Network Telemetry

Easy-to-troubleshoot

3/23/2018 4

Ping / Traceroute Reinventing the wheel?

3/23/2018 5

Old-School Passive Monitoring

Counters Per port

Queue State

Latency

Per flow

Per queue

… …

Reinventing the wheel?

3/23/2018 6

Carrier Network OAM Reinventing the wheel?

OAM Protocols

IETF ICMPv4

IEEE 802.1agITU-T Y.1731

IETF ICMPv6 IETF IPPM IP OAM

Higher

Layers

Layer 3

Layer 2

Layer 1

ITU-T Y.1711

MPLS OAM

IEEE 802.3ah

MPLS / PWE3

OAM

Ethernet

OAM

ITU-T G.8113.1MPLS-TP OAM

IETF MPLS-TP

OAM

IETF LSP-Ping

MPLS OAM

IETF PWE3

VCCVIETF

BFD

Active measurement / monitoring:

Control Message

Control Message

3/23/2018 7

Piggybacked Measurement Reinventing the wheel?

Telemetry is piggybacked onto data packets

IOAM / INT AM-PM

3/23/2018 8

Piggybacked Metadata – IOAM / INT Reinventing the wheel?

IOAM / INT Domain

Analytics Server

Switches push local

metadata into header:

delay, queue state, …

Telemetry

Info

IOAM (IETF)

INT (P4)

+SAI TAM

PacketsSent: 10,000

Packets Lost: 500

Packets Received: 9,500

3/23/2018 9Analytics Server

Data Center Network

Servers Servers

Reinventing the wheel?AM-PM: Alternate Marking –Performance Measurement

(RFC 8321)

Counts number of

packets received

Counts number

of packets sent

3/23/2018 10

Reinventing the wheel?Network Telemetry: An Evolution

Ping

TraceroutePassive

Monitoring

Carrier

OAM

IOAM / INT

AM-PM

NO !

Active Telemetry

3/23/2018 11

Marvell’s Network Telemetry Solutions

Passive Telemetry

CountersFlow-based

Probes IEEE 1588

Metadata

Grab-’n-GoTimestampingSelective

Probing

Per-hop Telemetry

Programmable

Logic

Packet

Metadata

Incoming

PacketModified

Packet

Incoming

Packet

Packet Processing

Packet

Metadata

Fixed

Programmable

3/23/2018 12

Metadata: Grab-’n-Go

Marvell Prestera

Packet

Processing

Packet

Metadata

Incoming

PacketOutgoing

Packet

3/23/2018 13

Timestamping Everything

✔ Periodic probing

✔ AM-PM

✔ TimeFlip

IEEE 1588 Synchronized Clock

Marvell Prestera

3/23/2018 14

Time.Sec Time.Frac1 * … ** … *

Periodic range

... timestampvalue

1 secondaction

timestamp field2 field3 field4 …

TCAM

Marvell Prestera Switch

header /

metadata

AM-PM using TimeFlip

Per-Flow Congestion Detection using AM-PM

loss and delay è congestion is detected

3/23/2018 15

3/23/2018 16

Selective Exporting

✔ Drop

✔ Congestion

✔ 1 of N

✔ Periodic

Analytics Server

Telemetry

Info

Marvell Prestera

IOAM / INT Domain

Analytics Server

3/23/2018 17

IOAM/INT: Periodic Exporting

Export one packet per second

Export last ms of every second

Telemetry

Info

Per flow / per port

IOAM / INT Domain

Analytics Server

3/23/2018 18

IOAM/INT: Adaptive Exporting

Event-driven:

✔ On drop

✔ On congestion

✔ On high rate

IOAM / INT Domain

Analytics Server

3/23/2018 19

Combining IOAM/INT with AM-PM

AM-PM as a trigger for exporting telemetry info.

Per-hop Telemetry

Challenge: export telemetry info without the expensive overhead of IOAM/INT.

3/23/2018 20

The Network Telemetry Toolset

CountersFlow-based

Probes IEEE 1588

Metadata

Grab-’n-GoTimestamping

Selective

Probing

Marvell Prestera

3/23/2018 22

References

[1] Mizrahi, T., Vovnoboy, V., Nisim, M., G. Navon, and A. Soffer, “Network Telemetry Solutions for Data Center and

Enterprise Networks”, Marvell white paper, 2018.

[2] Brockners, F., Bhandari, S., Pignataro, C., Gredler, H., Leddy, J., Youell, S., Mizrahi, T., Mozes, D., Lapukhov, P.,

Chang, R. and D. Bernier "Data Fields for In-situ OAM", draft-ietf-ippm-ioam-data-00, work in progress, 2017.

[3] C. Kim et al., “In-band network telemetry (INT)”, P4 consortium, 2015.

[4] Fioccola, G., Capello, A., Cociglio, M., Castaldelli, L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi, “Alternate

Marking method for passive and hybrid performance monitoring”, RFC 8321, 2018.

[5] Mizrahi, T., Rottenstreich, O. and Y. Moses, “TimeFlip: Scheduling Network Updates with Timestamp-based TCAM

Ranges”, IEEE INFOCOM, 2015.