timing over packet demarcation

The Access Company

White Paper

Timing over Packet Demarcation An Innovative Approach to

Synchronization Performance Monitoring

in Packet Networks

Alon Geva, Advanced Technologies Manager

Yaakov Stein, Chief Scientist

RAD Data Communications

Abstract

There is a pressing need to distribute accurate timing, i.e., frequency and/or

Time of Day (ToD), across Packet Switched Networks (PSNs) for applications

such as cellular backhaul. Highly accurate delivery of timing over PSNs is

challenging, due to their asynchronous nature and the need to overcome

impediments, such Packet Delay Variation (PDV). This challenge increases for

operators leasing transport services from carriers, as operators have little

control over the timing packets as they traverse the carrier’s network and, as

a result, timing recovery accuracy cannot be guaranteed. While Ethernet OAM

and performance monitoring tools are available for carriers and wholesale

provides, no such demarcation functionality is presently defined specifically

for the timing distribution service.

This paper reviews the main issues involved in timing over packet (ToP)

demarcation and provides best practices for ToP demarcation and

performance monitoring.

White Paper: Timing over Packet Demarcation

Contents

Introduction ........................................................................................................................... 2

Legacy Synchronization Performance Monitoring ............................................................ 4

Synchronization over Packet Networks using PTP ........................................................... 5

PTP Performance Monitoring Today ................................................................................... 7

PTP Demarcation - The Future in PTP Performance Monitoring .................................... 9

Useful Practices of ToP Demarcation ............................................................................... 12

Summary .............................................................................................................................. 15

References ........................................................................................................................... 16

© 2011 RAD Data Communications Ltd 1


Introduction

There is a pressing need to distribute accurate timing, by which we mean frequency and/or Time of

Day (ToD), across Packet Switched Networks (PSNs) for applications and devices that require it.

One of the most demanding cases relates to cellular backhaul. The second and third generations of

cellular networks’ base stations require provisioning of frequency with error less than ±50 ppb

(parts per billion) compared to the network’s Primary Reference Clock (PRC). Base stations

belonging to the next generation of cellular networks, such as the UMTS-TDD and the TD-SCMA,

further require provisioning of ToD with error less than 1.5 microseconds compared to Universal

Time Coordinated (UTC) (see [1]). Future cellular networks will have even stricter requirements in

order to enable new features such as soft handoff between neighboring base stations and Multiple

Input Multiple Output (MIMO). ToD accuracy here will be required to be at a level of hundreds of

nanoseconds!

Delivery of timing of such accuracy over PSNs is challenging, due to their asynchronous nature. In

particular, PSNs based on a conventional Ethernet physical layer are not designed with timing

distribution in mind, and require additional mechanisms for its distribution. Packet-based timing

distribution protocols send packets containing timing information from a timing master, usually

referred to as Primary Reference Clock (PRC) or Primary Reference Time Clock (PRTC), to a timing

slave located close to the application device requiring the precise timing. The Network Time

Protocol (NTP) is the most prevalent synchronization distribution protocol when high accuracy over

a well engineered network is required (e.g., for cellular backhaul).

A major impediment to the proper function of packet-based timing distribution protocols is the

Packet Delay Variation (PDV) that timing packets experience when traveling through the PSN. This

PDV is primarily due to the timing packets being queued in network elements along their path. The

queuing delay at each network element depends on the momentary traffic load when the timing

packet needs to be queued, and thus varies randomly over time. Hence, the overall PDV, being the

sum of all the queuing delays, varies non-deterministically. In particular, the aforementioned

symmetry assumption cannot hold at any given time, and special filtering mechanisms must be

employed to mitigate this effect. Despite these mechanisms, end-to-end frequency and time

distribution performance cannot always be guaranteed to meet end-application requirements.

2 © 2011 RAD Data Communications Ltd


The problem becomes more severe when the service provider relies on a carrier’s infrastructure to

deliver its traffic to the end application. In such cases, the service provider has little control over the

timing packets as they traverse the carrier’s network and timing recovery accuracy cannot be

guaranteed.

Service providers continuously monitor Quality of Service (QoS) parameters experienced by user data

traffic in order to ensure conformance to an agreed upon Service Level Agreement (SLA). This is

usually accomplished by network demarcation devices running Operation Administration and

Maintenance (OAM) protocols (such as ITU-T Recommendation Y.1731 for Ethernet) to monitor the

provided service. OAM functionality may consist of Fault Management (FM) and /or Performance

Management (PM) functions. Fault Management consists of defect detection and reporting

mechanisms, such as Continuity Check (CC), Connectivity Verification (CV), Forward Defect Indication

(FDI), and Backward Defect Indication (BDI). Performance Management involves measurement of

metrics such as Packet Loss Ratio (PLR), one-way delay (1wD), Packet Delay Variation (PDV),

throughput, and availability.

OAM mechanisms have been defined for many network and service types, from low-rate TDM links,

through SONET/SDH networks, to Ethernet Virtual Connection (EVC) services. However, no such

demarcation functionality is presently defined specifically for the timing distribution service. As a

result, service providers rely on performance metrics derived from the timing slave to infer the timing

service’s performance. This can be problematic for several reasons. First, timing slaves, employing

different timing recovery algorithms, may display different performance, leading to inconsistent

results. Second, timing slaves are typically located at the far edge of the service provider’s network,

and thus their performance does not capture the individual behavior of specific segments in the

service provider’s network, but rather a sum of all degradations along the path from master to slave.

Third, the timing slave may be embedded in a network element (e.g., a switch or a router), or end-

application device (e.g., a cellular base station), and not provide visibility to the performance

functions that need to be monitored.

An alternative that addresses all of the aforementioned shortcomings would be to place a

standardized timing slave at, or close to, the required demarcation points in the network, and use it

to monitor the performance at the desired locations. However, as will be described in the next

section, this option would necessitate provisioning a dedicated timing distribution service from the

master to each such location, incurring additional bandwidth and logistics support. Furthermore, this

dedicated timing distribution flow will, in general, experience network impairments different from the

timing flow that it is intended to monitor, introducing an inherent error and inconsistency.



Legacy Synchronization Performance Monitoring

Timing distribution, just like any other service delivered over a telecommunication network, needs to

be continuously monitored in order to validate its correct behavior and conformance to the required

performance level. The on-going migration from synchronous Time Division Multiplexing (TDM)

networks, which inherently distribute frequency, to asynchronous Packet Switched Networks (PSNs)

introduces major challenges for service providers. Not only must they now employ protocols to

distribute frequency, and in many cases ToD, they must also be able to monitor the quality of such

timing services.

Frequency distribution in synchronous networks is a mature technology that relies on the stringent

frequency accuracy requirements of data symbols transmitted over physical links. Frequency

information is automatically distributed downstream from a master clock to the end-application on a

hop-by-hop basis. Each network element along the path recovers the frequency of the symbols it

receives, enslaves a local oscillator to this frequency, and uses this oscillator as its frequency

reference for downstream transmission. This results in a timing (or synchronization) chain, consisting

of local node clocks connected by physical links. At the top of the timing chain usually stands a highly

accurate and stable autonomous clock called the Primary Reference Clock (PRC), usually an

autonomous atomic clock. Thus, the frequency at every point along the timing chain is said to be ‘PRC

traceable’.

Monitoring frequency accuracy and stability at a specific network element may be accomplished by

comparing the frequency of its local clock to a trusted frequency reference, such as another stand-

alone atomic clock or frequency output from a GPS receiver. The comparison is often carried out by

continuous phase comparison of the two signals. The outcome of this continuous phase comparison

can then be used to calculate frequency quality metrics, such as the Maximum Time Interval Error

(MTIE) and Time Deviation (TDEV) ( [2]). Measurement equipment designed for this task is readily

available, and is used by service providers to monitor and validate the quality of frequency

distribution at various locations within their network, and thus to trigger corrective actions whenever

anomalous behavior is detected.

A legacy SONET/SDH frequency monitoring system is presented in Figure 1. The monitored clock

signal is introduced into a frequency monitoring device (wander analyzer) that is also referenced with

an accurate reference clock (i.e. a PRC). The outcome phase error measurement (Time Interval Error -

TIE) is offloaded to a PC (or any other metric calculation device) for deriving MTIE and TDEV frequency

accuracy/stability metrics. These metrics may then be used to verify the performance of the

frequency distribution system and detect abnormalities.



Figure 1: Typical SONET/SDH frequency monitoring system

Synchronization over Packet Networks using PTP

Distribution of timing (frequency and/or time of day) in packet networks is achieved by a timing-over-

packet protocol. Such protocols function by sending timing packets between a timing master and a

timing slave. The timing master is usually connected to a highly accurate and stable frequency/time

reference (PRC or PRTC). The timing packets must be formatted as valid packets for the network, and

are transported from master to slave in the same fashion as all packets in the network.

When it comes to very accurate distribution of frequency and time over packet networks, the most

prominent timing-over-packet protocol is the Precision Time Protocol (PTP) version 2 (also known as

IEEE 1588v2 or IEEE 1588-2008 - [3]). The 2nd version, ratified in 2008, is Telecom oriented in the

true sense and includes various features and mechanisms making it an attractive apparatus for

Telecom applications.



In PTP, timing information is carried by three aspects of the timing packets, namely timestamps

carried explicitly in the packets, the precise transmission time, and the precise reception time of the

packets (by the master or slave device). The difference between the latter two properties is sensitive

to Packet Delay Variation (PDV) that obscures the master-to-slave and slave-to-master propagation

latencies, complicating the timing recovery process.

Figure 2 depicts the packets transactions for PTP. The transaction is initiated by the PTP master

(which is timed by a PRC/PRTC) sending a timing packet to the PTP slave. Timestamp t1 is recorded by

the PTP master as the first timing packet passes a well-defined (master) reference point on its way

out to the network. Timestamp t2 is recorded by the PTP slave as soon as the first timing packet

passes the slave’s reference point after reaching the PTP slave. It should also be noted that t1 is

explicitly carried by the first timing packet, and thus becomes known to the PTP slave (if the PTP

master is incapable of accurately timestamping the outgoing packet on-the-fly, a follow-up packet

containing t1 can be sent).

Figure 2: PTP packets transaction

In the reverse direction, timestamp t3 is recorded by the PTP slave as the second timing packet passes a

well-defined (slave) reference point on its way out to the network, and timestamp t4 is recorded by the

PTP master as the second timing packet passes a well-defined (master) reference point on its way in

from the network.



After the second timing packet has been received (by the master) and timestamp t4 is recorded, a third

timing packet is sent by the PTP master to the PTP slave, informing it with the value of t4. At this time

the PTP slave knows all four timestamps {t1, t2, t3, t4} and is able (under symmetry assumptions) to

calculate the network delay between the master and itself, and thereafter derive the instantaneous

time-error of its local clock. As before, the slave can correct its local clock’s frequency and/or time to

match that of the master.

PTP Performance Monitoring Today

Similar to what we discussed for TDM networks, in order to monitor timing performance at some point

along the network path between master and slave, a comparison must be carried out between a

physical timing signal (i.e., a clock signal) recoverable at that point and a trusted timing reference.

However, as timing distribution over PSN is not necessarily performed on a hop-by-hop basis,

intermediate network elements do not necessarily perform timing recovery and do not necessarily

maintain local clocks. To produce the required physical timing signal at the desired location, the

monitoring system itself could perform timing recovery, essentially reproducing the process carried out

by the slave. Alternatively, the timestamp information collected at intermediate locations could be used

to directly calculate packet network performance metrics (specifically PDV-based metrics) that could be

used to predict the attainable timing performance. The use of such PDV-related metrics is relatively

new, and is presently the subject of intensive research. Several promising metrics such as minTDEV and

MAFE (currently planned to be incorporated into the next version of ITU-T Recommendation G.8260

( [4]) have been proposed and shown to possess desirable properties.

Unlike synchronization monitoring in TDM networks that relies on the existing timing flow, producing a

physical timing signal at desired intermediate locations cannot be performed by simply tapping onto the

content of existing timing over packet flows. One way to overcome this is to introduce a dedicated

timing packet flow from the timing master to the intermediate location, since existing timing-over-

packet protocols require active interaction between master and slave. Such a PTP timing monitoring

arrangement is depicted in Figure 3, where the PTP master serves both the PTP slave and the ToP

monitoring device using two separate timing flows. In this scheme, the ToP monitoring device appears,

from the timing master’s point-of-view, to be just another PTP slave. The overall timing bandwidth

increase is directly proportional to the number of active PTP monitors installed in the network.

Moreover, as separate independent timing flows are used for the PTP slaves and the ToP monitoring

devices, the true PDV experienced by the PTP slaves is not really being monitored.



Figure 3: PTP performance monitoring using an intermediate slave device

A second approach would be to use standard network OAM protocols (e.g. ITU-T Y.1731 [5]) to

monitor network PDV between two demarcation points. Such an approach is presented in Figure 4

where standard OAM packets sent from one MEP (Management Entity Group End-Point), located at

one edge of the network, to a seconds MEP, located at the opposite end, are used to measure

relevant PDV information and compute the required PDV metrics based on that. The problem is that,

here too, the measurements are being carried based on synthetic OAM packets rather than the

original PTP flow we want to monitor. Moreover, any Transparent Clock function located between the

demarcation points will not be taken into account as it will be completely agnostic to the OAM

packets.

Figure 4: PTP performance monitoring using standard OAM techniques



PTP Demarcation - The Future in PTP Performance Monitoring

The new patent pending approach enables monitoring quality and achievable performance of a timing

distribution service in packet networks at any given point in the network, without requiring

introduction of additional traffic.

At the heart of the new approach is a network device, hereinafter referred to as a Timing-over-Packet

Demarcation Function (ToPDF). The ToPDF contains two network interfaces, to be called the Master-

Facing Interface (MFI) and the Slave-Facing Interface (SFI), which enable it to be connected as a

“bump on the wire” at any desired location in the network. The network interfaces can be of any

type depending on the specific physical and link layer technologies employed. The ToPDF is

completely transparent and does not modify monitored packets in any way. Furthermore, the ToPDF

does not interact with other network elements, and does not generate any additional traffic load on

the network.

The ToPDF has the ability to identify incoming and outgoing timing packets (from both its network

interfaces), to extract any information from these packets, and to record accurate timestamps. Based

on the extracted information and the timestamps, the ToPDF can perform calculations in order to

derive the following parameters:

• Timing status of the network

• Timing metrics that might be useful in predicting the currently achievable timing distribution

quality over a given path

• Actual recovered frequency and time achieved in this specific location of the network

Such information can be then used by the service provider to better assess the condition of the

timing distribution in its network.

The ToPDF can be placed at any desired location in the network. In some cases the ToPDF would be

located at (or near) a Customer Edge (CE) switch or router, forming a demarcation point between the

provider network and the customer network. In others, it will be placed at the border between two

service provider networks, or at strategic points inside a single network.



We further assume that one or more timing packet flows traversing the ToPDF in both directions

(going from master to slave and back). The ToPDF is able to detect the dedicated timing packets that

traverse it in both directions, but does not alter them in any way (and certainly does not alter non-

timing packets). However, the ToPDF performs the following non-intrusive operations on timing

packets:

• Extract information from the timing packets, in particular, flow identifiers and timestamps

• Timestamp the timing packets as they traverse well-defined ingress/egress reference points,

based on its local clock

It should be noted that such non-intrusive detection, data-extraction and timestamping mandates the

use of dedicated hardware processing, in order to minimize the additional delay introduced by the

ToPDF and make it completely fixed and symmetrical (This requirement can be easily accomplished for

the PTP case, using an End-to-End Transparent Clock).

In the following discussion, a “timing flow” will mean any persistent flow between a timing master

and a timing slave that distributes timing information from master to slave. Moreover, a “raw

timestamp set” will mean a specific combination of four timestamps {t1, t2, t3, t4} that contains the

complete timing information exchanged between a given timing master and a given timing slave.

Figure 5 presents the ToP demarcation concept with a PTP distribution system for a typical cellular

backhaul scenario where the cellular service provider is buying wholesale transport services from a

local carrier. In this example, the PTP master is part of the carrier network (part of the wholesale

service) and the PTP slave is located at the base station (part of the customer network). The ToPDF is

embedded in the carrier network’s PE router that delimits the two networks, with access to the PTP

distribution timing flows. Thus, it can identify a specific timing flow and generate its own local

timestamps (based on its local clock) referenced to some pre-defined timestamping reference point

on its MFI. Thus, when the entire packet transaction between the PTP master and PTP slave is over

and the slave has the required set of timestamps {t1, t2, t3, t4} to calculate instantaneous time-error

between itself and the master, the ToPDF will hold the required set of timestamps {t1, t’2, t’3, t4} to

calculate instantaneous time-error between itself and the master. Additionally, using this set of

timestamps, the ToPDF can now derive all the required 1-way/2-way network delay and PDV metrics.



Figure 5: PTP ToP demarcation

The ToPDF will also easily handle any intermediate TC nodes by looking at the Correction Field (CF) in

the header of the traversing PTP packets. As described in Figure 6, by extracting the CF values from all

the PTP messages comprising a complete PTP transaction (Sync, Delay Request and Delay Response),

the ToPDF has all the information it needs to take the TC delay compensation for both the

master ToPDF ( ) and ToPDF master ( ) network paths.



Figure 6: PTP ToP demarcation with TCs

Useful Practices of ToP Demarcation

In many cases, end-to-end PTP frequency and time distribution is the only real possibility for the near

and interim future, resulting in a PDV-sensitive solution whose bottom line performance cannot be

always guaranteed. This poses a nasty ‘headache’ for the service provider: how can the

time/frequency distribution service be used in such a way that would guarantee the required level of

performance? More importantly, how can we monitor the service in order to be able to warn that the

service has been degraded, isolate problems and tell where in the network the degradation occurred?

The answer to these questions might seem difficult at first glance. Clearly, as was already discussed,

packet timing flows cannot be naturally monitored by legacy time/frequency-error monitoring devices.

Moreover, with standard end-to-end PTP distribution, only the slave device at the end of the

distribution chain can be monitored, leaving the entire network section under a thick curtain of

uncertainty, and in many other cases even that is not true in reality (for example, when the cellular

base station does not offer any external monitoring/probing capabilities for the integrated PTP slave).



The good news is that using the new ToP demarcation concept the above questions can be mapped

into the following almost equivalent one: Where in our network should we deploy ToP demarcation

entities in order to get:

a. Best coverage of the network (the timing distributing part)

b. Optimal dissection of the network for best detection of fault conditions

Understandably, ToP demarcation entities may be located in many different network locations along

the timing flow(s) distribution path and, indeed, such practice may be extremely useful in order to

isolate timing problems. For example, Figure 7 depicts a PTP timing flow (from a PTP master located

at the 1st provider network to a PTP slave located in the customer network) that traverses three

networks: The first network is operated by provider #1, the second network is operated by provider

#2, and the third network is operated by the customer. By placing ToPDFs at the (two) demarcation

points of these networks, as well as another ToPDF at the edge of the customer network (its SFI

directly connected to the ‘First-Mile’ link), one can estimate the contribution of each of the networks

to the timing distribution degradation, and in particular one can isolate the source of major

performance degradations.

The last ToPDF, located close to the end-application device that comprises the PTP slave, experiences

PDV similar to the PTP slave and thus should provide a reliable estimate of the overall end-to-end

timing recovery performance. This can be exploited by the cellular service-provider to appreciate the

current condition of the PTP slave even when it is embedded in the end-application device (cellular

base station in this case) and its status/recovery outcome is not directly accessible.

The second ToPDF, located at the demarcation point between the provider networks and the

customer network, experiences the PDV introduced by the provider networks only. Bidirectional PDV

metrics computable by ToPDF2, at this point, can be used by the providers to ensure the quality of

service they provide, and by the customer to monitor the service it is being provided.

The first ToPDF, located at the demarcation between the first and second provider networks,

experiences PDV introduced by provider #1’s network only. Bidirectional PDV metrics computable by

ToPDF1, at this point, can be used by the two providers to predict the timing performance that would

have been attained by a PTP slave served only by the first provider.



Figure 7: Multiple ToP demarcation entities along a synchronization distribution chain

The PTP performance monitoring approach described here is only one possible practice out of many

that could be used to closely monitor PTP time/frequency distribution. Clearly, the selected

approach needs to match the underlying network and timing distribution architectures. The

appealing thing about the ToP demarcation technology is its high adaptation capability to almost

any given network device, from large aggregation switches down to SFPs (see Figure 8), as well as

to any timing distribution topology. All that is really needed is to connect the network-

element/module containing the ToPDF to a ‘bump on the wire’ on the desired network link.

Figure 8: ToPDF elements



From this point on, the ToPDF will continuously monitor the selected timing flows, sending its

findings and collected data to a central NMS/PM portal that will synchronize, integrate and analyze

all collected data (from all the ToPDFs), and supplying the user with concurrent, on-going and

comprehensive information about the synchronization distribution status in its network (Figure 9).

Figure 9: Multiple ToP demarcation entities along a synchronization distribution chain

Summary

RAD has developed an innovative PTP performance monitoring technique allowing carriers and service

provides to closely monitor the Timing over Packet distribution in their networks, making sure it is

working well and supplying the required level of performance.

Contrary to existing solutions that are only end-to-end, RAD’s PTP PM solution can provide deep

insight to the contribution of each network segment to the overall service degradation, allowing the

user to quickly isolate problems and take proactive measures to solve them. For more information,

please contact [email protected].


mailto:[email protected]



References

[1] 3GPP TS 25.402 version 5.2.0 Release 5,

[2] ITU-T Recommendation G.810 (08/96), definitions and terminology for synchronization networks.

[3] IEEE 1588-2008 (July 2008), IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems.

[4] ITU-T Recommendation G.8260 (08/2010), Definitions and terminology for synchronization in packet networks.

[5] ITU-T Recommendation Y.1731 (07/2011), OAM functions and mechanisms for Ethernet based networks.

North America Headquarters RAD Data Communications Inc. 900 Corporate Drive Mahwah, NJ 07430 USA Tel: (201) 529-1100, Toll free: 1-800-444-7234 Fax: (201) 529-5777 E-mail: [email protected] www.radusa.com

International Headquarters RAD Data Communications Ltd. 24 Raoul Wallenberg St. Tel Aviv 69719 Israel Tel: 972-3-6458181 Fax: 972-3-6498250 E-mail: [email protected] www.rad.com

The Access Company

The RAD name and logo is a registered trademark of RAD Data Communications Ltd. © 2011 RAD Data Communications Ltd. All rights reserved. Subject to change without notice. Catalog no. 802478 Version 11/2011

www.rad.com