AAV Performance Report Readout

National E2E RAN Performance

Tim Zhang

Ashish Dhital

6/8/2010

AAV Architecture Overview

ETHERNET BACKHAUL CONFIGURATIONS

•Hybrid Iub architecture involves deploying parallel ATM/T1 and IP/Ethernet backhaul bearers to a given NodeB

•In the next two diagrams the traffic paths are illustrated

•Notice how the NodeB OAM traffic is routed in the two solutions

•This is critical in understanding the impact of different hybrid alarm constellations

•Also notice the different paths the HDSPA/HSUPA traffic takes in the two solutions

Confidential and Proprietary Information of T-Mobile USA 3

ATM IUB

RNC 1

RNC n

All RNCsat MSO

NMNET

RSP 1

RSPs supporting ATM NodeBs at

MSO

NodeB 1

NodeB k

All ATM NodeBs homed

to MSO

NSN NetAct servers

RSP mTDM

backhaul SONET

HSDPA/HSUPA

R99 + signaling

NodeB OAM (carries NodeB alarms)

Confidential and Proprietary Information of T-Mobile USA 4

Notice how the NodeB alarms are routed via the RNC using IPoATM over the T1s

HYBRID IUB

RNC 1

RNC n

All RNCsat MSO

NodeB 1

All hybrid / all IP based NodeBs homed to MSO

7609 #1

7609 #2

AAV Ethernetbackhaul

NMNET

RSP 1

RSPs supporting ATM NodeBs at

MSO

NSN NetAct servers

RSP mTDM

backhaul

NodeB n

SONET

Eth

ern

et

Ethernet Ethernet

JDSU VisionEMS/NetComplete servers

HSDPA/HSUPA

R99 + signaling

NodeB OAM (carries NodeB alarms)

CellSiteLAN OAM (carries NID alarms/KPIs)

Notice how the NodeB and NID alarms take different

paths with hybrid Iub.

Confidential and Proprietary Information of T-Mobile USA 5

HYBRID IUB HW CHANGES

Confidential and Proprietary Information of T-Mobile USA 6

NetEx

T-PIM

Cell site n

Cell site 2

Cell site 1

MSO

RSP

ES

C

ES

C

ES

C

ES

C

DS1 based existing TDM transport service

DACs

Optical SONET (OC3/12 channelised / concatenated)

RNC 1

NIS

NIS

NodeB n

FTIA

NodeB 1

FTIA

NodeB 2

FTIA

switchedethernet L2

transport service

Cell site n

Cell site 2

Cell site 1

MSO

E-OAMMSO NID

RSP

NP

GE

(A)

NP

GE

(P)

NP

GE

(A)

NP

GE

(P)

Cisco 7609 #2

ES

C

ES

C

ES

C

ES

CDS1 based existing TDM

transport service

DACs

Optical (or electrical) GigE/FE between TM and AAV’s NIDs

2 X 2 lag

Electrical 100BaseT between TM NID and NodeB

E-OAMNID

(JDSU)

E-OAMNID

(JDSU)

E-OAMNID

(JDSU)

ES

20E

S20

6548

6548

Optical GigE

ES

20E

S20

Cisco 7609 #1

Optical SONET (OC3/12 channelised / concatenated)

E-OAMMSO NID

QT600(1)

6548

6548

QT600(2)

RNC 1

NP

GE

(A)

NP

GE

(P)

NIS

NIS

NodeB n

FTIA

NodeB 1

FTIA

NodeB 2

FTIA

HYBRID IUB HW CHANGES

7

JDSUVisionEMS

NetEx

T-PIMHSDPA traffic flows from RNC’s NPGE IP addresses the NodeB UP IP address

HSUPA traffic flows from the NodeB UP IP address to RNC’s NPGE IP addresses

JDSUNetOptimise

NodeB’s user plane IP address

RNC’s user plane IP addresses

(1 per NPGE pair)

switchedethernet L2

transport service

Cell site n

Cell site 2

Cell site 1

MSO

E-OAMMSO NID

RSP

NP

GE

(A)

NP

GE

(P)

NP

GE

(A)

NP

GE

(P)

Cisco 7609 #2

ES

C

ES

C

ES

C

ES

C

DS1 based existing TDM transport service

DACs

Optical (or electrical) GigE/FE between TM and AAV’s NIDs

2 X 2 lag

Electrical 100BaseT between TM NID and NodeB

E-OAMNID

(JDSU)

E-OAMNID

(JDSU)

E-OAMNID

(JDSU)

ES

20E

S20

6548

6548

Optical GigE

ES

20E

S20

Cisco 7609 #1

Optical SONET (OC3/12 channelised / concatenated)

E-OAMMSO NID

QT600(1)

6548

6548

QT600(2)

RNC 1

NP

GE

(A)

NP

GE

(P)

NIS

NIS

NodeB n

FTIA

NodeB 1

FTIA

NodeB 2

FTIA

Ethernet Y.1731 CCM (Continuity Check Message) frames sent every 1 sec between MSO and cell site NIDs on CORENET VLAN monitors

live Ethernet performance.

E-OAM PM

8

T-PIM

JDSUNetOptimise

KPIs

AAV CONNECTIVITY ISSUES

• To identify NMNET/CORENET AAV VLAN/layer 1 connectivity issues to a cell site use

• E-OAM cell site NID and Node B ET-MFX11 HW alarms

• E-OAM FM and PM alarms

• RNC does not see the Ethernet circuit and 7609 AAV VLAN alarms will be disabled

• For investigation and fault isolation• Ping the cell site’s IP hosts

• Run rapid RFC2544 test to cell site NID

• Please refer to following links for detail AAV Troubleshooting procedure.

• Ericsson: Click here

• NSN: Click here

Confidential and Proprietary Information of T-Mobile USA

AAV Performance KPIs(from JDSU)

Intro to JDSU Performance Monitoring CFM Y.1731 Test Packets

TMO has chosen CFM Y.1731 as the standard for real time circuit testing.

The E-OAM NIDs send DL frames every second with a time stamp; in the other direction, the cell site NIDs send UL frames with time stamps.

When frames are received, the NIDs calculate the delay, evaluate loss, etc. This is on-going all the time. These tests are well defined in the Y.1731 standards and are called Continuity Check Message (CCM) checks.

DL and UL frames are sent and monitored on CORENET VLAN (not on NMNET VLAN). We only want to monitor user traffic.

NID’s will raise an alarm if there is a high error rate or loss of packets, and gives us KPIs that can be studied later.

If it finds problems it will generate an alarm for NetExpert and send data to T-PIM via NetOptimize.

CFM Packets Travel Between MSO NID and Cell Site NID’s.

AAV

Provider NIDTMO NID

TMO MSO NID

QT-600

RNC

Bearer Traffic

CFM Traffic

KPI Value Description

Frame Loss Rate: SLA threshold less than 0.0092%Max Frame Delay: SLA threshold less than 8msMax Frame Delay Variation: SLA threshold less than 2msAvailability: SLA threshold great than 99.995%

KPI Description

Throughput Average Traffic Throughput over Ethernet

Throughput/CIR (%)(Utilization)

Percentage of throughput relative to the CIR – 0 to 100%.CIR: Committed Information Rate in Megabytes per second

Availability (%) Derived from the report time calculation for the flow. The percentage of time available is the cumulative total of time in the date/time period minus the Unavailable Seconds (UAS).Availability is calculated based on Frame Lost Rate. If there is no Frame lost, then Availability is based on FD and FDV if they exceed the SLAs metrics.

Frame Loss Rate (%) Calculation based on the cumulative totals for frames lost and frames transferred over the date/time period.

Minimum Frame Delay (FD) Report time minimum FD value for all queried values over date/time period.

Average Frame Delay (FD) Report time average FD value for all queried values over date/time period.

Maximum Frame Delay (FD) Report time maximum FD value for all queried values over date/time period.

Minimum Frame Delay Variation (FDV) Report time minimum FDV value for all queried values over date/time period.

Average Frame Delay Variation (FDV) Report time average FDV value for all queried values over date/time period.

Maximum Frame Delay Variation (FDV) Report time maximum FDV value for all queried values over date/time period.

These SLA thresholds are designed for voice traffic over Ethernet. With the current implementation (only HS user plane payload over Ethernet), Frame lost rate is more critical than the other two KPIs.

Typical Worst Flows

Example of SLA Alarming

RAN Performance KPIs

Traffic

Throughput

Retransmission

RetainabilityAccessibility

Channel Switching

Latency

Confidential and Proprietary Information of T-Mobile USA 16

• HSDPA/EUL ThroughputThroughput increase

• PS LatencyPS Latency decrease

• HSDPA/EUL TrafficNo impact or slightly increase

• HS/EUL RetainabilityNo impact

• HS/EUL AccessibilityNo impact

• Voice/PS R99 PerformanceNo Impact

AAV Performance Report Demonstration

TPIM Report (1)

There is AAV JDSU report under TPIM Content Browser.

Confidential and Proprietary Information of T-Mobile USA 18

Search for “JDSU”

TPIM Report (2)

Confidential and Proprietary Information of T-Mobile USA 19

Report Name

Specify the report duration

Select level of the object

JDSU HierarchyRegion –> MSO-> AAV Vendor -> NID

Select report level and time level

BH is defined based on highest Ethernet throughput

AAV RAN KPI Summary Report (Beta Version)

Available in Liberty Reporting Server. Provide side by side RAN and JDSU report with RAN system hierarchy. Link:

Confidential and Proprietary Information of T-Mobile USA 20

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+RAN+KPI+SUMMARY+DAILY&rs:Format=HTML4.0

Filter by RNC or Cluster

Specify Region/Market

Select the Sites, Specify the report level

Specify Report Duration

Green: RAN KPIs Orange: AAV KPIs

Click the date to get hourly drilldown

Tip:Report can be exported as Excel

Sample: SE01680D

Confidential and Proprietary Information of T-Mobile USA 21

The side by side RAN and JDSU report provides an easy way to correlate the Ethernet issue with RAN performance.

AAV JDSU KPI Summary Report (Beta Version)

Available in Liberty Reporting Server. Provide side by side RAN and JDSU report with JDSU system hierarchy. Link:http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+JDSU+KPI+SUMMARY+DAILY&rs:Format=HTML4.0

Confidential and Proprietary Information of T-Mobile USA 22

Specify Report Duration

Specify the region and report level

Select object

JDSU system hierarchy:Region -> MSO -> AAV Vendor -> NID

Green: RAN KPIs Orange: AAV KPIs

OSS Vs. Insite Audit Delta ReportAvailable in Liberty Reporting Server. Provide AAV site audit for any discrepancy between Insite database and OSS parameter.http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fOSS+VS+INSITE+AAV+AUDIT+DELTA&rs:Format=HTML4.0

• AAV site in Insite: The site is defined as “Ethernet Completed” in Insite database.

• AAV Site in OSS: The site is configured to carry HS user plane on Ethernet in RAN system. (based on actual parameter setting)

Confidential and Proprietary Information of T-Mobile USA 23

Action items

AAV Performance DashboardAvailable in Liberty Reporting Server. Provides the AAV performance overview on region and market level.National Report (region level)

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+National+Dashboard&rs:Format=HTML4.0

Regional Report (market level)

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+Regional+Dashboard&rs:Format=HTML4.0

Confidential and Proprietary Information of T-Mobile USA 24

Known LimitationsSome reports on Liberty server are beta version. The purpose is to validate the report, receive market feedbacks, then get final implementation in TPIM.

Here are some known limitations in these reports.• There are some missing NSN KPIs in RAN summary report and JDSU summary report (Liberty server) due to the NSN old counter limitation in Common table. New common table will be deployed soon with new NSN counters.

• There is no Latency & Throughput distribution KPIs for NSN, due to lack of TPIM aggregate tables. Action has been taken by TPIM team.

• We are experiencing some JDSU missing data issue. It is the reason you will see some blank in JDSU KPIs. JDSU and TPIM team are working on this issue.

Confidential and Proprietary Information of T-Mobile USA 25

Question & Feedback Contacts

If you have questions or feedbacks, please send to your regional POCs.

South Region: Kien Tang

West Region: Christopher Lugos & Vivencio Luna

NE Region: Kartar Wadhwa

Central Region: Crisanto Espinoza

Confidential and Proprietary Information of T-Mobile USA 26

Experience Sharing

Experience SharingUnderstand the network behavior when Ethernet has problem.

When Ethernet is complete down for long period

Ericsson:

FMX will be triggered to flip the site from AAV to ATM after Ethernet alarm has been received for longer than 1 min.

• May not see the performance impact on KPIs, if FMX works properly.

• May cause 100% access failure if FMX doesn’t work.

• If you have to manually lock the AAV in case FMX doesn’t work, you have to lock the AAV on RNC side, NOT NODEB.

NSN:

NSN system will still try to allocate the HS over Ethernet. Obviously, the HS allocation will be failed. Then the PS will be downgrade to R99, then setup this R99 call over ATM. After a few seconds on R99 RAB, the system will up switch the call to HS again with ATM backhaul.

• HS Access Failure increased up to 100%

Confidential and Proprietary Information of T-Mobile USA 28

Experience Sharing

RAN Performance degradation due to unstable Ethernet

HS Drop rate may increase. The impact on RAN performance depends on the severity of Ethernet issue and how frequent the issue is happened (Continuously frame lost, or occasionally)

Confidential and Proprietary Information of T-Mobile USA 29

Useful Links

Documentation

AAV Performance SharePointhttp://snpshpc1/functional/npp/NP/Pages/AAV.aspx

• RAN/JDSU KPI Specs

• TPIM Database Specs

• JDSU PM Statistic Description

• Performance Discussion Board

Ericsson/NSN AAV Training Material (From Andrew): http://snpshpc1:8687/SiteDirectory/UMTS/NetworkReadiness/LastMile/DE/Shared%20Documents/Presentations/Presentations%20under%20construction/

• AAV Architecture

• Configuration Management

• Fault Management

• Performance Management

AAV Online TrainingCourse Name: Ericsson Hybrid Iub CM/FM/PM in UMTS

NSN Hybrid Iub CM/FM/PM in UMTS

AAV Design Documenthttp://snpshpc1:8687/SiteDirectory/UMTS/NetworkReadiness/LastMile/DE/default.aspx

AAV CIQ Documenthttp://snpshpc1:8687/SiteDirectory/UMTS/NetworkReadiness/Transport/default.aspx

  Confidential and Proprietary Information of T-Mobile USA 31

Performance Report

TPIM Serverhttp://sqpn1mcl.eng.t-mobile.com/tcm_portal/TPIMLogin.jsp#

Liberty Server

AAV RAN KPI Summary Report (Beta)http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+RAN+KPI+SUMMARY+DAILY&rs:Format=HTML4.0

AAV JDSU KPI Summary Report (Beta)

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+JDSU+KPI+SUMMARY+DAILY&rs:Format=HTML4.0

OSS Vs. Insite Audit Report

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fOSS+VS+INSITE+AAV+AUDIT+DELTA&rs:Format=HTML4.0

AAV Performance Dashboard

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+National+Dashboard&rs:Format=HTML4.0 (National Report)

http://10.39.8.149/ReportServer/Pages/ReportViewer.aspx?%2fNational+Performance+Reports%2fRAN%2fAAV+REPORTS%2fAAV+Regional+Dashboard&rs:Format=HTML4.0 (Regional Report) Confidential and Proprietary Information of T-Mobile USA 32

Thank you!