High Speed Data NetworksMobile Device
Application Testing Challenges
Ensuring Mobile Applications Success!
Spirent Wireless SeminarApril 2007
Proprietary - © 2006 Spirent Communications.
Agenda
Part I – Applications Enablers - Mobile Data Performance Analysis
Understanding how handset and device development can impact your upgrade to HSPAMethodology for analysing the performance of devices required to support HSDPA Identifying timescales for availability and widespread deployment of HSDPA and HSUPA enabled devicesDetermining realistic performance of devicesSpirent’s APEX UMTS Data Performance Module
Part II – Mobile Multimedia Application Performance
Proprietary - © 2006 Spirent Communications.
Need to answer questions such as, what is the …Actual time for data transfer?
Ability of the Device to accurately measure downlink channel quality and report CQI?
ApplicationApplication GoodputGoodput vs. Physical LayerPhysical Layer throughputthroughput rate?
Device performance under long service usage period?
Mechanisms to help benchmark UE data performance?
Statistical data needed to help establish “golden UE” performance?
Characterizing Data Performance
Proprietary - © 2006 Spirent Communications.
Why Test Goodput/Throughput Performance?
Layer 1 Layer 1 Throughput Throughput
Rates (Mbps)Rates (Mbps)
Air Air InterfaceInterfaceCDMA 2000
CDMA 2000UMTS
UMTS
Release 99
Release 99
HSDPA with
HSDPA with
QPSK only
QPSK only
HSDPA with
HSDPA with
16QAM16QAM
Mob
ile A
pplic
atio
nsM
obile
App
licat
ions
153 Kbps153 Kbps
2.42.4 3.13.1
384 Kbps384 Kbps1.81.8
14.414.4
SMSSMS MMSMMS
File File
SharingSharing
Image Image
SharingSharing
Short Video
Short Video
burstburst
Streaming
Streaming
VideoVideo
key enablerkey enabler of datadata--centric centric applications applications such as Mobile Videosuch as Mobile Video
major drivermajor driver for faster air interface technologies
no current benchmarksno current benchmarksto test Mobile Video throughput
GoodputGoodput/Throughput /Throughput ……
EVDO
EVDO
Release 0
Release 0EVDO
EVDO
Release A
Release A
Methodology for analysingthe performance of devices required to support HSDPA
Proprietary - © 2006 Spirent Communications.
Conformance testingConformance testing offers a starting point with some limitations
The realreal--world deploymentworld deployment scenarios require adapting to variablevariablenature of the networks, and the UE test conditions!
MeasurableNot Addressed
CriticalMulti-hour stress tests
Quantified Empirical Data
Pass/FailCriticalUE-to-UE comparative data
MeasurableNot Addressed
Critical“Safety margin”
MeasurableNot Addressed
CriticalL1, L7 throughput
MeasurableNot Addressed
CriticalThroughput sensitivity to key parameters
VariableFixedVariableTransport Block size
VariableFixedVariableMAC-hs parameters
Performance Testing
Conformance Testing
Real World Testing
Real-world conditions are constantly changing.
Conformance testing has limited value in assessing QoE.
Performance Testing simulates real-world test scenarios.
HSDPA Testing Challenges - Conformance Limitations
Proprietary - © 2006 Spirent Communications.
HSDPA is a highly dynamicdynamic and adaptiveadaptive technologyData rate variesvaries continuously as a function of the channel qualitychannel quality
Mobile devices constantly competecompete for a shared channelshared channel
Network has multiple parametersparameters for specific implementation
Data throughput volume is affected by protocol layer interdependenceinterdependence
ReliableReliable and repeatablerepeatable lablab--basedbased UE testing is needed to ensure necessary HSDPA test coverage prior to deployment …
HSDPA Testing Challenges – Variable Technology Characteristics
Identifying timescales for availability and widespread deployment of HSDPA and
HSUPA enabled devices
Proprietary - © 2006 Spirent Communications.
The Importance of Data Services
By all measures, data continues to grow as a percentage of totalOperator revenue
Timely deployment of HSPA networks will further fuel this growth
Source: GSA
Proprietary - © 2006 Spirent Communications.
HSPA Deployment: The Operators
121121 Operators deploying HSPDA in 5555 countries
63 commercial HSDPA networks in 38 countries have already launched
Deployment commitments are truly global:AmericasAmericas: 33 countries/territories
APACAPAC: 12 12 countries/territories
EuropeEurope: 3131 countries/territories
Middle East & AfricaMiddle East & Africa:99 countries/territories
Source: GSA GSM/3G Market/Technology Update, October 1, 2006Source: GSA GSM/3G Market/Technology Update, October 1, 2006
Proprietary - © 2006 Spirent Communications.
HSPA Deployment: The Mobile Devices
407 WCDMA/HSDPA devices launched by 44 suppliers
62 devices in the market (from 19 suppliers) that support HSDPA
Source: GSA GSM/3G Market/Technology Update, September 30, 2006Source: GSA GSM/3G Market/Technology Update, September 30, 2006
Proprietary - © 2006 Spirent Communications.
Deployment: Faster and Better, HSUPA
HSDPA 7.2 MbpsRX DiversityRX Diversity – Integral part of higher data rates
7.2 Mbps-capable devices already in the market
Firmware upgradeFirmware upgrade required to enable 7.2 Mbps
Commercial driver for network availability still unclearLate 2007? 2008?Late 2007? 2008?
HSUPAPre-commercial devices expected at beginning of 2007beginning of 2007First commercial data card launches in mid 2007mid 2007“Handheld” HSUPA devices begin to appear in 20082008
Determining Realistic Performance of Devices
Proprietary - © 2006 Spirent Communications.
Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Static Conditions
Downlink rate of four HSDPA CAT 6 devices tested using APEX UDP under two sets of static channel conditions
Physical data throughput measurements shows expected results:Under favorable conditions all devices performed close to maximum rate (3.6 Mbps)
Under weakened conditions there is a drop in throughput rate ranging between (2.2 – 2.5) Mbps
Physical data throughput is not enough to provide a root-cause of poor performance.
Additional metrics such as Median CQI or the Mac-HS statisticsare helpful
Proprietary - © 2006 Spirent Communications.
Device (UE) physical layerhysical layer performance is consistent under strong RFstrong RF conditions and it begins to diverge as RF weakensRF weakens
under staticstatic conditionsconditions
Case Study - Applications Performance Differences under Static Conditions
Ask for a copy of our latest WhitepaperAsk for a copy of our latest Whitepaper……
APEX UDP resultsAPEX UDP results……
0
500
1000
1500
2000
2500
3000
3500
4000
HS-PDSCH = -3 dB / Ior = -60 dBm HS-PDSCH = -6 dB / Ior = -80 dBm
Static Channel Conditions
Configuration = HS-PDSCH = -3 or -6 dB, Ior = -60 or -80 dBm (No Noise)TTI = 1, # of H-ARQ = 6, CQI = Fixed at 22, TBS = Set per TR 25.214
Data
Thr
ough
put (
Kbps
)
UE AUE BUE C UE D
Throughput Differences @ weakened channel conditions(>10% when UE B is compared to UE D)
Reduction in Maximum Throughput @ weakened channel conditions (>30% drop on all CAT 6 UEs tested)
Proprietary - © 2006 Spirent Communications.
Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Dynamic Conditions
Downlink rate of four CAT 6 devices tested using APEX UMTS Data Performance Module with four different fade models (PA3, PB3, VA30, PA3, PB3, VA30, VA120VA120)Performance of devices varied significantly under different fade modelsfade modelsand varying code powersvarying code powers
E.g. the throughput rate difference between UE D and other UEs is increased to 30%30% under PB3 & VA30PB3 & VA30 fading conditions
Variations observed between devices with the same chipsets, and between devices with different chipsets.
171344.05UE D
15932.5UE C
15930.51UE B
14878.68UE A
Median CQI
Throughput (kbps)
Device Under Test
UDP Downlink TestVA30 Fade Model with Ior/Ioc = 10 dB
Commercial devices are providingHSDPA capabilities, but their
performance differencescan be significant
Proprietary - © 2006 Spirent Communications.
Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Static Conditions
End userEnd user’’s quality of experiences quality of experienceExample: UE B with 37.9% NAK at Ior = -80 dBm could lead to blank frames and/or blockiness in an application such as VT or Video Streaming
Accurate channel quality detection and reporting (CQI) is critical to:
Mac-HS Statistics (ACK / NACK / Stat DTX Performance) for CAT 6 UEs
Table 1. Mac-HS Statistics (ACK / NACK / Stat DTX Performance) for CAT 6 UEs
030.0369.97202507.98D
031.468.6212459.44C
037.9562.05212223.86B
032.9767.03202402.26A
Static Conditions – HS-PDSCH = -6 dB / Ior = -80 dBm (weakened conditions)
00100243583.99D
00100243583.99C
00.2599.75243575.08B
00.199.99243583.76A
Static Conditions – HS-PDSCH = -3 dB / Ior = -60 dBm (favorable conditions)
Stat DTX %NACK %ACK %Median CQI
Throughput
(Kbps)
UE Under Test
Proprietary - © 2006 Spirent Communications.
Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Static Conditions Cont.
Overall network optimization and experience of all usersOverall network optimization and experience of all users
Accurate channel quality detection and reporting (CQI) is critical to:
Example: UE BUE B’’s s CQI compared to UE DUE D is asking for more resources while its higher NACK % NACK % indicates that it won’t be able to handle it. Under live conditions, this could mean that network is sending too much data to a user that might be in poor coverage resulting in high error rates, wasting shared resources.
Mac-HS Statistics (ACK / NACK / Stat DTX Performance) for CAT 6 UEs030.0369.97202507.98D
031.468.6212459.44C
037.9562.05212223.86B
032.9767.03202402.26A
Static Conditions – HS-PDSCH = -6 dB / Ior = -80 dBm (weakened conditions)
00100243583.99D
00100243583.99C
00.2599.75243575.08B
00.199.99243583.76A
Static Conditions – HS-PDSCH = -3 dB / Ior = -60 dBm (favorable conditions)
Stat DTX %NACK %ACK %Median CQI
Throughput(Kbps)
UE Under Test
Proprietary - © 2006 Spirent Communications.
Device (UE) Device (UE) application layerapplication layer performance varies significantly performance varies significantly under the under the dynamicdynamic fadingfading and and noisenoise conditionsconditions
Ask for a copy of our latest WhitepaperAsk for a copy of our latest Whitepaper……
APEX UDP resultsAPEX UDP results……
Case Study - Applications Performance Differences under Dynamic Conditions
0
200
400
600
800
1000
1200
1400
1600
1800
2000
PA3 PB3 VA30 VA120
Fade Model (Per 34.121 - Appendix D)
Configuration = HS-PDSCH = -3 dB, Ior = -60 dBm, Ior/Ioc = 10 dBTTI = 1, # of H-ARQ = 6, CQI = Based on UE Reports, TBS = Set per TR 25.214
Dat
a Th
roug
hput
(Kbp
s)
UE AUE BUE C UE D
Clear Performance DifferencesClear Performance DifferencesUE D shows a 30% increase in data throughput rates when
compared to the other UEs tested.
Proprietary - © 2006 Spirent Communications.
Dynamic Dynamic conditions such as varying code powervarying code power levellevel directlydirectlyaffectsaffects the device application layer performance.
0
200
400
600
800
1000
1200
1400
1600
HS-PDSCH = -3 dB HS-PDSCH = -4 dB HS-PDSCH = -5 dB HS-PDSCH = -6 dB
Code Power Level (HS-PDSCH Level) - For Fade Model PB3 (Per 34.121 Appendix D)
Configuration = HS-PDSCH = -3 to - 6 dB, Ior = -80 dBm, Ior/Ioc = 10 dBTTI = 1, # of H-ARQ = 6, CQI = Based on UE Reports, TBS = Set per TR 25.214
Dat
a Th
roug
hput
(Kbp
s)
UE AUE BUE C UE D
Code Power Reduction yields Throughput Degradation1 dB = 16% (on average) & 2 dB = 29% (on average)
Case Study - Applications Performance Differences under Dynamic Conditions Cont.
APEX UDP resultsAPEX UDP results……
Spirent’s APEX UMTS Data Performance Module
Proprietary - © 2006 Spirent Communications.
APEX Test Manager
Building Blocks
WCDMA/HSDPA CDMA 1X/EV-DO
Platform
Test ModulesUMTSData
Performance
Wireless Multimedia
Analysis
ReliableReliable and repeatablerepeatable lablab--basedbased device testing to ensure necessary HSDPA & R99 throughput performance prior to deployment …
Ensuring Mobile Applications Success with APEX UMTS Data Performance Module
ModularModular CustomizableCustomizable AutomatedAutomated
Proprietary - © 2006 Spirent Communications.
Answering Key Performance Questions
Packet Data RoutePacket Data Route
Fading
Multi-path
Delay
AWGN
UDP
TCP
FTP
User Defined
APEX UMTS DATA Performance ModuleAPEX UMTS DATA Performance Module
SpirentTest Manager
SR5500Channel Emulator
SR3420Network Emulator
RadioRadioAccessAccessNetworkNetwork
How well does application
layer goodputperform?
How well does application
layer goodputgoodputperform?
How does fading impact throughput/ goodput?
How does fading impact throughput/ goodputgoodput?
How well does physical layer
throughput perform?
How well does physical layerphysical layer
throughput perform?
For HSDPA & R99 DevicesFor HSDPA & R99 Devices……
Proprietary - © 2006 Spirent Communications.
Spirent Test Manager & UMTS Data Performance
Test Module
FTP ServerUDP Server
LAN
Test Client Laptop
SR3420BSS Simulator
RFRF
Architecture & Instruments
Controller PC
SR5500Fading Simulator
RXDiversity
Proprietary - © 2006 Spirent Communications.
The SR3420 WCDMA Network Emulator is …Built on the latest platform based on SpirentSpirent’’s global leadership in s global leadership in lablab--based wireless network emulationbased wireless network emulation
Designed specifically to address highhigh--speed dataspeed data
Upgradeable to track evolving industry requirements
Real-Time Network Emulation
SR3420 is built upon a real-time state machine
Resulting in a “Network Emulation much closer to a real network than others”
Provides greater HSDPA performance at all layers of the protocol stack
Proprietary - © 2006 Spirent Communications.
The SR5500 Wireless Channel Emulator is …Built on SpirentSpirent’’s bests best--inin--class channel class channel emulationemulation solution for real-life deployment scenarios …
RF impairments fully exercise, for example…Adaptive coding and modulation
HARQ with soft combining
Real-Time Channel Emulation
Allows integrated RX Diversity within the APEX UDP system
Supports all 3GPP fade models
Dynamic Environment Emulation (DEE)- provides real-network modeling capability
Proprietary - © 2006 Spirent Communications.
Spirent’s APEX Platform Expert Tools
NINI’’ssTestStandTestStandSoftware Software EngineEngine
SpirentSpirent’’s s Enhanced Enhanced Front EndFront End
Open & Open & CustomizableCustomizable
Easy & Easy & Comprehensive Comprehensive Test CreationTest Creation
Proprietary - © 2006 Spirent Communications.
Reduced ramp-up time
EasyEasy--toto--useuse, interactiveinteractive GUI requires minimal training
APEX UMTS Data Performance Module Offers Major Savings
Reduced test execution time
Repeatable, realRepeatable, real--worldworld conditions in a controlled lab environment
Reduced test implementation time
IntegratedIntegrated, turnturn--key solutionkey solution and powerful Test Manager environment allows rapid test implementation