WPP Study Group Tutorial

• Group created at Vancouver meeting to create test definitions and methods for 802.11

• Presentations today will cover WPP from several points of view– Large user– IC vendor– Testing community

Agenda

• Bob Mandeville: Iometrix

• Don Berry: Microsoft

• Mike Wilhoyte: Texas Instruments

• Kevin Karcz: UNH-IOL

• Jason A. Trachewsky: Broadcom

• Fanny Mlinarsky: Azimuth Systems

• Round Table Discussion and Q&A

802.11 Requirements forTesting Standards

Bob Mandevillebob@iometrix.com

WPP

• What is the need for 802.11 metrics

• What problems will they help solve?

• Who will the primary users be?

• How do we go about creating new metrics for wireless?

Two Approaches to CreatingTesting Standards

• IETF (BMWG)– Based on two-step

approach to definitions:

• Terminology document (all relevant functional characteristics are defined)

• Methodology document

• This method is most appropriate for performance testing

• ATM Forum, Metro Ethernet Forum– Based on ratified

standards documents– Each test definition is

referenced to standards source text

• This method is most appropriate for conformance testing

IETF BMWG Test Standards Templates

• Terminology Definition Template:– Term to be defined. (e.g., Latency) – Definition: The specific definition for the term. Discussion: A brief

discussion about the term, it's application and any restrictions on measurement procedures.

– Measurement units: The units used to report measurements of this term, if applicable.

– Issues: List of issues or conditions that effect this term.– See Also: List of other terms that are relevant to the discussion of this

term.• Methodology Definition Template:

– Objectives– Setup parameters– Procedures– Measurements– Reporting formats

Conformance OrientedTest Methods Template 1/2

Test Name Name derived from reference document

Test Definition ID

A punctuated alphanumeric string assigned to each defined requirement and test procedure couple using the following convention: ‘one to three letter abbreviated source document name’ . ‘section number’ - ‘paragraph number in the section from which requirement is derived’. This number always figures as the last number of an ID. Ethernet Services Model=M; Ethernet Services Definitions=S; Traffic and Performance Parameters for SLSs=T. Example: M.6.1-4

Reference Document

Source

Reference document and section (and paragraph when useful for clarity)

Test Type Functional, Conformance, Interoperability or Performance

Test Status Normative, optional, additional

Requirement Description

Brief description of the service requirement that the device must or should satisfy

Description of DUT/SUT

Type of Ethernet frame forwarding Device Under Test (DUT). Common designations used in this document are: CE Device (Customer Equipment Device); UNI Device (User Network Interface Device); MEN Device (Metro Ethernet Network Device). A UNI device may be considered as a kind of MEN ‘border’ device.

Conformance OrientedTest Methods Template 2/2

Test Object Succinct description of test purpose

Test Bed Configuration

Succinct description of test bed configuration

Test ProcedureSuccinct description of the test procedure with mention of the test stimulus and expected output

Units

Units can be time units, rates and counts in integers such as milliseconds, frames per second and numbers of frames transmitted or received. For the most part units used are defined in RFCs 2285, 2544, 2889

VariablesVariables such as frame length, test duration, numbers of interfaces under test must be described

ResultsDescription of the textual, numerical and/or graphical format in which to display test results

RemarksDescription of any particular observations that might effect the test result

Sample List of 802.11 Terms to be Defined by Category:

Requirements forTesting Standards

• Roaming Test– A practical example which shows poor performance

related to a lack of test standards definitions• To roam a 802.11 a/b/g device will:

1. disassociate from one AP

2. search for a stronger RF signal from another AP

3. then associate and authenticate with that A

4. resume normal data transmission

• Roaming can fail due to:– transient RF conditions

– the time that APs and devices take to complete the four step roaming process

Test Configuration

Azimuth Director®

RF Shielded Enclosure

Access Point 1

RF Shielded Enclosure

Access Point 2R

R

R

R

STM with roaming client(DUT)

Channel 2Monitor STM(Airopeek)

Channel 1Monitor STM(Airopeek)

Combiner

Azimuth W800 Chassis

RFPM-1 RFPM-2

Hub

Fiber OpticRepeater

Test Network

Test Procedure• Roaming Test recorded:

Total Roaming Time = Decision Time + Switch Over Time

– The Decision Time is the time it took the NIC to stop attempting to transmit packets to AP 1 after the attenuation of the RF signal

– The Switch Over Time is the time it took the NIC to complete its association with AP2 after it stopped attempting to transmit packets to AP 1

– During the Decision Time cards recognized the signal attenuation and invoked proprietary algorithms to adapt their rates to slower speeds.

– Switch Over Time ends when the NIC receives the AP’s acknowledgement to its association request.

– This time should only be recorded as valid if data traffic from the NIC to the AP successfully resumes transmission after association.

Test Results

802.11a/b/g Roaming Times

0

5

10

15

20

25

30

35

802.11a APa-NICa

802.11a APa-NICb

802.11a APa-NICe

802.11g APb-NICb

802.11g APb-NICc

802.11g APc-NICb

802.11g APc-NICc

802.11b APd-NICa

802.11b APd-NICc

802.11b APd-NICe

Sec

on

ds

Total Roaming Time

Switch Over Time

Best to Worst Roaming Times

0

5

10

15

20

25

30

35

Best Case SingleVendor (802.11a)

Best Case VendorMix (802.11b)

Average (802.11a) Average (802.11b) Average (802.11g) Overall Average(802.11a/b/g)

Worst case(802.11g)

Sec

on

ds

Test Conclusions• Need to break out Decision Time and Switch over Time• Switch Over Times are as low as 60 milliseconds and averages a

little over 700 milliseconds over all the combinations excepting two outliers which took over 8 seconds.

• In the majority Decision Time is the largest contributor to the overall Roaming Times.

• Packet traces show that most implementations of the rate adaptation algorithms maintain transmission at the lowest 1 Mbps rate for several seconds after loss of RF signal has been detected.

• These algorithms will need to be revisited to deliver quality roaming.• Test standards for measuring roaming times can make a significant

contribution by aligning all vendors and users on a common set of definitions

• This applies to roaming but also to a large number of other undefined terms

Challenges of Operating an Enterprise WLAN

Don Berry

Senior Wireless Network Engineer

Microsoft Information Technology

Microsoft’s CurrentWLAN Network

– 4495 Access Points– 1 AP per 3500 sq/ft– ~15 Million sq/ft covered in 79 countries– 70,000 users– 500,000+ 802.1x authentications per day –

EAP-TLS– Supports 5.5 and 11Mbps only

Wireless Service Challenge

• What is “Wireless Service”?• How is it measured?• What factors impact Wireless Service?• How do you improve Wireless Service?

Wireless Service and Support

• Service Goals– Make Wireless service equivalent to wired– Offer unique mobile services

• Operational Goals– Reduce operational costs– Minimize support personal

How can WPP Help?

• Produce criteria that reflect the client experience

• Offer information that can compare different environments – Enterprise, SOHO, home

Desired Outcome of WPP:A Perspective From a Si Provider

Mike Wilhoyte

Texas Instruments

Key Issues We Face Relevant to WPP

• Supporting Customers w/ Custom Test Fixtures– Are often uncontrolled and therefore repeatability is questionable– May introduce unintentional impairments and therefore don’t

effectively isolate the feature under test– May unintentionally push beyond the boundary of the

specification– May stress the system beyond what any user or evaluator will do– May overlook other critical areas of system performance

• The complexity of the specification has grown since the days of 11b and more than ever, performance is setup dependent– Are tests really apples-to-apples?

These Issues Result in:• Confusion over unexpected test results

• Resource drain

A Real Example: Customer ACI Test Fixture

RF Shield Room – non Anechoic

AP

DUTSMIQ

People Observing the Results

AP

AP

AP

AP

Over 30 Active APs

TCP/IP

Test:Plot TCP/IP throughput with increasing levels of interference from the SMIQ

Issues With This ACI Test Fixture

• Metal walls in shield room producing multipath making the test results depend even more on the position of the laptop (in a fade?)

• People (2.4 GHz RF absorbers) in the room• Over 30 AP’s active which may couple into the RF front-

end (even though it’s cabled) of the test AP• SMIQ produces a non-realistic signal since the carrier is

always present even though it may be triggered externally– There are ways around this

• The test AP is not isolated from the interference and its behavior will affect the test result of the DUT– Rx performance in the same interference– Deferral behavior in the Tx (CCA) is affected– Rate adjustment behavior

Can you imagine trying to repeat any test result from this fixture in YOUR lab?

A “Better” ACI Test Fixture Testing the STA

APSTA

out

PA

DUTAP

Attenuator

30 dB20 dB

Anechoic Chamber 1 Anechoic Chamber 2

in

clp

30 dB

Interferingnetwork swepton channels1-11

Anechoic Chamber 4

Channel 6

PA isolates interfering network and isnot affected by traffic in chambers 3,4

clp

out in

Anechoic Chamber 3

Atten

pad pad

A “Better” ACI Test Fixture Testing a Network (AP+STA)

APSTA

out

PA

DUTAP

Attenuator

30 dB20 dB

Anechoic Chamber 1 Anechoic Chamber 2

in

clp

30 dB

Interferingnetwork swepton channels1-11

Anechoic Chamber 3

Channel 6

PA isolates interfering network and isnot affected by traffic in chamber 3

Desirable Outcomes of WPP

• Develop a minimal set of test metrics that are relevant to key performance parameters such as:– Robustness– Throughput/Capacity– Range

• Develop a Set of Test Best Practices that:– Produce repeatable results– Achieve the right balance between complexity and cost– The industry will use

UNH-IOL perspective on WLAN Performance testing

Kevin Karcz

March 15, 2004

A Quick Background

• UNH-IOL Wireless Consortium primarily has focused on interoperability and conformance tests for 802.11, not performance testing

• Have generated traffic loads to observe a DUT’s behavior under stress, but not specifically to measure throughput or related parameters.

• However, QoS is inherently about providing performance while sharing limited resources– Optimization of: Throughput , Range, Delay & Jitter– Constrained by:

• User resources: CPU load, DC power• Community resources: Available spectrum, aggregate users

–

Examples of performance tests

• PHY layer– Multi-path fading channel emulation using a

Spirent SR5500 fader. • What fading models should be used?

• MAC layer– Throughput varies with Traffic Generator used– CPU load differs significantly for between

different vendors. Much greater than CPU load for a typical Ethernet device.

Clear methods of testing are needed…

• As we start measuring more performance metrics

• Can each layer of the network be measured independently?

• Which metrics need to look at the interaction of multiple layers?

• Hassle of real world scenario testing vs. a PHY test mode?– Black box testing requires DUT to authenticate and

associate with test equipment and interact at the MAC layer, not just the PHY layer.

Some gray areas of testing• Throughput

– Is throughput measured as a MAC layer payload? At IP layer? TCP or UDP layer?

– One DUT may have better PER measurements at the PHY layer than a 2nd DUT, but may get worse throughput if it’s rate selection algorithm is poor.

– Difficult to maintain consistency in an open (uncontrolled) environment• Can throughput be measured in a cabled environment without an antenna?• What if the DUT has a phased array antenna?• What if the device is mini-PCI and inherently has no antenna?

• Range test– What if a higher TX level causes higher adjacent channel interference and brings

the aggregate throughput down for a neighboring BSS? • Power consumption

– Is this just the DC power drain at the cardbus card interface?– Should CPU load be included if the DUT implements much of it’s MAC

functionality on a host PC?• Roaming

– Quickest time: 1 STA, 2 APs on same channel – More realistic: AP reboots, Multiple STAs roam to new AP on new Channel

Why WPP role is important to UNH-IOL?

• Open standards are desired for the basis of test suite development

• Defining test parameters and standardization of Test Scenarios makes comparison of ‘Apples to Apples’ easier

• IEEE focuses on the technical content • Our interest is the testing, not determining

how results are utilized

Why WPP should define the tests

• UNH-IOL follows IEEE PICS for test cases

• More detailed info for test results– Cases: PDA/laptop/AP weight test results

differently

Example criteria weighting

Throughput

Delay & Jitter

DC power

Roaming time

Laptop ++ + ++ +

AP ++ ++ N/A N/A

VoIP phone

0 +++ ++ ++

Comments on Wireless LAN Performance Testing And

Prediction

Jason A. TrachewskyBroadcom Corporation

jat@broadcom.com

Topics

• Test Categories for WPPTest Categories for WPP

• Some Test Configurations

Test Categories for WPP

• Deciding what parameters are to be considered is the challenge.

• How do we transform user perception of performance into a set of repeatably-measurable quantities?–Throughput and Range (what environments?)–Frame latency–Visibility of APs

Test Categories for WPP

• How do we transform user perception of performance into a set of repeatably-measurable quantities?–Delays in association/authentication–Host CPU utilization–Ability to roam without loss of connections–Etc.

Test Categories for WPP

• Basic PHY/RF Measurements–Transmitter Parameter Measurements

• TX EVM or Frame Error Rate (FER) with Golden/Calibrated Receiver

• Carrier suppression• Carrier frequency settling

Test Categories for WPP

–Receiver Parameter Measurements• RX FER vs. input power

–Flat channel (controlled through cabled/shielded environment)

–Controlled frequency-selective channels (known multipath power-delay profile)

–Antenna measurements• cable/feed losses (S11 and S21)

• gain vs. azimuth and elevation angle

• One can easily take a great receiver design and blow away all gains with a bad antenna or lossy feed!

Test Categories for WPP

• MAC Layer Measurements–rate adjustment behavior

• specific parameters? test conditions?

–association and roaming behavior• specific parameters? test conditions?

–frame latency–layer-2 throughput with encryption–host CPU cycles consumed?

Test Categories for WPP

• Layer-4 Measurements–UDP frame loss rate and latency vs. received

power• flat or frequency-selective channels?

–TCP throughput vs. received power• flat or frequency-selective channels?

Test Categories for WPP

• Open-air Measurements–Open-air measurements are always subject to

imperfectly-known time-varying multipath power-delay profiles.

• There is substantial variation at 2.4 and 5.5 GHz over 10s of msec.

Test Categories for WPP

–We have established that frequency-selectivity due to multipath can result in higher-loss channels having higher capacity than lower-loss channels.

• The capacity of the channel can vary rapidly.• This is a more significant problem for systems like

802.11a and 802.11g which include a large number of possible rates to better approach the practical capacity.

• (The problem won’t get easier for future WLAN standards.)

Test Categories for WPP

• Open-air Measurements–What can we do?

• Try to perform as many measurements as possible with cable networks.

• Perform open-air measurements in an area in which the distance between transmitter and receiver is small compared with the distance between either transmitter or receiver and any other object. I.e., strong LOS.

–Helpful but not sufficient, as even small reflections affect channel capacity.

Test Categories for WPP

• Open-air Measurements–What can we do?

• Give up and perform a large ensemble of measurements and gather statistics.

Channel Measurement Block Diagram

• Scope provides 10Mhz reference clk for all systems• 3 long interconnect cables connect tx and rx side• Filt module includes LNA

Scoperx

Filt

Filt

Filt

Filt

10 Mhz clk

linuxcontroller

DC PWR

10/100 hub

ARBtx0

ARBtx1

ARBtx2

ARBtx3

10

Mhz

clk

ARBtxctl

I / Q

10/100 hub

gpib

ethernet

UPSUPS

ethe

rnet

trigger

Time- and Frequency-Selective Fading

-25 -20 -15 -10 -5 0 5 10 15 20 25-76

-74

-72

-70

-68

-66

-64

-62

-60

-58

subcarrier index

gain

(dB

)

time evolution of TX_ANT0 to RX_ANT0 channel

0 msec.10 msec.20 msec.30 msec.40 msec.50 msec.60 msec.

> 10-dB change in received signal power in some bins over 60 msec.

Topics

• Test Categories for WPP

• Some Test ConfigurationsSome Test Configurations

System Block Diagram

–Main boardtest system–Transmit test system

boardtestsystem

TransmitTest Filter

Mux

MicrowaveSpectrum Analyzer

controller(attenmach)

GPIB or DIO line

ethernet

RF cable

GPIB

Boardtest System

DUT mux-2

power meter

atten REFcoupler coupler mux-3

mux-6

SA

5V,12V,GND

controller(attenmach)

GPIB or DIO line

Ethernet

RF cable

GPIB

RF “Loop” Block Diagram

Split /

Combine

6-dB Loss

Split /

Combine

6-dB Loss

Golden Node

DUTATTEN

ATTEN

ATTEN

ATTEN

ATTEN

Does WPP specify RF test fixtures?

Does WPP specify fading channel emulators (no!) or a set of fading channel profiles (maybe!).

Multipath Channel 2

0 50 100 150 200 250 300 350 400 450 500-130

-120

-110

-100

-90

-80

-70

-60

-50

-40

-30

delay (nsec)

path

gai

n (d

B)

channel 2 (t = 133 nsec.)

Example fixed multipath channel power-delay profile.

Comments on Wireless LAN Performance Testing And

Prediction

Fanny MlinarskyAzimuth Systems

fanny_mlinarsky@azimuth.net

Ethernet vs. WiFi

Wired test metricsWired test metricsRFC 2285,2544,2889RFC 2285,2544,2889

Wireless test metricsWireless test metrics

Security Security

Test Metrics

OfferedLoad

PacketSize

# ofClients

% ofPower Save

Clients

% LoadOf Assoc/De-assoc/Re-assoc

Encryption

RTS/CTSFragmentationAuthentication

Packet forwardingPacket forwarding

RoamingRoaming

Rate adaptationRate adaptation

Behavioral Behavioral

WEPTKIPAES

EAP-TLSEAP-TTLSPEAPLEAP

QoS QoS

Forwarding Rate Measurement

1500

1750

2000

2250

2500

0 120 240 360 480 600

Time (Seconds)

Fo

rwa

rdin

g R

ate

(F

ram

es

/Se

c)

1500

1750

2000

2250

2500F

orw

ard

ing

Ra

te (

Fra

me

s/S

ec

) Open airOpen air

Controlled RFControlled RF

Controlled Test Environment

• If measurements are not repeatable then the test is invalid

• Open air creates unpredictable test conditions due to interference and multipath

• Shielded and cabled test environment may be necessary for some measurements

Summary• The IT manager’s question: “How

well do mobile computing solutions perform in the enterprise?”

• The dilemma: Standard ‘wired’ benchmarking techniques won’t give the right answer

• Verifying function and performance in the inherently unstable wireless space calls for a new methods & metrics

• The answer: New methods to test and measure every aspect of wireless protocols

• Wireless metrics outnumber traditional Ethernet metrics 5:1