lte drive-test-process

© 2010 Comba Telecom. All Rights Reserved

www.comba-telecom.com

1

LTE Drive Test Process

Paul HO

MBA Macq., B.Eng(Hons), M.I.E.T.

TMS

2

Agenda

Network Roll-out Flow

LTE drive Test process

Define Scope of work (SOW)

Define Methodology

Project Planning

Kick-off Meeting

DT Tools Preparation

DT Tools Testing and Mock Report

On-site Pre-DT Checks

eNB On-air & Alarm report

Delivery of DT work

Filing and nomenclature

Post Processing (Reports)

Confirmation / Acceptance

3


LTE

Business

Case

LTE

Planning and

Design

Nominal Site

selection and

RF Testing

Site

Acquisition

Site

Installation

Works

Cluster of

sites

completed

eNB testing

& on-air Drive Testing

Drive

Testing

Reporting

Optimisation

Commercial

Launch

Legend:

Planning Stages

Installation Stages

Drive Testing and Optimization

Commercial Launch

1 2 3 4 5

6 7 8 9 10

Planning Stages

• This stage is done by the Operator or vendor

Installation Stages

• Comba can potentially be involved in site preparation and

some form of civil works

Drive Testing and Optimization

• This is an area where Comba can participate actively.

4

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




5

Drive Test Process Drive Test Reporting Legend

Define Scope

of Work

(SOW)

Define

Methodology

Kick – Off

Meeting

DT Tools

Preparation

On – Site

Pre - DT

checks

Project

Planning

LTE Drive test process

Drive

Testing

8

DT Tools

Testing and

Mock report

Filing and

nomenclature

Post

Processing

(Reports)

eNB On-Air &

Alarm report

Delivery of

DT work

8a

8b

8c

8d

8e

8f

8g

8h

8i

8j

9

For

Appro

val and F

ine t

unin

g

Comba

Operator

+ Comba

Operator

Aft

er

DT

, check n

o a

larm

s d

uring D

T

Confirmation/

Acceptance

Planning Preparation Execution Documentation

9a

Drive

Testing

Reporting

9

6

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




7

8a Define Scope of Work (SOW)

Defining Scope of Work

Understand the nature of the project. (i.e. Total greenfield, greenfield

overlay, re-farming or vendor swap)

What is operator’s objective?

To measure for PASS/FAIL criteria post optimisation?

To measure as a base-line as part of their delivery BEFORE/AFTER a

delivery?

To measure the performance of an existing

Size of project. (i.e. number of sites, macro or micro, and even IBS)

Size of geographical coverage

Size of cluster of sites (How many clusters and how many sites in

clusters)

Number of times of drive test required per clusters. (i.e. to cover what

roads and how many times)?

What time to do drive test?

What is the duration of each drive test.

What type of applications to test? What throughput to test?

8


Preparing a Scope of work (SOW) document

All work to be delivered must be clearly spelt out in detail, with inclusions

or exclusions.

Comba’s role in this SOW must be detailed

Operator’s role in this SOW must be detailed

3rd Party’s role in this SOW must be detailed

Key person Point of contact between (Comba – Operator – third party)

must be detailed.

Working level interaction must be detailed

Escalation channels must be detailed (Example of a grading system)

Critical problem – Escalate to respective key person within 6 hours.

Major problem – Note down and bring up in project meetings, or within 2 days.

If un-resolved within 1 week, raise it to Critical status. All key person must be

notified.

Minor problem – Improvise and try to solve, else note and bring up during

meetings.

Comba (TMS can help) to prepare a checklist of these SOW so

as to be prepared for technical discussions with operator.

9


Escalation channel

Operator Comba 3rd Party

GM/ac Dir VP Planning Proj. Mgr

Cluster mgr Proj. Mgr

OSS mgr

Team

Team

Team

Esc

alat

ion c

han

nel

10

LTE Architecture

Source: UMTS LTE Network, Services, Technologies, and

Operation; Althosbooks; Lawrence Harte

Having an overview of LTE architecture is key to understanding data

services and it’s respective testing methodology.

11

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




12

8b Define Methodology

Executing on the prepared Scope of work (SOW) document

Methodology is basically a framework to spell out the way in which to

carry out the work.

Execute

SOW

Map route

planning

Prepare

Tools

Define DT

rules

Define Services

and Testing

Procedure

Config tools

Tools set up

Define

Reporting

format

Define

acceptance

Criteria

13

Define Methodology

Executing on the prepared Scope of work (SOW) document

Ranging of 15~25 sites per cluster excluding in-building sites

Each cluster must belong to the same S-GW

Every cell should only belong to one cluster

Obtain the customer’s inputs on what is customer’s objective. (i.e. KPIs,

etc.)

Map route planning

Clusters to be clearly demarcated on map. Routes to be numbered.

Use Mapinfo or other tools to have the clusters/routes clearly identified.

Boundaries of clusters to be clearly described, in words.

Usually main road and streets with 2 car lanes wide (both directions)

All cells in cluster need to be driven through

Route intersection between borders needed

Timing of drive including traffic jam not exceeding 6 hours

Timing of drive tests to be pre-determined and agreed, i.e. (Morning,

afternoon, Evening, night, midnight)

14

Define Methodology

Map route planning

These descriptions should be carefully prepared, kept and

complemented with map drawings.

Each cluster drive test team should be given these maps and drive

routes descriptions.

This is a sample spreadsheet table that must be prepared or obtained from customer

Route number Starting Ending

Comments

1 Amk Ave 3 intersection with Amk Ave5 Drive along Amk Ave 3, Turn right

when reaching Amk industrial park

Roadworks, be small

detour.

Drive along Amk industrial park (for

200m)

Turn left at Blk 305 on xxx road.

2

15

Define Methodology

Define services to be tested

Voice, Video, FTP, HTTP, FTP, Ping, etc.

Mode of service, HSPA, HSDPA, LTE.

These tables and pass/fail criteria must be established.

This is a sample spreadsheet table that must be prepared or obtained from customer

Services Data

rate

Latency Set up time Pass / Fail

criteria

Long

call/Short

call

MOC /

MTC

Comments

Voice

HTTP

(internet)

Video

FTP (DL or

UL)

Ping RTT

16

Define Methodology

Testing during routes

Per route – 1 set of Nemo/TEMs to be set up in the following config: -

1 x Voice port – with UE1

1 x Http port – with UE2

1 x Video port – with UE3

Per route – 1 set of Ping and FTP

This is a sample spreadsheet table that must be prepared

Services Route Number Drive number Comments

Voice 1 1 In other words, due to number of services required

for testing, it is either 2 sets of equipments are

required or 1 set of equipment is configured for 2

drives per route.

Project manager must consider the trade-offs or

feedback these to customer.

HTTP (internet) 1 1

Video 1 1

FTP (DL or UL) 1 2

Ping RTT 1 2

17

Define Methodology

Define Testing methods

Tools must be calibrated, and compared against customer’s trusted tool!

How fast to drive the vehicle.

How many services to test per drive test? (i.e. in one drive, what is the

set-up? One voice, one data? How many ports?)

Preparation of Field Measurement Tool

Procure tools if required.

UE mobile – Not just any mobile, be specific (i.e. model number, etc)

Scanner

GPS

DT Tool : Nemo/Tems/Agilent/Genex (check with customer the version to

use, else we can propose to customer what to use)

Cell plan and basic configuration and parameters (Very Important!!!)

Creation of script for different call functionality

Long call / Short call / Video call / R99 DL / R99 UL / HSDPA / HSUPA

18

Define Methodology

Neighbour List planning review

For some cases, (under SOW), Operators want to make use of DT to

review neighbour lists. So DT engineers must be taught how to make the

relevant observations.

Miscellaneous Tasks – Such as making Observations on DT

If these scope are included, then there should be a methodology

developed for how an engineer can observe these DT to capture

optimization problem symptoms.

That is why, understanding the REASON (Objective) of the Drive Test is key

to having a good customer service and relationship. (Doing DT for DT

sake adds little value for the operator/customer)

19

GPS Antenna

Scanner Antenna Scanner

Laptop

Define Methodology

Each drive test team comprises the following (Example): -

1 x UE for short voice test call

1 x UE for Long voice test call

1 x USB model for packet data test call

1 x Laptop for data collection (Specify OS version)

1 x Nemo or Tems (specify version)

1 x GPS

1 x Scanner (Specify version)

3 x SIM to be provided by customer

20

Define Methodology

Placement of tool requirement;

Ensure UE to be firmly attached in order to capture the best signal

Always use back the same UE for same call function test

Ensure UE is always with battery as not to lose record of log file

(As long as the method is consistent and agreed with the customer, all future drives must adopt the same method and placement)

21

Define Methodology

Script is important during test as we need to test for different

scenario. (Exact call script needs to be developed for different tools)

Different call script can be useful to identify different issue.

Short call duration = 90sec

Call setup time = 15sec

Wait time = 15sec

Long call duration = 3600sec


Wait time = 15sec

Short call identified the problem

like:

• Call setup success rate

• Fail call reason

• Call setup time

• Network issue

Long call identified the problem

like:

• Drop call reason

• RF environment

• Poor coverage area

22

Define Methodology

1. GPRS Attach

2. Activate PDP Context

3. FTP Logon 10.8.20.3

4. FTP Transfer(Receive)

5MB.TXT,100

5. FTP Logoff

6. Deactivate PDP Context

7. GPRS Detach

8. Wait 5s

R99 DL/UL/HSDPA/HSUPA

identified the problem like:

• File transfer rate

• RF environment

• SGSN or GGSN issue

• Network problem

• Transmission limitation

MOC Video call duration = 120sec


Wait time = 15sec

Video call identified the problem

like:

• Call setup success rate

• RF environment

• Network issue

• Power control

• Streaming quality

MTC = till MOC duration end

Call setup time = Auto answer call

Wait time = Idle wait for MOC

23

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




24

8c Project Planning

List out the methodology to be used and it’s impact on time-line

List the number of sites and clusters to be covered

List the number of routes to be driven

List the number of times the routes must be driven

Note the time of drive and logistics involved

Procurement of tools

Note the operational timing with OSS team and plan around it

Prepare the project for approval by customer

25

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




26

8d Kick off Meeting

Why have Kick off meetings?

Kick off meetings are important as it makes sure that we get the

customers involved into the delivery of the project with the assignment of

key personnel to be involved in the project.

Finalize Scope of work (SOW)

Finalize Methodology

Finalize Project planning

Assign key personnel and delivery teams

Try to develop an agenda prior to meeting up

Prepare minutes and make sure everyone signs off on the action

plans or agreed outcomes, especially pertaining to acceptance and

confirmation. This ensures minimum problems during acceptance

stage.

27

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




28

8e DT Tools Preparation

Depending on the services to be used, different tools will need to be

prepared or procured. Different operators may favour different tools.

This section is to be followed up and executed as part of the project

plan.

Specify all the maps required, cell plans, Ues and so on.

29

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




30

8f DT Tools Testing and Mock Report

Test the tools for each of the services. Iron out any potential issues.

Generate mock reports using the pre-defined reporting

standards/formats and present these to operator for confirmation, or

check against agreed delivery format defined in methodology.

31

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




32

8g On-Site Pre-DT Checks

A step-by-step Pre DT checklist needs to be developed and

distributed to Drive Test engineers. A document can be generated for

steps 8g, 8h, 8i and 8j together.

See Appendix A for alternative test procedure

LTE Initial Site functionality test procedure

When someone goes to the site, to the cluster or the region, he needs

to make sure the following (create a list): -

Make sure vehicle has all the adapters, power supply, inverters, etc.

(make a list)

Make sure DT tools are operational for all the required services. (See

detailed examples)

33


A step-by-step Pre DT checklist needs to be developed and

distributed to Drive Test engineers.

When someone goes to the site, to the cluster or the region, he needs

to make sure the following (create a list): -

Make sure vehicle has all the adapters, power supply, inverters, etc.

(make a list), ask OSS team to send out an alarm report for the clusters

being tested.

Make sure DT tools are operational for all the required services. (See

detailed examples)

34


Sample Pre-DT checklist and step-by-step document.

Introduction (Reference: Drive Test & IBC Checklists, Ericsson, sep 2011)

There are 3 different types of tests: -

Cluster test (Setup 1 & 2)

Single Site test (Setup 3 & 4)

In-building test (Setup 5)

35



36



37



38



Test Information

Step 1 Earfcn = 3210, EarfcnUL = 21210

Step 2 Scan DL frequency band: 2660Mhz to 2672Mhz, centre :

2666Mhz.

RBW = 40Khz

Sweep Time = 8

FTP Server for UL/UL tests at IP address 10.254.249.58

User TEMS scripts to control the UE activities: -

DL filesize: 1GB, UL filesize: 500MB.

Ping message: 32 bytes.

TEMS file setting: Swap after 65,000 lines

File naming Convention

LTE_Cxx_S01_testername_2001MMDD_xxx.log

(Where Cxx = cluster name, S01 = Test setup)

39



Cluster Pre-Drive

System Check:

Get latest cluster drive routes

90% cluster readiness (You can set whatever readiness your methodology

allows for)

No major alarms (From OSS – So you can make sure that OSS team must

send a copy of the BTS or eNB alarm reports at the beginning of each day, if

possible)

Ensure the sites in the test clusters are unlocked and put on “cell-reserved”

Activate the UETR tracing in OSS (if available)

(Let’s add one more term, make sure to activate UETR for the correct number)

40



Equipment Setup check

Always follow the same setup configuration (e.g. in Setup 1, MS1 is

always scanner & MS2 is always UE.)

Ensure property care on the GPS and scanner antenna cables (going out

via the car window.)

Make sure that the windows don’t Crimp the cable!!!

Note down the IMSI used for the different test UE.

Do a Pre-drive test ping and DL/UL to ensure FTP server is functioning

well. (make sure people manning the server is available before and

during drive, so that resets can be possible)

Drive in the same direction as the previous drive route (if applicable)

41



Check List during Drive

Maintain reasonable speed of 70 to 90Km/hr.

Check that logfiles are active and recording while driving.

If DL/UL sessions stop or hang unexpectedly during testing, stop and re-

start session.

Note problematic areas such as those below: -

Poor RSCP (<-100dBm), Poor CINR (<0dB), Low DL throughput (<5Mbps)

Sites not transmitting even when UE is at close range.

Swapped feeders (PCI not present at expected coverage area)

Laptop/ UE connection problems, etc.

Take note of inaccessible drive routes

Drive round the site in both clockwise and anti-clockwise direction (Only

applicable for single site test)

42



This is the spreadsheet or form each engineer should carry when out in the field

doing drive test.

Cluster

Information

Date

Cluster

Drive tester

MS1 model & serial no

MS1 IMSI

MS2 model & serial no

MS2 IMSI

S/N Time Location Problems Encountered, Observations

43



Check list post - drive

Change back the sites in the test clusters to “Not Cell-reserved”.

Re-plan inaccessible drive routes

Upload all the TEMS logfiles and Cluster drive log into a FTP server

(vendor to provide) immediately after the drive test.

44



Worksheet to use for DL

45



Worksheet to use for UL

46



47



Contact List (This is perhaps the most important – make sure that they are available at the beginning of the

drive and book their time)

Radio: - Name – number

eNB: - Name – number

OSS: - Name – number

Data services (Data server) Name - number

48

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




49

8h eNB On-Air & Alarm Report

This section may be already included in a step-by-step document

issued to drive test team.

Make sure to have the OSS or eNB contact person responsible for

each cluster and if possible. Alarm reports ideally should be received

from operator before and after each day’s drive tests.

The alarm must be checked before and after drive test.

50

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




51

8i Delivery of DT work

During the delivery of DT work, you could list the precautions or

observations that you will need the drive test team to note. 8i can be

part of 8g and 8h. A combined document can be delivered to drive

test team and DT team to be trained on these Standard operating

procedures (SOP).

Make sure to do a replay of each DT file after collection and save one

copy onto a portable hard-drive as well as on the laptop.

The files must be uploaded daily into the server.

52

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




53

8j Filing and Nomenclature

Identify the parameters and include them in the nomenclature: -

Clusters (Suggested code = Cxx, i.e. C01, C02, C03, etc.)

Or By Route (Suggested Code = Rxx)

Types of set up or tests. (Suggested code = Txx, T01, T02, T03, etc.)

Network type (LTE suggested code = LTE.)

Date (Very important, format = ddmmyy)

Name (suggested Code = 2 characters – i.e. PH = Paul Ho)

LTE_Cxx_Rxx_Txx_PH_ddmmyy_number

An example: -

LTE_C03_R07_T01_PH_251111_01 This refers to cluster 03, Route 07, Test 01, by Paul HO, 25th Nov 2011, and 1st test log.

54


Setting up server space – make sure it’s sufficient and make sure

documentation staff or IT staff will back-up the folders daily.

Make sure that the routes do not repeat. (i.e. cluster 1 and cluster 2

cannot both have a route 7. If cluster 1 has 10 routes, they will be 1 to

10 and cluster 2 goes from 11 onwards.) Randomly run the logfiles to

make sure they are not corrupted.

Cluster 01

LTE_C03_R07_T01_PH_251111_01

Cluster 02

Cluster XX

Route 01

Route 02

Route 01

Test 01

Test 03

Test 02

55


Overall Filing Structure (Sounds simple, but Take it Seriously!)

Cluster 01

Cluster 02

Cluster XX

Route 01

Route 02

Route YY

Test 01 Test 01 Test 02

Drive Test

logs

Scope of

Work

Methodology

SOW docs KPI criteria

Mapping

format

Clutter

format/data

Tools Inv. Reporting

format Cluster

definition

Route

Definition

Commercial

Tools Script

Template/file Mapping files

Master

Methodology

Cell / Site

plan files

Service

testing flow

Pre/post

Alarm report

Post

processing

Reporting

Summary

Report Cluster

reports By Routes By Tests

Legend: (Defined /given by)

All filing by

Comba

Comba

Operator

+ Comba

Operator

56

Agenda




Define Methodology

Project Planning

Kick-off Meeting





Delivery of DT work




57

9 Post Processing (Reports)

There are many ways to generate reports, some of the formats are: -

TEMS reports – Automatically generated.

Nemo reports – Automatically generated.

Actix reports – Runs off a template created in Actix with benchmark

drives or logs

Crystal reports

Export to Mapinfo

Usually the post processing will be done by 1 senior engineer for smaller

projects. But it is important to document a step-by-step guide so that

practically anyone can do it on a per cluster basis, so as to reduce

bottleneck on senior engineer.

Mapinfo export procedure TEMS report procedure

58

9a Confirmation and Acceptance

Post Processing and Reports may be embedded with a sign-off

section where the relevant parties sign it off.

This process may or may not be included.

It is best to put this process at the project kick-off stage. And once the

scope of work and the methodology and reporting samples are

accepted, then we should try to avoid having an extra confirmation

and acceptance. However it may not be avoidable in every case. So

the check list of items to be delivered needs to be compiled and a

sign-off needs to be incorporated into the reports.

60

Appendix A -LTE functionality Test Procedure

Stationary Functionality Test – This stage is to test the data service

Preparation of

Site

information

Sites on air

(Check Alarm

report)

Verify Location

of Serving Cell

Search good

dominant RF

environment

Map, cell plan, etc

RSRP > -65dBm

RSRQ > -10dB

SINR > 20

Single cell

Functionality

checks

Ping Test RTT <

15ms

(local server ONLY)

DL Throughput

(using FTP only)

> 50Mbps

UL Throughput

(Using FTP only)

> 30Mbps

If Ping fails, no need

to do other tests

FAIL – Report to

customer

PASS –

Reporting &

Acceptance

May generate mock report

If DL too slow,

no need to test UL

Note observations or log the

files and make sure to

keep messaging collection ON.

61

Appendix B – PING Test Screenshot

Huawei GENEX Probe for PING testing

Ping at least 20 times to a designated local server.

If you want to ping an overseas server, then you cannot FAIL the test if RTT>15ms

62

Appendix C – DL Throughput testing

Huawei GENEX Probe for DL Throughput testing

Continuous download from designated local server for 1min with avg. thruput >

50Mbps (Make sure to measure only during FTP in Session) – Visual inspection

63

Appendix D – UL Throughput testing

Huawei GENEX Probe for UL Throughput testing

Continuous upload from designated local server for 1min with avg thruput > 30Mbps

lte drive-test-process

Engineering