UNIVERSITY OF WATERLOO

Faculty of Engineering

Management Science 443: Telecommunications Management

Performance Metrics for Quality of Service in

a Cellular Network

Prepared by:

Ian Hung (iXXXXXX), 99XXXXXX Bobbin Lo (bXXXXXX), 99XXXXXX Yan Ma (yXXXXXX), 99XXXXXX

Louis Szeto (lXXXXXX), 99XXXXXX

March 29, 2004

ii

Table of Contents List of Figures .................................................................................................................... iii List of Tables ..................................................................................................................... iv Abstract: ...............................................................................................................................1 1.0 Introduction..............................................................................................................1

1.1 Motivation............................................................................................................2 1.2 Purpose.................................................................................................................3

2.0 Background..............................................................................................................4 2.1 Definition of Performance Metrics ......................................................................4 2.1.1 Bandwidth ................................................................................................... 4 2.1.2 Delay........................................................................................................... 5 2.1.3 Jitter............................................................................................................. 5 2.1.4 Loss ............................................................................................................. 6 2.2 Critique of Literature ...........................................................................................6 2.2.1 Bandwidth ................................................................................................... 6 2.2.2 Delay........................................................................................................... 7 2.2.3 Jitter............................................................................................................. 7 2.2.4 Loss ............................................................................................................. 7

3.0 Research Model .......................................................................................................8 4.0 Hypothesis................................................................................................................8 5.0 Experimental Design................................................................................................9

5.1 Network Side Controlled Variables .....................................................................9 5.2 Customer Side Controlled Variables .................................................................11 5.3 Questionnaire Design.........................................................................................11

6.0 Analysis..................................................................................................................13 6.1 Data Analysis of Customer Application Usage .................................................13 6.2 Data Analysis of Customer Application Preference ..........................................13 6.2.1 Observations ............................................................................................. 13 6.2.2 Null Hypothesis ........................................................................................ 13 6.3 Data Analysis of Customer Application Demand..............................................16 6.3.1 Observations ............................................................................................. 16 6.3.2 Null Hypothesis ........................................................................................ 16 6.4 Data Analysis of Application Technology Preference.......................................17 6.4.1 Null Hypothesis ........................................................................................ 17 6.5 Correlation between Performance Metrics with Customer Demand .................18 6.5.1 Investigation of Current Customer Usage & Preference .......................... 18 6.5.2 Investigation of Current Customer Usage & Future Customer Demand.. 19

7.0 Conclusion .............................................................................................................20 References..........................................................................................................................21

iii

List of Figures Figure 1: Example of Buffer ............................................................................................... 7 Figure 2: Web application used to obtain the ANOVA test values for H0-2 ................... 14 Figure 3: ANOVA test values for Null Hypothesis 4 ....................................................... 16

iv

List of Tables Table 1: Application Traffic Categories ........................................................................... 10 Table 2: Application Performance Dimensions................................................................ 10 Table 3: Current Usage of Different Wireless Applications.............................................13 Table 4: T-Value for Null-Hypothesis 1........................................................................... 14 Table 5: ANOVA Test Results for Null-Hypothesis 2 .....................................................14 Table 6: T-Test Results for Null-Hypothesis 3................................................................. 15 Table 7: Comparison between Current Customer Usage and Customer Preference ........ 18 Table 8: Application Performance Dimensions with Ratings for Voice Call and SMS... 19 Table 9: Comparison between Current Customer Usage & Customer Demand in Future19 Table 10: Application Performance Dimensions with Ratings......................................... 19

1

Performance Metrics for Quality of Service in a Cellular Network

Abstract: Quality of service parameters of the cellular network is examined in this paper. The relationship between network performance metrics with customer performance metrics is investigated. Network performance metrics are restricted to Bandwidth, Delay, Jitter, and Loss. Conversely, customer performance metrics are limited to customer application utilization. A questionnaire is circulated among university students that solicit customer application utilization information. Data collected about application usage, preference, demand, and technology are mapped back to the network parameters via a predetermined set of weights. Statistics is used to analyze the relevance of the relationship between customer and network performance metrics along with the relative level of importance among the network parameters. Keywords: Quality of Service, Performance Metrics, Cellular Network

1.0 Introduction Quality of Service has been the focus of recent research and has generated an increasing amount of interest. Its business implications are particularly profound in many areas of wireless network design. The definition of Quality of Service is ambiguous as it can refer to the measurement of customer satisfaction, such as the cost of subscription or the waiting time required for customer support, to technical aspects of the cellular network, such as call setup time or the rate at which calls are dropped. The term “quality” is a generic performance measure and has many meanings. The ETSI definition of QoS in the context of wireless networks states that it is the “collective effect of service performances which determine the degree of satisfaction of a user of the service.”[1] A user most often does not need details about how a service is provided; their primary concern is the resulting quality. The collective effect of service parameters in determining customer satisfaction suggests that there exist parameters in the network that must be met for an acceptable customer experience. In other words, the definition of Quality of Service can be viewed as either a top-down customer driven statement at which quality is measured by their ultimate satisfaction or a bottom-up network driven statement at which quality is determined by satisfying various network parameters. In the report, UMTS QoS and Network Performance, the specific network performance metrics are questioned for their effectiveness in measuring network quality [2]. Consequently, in the article, Understanding Consumer Concerns about the Quality of Wireless Telephone Service, Quality of Service is measured by the degree at which consumer concerns are addressed [3]. Throughout all the articles regarding QoS, there has been little research in connecting customer

2

satisfaction with network performance metrics; whereas, the performance metrics concerning customer service quality and network quality have been meticulously dealt with as two independent issues. It is noted that for articles related to network design in meeting QoS, customer satisfaction resulting from various network parameters has been implicitly assumed and that their level of importance is the same. In this report, the definition of Quality of Service shall be dichotomised into two distinct categories that are customer performance indicators and network service indicators. Customer performance indicators contain a myriad of performance metrics used to measure user satisfaction, but in this report, it shall be limited to customer application utilization. This is composed of application usage frequency, application preference, application demand, and application technology preference. Network performance indicators have been abstracted into four sections namely Bandwidth, Delay, Jitter, and Loss. The network under examination is chosen to be the digital cellular network. This study will first examine the link between network performance indicators and customer performance indicators and to statistically determine whether a correlation indeed exists. Secondly, the network performance indicators are examined to statistically identify any differences among their level of importance. A questionnaire has been created to specifically solicit customer application utilization information and is distributed among university students. This data is first statistically analyzed to derive any patterns among the applications’ usage, preference, demand, and technology criteria. Next, the data is mapped back to the network performance metrics according to a set of predetermined weights for the specific application to observe whether a relationship exists between network and customer application metrics and to determine whether there is significant difference in the level of importance for each network parameter. Results from this research will reaffirm the importance for proper network design if such correlation between network and customer performance metrics exists. Subsequently, if performance metrics are found to be valued differently, it will also guide resource allocation to specific areas of network development with greater perceived customer service impact.

1.1 Motivation Research in the area of Quality of Service in a wireless network has been generally divided between the optimization of network performance metrics and the study of perceived Quality of Service experienced by the customer. Few articles have explored the actual effects of changing network parameter with final customer satisfaction. On contrary, network performance metrics and customer performance metrics have been often studied independently. This report will attempt to establish a link between network performance metrics with customer performance metrics. If successful, findings from this research will validate the importance of network performance metrics and emphasize aspects of the network with the greatest customer impact.

3

1.2 Purpose Network and customer performance metrics are Quality of Service measurements. Network Performance Metrics are Bandwidth, Delay, Jitter, and Loss. Customer Performance Metrics are limited to customer application utilization that consists of application usage, application preference, application demand, and application technology. P1: The first objective of this report is to determine whether a relationship exists between customer and network performance metrics. The questions addressed are the following: Is customer application utilization substantially affected by network allocation of resources? Corollary, do changing network parameters substantially affect customer application utilization? P2: The second objective of this report is to determine whether the level of importance for each network performance metrics is different. The question asked is: Are all network performance metrics equal in their level of importance for Quality of Service at a given time? P3: The third objective of this report is to rank the level of importance of each network performance metric if applicable. The question becomes: Which network performance metric is most indicative of Quality of Service at a given time?

4

2.0 Background Quality of Service (QoS) in cellular networks is one of the most significant issues in the wireless industry today. QoS is defined as the level of network performance that a wireless user receives. There are several major factors that affect the level of network performance. These are Loss, Delay, Jitter, and Bandwidth. [4] Each of these components of QoS refers to specific technical parameters in measuring performance. The resulting performance translates to a certain level of customer satisfaction. If a link can be established, resources can be properly diverted to areas that maximize customer satisfaction. Today, most cellular networks are End-to-end IP-based networks allowing for network equipment with less complexity. [5] In return, there are less initial setup and maintenance costs incurred on the service provider. Hence, the service provider can become more competitive. However, IP-based networks provide a challenge to QoS, as IP-based networks operate on the premises that packets are sent connectionless. [6] This means the routing path of a packet is not pre-determined, but rather each packet is routed separately according to the shortest path. This can create problems because the shortest path may not be the least costly path. Some paths may be traffic-congested, slow (low bandwidth), and be subjected to security issues.

2.1 Definition of Performance Metrics To meet the four criteria of QoS in a wireless environment, there must be some way to measure quantitatively how well a service is doing. Many parameters have been developed specifically to target the four areas. These forms of measurement are what are defined as Performance Metrics. Throughout the recent decades of high growth in the wireless industry, a lot of literature has been written with focus on the development of such Performance Metrics. For example, the Call Success Rate (CSR) and Drop Call Rate (DCR) are just some of the parameters used for describing the Loss of a wireless service. This paper will add onto the work of previous literature, and attempt to find a relationship between QoS described with Performance Metrics and customer satisfaction. This is a topic of interest since not all forms of metrics are equally weighted in the customer’s perspective. To find these relationships are by no means trivial, but would provide guidance for resource allocation.

2.1.1 Bandwidth There are two definitions of bandwidth [9]. For the purpose of this paper, the second definition would be explored in greater depth.

1. The range of frequencies usable by a device. 2. The amount of information transferable within a given time. In a wireless environment,

bandwidth is often expressed in bits per second (bps). How does bandwidth relate to QoS? It would be ideal if bandwidth is infinite across all intra- and inter-networks. Unfortunately, this is never the case in the real world. For multiple reasons, there are different bandwidth allowances in different sections of a network. Interfaces from high to low bandwidth can become a point of congestion and unpredictability. [5] The truth is, no service provider can possess the infinite spectrum for transmission. There are only so many

5

allowable frequencies for wireless transmission which are regulated by government agencies. This is necessary to define servicing territories and prevent unpredictable interference. A common wireless technology has an ideal maximum bit rate of 384 kbps. [7] Due to the nature of technology, CDMA does not operate on a certain band as assigned by the government, but rather it spreads information across all channels.

2.1.2 Delay Delay in a network is defined as the transmit time an application experiences from the ingress point to the egress point of the network. Delay can cause significant QoS issues with applications such as voice and video, and applications such as fax transmission that simply time-out and fail under excessive delay conditions. Some applications can compensate for a smaller amount of delay but once a certain amount is exceeded, the QoS becomes compromised. [5] In the Global System for Mobile Communications (GSM network), it uses the existing TDMA technique, a method in which each conversation uses only a frequency for part of the time. Each technology has its own codec and with its compression. There is a tradeoff between quality and compactness. GSM’s codec gives the best results, producing a 260-bit packet every 20 milliseconds for a data rate of 13Kbits/sec at ‘full-rate’. Recently, ‘half-rate’ codec has been used to aim to double the voice capacity of a network and hence to reduce the possibilities of delay. Finally, delay can be both fixed and variable. Fixed delays are:

- application-based delay e.g. voice codec processing time and IP packet creation time by the TCP/IP software stack

- data transmission (queuing delay) over the physical network media at each network hop - propagation delay across the network based on transmission distance

Variable delays are:

- ingress queuing delay for traffic entering a network node - contention with other traffic at each network node - egress queuing delay for traffic exiting a network node

2.1.3 Jitter Jitter is the measure of delay variation between consecutive packets for a given traffic flow. Jitter has a pronounced effect on real-time, delay-sensitive applications such as voice and video. These real-time applications expect to receive packets at a fairly constant rate with fixed delay between consecutive packets. As the arrival rate varies, the jitter impacts the application’s performance. A minimal amount of jitter may be acceptable but as jitter increases, the application may become unusable. Some applications, such as voice gateways and IP phones, can compensate for small amounts of jitter. Since a voice application requires the audio to play out at a constant rate, if the next packet does not arrive within the playback time, the application will replay the previous voice packet

6

until the next voice packet arrives. However, if the next packet is delayed too long, it is simply discarded when it arrives, resulting in a small amount of distorted audio. [5] All networks introduce some jitter because of variability in delay introduced by each network node as packets are queued. However, as long as the jitter is bounded, QoS can be maintained. Performance degradation due to Jitter can be improved with buffering. The larger the buffer size, the less likely that an application will stall. The buffer size is a tradeoff between performance and cost.

2.1.4 Loss Loss can occur due to errors introduced by the physical transmission medium. For example, landline connections have very low loss as measured in the Bit Error Rate (BER). However, wireless connections such as satellite, mobile, or fixed wireless networks have a high BER that varies with the environment or geographical conditions such as fog, rain, RF interference, cell handoff during roaming, and physical obstacles such as trees, buildings, and mountains. Wireless technologies often transmit redundant information since packets will inherently get dropped some of the time due to the nature of the transmission medium. Loss can also occur when congested network nodes drop packets. Some networking protocols such as TCP (Transmission Control Protocol) offer packet loss protection by retransmitting packets that may have been dropped or corrupted by the network. When a network becomes increasingly congested, more packets are dropped and hence more TCP retransmissions. If congestion continues, the network performance will significantly decrease because much of the bandwidth is being used to retransmit dropped packets. TCP will eventually reduce its transmission window size, resulting in smaller and smaller packets being transmitted. This eventually will reduce congestion, resulting in fewer packets being dropped. [5] Congestion has a direct impact on packet loss; hence, congestion avoidance mechanisms are often deployed. One such mechanism is called Random Early Discard (RED). RED algorithms randomly and intentionally drop packets once the traffic reached one or more configured thresholds. RED takes advantage of the TCP protocol’s window size throttling feature and provides more efficient congestion management for TCP-based flows. Note that RED only provides effective congestion control for applications or protocols with ‘TCP-like’ throttling mechanisms.

2.2 Critique of Literature This section will critique the existing forms of Performance Metrics as mentioned from 2.1.1 to 2.1.5.

2.2.1 Bandwidth As a standard, bandwidth is mentioned in kbps, the amount of data transferable per second. There is an argument by some that the cost increasing bandwidth exceeds the benefits increased customer satisfaction. For instance, CDMA uses 4.7 to 14kbps of bandwidth for voice, depending on network conditions. The rest of the 384 kbps is can be used for data services. The amount of bandwidth for voice does not change for even greater bandwidth. Hence, increasing bandwidth benefits only those using data services. This is a question of what percentages of

7

customers actually use these services. Hence, there is a “knee” that minimizes the bandwidth provided while maximizing the customer satisfaction.

2.2.2 Delay Common to most literature is the need to decrease delay between packets. Different application has different tolerances for delays. For example, the success of a file transfer without error is more important than the transfer being completed in a timely manner. However, real-time applications are more susceptible to such delays. For instance, long delays would cause a video conference to drop off. To the average wireless user, any stalling in an application is unsatisfactory.

2.2.3 Jitter Jitter is often caused by bottlenecks, or interfaces between paths of high and low bandwidth. These bottlenecks cause congestion in traffic and are the source of uncertainty in delay. Jitter can also vary depending on the time of day because of varying traffic. One method that improves performance caused by Jitter is the use of buffering at the receiver. The use of buffering can be described with the Figure 1 below. In the diagram, n + 1 incoming packets are buffered for processing. As long as Jitter and other causes of delay do not affect the number of buffered packets to fall to zero, the application will not stall.

Figure 1: Example of Buffer

From a glance, a large enough buffer size would ensure that an application would not time out on the receiver end. However, in the case of real-time applications, the larger the buffer size, the less “real-time” the application becomes. An application would be n+1 multiplied by the time it takes to process each packet delayed between receptions of a packet and when it’s actually processed. Hence, it is evident that maintaining a buffer size appropriate to the type of application running and current network conditions would minimize Jitter. This would result in maximized customer satisfaction.

2.2.4 Loss Packet loss is measured in BER. Wireless channels have higher BER than wired channels. This is true because of interference, noise, and other types of transmissions within the same channel. Protocols such as TCP compensates for lost packets by retransmitting the lost packet. Packets are retransmitted until a good packet is received. However, as BER increases, more bandwidth is diverted to retransmissions. This decreases the bandwidth used for useful transmissions. Hence, lost packets do not directly affect QoS, but its external affects decreases the bandwidth. Problems with decreased bandwidth and how to measure it are discussed in 2.1.1 and 2.2.1.

n n+1 …

Packets

2 3

8

3.0 Research Model In the customer’s perspective, wireless services should resemble that of wired services. [4] However, this is not achievable due to the inherent topologies of the two technologies. For wired services that employ fibre optics, there is little error and high bandwidth. However, in the case of wireless services, a wireless channel has more than one device accessing the medium for a given time. Hence, a signal in a wireless channel is subjected to interference (natural and intentional), environmental noise. Today, the voice quality of digital wireless services is comparable to that of wired services due to efficient codec and increased bandwidth. Voice usually occupies anywhere from 4.7 to 14 kbps of bandwidth. Furthermore, increasing bandwidth would not significantly increase QoS. In fact, bandwidth requirements are more demanding for data services. This is because customers would like to have data services to be instant for any type of task. Since customers expect that all wireless services be instantaneous, any delay would be viewed as degraded QoS. In this sense, any delays and jitter would not be acceptable. In addition to zero delays, customers also expect the service to be reliable. This is in relation to the factors affecting Network Availability. It is obvious that these desires cannot be satisfied economically. To do so would require an enormous amount of resources by the service provider to the point where no profit is made. Hence, it is important to balance customer desires and the amount of available resources to improve network performance. The research model to be developed in this paper will address this issue.

4.0 Hypothesis The high-level hypothesis of this report addresses objectives P1, P2, and P3 of this report. These are stated below: H1. There exists a relationship between customer performance metrics, namely customer

application utilization, with network performance metrics. H2. Network performance metrics are equally important. By contradicting this hypothesis, the

opposite is true: the level of importance of each network performance metric varies among each other.

9

5.0 Experimental Design This section briefly describes the controlled variables that are important to this research. These controlled variables are divided into two main categories: Network Side Controlled Variables and Customer Side Controlled Variables.

5.1 Network Side Controlled Variables All the network applications can be characterized based on end-user expectations or applications requirements. They can be characterized into four different traffic categories. They are Interactive, Responsive, Timely and Network Control that are discussed below. For Interactive Applications, where two or more people actively participate, they would expect the application to respond with minimum delay and jitter between sender and receiver. Hence, these characteristics are similar among applications that fall into this category. Voice call and video conferencing are applications in this category. Therefore, they are very sensitive to delay, jitter, and loss. The difference between these two applications is the required size of bandwidth. Voice call is audio based, and therefore, its demands are quite small, while video conferencing is both video and audio-based, therefore, it requires a greater amount of bandwidth. [5] For Responsive Applications, where a person is connected to a networked device, performance requirements are expected to be similar to those of interactive applications; therefore, they would also require relatively low packet delay, jitter and data loss. However, QoS requirement for responsive applications are not as stringent as interactive applications. Responsive applications can use either UDP or TCP-based transport. Web-based applications are based on the HyperText Transport Protocol (HTTP) and always use TCP. The lost media packets are typically handled by application-level protocols if the media is sufficiently buffered; otherwise, the lost packets are discarded, resulting in some distortion in the media. Internet Browsing and Messenging are applications in this category. These applications require a medium sized bandwidth. [5] For Timely Applications, where a person is connected to a networked device but do not need real-time performance, information is expected to be delivered in a timely manner. These applications require packets that arrive within a bounded amount of delay. Delay and jitter are inconsequential because the transfer often takes minutes to complete. They use TCP-based transport and lost packets are retransmitted resulting in no packet loss in the transfer. Therefore, delay in timely applications is not important as long as the delay is within a bounded amount of time. Jitter has a negligible effect on these types of applications and loss is reduced to zero due to the loss recovery mechanism of TCP. SMS, E-mail and File Sharing are the examples for timely applications. The only concerns for these applications are the bandwidth and the data loss. It is absolutely unacceptable for a E-mail to be delivered in 30 minutes after the sender sends that out. So, it requires a medium size bandwidth which can transfer the data in a quick manner. TCP is used for the transfer of data for these applications; therefore, no data will be lost. [5] For Network Control Applications, which controls the network, they require relatively low amount of delay. Similar to timely applications, jitter has a negligible effect on network control applications. However, due to some applications are not transported via TCP, there is no loss

10

recovery mechanism. A summary of the four different types of traffic categories and respective examples are listed in Table 1. [5]

Table 1: Application Traffic Categories

Traffic Category Applications Interactive Voice Call

Video Conferencing Responsive Internet Browsing

Messenging Timely SMS

E-Mail File Sharing

Network Control Routing Table 2 illustrates the QoS performance dimensions required by some common applications. They are Voice Call, SMS, Internet Browsing, E-mail, Messenging, Video Conferencing and File Sharing. For each of these applications, a grade of low (1), medium (2) or high (3) will be assigned based on its usage and requirement of the specific performance dimensions. Delay-tolerant applications such as E-mail would be configured to have a lower emission priority than delay-sensitive real-time applications such as voice call or video conferencing. These delay-tolerant applications maybe buffered while the delay-sensitive applications are being transmitted. However, the downside of this approach is that low emission priority queues may never get serviced if there is always higher emission priority traffic with no bandwidth rate limiting. In the table, a value of 1 will be assigned to low, a value of 2 will be assigned to medium and a value of 3 will be assigned to high. For example, voice call requires a low bandwidth; however, it is very sensitive to delay, jitter and loss of data, and for SMS, it requires a low bandwidth, not sensitive to delay and jitter but very sensitive to data loss. So, both voice call and SMS demands the same level of bandwidth in order to complete the application.

Table 2: Application Performance Dimensions

Performance Dimensions Sensitivity to Application Bandwidth

Delay Jitter Loss Voice Call Low High High Medium

SMS Low Low Low High Internet Browsing Medium Medium Low High

E-Mail Low Low Low High Messenging Medium Medium Low High

Video Conferencing High High High Medium File Sharing Medium Low Low High

11

5.2 Customer Side Controlled Variables Currently, there are five cellular service providers in the East Coast of Canada. They are Rogers Communications Inc (Rogers), Microcell Communications Inc (Fido), Microcell Connexions Inc (Simpro), Bell Canada (Bell) and TELUS Corporation (Telus). Rogers, Fido and Simpro provide GSM services while Bell and Telus provide CDMA services. Other than the traditional voice services, all service providers provide Short Message Service (SMS), internet and E-mail access with General Packet Radio Service (GPRS) and file sharing with cable or Infrared (IrDA). In the coming future, video conferencing will be implemented into the wireless network. For the survey, the target participants are set to be university students in Ontario aged 19-23. The lower bound of the target sample size is set to be 30 students for statistical reasons. This group of target participants is set in order to obtain a region-specific, occupation-specific, and age-specific group for the customer side controlled variables. First of all, the selected group was narrowed down to students at the University of Waterloo since only a small sample size can be obtained within a short period of time. Furthermore, students aged 19 to 23 were chosen because these participants will be knowledgeable of some wireless applications. A questionnaire is created containing only the desired application name and the qualitative aspects influencing these applications. Based on the results from the questionnaire, it is possible to find whether a correlation between customer performance metrics and network performance metrics exist along with whether there exist a different level of importance among the latter set of metrics.

5.3 Questionnaire Design The questionnaire is divided into four sections as shown in Appendix A. The first question column addresses how frequently customers use the applications on their current cellular phone or PDA. This column measures the usage of each individual available application on their current cellular phone or PDA in the form of marking ‘1’ to ‘5’, where ‘1’ means never use the application and ‘5’ means often use the application. Each application is measured independently and is not affected by the degree of usage of other applications. The result obtained from this column can show the actual usage of each application; however, since the applications are rated independently, ANOVA test or t-test could not be conducted. Therefore, only observations were made based on the calculated mean values. The second question column addresses how important each application means to the customer. This column measures the importance ranking of each individual available application in the form of ranking ‘1’ to ‘7’, where ‘1’ means least important and ‘7’ means most important. All applications are measured dependent of each other, so only one application will have a value of ‘1’ and so on. The result obtained from this column can show the preference of today’s customers’ demand on a specific application. Since each application is ranked with respect to other applications, a t-test and ANOVA test could be performed to find the correlation between each application and if there is any difference between user usage and preference in wireless applications. The third question column addresses how customers rank the importance of each application five years from now. This column measures the importance ranking of each individual application

12

available five years from now in the form of ranking ‘1’ to ‘7’, where ‘1’ means least important and ‘7’ means most important. All applications are measured dependent of each other. The result obtained from this column can show the difference in user preference and demand of a network resource distribution. The data collected from this question will enable the analysis of how the network resource should be distributed and what changes need to be made in order to satisfy the demand of the projected behaviour. The fourth column addresses which technology customers think that are the most important to them in each application. For each application, participants were required to pick one aspect out of the two listed aspects available for them to choose. This question will determine the network aspect that is most important for customers, and therefore, from the results obtained, suggestions can be made to improve on existing technology and network resource distribution.

13

6.0 Analysis After collecting the response from 37 participants, the data was recorded in an excel sheet as shown in the Appendix. Appendix A contains the Customer Survey circulated to the participants. Appendix B depicts the user responses and data analysis for question 1; Appendix C depicts the user responses for question 2. Appendix D shows the responses for question 3. Appendix E shows the responses for question 4.

6.1 Data Analysis of Customer Application Usage An interesting observation can be made from the data collected for question 1, as shown in Appendix B. Because the Voice Call application mean value is the highest (4.135) as shown in Table 3, it is possible to conclude that the Voice Call application has the highest usage in the present time.

Table 3: Current Usage of Different Wireless Applications

Application Mean Value Voice Call 4.135

SMS 2.838 Internet Browsing 1.946

E-mail 2 Messenging (MSN, ICQ, AIM) 1.946

Video conferencing 1.351 File sharing (video, music, picture) 2

Index: 1 = Very low usage; 2 = Low usage; 3 = High usage; 4 = Very high usage

6.2 Data Analysis of Customer Application Preference

6.2.1 Observations From the collected data from question 2, three observations can be made:

1. The mean value of Voice Call remains the highest at 5.919 2. The mean value of Internet Browsing, E-mail and Messenging are relatively close to each

other equals approximately 4.0 3. Video conferencing and File Sharing have similar mean score of 2.5

There is a significant increase in SMS (4.973) compared with customers current usage (2.838), it is then necessary to examine if it is equally preferred as Voice Call.

6.2.2 Null Hypothesis H0-1: Customers have equal preference for Voice Call and SMS A t-test was conducted and the results are shown in Table 4. At a degree of freedom equals to 2 and p = 0.05, the acceptable t-value is 2.919986. Since the calculated experimental t-value 8 > 2.92, the null hypothesis is rejected; meaning Voice Call and SMS are having different customer preferences and Voice Call remains the most preferred wireless application.

14

Table 4: T-Value for Null-Hypothesis 1

Application Mean T-Value Voice Call 5.919

SMS 4.973 8.007

H0-2: Participants consider Internet Browsing, Email and Messenging equally preferred A NOVA test was conducted because more than three groups of correlated data had be to compared. At a degree of freedom equals to 2, the calculated F-value equals 0.0301 as shown in Table 5. Since F-value <1, it is possible to conclude that there is no statistical difference between the mean; thus, these applications are equally preferred.

Table 5: ANOVA Test Results for Null-Hypothesis 2

Application Mean F-Value Internet Browsing 3.973

E-mail 3.946 Messenging (MSN, ICQ,

AIM) 4.027

0.0301

Figure 2: Web application used to obtain the ANOVA test values for H0-2

15

H0-3: Participants Consider Video Conferencing and File Sharing equally preferred

A T-test was conducted to determine if customers prefer equally video conferencing and file sharing. At a degree of freedom equals to 2, the resulted t-value was found to be 1.84 as shown in

Table 6 and the acceptable t-test value is 2.919986. Since 1.84 < 2.919986, this proves that these two applications are equally preferred; therefore, the null-hypothesis can be accepted. However, the p-value equals to 0.207 > 0.05.

Table 6: T-Test Results for Null-Hypothesis 3

Application Mean T-Value Video conferencing 2.405

File sharing (video, music, picture) 2.757 1.84

16

6.3 Data Analysis of Customer Application Demand This section investigates the data collected for Question 3 from the survey as shown in Appendix D. The goal is to determine if there will be any increase or decrease in application demand 5 years from now by comparing the mean values of Question 3 with that of Question 2. In addition, if there is a change in customer demand, which application will be more likely to be in demand.

6.3.1 Observations Three observations can be made by comparing the data collected from Question 2 and Question 3:

1. Compared with participants' future preferences again their current preferences, Voice Call has a decrease in future demand since its mean value (5.189) is lower than that of current demand (5.919).

2. Compared with participants' current preferences, SMS has a lower future demand since

its mean value of 4 is lower than that of current demand (4.973).

3. There is a significance increase of demand for Video Conferencing and File Sharing in future because the future demand mean values (3.541, 3.297) are much higher than the current desired preference mean value (2.405, 2.757).

6.3.2 Null Hypothesis H0-4: Future usage of SMS, Internet Browsing, E-mail, Messenging, Video Conferencing and File Sharing will be equally likely in demand. ANOVA test was conducted to determine if SMS, Internet Browsing, E-mail, Messenging, Video Conferencing and File Sharing will be equally likely in demand 5 years from now. With a degree of freedom of 5, the F-value is found to be 0.7664. Since F < 1, this proves that the null hypothesis should be accepted and that the above listed 6 applications are statistically similar to each other. Figure 3 depicts the resulting F-value and p-value.

Figure 3: ANOVA test values for Null Hypothesis 4

17

6.4 Data Analysis of Application Technology Preference This section examines the feedbacks from participants to determine which wireless aspect is most important for each wireless application. From the data collected, as show in Appendix E, the following six Null Hypothesis were tested using the chi-test

6.4.1 Null Hypothesis H0-5: Participants Call Success is equally important as Distortion Free for Voice Call With 2 degrees of freedom, the Chi-test result was found to be 0.68, which is far less than the acceptable value of 5.99. This means that the null hypothesis is accepted and that Call Success is equally important as Distortion Free. However, the p-value was found to be 0.713310954, much higher than 0.05. H0-6: Message Send/Receive success is equally important as Short Delay for SMS With 2 degrees of freedom, the Chi-test result was found to be 9.76, which is above the acceptable value of 5.99. This means that the null hypothesis should be rejected. The new conclusion is that Message Send/Receive is more important than Short Delay. The p-value was 0.008 < 0.05. H0-7: Message Short Delay is equally important as Page Load Success for Internet Browsing With 2 degrees of freedom, the Chi-test result was found to be 6.081, which is above the acceptable value of 5.99. This means that the null hypothesis should be rejected. The new conclusion is that Short Delay has higher importance than Page Load Success. The p-value was found to be 0.048 < 0.05l. H0-8: Message Send/Receive success is equally important as Short Delay for E-mail With 2 degrees of freedom, the Chi-test result was found to be 16.89, which is greater than the acceptable value of 5.99. This means that the null hypothesis should be rejected. The new conclusion is that Message Send/Receive has higher importance than Short Delay. The p-value was 2E-04 < 0.05. H0-9: Message Send/Receive success is equally important as Short Delay for Messenging (MSN, ICQ, AIM) With 2 degrees of freedom, the Chi-test result was found to be 1.324, which is far less than the acceptable value of 5.99. This means that the null hypothesis can be accepted. However, the p-value was found to be 0.516, much higher than 0.05. H0-10: Short Delay and High Audio & Video Quality are equally for Video Conferencing With 2 degrees of freedom, the Chi-test result was found to be 0.027, which is far less than the acceptable value of 5.99. This means that the null hypothesis can be accepted. However, the p-value was found to be 0.987, much higher than 0.05. H0-11: Fast Transfer and Successful Transfer are equally important for File Sharing With 2 degrees of freedom, the Chi-test result was found to be 0.027, which is far less than the acceptable value of 5.99. This means that the null hypothesis is accepted. However, the p-value was found to be 1, much higher than 0.05.

18

6.5 Correlation between Performance Metrics with Customer Demand In Section 6.1, it was concluded that the current highest usage in wireless application is in Voice Call. Section 6.3 concluded that there will be an increase of usage in SMS, Messenging, Internet Browsing, File Sharing, and Video Conferencing wireless applications. Knowing the trend in wireless application and the wireless aspects that matter to customers, it is then possible to determine which area in wireless network requires further investment for research and development to satisfy customer demand.

6.5.1 Investigation of Current Customer Usage & Preference From the observations made in Section 6.1 and the proven null-hypothesis in Section 6.2, the new question to ask is what is the difference between the wireless applications that customers are currently using and the applications they would prefer to use. These differences are shown in Table 7.

Table 7: Comparison between Current Customer Usage and Customer Preference

Application Current Usage

Customer Preference

Voice Call 4 4 SMS 1 3

Internet Browsing 1 2 E-Mail 1 2

Messenging 1 2 Video Conferencing 1 1

File Sharing 1 1 Index: 1 = Low usage or low preference; 2 = Medium usage or medium preference; 3 = High usage or high preference; 4 = Highest usage

Since there is a significant customer preference increase in SMS, by comparing with the performance metrics parameters for Voice Call and that of SMS, it is possible to conclude that when the number loss of packets can be reduced in wireless applications, the number of customer use SMS will increaseTable 8 depicts the different performance matrix for Voice Call and SMS based on Table 2. This conclusion is consistent with the findings from Question 4, namely that users rated Message Send/Receive Success highest for SMS and Call Success the highest for Voice Call. Both aspects are tightly related to the loss packets performance criteria. This proves that both hypotheses are true, namely that:

- There exists a relationship between customer performance metrics, namely customer application utilization, with network performance metrics.

- Network performance metrics are equally important. Loss of packets affect how customers perceive the Message Send/Receive Success aspect and Call Success aspect of a wireless network.

19

Table 8: Application Performance Dimensions with Ratings for Voice Call and SMS

Performance Dimensions Sensitivity to Application Bandwidth

Delay Jitter Loss Voice Call 1 3 3 2

SMS 1 1 1 3 Index: 1 = Low dependency; 2 = Medium dependency; 3 = High dependency

6.5.2 Investigation of Current Customer Usage & Future Customer Demand From the observations made in Section 6.1 and the proven null-hypothesis in Section 6.3, it is possible to investigate if there is a difference between the wireless applications that customers are currently using and the applications they will more likely to use 5 years from now. These differences are shown in Table 9.

Table 9: Comparison between Current Customer Usage & Customer Demand in Future

Application Current Usage

Customer Demand

Voice Call 3 3 SMS 1 2

Internet Browsing 1 2 E-Mail 1 2

Messenging 1 2 Video Conferencing 1 2

File Sharing 1 2 Index: 1 = Low usage or low demand; 2 = High usage or high preference; 3 = Highest usage or highest demand

Table 10: Application Performance Dimensions with Ratings

Performance Dimensions Sensitivity to Application Bandwidth

Delay Jitter Loss Voice Call 1 3 3 2

SMS 1 1 1 3 Internet Browsing 2 2 1 3

E-Mail 1 1 1 3 Messenging 2 2 1 3

Video Conferencing 3 3 3 2 File Sharing 2 1 1 3

Index: 1 = Low dependency; 2 = Medium dependency; 3 = High dependency

20

Clearly as shown in Table 9, customers will use SMS, Internet Browsing, E-Mail, Messenging, Video Conferencing and File Sharing more frequently in future. Based on Table 10, it is possible to conclude that wireless bandwidth and loss packets are the performance parameters require additional research and development more urgently to improve on the existing technology in order to satisfy the future demand. This conclusion is consistent with the findings from Question 4, namely that users rated Short Delay and Fast Transfer Call Success the highest aspects for these applications. Both aspects are tightly related to the wireless bandwidth and loss packets, proving that customer preferences matches with actual technical performance metrics for these more advanced wireless applications.

7.0 Conclusion The focus of this research paper is to examine the link between network performance indicators and customer performance indicators and to statistically determine whether a correlation indeed exists. Secondly, the network performance indicators are examined to statistically identify any differences among their level of importance. Based on the survey feedback collected from 37 participants, it was concluded that technical network performance parameters indeed correlates to customer qualitative preference indicators. It was also concluded that users that perceive high bandwidth correspond to fast transfer rate and slow delay; while low loss of packets correspond to message send/receive success and call success. Furthermore, it was also found that the customers will have a higher demand for SMS, Internet Browsing, Messaging, File Sharing and Voice Conference applications in the future once the technical barriers can be reduced. This means that wireless network bandwidth and the loss of packets require additional research and development in order to satisfy the market demand 5 years from now. Customers are satisfied with the current wireless technology affecting the delay and jitter aspects; therefore, it appears that there is no urgent need to improve on these areas. In satisfying the objectives of this report, a correlation between network and customer performance metrics has been statistically confirmed; hence, objective P1 is validated. Moreover, there is a statistical difference in the level of importance among network parameters; hence, objective P2 is validated. Lastly, the level of importance of each network parameter is stated; hence, objective P3 is validated.

21

References [1] “Measuring Quality of Service in an Experimental Wireless Data Network,” Internet Search, http://atnac2003.atcrc.com/ORALS/Ivanovich.pdf [2] “UMTS Quality of Service and Network Performance,” Internet Search, http://www.3gpp.org/ftp/tsg_cn/WG4_protocollars/Temp/SMG%2323/TDocs/P-97-657.pdf [3] “Understanding consumer concerns about the quality of wireless telephone service,” Internet Search, http://www.kintera.com/AccountTempFiles/cf/%7B39C94822-4A0C-4245-B0E5-22D474242D71%7D/AARP%20Study.pdf [4] C. Liu, “Quality of Service and Mobility for the Wireless Internet,” Oct 27th 2003. [5] “Introduction to Quality of Service,” Nortel Networks, Article #56058.25-201803. [6] K. Naik, “Data Link Layer,” University of Waterloo E&CE 428 Lecture Notesy, http://www.ece.uwaterloo.ca/~ece428/dlc.pdf [7] Andrew J. Viterbi, CDMA: Principles of Spread Spectrum Communication, Prentice Hall PTR, 1995 [8] M. Ivanovich, P. Fitzpatrick, J. Li, M. Beresford, A. Saliba, “Measuring Quality of Service in an Experimental Wireless Data Network,” Telstra Research Laboratories, Mar. 2003 [9] “Bandwidth,” Internet Search, http://www.computer.org/author/style/refer.htm [10] “Interactive Statistics,” Internet Search, http://members.aol.com/johnp71/anova1sm.html [11] Newman, W.M. & Lamming M.G, Interactive System Design, Reading MA: Addeson-Wegle, 1995