customer experience index (cei) solution for net promoter

© 2018 Nokia CEI as NPS prediction - Solution Paper

of 22

Customer Experience Index (CEI) solution

for Net Promoter Score prediction

July 2018

http://www.nokia.com/


of 22

This material, including documentation and any related computer programs, is protected by

copyright controlled by Nokia. All rights are reserved. Copying, including reproducing, storing,

adapting or translating, any or all of this material requires the prior written consent of Nokia. This

material also contains confidential information, which may not be disclosed to others without the

prior written consent of Nokia.

© Nokia 2018. All rights reserved.

About Nokia

Nokia is a global leader in the technologies that connect people and things. Powered by the innovation of Bell Labs

and Nokia Technologies, the company is at the forefront of creating and licensing the technologies that are

increasingly at the heart of our connected lives.

With state-of-the-art software, hardware and services for any type of network, Nokia is uniquely positioned to help

communication service providers, governments, and large enterprises deliver on the promise of 5G, the Cloud and

the Internet of Things.

http://www.nokia.com || https://networks.nokia.com/software




https://www.facebook.com/Nokia

http://www.linkedin.com/company/Nokia

https://twitter.com/Nokia


of 22

CONTENTS

1 INTRODUCTION 4

2 CEI INSIGHT DESCRIPTION 6

BUSINESS PERSPECTIVE 8

USER INTERFACE & REPORTING 10

3 DATA MODELS 14

CUSTOMER EXPERIENCE INDEX DATA MODEL 14

DATA COLLECTION 15

4 CONCLUSION 17

APPENDIX A: NOKIA’S CUSTOMER INSIGHTS SOLUTION OVERVIEW 18



of 22

1 Introduction

Summary: A sophisticated set of Machine Learning algorithms use the network and non-

network data to predict the Net Promoter Score (NPS) for the entire subscriber base, and

on a per end user basis. The model is calibrated using periodic outside-in survey results to

provide an accurate proxy for NPS on an ongoing basis.

The key benefits are being able to

• Segment end users across clearly identified categories that reflect the level of

satisfaction of their experience with the service provider

• Drill-down (which end users, which locations, which devices, etc.) and serve multiple

stakeholders of the service provider organization, such as Operations with reasons/root

cause analysis for detraction enabling rapid and proactive corrective actions to improve

NPS, as well as Marketing and Strategy teams with insights on how to maximize

promoters end user usage and loyalty and generate new revenues opportunities

The traditional method of calculating Net Promoter Score (NPS) through customer surveys has

limitations in terms of the number of customers who can be surveyed, the cost of conducting the

survey, the time taken to get the survey results and the availability of detailed information on the

reasons for detraction. While the wide adoption of NPS across industries is an indicator of the

relevance of this measure, the problems listed lead to ask the following questions:

▪ Can NPS be calculated in near real-time instead of waiting for the results from a weekly

survey?

▪ Is it possible to cost-effectively calculate NPS for all customers instead of a sample of a few

thousand customers at a time?

▪ Can the reasons for detraction be proactively identified without having to ask the

customer?



of 22

▪ When corrective action is taken could the improvement in NPS also be measured

immediately?

▪ How to best capitalize on NPS scores that are satisfactory?

Communication Service Providers have the advantage of collecting a large number of KPIs at

different touch points such as each voice call, data session, customer service interaction and bill

sent. If this information is aggregated on single subscriber basis using a sophisticated algorithm it

can give a reasonably comprehensive view of customer experience at the individual subscriber

level. The algorithm would generate on a daily basis a single number on a 0-100 scale representing

how satisfied each customer is likely to be based on delivered service levels at various touch points.

Taking this one step further the algorithm can be calibrated so this single number reflects the

promoter score that a subscriber is likely to give i.e. predict how each customer would rate their

service provider on the 0 to 10 NPS scale using accurate historical information on the experience

for all individual subscriber service events with the operator over an extended period of time.



of 22

2 CEI Insight Description Nokia has designed and deployed a Customer Insights solution that predicts the promoter score a

customer is likely to give in response to the NPS survey based on that individual’s delivered service,

by calculating a near real-time overall NPS score for the entire subscriber base.

The prediction, referred as Customer Experience Index (CEI), is based on data that is captured non-

intrusively from the various network data sources collected via probes (transactional data / call

detail records), Billing and Care insights from IT systems (number of interactions with CC over

certain period, etc), CRM information (end user profile, tariff plan, etc). Other data sources such as

sentiment analysis, social media analysis, etc. can be added flexibly into the model. Network data

collected relates to both connectivity/accessibility (Control Plane), Usage and Quality of the service

(User Plane) as shown in Figure 1 below.

Methods have been developed to validate the quality of data arriving from the data sources,

process the outliers and impute the missing values to make it robust to imperfections of the

incoming data. In addition, elaborate methods and metrics to validate the performance of the

machine learning models result in consistent prediction performance. Building and deployment of

the machine learning model is automated (AutoML) to a large extent resulting in reduced time to

deployment as indicated in Figure 1 below.

Figure 1: Most complete Customer Experience Index (CEI) in the industry powered by Auto ML

The accuracy of the calculated NPS has been verified by comparing system calculated proxy-NPS to

actual NPS survey results for the same time-period. An actual comparison between the Net

Promoter Score given by the customer in a survey and the proxy-Promoter Score calculated using

Machine Learning is shown in Figure 2.

C u sto m e r S u r vey

�

CRM & location dataAge, GenderSubscription PlanLocation, DeviceLife time value, … �Network DataVoice, VoLTESMSData ÂOther SourcesCall center trouble ticketsBilling & Charging tickets

�

Data Auto ML

Model definition

Data Ingestion & pre-processing

Model building & training

Model evaluation & best model selection

Model deployment

C E I

Customer experience index is calculated

for all subscribers

Promoters

Passive

Detractors

One score is computed per subscriber

reflecting actual experience correlated with NPS surveys

9 0 8 0 6 7 5 4 2 0 3 4

Further aggregated by time, location,

device, segment, etc. to allow detailed analytics



of 22

Figure 2: Promoter Score versus Customer Experience Index

Please note that Nokia’s experience in creating CEI score has been a long-standing experience

acquired over the years. The model has been constantly evolving to increase accuracy first by

ingesting other type of data than network data, and second by using more sophisticated

technology until machine learning applied today. The combination of knowledge, experience and

constant innovation is what differentiates Nokia in our ability to produce an accurate reflection of

the customer experience. Our Machine Learning model has been deployed with a high level of

accuracy with several Tier 1 Service Providers across the globe.

The CEI framework is productized and adaptable such that service providers be able to benefit

from it right away while being able as well to consider and test different types of machine learning

models.

Figure 3: Customer Experience Index evolution



of 22

Business Perspective

Data collected from the network, customer contact points as well as many other data sources are

analysed and translated into concrete actions to improve the perceived experience, the customer

life time value, as well as identify new revenue opportunities.

Nokia is committed to build an outcome based approach when producing our CEI framework.

Types of actions that can be generated are depicted in Figure 4.

Figure 4: The Power of the Customer Experience Index

In the following customer reference example, the calculated CEI score acting as NPS proxy has been

used as the starting point for NPS improvement use cases. A drill-down on detractors could

identify dropped calls as the reason for detraction and specific actions were taken at these

locations on improving call continuity.



of 22

Figure 5: CEI Contributing factors drilled down to and corrective actions recommended

In this case voice quality was chosen as the drill-down criteria. Sites with more than 30% of

subscribers having Mean Opinion Score (MOS) less than 3 were identified. The Net Promoter Score

of these sites was 19.7% and the voice CEI was 82. The identified sites and the corresponding

scores are shown in Figure 4 below.

Figure 6: Detractors due to voice quality problems. Identifying location with a large proportion of

subscribers having low MOS

Further investigation was made at these sites by the network operations team and actions such as

frequency change, mechanical tilt change, TRX addition, neighbour cell optimization, jammer

closure was taken. The MOS improved and as a result voice CEI and overall CEI increased. As a result

of improved KPI’s for individual subscribers and therefore a better promoter scores, the NPS for

these sites improves significantly.



of 22

Figure 7: Detractors due to voice quality problems. Actions: frequency and tilt change, TRX

addition, neighbor optimization, jammer closure were taken

User Interface & Reporting

User interface reporting presents data to the end user in meaningful report format which can be

used for quick decision making. The result set from data model is shown as graphs in reporting

module. Queries are optimized to improve reporting performance.

Figure 8.1: Predicted results from the ML model applied – per subscriber



of 22

Figure 8.2: Predicted results from the ML model applied – per subscriber

The Customer Experience Index (CEI) application provide for various reports such as (not

exhaustive):

▪ Customer

Experience Index by

region & city

▪ Customer

Experience Index with

time (trends)

▪ Comparison

▪ Top X details

Region and City Analysis

A location analysis shows the measure of Experience Index by Region and Cities. By selecting a

region in Region analysis, the analysis moves to analysis by Cities.

Numbers inside a bubble represents the Customer Experience Index for a particular location and

the size of the bubble represents the total number of subscribers. Color code is used to reflect the

quality of the experience per location, e.g. in this case a poor experience is affecting Perth area.

Refer to the figure below.



of 22

Figure 9: Location Analysis - Customer Experience Index by location

Comparisons

This report enables user to compare multiple dimensions e.g. models of devices, location, and

subscriber segments etc.

Figure 10: Comparison Report

Top X details



of 22

This report shows dynamically the top x subscriber having a high index for selected criteria in the

global filter. Similarly, analysis can be done for the top ‘x’ - subscriber segment, devices and

locations.

Figure 10: Top X Details Report



of 22

3 Data Models

Customer Experience Index Data Model

An overview of Nokia key KPIs used by Nokia’s common data model and Key KPIs for CEI calculation

is shown in the following figure. The picture depicts a Mobile network. Similar model applies and is

available as well as commercially deployed for Fixed network.

Figure 3: Customer Experience Index Data Model - Key KPIs (mobile domain)

The KPIs in the above model are supplemented by the following data in the data model:

▪ Radio conditions (signal strength, signal to interference ratio) in idle and connected mode

▪ Outbound roaming status (% of time spent as outbound roamer)

▪ Usage information (E.g.: number of calls, number of SMS, data volume, time of use)

▪ Mobility information (# of cells visited)

▪ Customer Static data (*)

- CRM (Corporation, Department, Segment, Gender, Age, Contract type, Plan)

- Device (OS, device model, vendor)

- Location topology (cell / address)

- Technology

- Customer life time value

- APN

CEI (Customer Experience Index)

NEI (Network Experience Index)

Voice

SMS Data

CSI

(Customer

Service Index)Voice CS VoLTE

Call Drop MO (%) - CoreCall Drop MO (%) - Subscriber

Call Drop MO (%) - Radio

Call Drop MT (%) - CoreCall Drop MT (%) - Subscriber

Call Drop MT (%) – RadioAverage Call Setup Time( s)

Call Setup Success Rate (%)

Average MOS (#)Average Echo Path Delay (ms)

Average Jitter (ms)Average Noise Level (dB)

Mute Call Rate (%)

VoLTE Registration Failure (%)VoLTE Call Setup Failure (%)

VoLTE Average Call Setup Dur (s)

VoLTE Default Bearer Act Fail (%)VoLTE Dedicated Bearer Act Fail (%)

VoLTE Call Drop MO (%)VoLTE Call Drop MT (%)

VoLTE Average MOS Tx (#)

VoLTE Average MOS Rx (#)VoLTE Average MOS (#)

VoLTE Min MOS Tx (#)VoLTE Min MOS Rx (#)

VoLTE Min MOS (#)

VoLTE Redial Call Rate (%)VoLTE Long Call Setup Rate (%)

VoLTE Mute Call Rate (%)

Receive Failure Network (%)Receive Failure Subscriber (%)

Send Failure Network (%)

Send Failure Subscriber (%)

Attach Failure Rate (%)Activation Failure Rate (%)

Default Bearer Activation Fail (%)

Handover Failure Rate (%)NI Detach Rate (%)

NI Deactivation Rate (%)Dedicated Bearer Activation Fail (%)

Average PDP Setup Time (s)

Average Default Bearer Setup Time (s)Average Dedicated Bearer Setup Time (s)

Throughput Browsing (Mbps)Throughput Streaming (Mbps)

Application Access Time Browsing (s)

Application Access Time Streaming (s)Service Access Failure Rate (%)

Average Repetitive Calls (#)Escalation (%)

Average Ticket Age (days)

First Level Resolution (%)Mean Customer Wait Time (s)

Completed Request (%)Abandoned Calls (%)

Average Resolution Time

(days)



of 22

(*) CEI also utilises important data for customer related information that is imported into the data

model, usually from Customer Relationship Management (CRM) systems. Its nature is “static”

compared to the network traffic (counter & KPI) data processed through the southbound network

interfaces. The customer related information is assumed to remain mostly unchanged, and no

historical records are kept for this data.

Data Collection

Nokia plan the data sets and build the model, after exploring what KPIs and data can be provided.

In partnership with the service provider, and based on the KPIs and/or other data from the list

provided above, the solution can use existing Data Lakes. In such a model where data from existing

data lakes are used, Nokia expects such data to be made available via a Hive connector on the Data

Lake. Data not available from existing lakes would be collected from the network or other sources

as mentioned in previous sections.

Hadoop Storage nodes will provide the storage for customer KPIs and customer event data. The

customer KPI data and event storage enable the customer analytics functionality; default reports

and content for background processing reporting.

The functions performed by data collection and analysis, in conjunction with the Data Lake are:

▪ Collection of the subscriber event data from the network

▪ Storage of the subscriber event data

▪ Forming of the subscriber individual quality and use counters

▪ Default content for analysis

▪ Dynamically updatable data analysis

The customer definition in the customer repository requires data from CRM systems to be

collected. This can be from a mixture of different systems deployed and different interfaces to

these systems.



of 22

Network Data Sources

The collection of Network Data sources that are included as input to calculate the network part of

the CEI are the following:

Service Connectivity/Accessibility Usage Quality

Voice 2G, 3G IuCS, A IuCS, A Nb/Nc

SMS 2G, 3G IuCS, A IuCS, A NA

VoLTE S1-MME, S11 Gm (logical interface) Mb

Data 2G, 3G Gb, IuPS Gn UP Gn User plane, DPI

Data 4G S1-MME S1U S1U, DPI



of 22

4 Conclusion

Nokia’s approach provides a powerful method to identify actions that will improve NPS and witness

results in near real-time after actions are taken. It addresses the traditional problem of NPS

surveys which may tell you that experience is deteriorating but cannot explain why. The NPS

surveys also identify only a small sample of detractors since the survey methodology has an

inherent limitation on the number of subscribers who can be surveyed and the frequency of

surveys. Nokia’s CEI NPS proxy on the other hand is available in near real-time for the entire

subscriber base and provides a detailed drill-down to the likely cause for a customer to be a

detractor. Nokia has achieved a level of accuracy that makes this NPS proxy usable for NPS

improvement initiatives from real customer deployments and the Machine Learning algorithms and

calibration approach are being continuously improved to achieve higher levels of accuracy.



of 22

Appendix A: Nokia’s Customer

Insights Solution Overview

▪ Reveal high value customers with low customer experience as input for retention

activities

▪ Analyse service usage and quality for dynamically defined customer segments

▪ Identify best candidates for targeted campaigns

The challenge

Evolving technology, more and mode devices connecting to the network, increasing amount of

generated data result in the following challenges:

▪ Information on customer experience is scattered across organisational silos

▪ No holistic view of customer experience or ability to measure impact of initiatives across the

organization

▪ Inadequate understanding of customer experience touchpoints and KPIs and how to

improve them over time

▪ Transitioning to analytics driven data organisation can be hampered by people, tools and

processes

▪ Working in reactive mode rather than using predictive analytics and automation

The solution

Nokia’s Cognitive Analytics for Customer Insight provides a platform to accelerate the customer

experience journey, from any starting point. It helps increase customer satisfaction by creating a

holistic view of fixed and mobile subscriber experience. It provides uniquely actionable insights

down to every customer or cell. It has a built-in analytics framework with machine learning

algorithms for tuning insights.



of 22

Customer Insight’s technology utilizes Artificial Intelligence, to provide:

▪ Holistic view of customer experience and a platform to drive actionable use cases

▪ Key focus on improving the customer’s experience in mobile & fixed networks, increasing

the levels of automation in processes and increasing revenue

▪ Analytics to understand and improve customer experience across different touch points

using all available data sources

▪ Secured data collection and remote access to the system

▪ Increasing number of applications and use cases that can be selected from the Nokia’s Use

Case library

Customer Insights Introduction

Insight into how users are experiencing their services and the capability to take action based on

this insight are more important than ever. To address this, we need key capabilities to gather real

time and non-real time data from multiple sources, generate holistic insight about the customer

experience, and use it to prioritize and trigger concrete actions to improve the customer

experience and their business results.

Nokia’s Customer Insight combines an innovative dashboard view of the customer-centric

performance indicators with application for specific departments. You are able to access and share

information across the entire organization from operations to marketing to customer care.

Because everyone uses the same product, they speak the same language, without the need for

specialist knowledge.

Nokia’s Customer Insight is not another data warehouse. It is ready to use, with industrialized

telecommunication knowledge available to collect the large amount of data from the network,

customer contact points and other data sources to enable a view of each customer. The

information retrieved is analysed and consequently translated into concrete actions to improve the

perceived experience, and the customer life time value. In addition, we can identify new revenue

opportunities. To this, it requires us to turn the retrieved data into value.

The initial step is to collect relevant customer data from different data sources. This can be for

example collecting traffic data on a corporate customer level. In the second step this data is

processed and analysed in order to convert the data into information and knowledge

Customer Insights Functionality Overview

Customer Insights provides a centralized control of rich, near real time data produced by multiple

data sources like the Nokia real-time network analytics, third party probes, billing systems, and

many more. The data collected from those multiple data sources is stored for long term, correlated

and is provided a way of translating enriched information into actionable insights. With this, we are

developing a way of allowing users accessing customer relevant data, and complementing it with

wider end to end view and greater historical perspective than the one provided by a real time

environment. We bring this intelligence for different user groups by having a set of applictaions to

address different needs in terms of information type and detail. Different departments can then

benefit from different functionalities and end user data by means of specific use cases in order to

reveal relevant insights and derive actions in order to blend and improve existing processes/

business practices.



of 22

Customer Insights architecture and functionalities are specifically designed to deal with the key

challenges mentioned, that go from the point where data is collected simultaneously from multiple

data sources to actionable insights tailored for different user groups. The data model is TMF SID

certified (Shared Information Data) while the dynamics of the market allows adopting new data

source quickly as modelling follows a bottom up design that facilitates to develop schemas and

data loading procedures for new integration. Data Integration services help in creating the business

logic for data loading, aggregation and data quality checks. Data Integration application offer many

off-the-shelf adapters including various database adapters (Hadoop HDFS, Oracle, MySQL, Sybase,

ODBC etc.,) and file adapters (ASCII flat file, XML file, CSV files etc.,) to load the data from external

sources. In addition, it also includes auditing services to ensure data quality, data reconciliation and

security.

Customer Insights is offered in small scale deployments to large scale deployments with many

nodes. A staging server to perform data quality checks like identification and deletion of duplicate

data, identification of missing data, identification and deletion of bad records and data source

identification (add source id to incoming records) etc., is offered for robust data loading. The data

after cleansing, quality checks is loaded into raw data base tables and stored for the defined period

of time. Data can be loaded either in real time or in batch for faster performance. The raw data can

be used later for data aggregation and also for reporting. Data reconciliation is available as part of

the platform and be able to report if there are issues identified with data availability.

Business Reporting services allow configuration of rich visual content packages. Customer Insights

also offers numerous off-the-shelf content applications like High Value Customer Insights. Reports

can be scheduled at periodic intervals and can be visualized via web access or distributed by email

in various formats like HTML, PDF, Word etc.,

Customer Insights uses Hadoop Big Data HDFS based database and it is tuned to meet the

performance, capacity and scalability needs and at the same time offers customized services for

further database fine tuning. The system can scale vertically and horizontally based on capacity

needs in runtime.

Customer Experience Index

The Customer Experience is the overall psychological impact of a customer's interactions with the

Communications Service Provider (CSP) and the services provided. This includes direct experience

from the quality of product and network services delivered.

One of the major findings from a number of surveys and studies carried out by Nokia Networks is

that a significant gap exists between perceived assumption of delivered customer experience and

the "real" experience of end customers. The perception gap is substantiated by the fact that, while

80% are convinced of delivering a superior customer experience, only 8% of customers believe

that they receive it.

At the same time, we collect a tremendous amount of data from the network, customer contact

points and other data sources, that would enable them to create an overall, yet detailed, view on

the customer experience. The information retrieved should be analyzed and consequently

translated into concrete actions to improve the perceived experience and retention of subscribers

and also identify new revenue opportunities.

Customer experience index (CEI) enables understanding “customer experience” from the

subscribers’ point of view.



of 22

Application

The Customer Experience Index application can be used by many including product management,

marketing and account management personnel. It provides:

▪ Customer segmentation

o Easy segmentation and micro-segmentation of subscribers

o Identification of the right candidates for targeted campaigns, increasing the hit-rate

and saving costs.

▪ Cross-functional customer experience index

o Summarizes the customer experience in one index that crosses the organization -

one business KPI.

▪ Value customer analysis

▪ For calculation of overall experience index, platform supports flexible weight setting for

premium customers. That means - we can give more importance to a certain set of

customer segment over others.

Customer experience is impacted by many services and processes, both technically and

organizationally. CEI calculates and collects customer specific measurements from these services

and processes. Example: “Trouble ticket resolution time, Dropped data session” etc. The

measurements are collected over the customer’s life cycle, from the order to the termination of

the contract.

Customer Experience Index Customer

Contract

Subscription

Quality

Usage

Invoicing

Customer Service

Timeframe

Different thresholds are applied on customer specific measurement and further normalized based

on their importance to the customer experience. Each service is weighted against each other and

the Customer Experience Index is calculated for all customers individually. The Customer

Experience Index is a uniform KPI reflecting the overall experience of each subscriber.

Following the trend and distribution of customers according to the Customer Experience Index

enables the CSP to understand the reasons for a perceived quality difference against the measured

quality. Adjusting the thresholds of the KPIs and weights for the service builds a more reliable

Customer Experience Index.



of 22

The Customer Experience Index uses only data that has an impact on the customer experience

itself, emphasizing the soft factors. For example, dropped calls are considered, but the number of

hand overs might be ignored.

Modeled customer experience can be displayed in the CEI distribution report where the number of

customers Vs the Customer Experience Index graph is plotted. The user access to the application is

provided by web clients. The solution is designed to support a multi user environment where the

system has multiple concurrent users performing actions.

Though data analysis is distributed, the system provides a single access point to the data so that

the user can gain uniform understanding of the points of interest over the full network and all

technologies. User access to the data is authenticated and can also be restricted at a user level.

The customer repository forms subscriptions based on the measurements’ relationship to the

customer in person and to the contract, age and gender of the customer based on the integrated

data sources e.g. CRM. This additional data is used to group and analyze customer level data by the

new aspects provided.

The services offered by the system functions are the following:

▪ User authentication

▪ User management

▪ Central point of access

▪ Raw & long-term storage

▪ Customer repository

▪ Integration templates

System wide functions are also scalable by distributed processing architecture. When the number

of users, data storage time, data volumes or number of customers managed by the system is

changed, the solution can scale accordingly. The customer repository therefore enables the flexible

analytics of the customers data based on points of interest without report and database changes.

The customer repository is populated with key data from existing systems such as CRM, invoicing

and trouble ticketing. Integration is done via Integration templates.


customer experience index (cei) solution for net promoter

Documents