presentation by dhruba jyoti chaudhuri

Dhruba Jyoti Chaudhuri Tata Consultancy Services

Mantra for Process ExcellenceDHRUBA JYOTI CHAUDHURISenior ConsultantTATA CONSULTANCY SERVICES LTD

Myth Busted – You Can Read Customer’s Mind; a Statistical Model to Stay Close to Your Customer

Dhruba Jyoti Chaudhuri1*, Vasu Padmanavan2, S Balakrishnan3

Tata Consultancy Services Limited, Mumbai, India

*Corresponding Author. Tel: +91 9831048870, Email: [email protected]

Theme

“Mantra for Process Excellence”

Key Words

Customer Satisfaction, Process Performance Model, Statistical Model, Logistic Regression

Abstract

The famous KANO Model suggests Today’s Delighter would become Normal Expectation Tomorrow

and may become Dissatisfier thereafter………needless to articulate why every service organization

strives to improve customer satisfaction. A loyal customer implies customer retention, repeat

business, increased share of wallet, positive referrals, new business opportunity etc. It is an essential

element to grow, remain competitive and build ability to create differentiators. Many organizations set

target for Customer Satisfaction Index based on history information, experiences and leadership

mandate. This is further linked to various units’ performance objectives, captured at individual

customer touch point (executing projects). Normally, Customer Satisfaction is captured by conducting

surveys, then measured, analyzed and action plan formulated aimed at future improvement. The cycle

being repetitive but a reactive one, based on lag measures that limits ability to act proactively and

make things different. Process Performance Model builds this ability by identifying various lead

indicators, establishing a cause & effect relationship to an outcome, and enabling what-if analysis.

This enables project managers with capability to monitor key influence factors continuously, assess

possible outcome well in advance and act to control the outcome; thereby stay close to customer. The

paper summarizes a unique model (patent filed) built upon sampling ~3000 customer feedback and

subsequent survey responses from 700+ practitioners. The novelty lies in the model’s predictability

within diversity (project, customer, location, geography etc.) and scalability (6000+ instances). It is

digitized and driving significant and quantifiable improvement at organization level. The opportunity for

replication and benefit multiplication is very high across industry.


mailto:[email protected]

Contents

Introduction............................................................................................................................. 3

Business Case...........................................................................................................................3

What is a Process Performance Model?...................................................................................4

CSI Prediction Model................................................................................................................4

Building Blocks of the Model....................................................................................................5

Model Building Steps................................................................................................................7

Novelty.....................................................................................................................................9

Model Factors........................................................................................................................ 10

Model Accuracy......................................................................................................................11

Business Results.....................................................................................................................11

Sample Model Output............................................................................................................12

Conclusion..............................................................................................................................13

Acknowledgement................................................................................................................. 13


Introduction

There is a population of projects, for which there has been a decrease in Customer Satisfaction Index

(CSI) but could not be prevented. Proactive decision making was not possible in the absence of a

Process Performance Model (PPM). The impact of the current state of a project (based on the

applicable weakness, events, or scenarios) on future CSI cannot be quantitatively ascertained.

Similarly, by planning or taking actions and improvements, one cannot determine how they can

improve the CSI performance and their sufficiency to meet the target customer satisfaction necessary,

to meet the unit and organization business objective. Given a strong organization mandate, it is

essential to proactively assess, monitor, and improve the customer satisfaction. A Process

Performance Model was the only option to achieve this.

Business Case

One of the key strategic drivers contributing to an organization’s growth is repeat business, which is

aligned to the strategic goal Customer Retention. Achieving this goal ensures repeat business with

potential to further increase the wallet share within the same customer organization and open new

business opportunities through customer referrals. However, all these are possible only when an

organization has a loyal and satisfied customer. In contrast, a dissatisfied customer means potential

loss of business and thereby posing a risk of losing the customer to a competitor. This explains why

customer satisfaction and loyalty improvement have always been the top priority, while being

competitive.

At organization level, there is a defined process as a mandate to conduct Customer Satisfaction

Survey (CSS) and measure the Customer Satisfaction Index. There is a target for each project to

achieve which is rolled up at various units and organization level. The survey is conducted at periodic

interval and data is analyzed at project, unit and organization level aimed at i) analyzing SWOT

(Strength Weakness Opportunity, Threat), ii) translating customer voice into OFI (Opportunity for

Improvement)s and iii) formulation of actions, as appropriate.

Currently, this process is repetitive but a reactive one, based on lag measure i.e. what could be done

better and differently in the future based on past evaluation and feedback.

Hence, the Process Performance Models (CSI –ve Model and CSI +ve Model) have been developed

to bridge this gap by i) linking project performance to organization business objective and ii) enabling

proactive assessment and action.


What is a Process Performance Model?

Process performance models (PPM) is used to represent past and current process performances, to

predict the future performance. They can also predict critical process and product characteristics (or

related measures) that are relevant and linked to identify process performance baselines, which are

essential to meet important business objectives. PPMs enable an organization to take proactive and

informed decisions (mathematically best possible prediction or judgment, based on the available

information), by monitoring these key performance measures.

A Process Performance Model (PPM) primarily:

Establishes a cause and effect relationship between dependent (Y) and independent

variables (Xs)

Models the variation in factors (x) and provide an insight into the expected range and the

variation in the predicted outcome or result (Y)

Comprised of controllable and un-controllable factors

Facilitates WHAT-IF analysis, to enable decision making

CSI Prediction Model

The CSI –ve Model provides capability to monitor key influencing factors (project level issues,

weakness, and so on), assess possible outcome (impact on CSI), and exercise informed decisions to

control the outcome. The CSI +ve Model provides capability to proactively evaluate actions and

improvements’ ability to elevate the CSI performance, to meet the desired target.

The CSI –ve Model uses present scenarios of a project and predicts the probability of possible (delta)

%CSI slippage in various bands. This helps projects to link project events or scenarios to probable

impact (-ve) on future CSI, by observing the probability distribution in various bands and use it as lead

indicators, to act proactively and take informed decision towards reversing, minimizing, or containing

the impact.

The CSI +ve Model provides continuous ability to evaluate planned (or implemented) actions (or

improvements) towards meeting the desired CSI target, by providing indicators on possible increase

in CSI. For high performers, the model helps to sustain the level of satisfaction and continuously add

value to the customer.


Building Blocks of the Model

The CSI –ve Model was built based on a population of ~3000 projects’ CSI information, for which

there was a drop (Delta %decrease) in CSI and subsequent survey responses obtained from ~350

practitioners (projects), to capture various applicable factors with severity that had probable (-ve)

impact on their respective Customer Satisfaction Index (CSI). Qualitative comment was also captured

to help data cleansing and factor grouping. The emphasis was to understand what went possibly

wrong and was visible to the customer.

This information was the building block of this process performance model, based on Logistic

Regression, to establish the Cause and Effect relationship between Input Factors and Outcome/

Response (Delta decrease in CSI). The outcome was converted into various bands such as 0-5%, 5-

10%, 10-15% etc. to make it discrete. The core model output was further normalized (based on

previous CSI), to arrive at the final model outcome. The Model uses present scenarios of a project

and predicts the probability of a possible (delta) slippage in various bands. This helps projects to link

project events or scenarios to probable impact (-ve) on the future CSI, by observing the probability

distribution in various bands and use it as lead indicators, to act proactively and take informed

decision towards reversing or minimizing the impact. Figure 1 illustrates the core model framework for

CSI –ve Model.

Core M odel – Ordinal L ogistic Regression, Band wise L ogit/ Probability equations, Normalized further (Conditional Probability)

CSI –veModel

Block Diagram – Predicting Probable CSI Slippage

Current StateOf Project

FactorsAndSeverity

Probable delta slippage (CSI) in various Bands

Suggested Actions/ Best Practices for selected weakness

•Issues•Weakness•Dissatisfaction•Complaints etc. Action Plan/ Refined

Action Plan

Overall band wise population (CSI decrease) distribution

Previous CSIPre

process

BP

CSI

Focus is to reverse the impact and contain the slippage

Figure 1: Core Model Concept – CSI –ve Model


The model also suggests a set of best practices or actions, to address the selected weakness (model

input factors), based on the experience captured through a survey from projects for which there was

an increase in %CSI (data used for CSI +ve Model).

Ordinal Logistic Regression is used in this model to first (1st pass) segment out key factors and then

(2nd pass) to establish the cause and effect relationship. This helps to predict the probability of

possible (delta) slippage in CSI across various bands (ranges) basis current weakness or opportunity

for improvements in project execution.

Similarly, for the CSI +ve Model, the approach remains the same, while data and model equations are

different. Few factors are also different, based on the survey feedback and key factors determined by

the model (logistic regression). Experience from 500+ practitioners, who were successful in improving

their CSI, was used to build this model. Actions, improvements, and best practices from these projects

were fed back to the CSI –ve Model, to guide users on suggested actions and various best practices,

which can be considered while finalizing their action plans. Figure 2 illustrates the core model

framework for the CSI +ve Model.

Core M odel – Ordinal L ogistic Regression, Band wise L ogit/ Probability equations, Normalized further (Conditional Probability)

CSI +veModel

Block Diagram – Predicting Probable CSI Improvement

Current StateOf Project

FactorsAndMagnitude

Probable delta increase (CSI) in various Bands

•Actions•Improvements•Strengths•Appreciations etc.

Overall band wise population (CSI decrease) distribution

Previous CSIPre

process

CSI

Focus is to sustain the current level and further improve (as predicted) CSI, also evaluate, if the potential jump would be sufficient to meet the Target

Figure 2: Core Model Concept – CSI +ve Model


Model Building Steps

The approach and brief modeling steps for the CSS –ve Model are as follows:

1. Organization data (customer satisfaction) was extracted for two consecutive half years.

2. Common projects were extracted.

3. Population of common projects was further segmented into three sub populations as follows:

i. Projects for which CSI dropped or decreased (Delta -ve change)

ii. Projects for which CSI increased or improved (Delta +ve change)

iii. Projects for which CSI remained the same (No change, status quo maintained)

4. Population 3(i): This was further considered for CSI –ve Model (to assess potential CSI

slippage or rating down grade, as a model outcome), for any project with certain issues,

weakness, events, or scenarios that may lead to customer dissatisfaction (as model input

factors).

5. Population 3(ii): This was further considered for CSI +ve Model (to assess potential CSI

increase or elevation, as a model outcome), for any project with certain strength or

improvements planned or carried out (as model input factors).

6. Population 3(iii): This was not considered further and is left as-is.

7. List of 40+ factors was finalized from the organizational experience, which primarily influence

(negatively or positively) the customer’s perception and has been reflected in the previous

CSI. These factors represent various scenarios, events, and situations during typical software

development, maintenance, and service life cycle of projects.

8. Population 3 (i) was used to build the CSI –ve Model, as explained in the following steps:

9. The Model was conceptualized to establish a relationship, to predict possible slippage (delta)

in CSI for a set of applicable factors and level of influence.

10. Delta CSI slippage was calculated for each project instance.

11. Based on the range of slippage, five possible bands were chosen, to provide better

predictability for users. Accordingly, the delta CSI slippage data was converted into six

possible bands (0-5%, 5-10%, 10-15%, 15-20%, and >20%). Higher the band of probable

slippage, greater is the risk that warrants for management attention and rigor in action

planning and monitoring.

12. An online survey was conducted, to capture relevant factor(s) and their level of severity or

weakness from Project Leads or Managers, for each project instance.

13. The survey data was cleaned and formatted, based on the qualitative feedback and

calculated delta CSI slippage.

14. Factor grouping was performed, based on logical relationship, mutual exclusivity, and so on;

thus, 40+ factors were grouped and transformed into 11 logical groups.


15. Logistic Regression was chosen to model the cause and effect relationship between factors of

influence (project scenario, event, issue, weakness, and so on) and outcome (Delta CSI

slippage). This is chosen, because it has the capability to identify key influencing factors (p

value), relative importance or Weightage (odds ratio) of various factors, probability distribution

in various bands, and model accuracy or fitment (~67% concordance).

16. The 1st Pass was executed, to identify seven groups that have significant impact to determine

the model output.

17. The 2nd Pass was executed, to derive model variables (Constants, Coefficients, and so on)

and formulate the Logit equation, which then calculates the band-wise probability.

18. For the entire population, as described in 3(i), the band-wise delta CSI probability distribution

was determined.

19. The probability calculated in the 2nd Pass of the model was further normalized (conditional

probability) with overall distribution probability (previous step), to predict band-wise probability

distribution for given or selected factors.

20. Actions, improvements, and best practices captured through the survey for population 3(ii)

were used, to form a knowledge base that can be referenced in the CSI –ve Model.

Appropriate actions or best practices from this knowledge base will appear in the –ve Model,

when a specific issue or OFI, are selected by the users.

Note: The CSS +ve Model has been built by following similar steps, based on the data and survey

feedback (540 respondents) for population 3(ii).


Novelty

Perception Modeling: The basic difference between a generalized prediction model and a

model to predict the customer satisfaction is in leveraging hidden factors, perception, situation

based impact, and so on for data-driven forecasting.

One solution catering to diverse projects: The current model caters to scale and diversity.

6000+ projects of different size and type with various customers across countries and

geographies.

Data and Experience: The model factors in both customer satisfaction data at the

organization level and perception (describing factors and scenarios), which can positively or

negatively influence the customer satisfaction.

Opportunities unlimited, enabling every single project contributing to an organization’s growth:

The model is a mandate for all projects within the organization, to sustain the current

performance and improve the customer satisfaction. This helps them to take part and

contribute to the organization’s growth.

Converting Lag to Lead indicators – ability to act a cycle ahead: The model has been

successful in driving all projects, to evaluate the model factors on a regular basis proactively

and initiate action or implement best practices as a continuous process, instead of waiting for

the customer’s feedback to act reactively.

Could predict tilt on both sides of the pan: The model has two variants—one to predict the

probable slippage from the current satisfaction level and another to predict the potential uplift

of customer satisfaction, by implementing actions or best practices.

~25+ factors building the cause and effect relationship: This was a ‘dare to try’ attempt to

model various project factors, scenarios, and events into a relationship with the outcome as a

potential influence (both +ve or –ve) on the customer satisfaction. 40+ factors are initially

selected for modeling and the final model maps ~25 factors to the outcome.

Driving organization change – All to improve: Since inception, the model has been a game

changer by adopting a culture of ‘all projects have improvement scope’. While projects not

meeting the target have a clear mandate to improve, other high performers strive to sustain

the level and improve the satisfaction by adding value and delighters.


Model Factors

The Models map ~25+ factors, comprising 7 groups, as shown in the table 1 and table 2. While most

of the factors are similar in CSI –ve and +ve Models, a few factors are different.

Table 1: Key Factors CSI –ve Model

Grp. No. Factor Group Theme Key Considerations or Critical Factors

1 Competence Domain, Technology, and Project or Program

Management

2 Project Management or

Governance

Ability to Flag Issues and Risks, Quantitative

Monitoring, Connect, Sharing, and Transparency

3 Political or Other Sensitive

Issues

Change in Functional Manager and so on

4 Responsiveness, Proactiveness,

and Problem Solving

Complaint or Escalation Management, Attending

Priority, Proactiveness, RCA, and Problem Solving

5 Quality of Deliverables and Test

Management

Quality of Deliverables and Performance of System,

Integration and Acceptance Testing

6 Resource Management Attrition, Availability, On-boarding, Ramp-up, Bench

Strength, Release or Turnover, and so on

7 Value Addition Value Addition beyond BAU (Contractual

Expectations), Ideation, and Thought Leadership

Table 2: Key Factors CSI +ve Model

Grp. No. Factor Group Theme Key Considerations or Critical Factors

1 Competence Domain, Technology, and Project or Program

Management

2 Project Management or

Governance

Ability to Flag Issues and Risks, Quantitative

Monitoring, Connect, Sharing, and Transparency

3 Responsiveness,

Proactiveness, and Problem

Solving

Complaint or Escalation Management, Attending

Priority, Proactiveness, RCA, and Problem Solving

4 Quality of Deliverables and Test

Management

Quality of Deliverables and Performance of System,

Integration and Acceptance Testing

5 Resource Management Attrition, Availability, On-boarding, Ramp-up, Bench

Strength, Release, Turnover, and so on

6 Value Addition Value Addition beyond BAU (Contractual Expectations),

Ideation and Thought Leadership


7 Schedule Compliance Ability to Deliver On Time


Model Accuracy

Model was developed with an accuracy estimation of ~67%. It has been observed with ~70%

compliance with +/- 3% of margin error. Model has been further calibrated with this margin error limits.

-1.45

Majority of the margin errors falls in the range delta -0.16 -3.89%

-0.16

-3.89

-1.11

-1.80

Business Results

Overall (for the population deployed CSI +ve Model) projects can improve their CSI by 5.51-

6.27%

Projects (population that does not meet the organization CSI target) can improve their CSI by

9.13- 10.63%

Projects (population that obtained greater than the organization target) can improve their CSI

by 3.28- 3.78%

Note: The analysis was carried out, based on the organization CSI performance between H2 FY2012-

13 and H1 FY2013-14. A similar trend was observed between H1 FY2013-14 and H2FY2013-14 CSI

performance. Statistical test results are not shown due to confidentiality.

The critical factors that were selected by majority of the population, to improve CSI are as follows:

Technology Competence

Domain Competence

Sharing Ideas, Suggestions,

Improvements, Thought Leadership,

or Best Practices

Detailed Planning and Regular

Sharing of Progress and Status

Proactive Sharing of Issues, Flagging

Risks

High Quality Service or Deliverables

Maintained


Resource Management : No customer

impact (induction, on-boarding, ramp-

up, sudden release or attrition)

Sample Model Output

CSI +ve Model: Shows a partial snapshot of some applicable factors (chosen by a sample project),

the model output as probability of occurrence (CSI increase) in various bands (magnitude of delta

increase) and the next probable CSI range (based on previous CSI).

CSI -ve Model: Shows a partial snapshot of some applicable factors (chosen by a sample project),

the model output as probability of occurrence (CSI decrease) in various bands (magnitude of delta

decrease) and the next probable CSI range (based on previous CSI).


Conclusion

This paper describes how a Process Performance Model can help to monitor key project indicators on

a regular basis, assess forward impact on customer satisfaction, evaluate and deploy timely actions

(to reverse –ve impact, if any and enhance +ve impact), and enable organizational entities (projects)

to meet and exceed satisfaction target. These factors are essential to enjoy the customer’s loyalty,

and retain and strengthen the relationship. Thus, it helps understanding the customer’s expectations

better and align an organization’s priorities and actions accordingly with a goal, to satisfy and retain

every customer. The theme is applicable for all organizations in the service industry and the concept

is replicable, to multiply benefits across industry and organizations.

Acknowledgement

Authors would like to thank everyone who supported this innovation. They would also like to sincerely

thank Ms. Aarthi Subramanian (Global Head & VP, Delivery Excellence) and Mr. K Subramanian

(Head - Enterprise Quality Management) for their sponsorship, guidance, and sharing invaluable

experience. They would also like to thank K Ramesh, Venkatasubramaniam Sundar, and the

COMPASS team for implementing the solution as a tool and making it accessible for all associates

within the organization.


Author Details

Mr. Dhruba Jyoti Chaudhuri, M.Tech. in Computer Science and Engineering, has 20+ years of IT

experience and is currently working as a member of Corporate Delivery Excellence Group, to support

Analytics and Modeling. He has received professional certifications such as PMP®, PRINCE2®

Practitioner, CSM (Certified SCRUM Master), ITIL(F), and Six Sigma Black Belt. He has authored the

following papers: i) Applying Statistics to Predict and Manage Software Project Schedule, ii) AGILE

Burn down Chart deviation - Predictive Analysis to Improve Iteration Planning, iii) AGILE Metrics –

Roadmap to Attain Process Maturity; Build Organization Ability and Benchmark Performance, iv)

Implementation of LEAN, the Broader Spectrum, v) Customer Alignment and Business Value

Cascade - the ITIL way. Dhruba also filed a patent on “System and Method to Provide Predictive

Analysis towards Performance of Target Objects Associated with Organization”.

Mr. Vasu Padmanavan leads the Delivery Excellence Group of TCS' Banking and Financial

Services vertical. He also leads a team that is involved in designing and deploying

process performance models at the organization level. He has over 25 years of industry

experience and works with TCS for the past 16 years.

Mr. S Balakrishnan is part of TCS' Corporate Delivery Excellence Group, handling

deployment of PPMs and providing the deployment support for delivery processes. He

works with TCS for the past 18 years.


presentation by dhruba jyoti chaudhuri

Documents

statistical model

model factors

processperformance model

model accuracy

customer satisfaction

csi prediction model

customer retention

loyal customer