Impact of Sales Force Structure Change on Products Performance

Pilot Study

Business Intelligence SolutionsJune, 2015

1

Objectives/Business Questions• Does 2-up promotion of Product A have a positive impact on

its sales relative to 1-up promotion?• The hypothesis behind 2-up promotion: Engaging a 2nd representative in the

promotion will accelerate product adoption and have a positive impact on product performance of relative to 1-up promotion

• Testing 1-up versus 2-up promotion will allow to assess the impact and relative value of a 2nd representative engaged in active promotion of Product A within selected customer segment

• Does the incremental revenue associated with the 2nd sale representative actively promoting Product A provide an acceptable return on the investment?

• Is promoting Product A the best short-term use of the SF1 sales force capacity?

2

Findings / Conclusions

• There is no statistically significant or practically important difference in Product A sales between Test 1 and Control 1 groups

• Promotion cost for Control 1 group is two times higher than for Test 1 group

• The 2-up structure does not produce desired/expected outcome for Product A

3

Structure of Test – Control Groups • Test 1: Product B and Product C• Test 2: Product B, Product C, and Product A• Control: Product B, Product C • Control groups are formed on the basis of the last 2014 quarter sales data• The Test and Control groups were selected to allow for a sufficient number

of matched customers across the two groups to account for other variables that may impact Product A sales

By matching locations with respect to other variables (DTC, business size, geography, etc.) we can effectively isolate the number of representatives actively promoting Product A as the differentiating factor between the groups

4

Product BProduct AProduct BProduct A

Product BProduct C

Product BProduct C

SF2SF2SF2SF2 SF1SF1SF1SF1

Product BProduct AProduct BProduct A

Product BProduct AProduct BProduct A

SF2SF2SF2SF2 SF1SF1SF1SF1

Product BProduct AProduct BProduct A

Product BProduct AProduct C

Product BProduct AProduct C

SF2SF2SF2SF2 SF1SF1SF1SF1

Test 1 Test 2 Control

Methodology• Form Test1- Control1 and Test2 - Control2 groups, using the data of the

last quarter of 2014 and propensity score technique with:– nonparametric nonlinear logistic model– greedy one-to-one matching technique

• Develop Stochastic Gradient Boosting regression models for the first quarter of 2015 for each pair of Test – Control groups, using the following dependent variables:– Product B sales– Product A sales– Product C Salescontrolling for all– “User demographics” variables (sales potential, milestone, state, business size,

etc.)– promotion variables in last quarter of 2014

• Estimate the difference in sales for different sales team

5

One-to-one Matching on Propensity ScorePropensity Score Basics

• Propensity score– is the predicted probability of receiving the treatment (probability of

belonging to a test group)– is a function of several differently scaled covariates

• Propensity_Score = f (Product_B_Sales_Pre, Product_A_Sales_Pre, Product B_Sales_Potential,

State , Product A_Sales_Potential, Product B_Potential_Decile,

Promotion variables, etc.) where f is a non-parametric non-linear

multivariate function, unique for each pair of Test – Control study– If State in ('MA', 'MI', 'MN', 'IL', 'FL', 'NJ') then DTC_Indicator = 1; else DTC_Indicator=0;– If State in ('NC', 'CA', 'NY', 'GA', 'VA') then Paper_Indicator = 1; else Paper_Indicator = 0;

• A sample matched on propensity score will be similar across all covariates used to calculate propensity score

6

Control Groups• Control groups are formed on the base of propensity score

methodology, using only the last 2014 quarter data

• Control1 (for Test1 group with 547 Users): – Users are from Product A 1 – 8 deciles and from the following States:

AL, FL, MI, MN, NC, NJ, WI • Total Unmatched Number of Users: 4,244• Matched Number of Users: 543

• Control2 (for Test2 group with 717 Users): – Users are from Product A 1 – 8 deciles and from the following States:

AL, FL, MA, MN, NC, NJ, TN, WI • Total Unmatched Number of Users: 6,784• Matched Number of Users: 717

7

Propensity Scores Calculation• Approaches/software on non-parametric logistic

regression:– SAS SEMMA (Sample, Explore, Modify, Model, Assess)

methodology within SAS Enterprise Miner– SPSS CRISP (Cross Industry Standard Process for Data

Mining)– Salford Systems CART, MARS, TreeNet, and Random Forest

• Approach selected: SAS SEMMA within SAS Enterprise Miner and Stochastic Garadient Boosting of Salford Systems– Test1 – Control1: (543 Product Users per group)

• Best model: Funnel architecture of Neural Net – Test2 – Control2: (717 Product Users per group)

• Best model: Cascade Correlation architecture of Neural Net

8

Propensity Score: Selection the Best Modeling Paradigm

9

Neural Net was the bestmodeling paradigm

Propensity Score for Test1 – Control1 Groups: Selection the Best Modeling Method

10

Neural Net with Funnel architecturewas the best modeling method

Misclassification Rate: Train Validation 0.11 0.12

Propensity Score for Test2 – Control2 Groups: Selection the Best Modeling Method

11

Neural Net with Cascade architecturewas the best modeling method

Misclassification Rate: Train Validation 0.09 0.10

Matched-Pair Samples Comparison

• Non-parametric tests:– For interval variables:

• Kolmogorov-Smirnov Two-Sample Test – For nominal variables:

• Chi-square test• Before matching there was a significant difference in predictor

distribution across all variables for– Test1 – Control1 – Test2 – Control2

• After matching there was no significant difference in predictor distribution across all variables for – Test1 – Control1 – Test2 – Control2

12

Sales Analysis by GroupTreeNet/Stochastic Gradient Boosting Modeling

• Total number of predictors: 42

• Non-parametric model structure:Dep_var_Post = f(Dep_var_Pre,

Promo_vars_Pre, … User_demographics_vars)

13

Dependent Variable: Product B Sales Post

14

Control1

Test1

Difference is staistically significant but practically not important

Product B Sales Post

Group Product B Mean Visits

Post

Test 1 8.3

Cntrl 1 7.8

Product B Sales Post

Group Product B Mean Visits

Post

Test 2 7.20

Cntrl 2 8.17

Control2

Test2

Dependent Variable: Product B Sales Post for Test1 – Control1

15

Group Product B Mean Visits

2011

Test 1 8.3

Cntrl 1 7.8

The most important 5 predictors of Product B Sales Post:Product_B_Sales_PreProduct_B_Sales_PotentialStateProduct_B_Visits_PreProduct _A_Sales_Potential

Control1

Test1

Difference is practically not important

Product B Sales Post

Dependent Variable: Product C Sales Post for Test1 – Control1

16

Group Product C Mean Visits

2011

Test 1 7.6

Cntrl 1 3.4

Product C Sales Post

Difference is practically not important

Control1

Test1

The most important 5 predictors of Product C Sales Post:Product_C_Sales_PreProduct_A_Sales_PotentialProduct_B_Sales_PotentialStateProduct_B_Visits_Pre

Dependent Variable: Product A Sales Post for Test1 – Control1

17

Group Product A Mean Visits

2011

Test 1 3.8

Cntrl 1 7.7

ConclusionsThere is no statistically significant or practicallyimportant difference in Product A sales between Test 1 and Control 1 groups, but promotion costfor Control 1 group is two times higher than forTest 1 group. In other words, 2-up structure does not produce desired/expected outcome for Product A

The most important 5 predictors of Product A Sales Post:Product_A_Sales_PreProduct_A_Sales_PotentialStateProduct_B_Sales_PreProduct _C_Sales_Pre