knappen smartid presentation_final

May 2015

Heather Knappen, MSResearch Analyst

Metrix Matrix Inc.

Founded in 1997.

Located in Upstate NY.

Extensive expertise and knowledge in survey design, administration, analysis and data submission.

National survey work includes B2B, B2C, government, medical, telecommunications and utility projects.

MetrixMatrix.com (888) 865-0065 2


In 2012, Metrix Matrix implemented the SmartID method by which outbound caller ID area codes were matched to target respondents’ area codes.

This SmartID method was expected to improve efficiency and survey completion rates.

In a 2014 poster presentation to AAPOR, preliminary research revealed that projects conducted with SmartID were less likely to result in a completed survey.


Can this method increase response rates within an individual project when respondents receive calls that are associated with local (i.e., familiar) area codes?

And…

Does the use of SmartID reduce cost(s) by improving performance?

Looking more deeply into the before-and-after effects of SmartID…


Utility Authority within the State of Oklahoma

Residential customers were asked about satisfaction with the utility service/pricing

Telephone survey, 38 questions

Wave 1: 9/5/2014-10/7/2014 SmartID was not used Records were attempted up to 3 times

Wave 2: 10/8/2014- 11/5/2014 SmartID was used in the 405 area code only Records with 3 attempts (from Wave 1) were re-called, with no

more than 2 additional attempts New phone records were added


Region Name Area CodeTarget Sample

CompletedSurveys

(Wave 1)

% CompletedSurveys /Target

(Wave 1)

South West 580 350 319 91%

South Central 405, 580 350 255 73%

Central405, 580, 918/539

450 188 42%

North580, 918/549

700 549 78%

TOTAL: 1,850 1,311 71%


20%

36%

45%

60%55% 57%

100%

51%

100%

70%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

City A City B City C City D City E City F City G City H City I City J

Midpoint Sample Assessment: Central Region Completes, by City

After Wave 1, completed surveys for three cities within the Central Region were still less than 50% complete based on their target sample size. All three cities were also within the 405 area code.


(Contact Made)

24%

6%

63%

7%

34%

11%

52%

4%

38%

9%

50%

2%0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Contact Fail Completed Survey Declined Survey Problem Record

Wave 1 Record Status, by Area Code

405

580

918/539

(9%)

Records within the 405 area code were more likely to result in a contact, when compared to records in the remaining area codes.

However, when contacts were made, calls made to the 405 area code were less likely to result in a completed a survey (9%) when compared to the 580 area code (17%) and the 918/539 area code (15%).

% Complete/Contact Made

(17%) (15%)


(Contact Made)

24%

6%

63%

7%

55%

4%

38%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Contact Fail Completed Survey Declined Survey Problem Record

Wave 1 and Wave 2 Final Record Status for SmartIDin the 405 Area Code

Wave 1

Wave 2

After implementing the SmartID method (Wave 2), records were more likely to result in a failed contact.

When a contact was made, SmartID calls were no more likely to generate a completed survey when compared to Wave 1.


(10%)(9%)


(Contact Made)

40%

5%

51%

4%

58%

3%

35%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Contact Fail Completed survey Declined Survey Problem Record

Wave 2, Record Type with SmartID

Wave 2: Re-loaded Records

Wave 2: NewRecords

In Wave 2, 15% of the SmartID calls were re-loaded sample records from Wave 1 and 81% were new-purchase records.

Overall, new-purchase records were more likely to result in a failed contact.

Re-loaded sample records were more likely to result in an overall contact.


(8%)(9%)


OutcomeMinutes

/CallUnit Cost

Unit Count

Total Project Cost

Wave 1 Dials 0.75 $ 0.19 3,861

Contact Made –Completed Survey

10 $ 2.50 116 $1,606.09

Contact Made –Declined Survey

2 $ 0.50 1,165

Cost per Completed Survey: $13.85

Wave 2 Dials 0.75 $ 0.19 19,825

Contact Made –Completed Survey

10 $ 2.50 264 $5,750.25

Contact Made –Declined Survey

2 $ 0.50 2,647

Cost per Completed Survey: $21.78


The overall cost per completed survey increased by 57.3% from Wave 1 ($13.85) to Wave 2 ($21.78)

Calling effort increased in Wave 2 with more dials needed to complete a survey. Meanwhile, the cost remained nearly equivalent for declined and completed surveys when individuals picked up the phone.

Wave 1 Wave 2

Do

llar

s

Cost per Completed Survey, Wave 1 and Wave 2

Dials Declined Surveys Completed Surveys


Overall, the use of SmartID did not have the anticipated benefits on completion rates and production costs.

Calls made with SmartID were more likely to generate a failed contact.

Records that were re-called in Wave 2 using SmartID were more likely to result in a contact.

When a contact was made, SmartID calls were no more likely to result in a completed survey.

The overall cost per completed survey increased by 57.3% after using the SmartID method (due to increased effort).

However, SmartID may help improve survey coverage…

14MetrixMatrix.com (888) 865-0065

Jim Antonevich, PresidentMetrix Matrix

Chris Horn, Chief Information OfficerMetrix Matrix

Thank you to those who also contributed to this research:

Ann Rogers Lane, Director of OperationsMetrix Matrix

Contact Information:

Heather KnappenResearch AnalystMetrix Matrix [email protected](585) 739-0661


mailto:[email protected]

knappen smartid presentation_final

Documents