evaluation report for electronic customer reporting (ecr) web viewevaluation report for electronic...

Running head: Evaluation report for eCR 1

Evaluation Report for Electronic Customer Reporting (eCR)Eric J. Ericson

Purdue UniversityEDCI 577

Evaluation report for eCR 2

Evaluation Report for Electronic Customer Reporting (eCR)

Electronic Customer Reporting (eCR) is a report generating division of the Ericson Employer and Consumer Services Company (EECSC). Annually, the company trains newly hired employees to use the specialized systems to assist clients who call with questions, problems, or technical guidance requests. One particular system is brand new and slated for launching in the 3rd quarter of 2013. This system is named after the division: eCR. It replaces an older, outdated, system: Manual Customer Reporting (mCR). The Design Strategy Planner is in charge of ensuring new training materials are ready for the 2nd quarter of 2013 for quality testing and eventually 3rd quarter training. This training course will fit into the same slot as mCR had in an already utilized large new hire curriculum. Julian Ericson, the owner’s grandson, is charged with developing the training materials. Eric Ericson, the owner’s son (and Julian’s father) is charged with evaluating the training materials using the Kirkpatrick Four Level Evaluation framework. The evaluation will provide metrics for future use in measuring the program’s effectiveness.

Evaluation Goals and Scope

To know the audience who will be completing the eCR training module, we can look at the Ericson Employer and Consumer Services Company’s hiring strategy. Traditionally, those hired to process mCR (eCR will use the same qualifications) have a minimum of the following:

A high school diploma or equivalent. 2 years’ experience in a call center environment. Able to type 50 words per minute (wpm).

The new hires will undergo training in the curriculum prior to this module covering: Company basics. Customer service skills. Customer sensitivity. Core system overview.

Each of the trainees were hired as full time employees. It is important to note that EECSC has a small staff of contracted temporary employees as well (that will not undergo the eCR training).

Goals

The goals of the evaluation of eCR is to prepare newly hired staff to operate at 75% proficiency once they “hit the floor.” This standard is compared to current floor figures and measured based on elements such as speed to resolution, average call time, quality, and customer satisfaction statistics. It is also expected for new hires to reach 95% proficiency (according to pre-determined standards) 6 months post training.

Purpose

The purpose of this evaluation is to provide Ericson Employer and Consumer Services Company evidence that eCR is being trained at an effective level as measured using Kirkpatrick’s 4 Level Evaluation Model:

Level 1 (Reaction).


Level 2 (Learning). Level 3 (Behavior). Level 4 (Results).

(Kirkpatrick & Kirkpatrick, 2006)

Primary Objectives

For any organization, transitioning to a new platform can be a grueling process. Ensuring the training yields positive results is of utmost importance. The primary objectives of this evaluation are to:

Measure the effectiveness of the training course to demonstrate the value, and Investigate ways to improve it for future new hire classes.

Evaluation Processes

My evaluation strategy utilizes Kirkpatrick’s 4 Level Evaluation Model. In it I measure the training program’s effectiveness at:

Satisfying the customers (new hire trainees). Educating them (knowledge retention). Altering their behavior and performance (preparing them to perform the work). Impacting Ericson Employer and Consumer Services Company’s bottom line (results

leading to gain).

See Appendix A for the evaluation plan and also a timeline for completing the evaluation project.

Evaluation Instruments

Level 1 Evaluation: Reaction. While planning, researching, and prioritizing, I determined that, while Level 1 results are a good indicator of customer (trainee) satisfaction, I placed my least amount of emphasis on this level of evaluation. I decided this after researching ASTD, which recommended spending more money on Level 3 and 4 (behavior and results) and less on Level 1 (reaction) (2009). Regardless of the lower emphasis, I still produced this level of evaluation to get a glimpse from inside the classroom, to ensure the environment and materials were conducive to learning. I understood the value of measuring the feedback from customers (trainees) and used it to ensure that, while efforts are made to incorporate the fundamentals, the course was also fun: fundamental + fun = fully functional. Truly capturing the interest of trainees can endorse a deeper level of learning that can hopefully transfer into the job place through results.

See Appendix B for a copy of the Level 1 (reaction) evaluation survey. It was created using Metrics That Matter software.

Level 2 Evaluation: Learning. Julian supplied me with the list of training objectives in order for me to write an adequate Level 2 evaluation:


After completing this training, you will be able to pass an evaluation at 80% proving you are able to:

1. Classify internal and external Electronic Customer Reporting (eCR) callers.2. Use the eCR Password Reset Standard Operating Procedure to process eCR password

resets for internal callers.3. Demonstrate using the XYZ eCR Job Aid to:

a. Determine which reports are available.b. Distinguish what access a user needs to view reports:

i. Confidential.ii. Non Confidential.

iii. Expanded.iv. Standard.

c. Provide instructions to an external caller about how to run a report. 4. Demonstrate routing internal callers to the help desk.5. Complete a help desk email for an external caller with required information including

his/her:a. Name.b. Phone number.c. Email address.d. Policy number.e. User ID.f. Full description of issue:

i. Date report was run.ii. Report with parameters.

iii. Comments to explain issue.

I created three Level 2 evaluations to be administered throughout the training to observe Gagne’s seventh event of instruction. I also created a final Level 2 evaluation. All evaluations were based on the training objectives. I used the evaluation instrument: Articulate Quizmaker 2009 and the company’s traditional LMS. I chose to use electronic assessment technology to harvest benefits such as accuracy, speed, excellent reporting capabilities, and to create some semblance of initial standard metrics.

See Appendix C for copies of each evaluation.

Level 3 Evaluation: Behavior. In order to assess the transfer of knowledge, and to ensure trainees were effectively performing tasks they learned in training, I interviewed the trainee’s supervisor to acquire information such as performance, quality, and customer satisfaction rating data 6 months after the end of the training program. I also sought to gain the supervisor’s impression of how well the training class had performed on average. The goal of this Level 3 evaluation was to see what they were still practicing on the job. Knowing that even if trainees excelled at demonstrating proficiency during training, that it doesn’t guarantee they would carry over that proficiency on the job, I sought this data to provide solid evidence that they had successfully learned eCR to the desired extent.


See Appendix D for the interview questions I asked to their supervisor.

Level 4 Evaluation: Results. I evaluated at Kirkpatrick’s 4th level one year after the completion of the eCR training. The criteria that I chose to evaluate the results impacted by the eCR training course were those that corresponded to the strategic goals determined by our pre-training stakeholder/strategy planning meeting. Our goal was that the training and utilization of electronic vs. manual customer reporting would:

Increase quality results (higher percentage of work completed without critical errors) from 88% to 95%.

Decrease the Turn around Time (TAT) of call resolution from 96 business hours (old mCR) to 48 business hours (new eCR).

Increase average calls/hour by 50% (from roughly 6 to 9). Increase customer satisfaction figures from 64% to 85%.

The rationale of evaluating these statistics in this way was because real, measurable data such as this would provide a clear direction and ability to make important business decisions regarding fully adopting eCR or reverting back to mCR until a better resolution could be established. Also, with success, these improvements would allow the company to eliminate the need for contracted temporary staff. This would result in a company savings of a projected $312,000/year over year (eliminating 10 positions @ an average of $31,200/yr. expenses [wages, insurance, additional training, other]). The statistics gathered from this first round of level 4 evaluation will be used to establish standards/goal figures that can be used as a benchmark for future eCR trainings.

Analyze Methods

Level 1: Reaction

Since the Metrics that Matter software collects and produces reports, the process of analyzing the results was simple. I received reports via email for each trainee with their individual responses numerical data and comments. I manually compiled these figures into an average and forwarded the comments to Julian to pass along to the training team for encouragement and brainstorming. This method was economical and didn’t require a great deal of time.

Level 2: Learning

Ascertaining level 2 data was equally as easy as level 1 data. The traditional LMS system automatically/electronically compiled individual data reports as well as class reports to provide an average. The reporting information was broken down into groups such as question, objective, and individual results. All reports were emailed to me and also accessible via our LMS direct website.

Level 3: Behavior

Retrieving Level 3 evaluation data was relatively simple considering the electronic measurement systems we have in place. Supervisors have access to results from performance tracking software that tracks the number of calls processors are able to handle, a quality team that takes a random sampling of processors’ work and measures the actions against our pre-established


standard operating procedures (a source of our original training objectives) and calculates a score based on their successful implementation of our rules/standards, and also a feedback phone survey function that some callers are asked to confidentially complete to report their level of satisfaction. All remaining to do was to sit down with supervisors and discuss and view the results, and record them. I placed the results into our evaluation database along with the summary of level 1 and 2 evaluations to make readily available for the stakeholders executive summary. Since the participants were all new hires, it is implied that there are no outside influences affecting their ability to perform, therefore no counter measures were performed to measure such influence. I also made it clear that if any additional training were to be performed, that I must be notified so I can consider these possible influences when evaluating.

Level 4: Results

One year following training, I viewed the overall quality, calls/hour, TAT, and customer satisfaction statistics. I accomplished this by utilizing the trainees’ supervisor once more to get updated, current data. Again, he had access to electronic performance software, I was able to acquire quality statistics from the quality team and view the electronic compilation of customer satisfaction data produced by our phone survey software. Combining this data into averages and measuring them against our standards yielded comparable results.

Stakeholder Involvement

Key stakeholders were involved in the evaluation plan first by being invited to a pre-program planning committee meeting. During this meeting, stakeholders were made aware of program and evaluation goals. They were also asked for their input and what they were willing to take ownership of to ensure the success and smoothness of the eCR transition. Stakeholders were empowered to be a part of the process and to collect data. Stakeholders were asked to ensure that the evaluation-specific data collection was accurate and ongoing throughout the year. Lastly, key stakeholders were invited to an executive summary meeting.

Executive Summary

While results were encouraging, not all results met the goals. We were all aware we had set very high standards with high hopes. I decided to hold a meeting with the stakeholders, designers, and training staff to discuss the evaluation project’s findings. This would provide an excellent opportunity to explain in detail some of the elements that often get lost in charts and graphs in a memo. This also is advantageous as it allows an opportunity for questions and interjections, potentially even valid suggestions for further investigation or future exploration.

View Appendix E for the executive summary meeting minutes.


ReferencesKirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Training Programs. San Francisco:

Berrett-Koehler Publishers, Inc.The value of evaluation: Making training evaluations more effective. (2009). Alexandria, VA.

ASTD.


Appendices

Appendix A

The following is the evaluation plan and next, the evaluation project timeline.

(Cont’d on next page)


Appendix B

The following is the Level 1 evaluation survey I created for measuring the ECR training program trainees’ reactions.


Appendix C

The following are all the Level 2 evaluations I created for measuring Learning.

Level 2 Evaluation 1 – Knowledge Check Questions

1. (Multiple Choice Question, 0 points, 1 attempt permitted)

Correct!


Incorrect

2. (Multiple Response Question, 0 points, 1 attempt permitted)


Feedback when correct: That's right!

The correct answers are 1txiong and mnahag2.

Neither 1txiong nor mnahag2 end with a u, so they are possible external user IDs.

hupto4u and jsalv3u both end in u so you know they are internal callers.

Feedback when incorrect: That is not correct.

The correct answers are 1txiong and mnahag2.

Neither 1txiong nor mnahag2 end with a u, so they are possible external user IDs.

hupto4u and jsalv3u both end in u so you know they are internal callers.

3. (Hotspot Question, 0 points, 1 attempt permitted)


Feedback when correct: That's right! You selected the Change Password link. From

here, you enter a user's last name.

Feedback when incorrect: You did not select the correct link. You select the Change

Password link. From there, you enter a user's last name.



Correct!

Incorrect




You selected Continue to this website (not recommended).


From here, you log in.

Feedback when incorrect: You did not select the correct area.

You need to select Continue to this website (not recommended).



1. Use the XYZ eCR Job Aid to answer the following question.

Which of the following reports IS NOT available under eCR?

(Multiple Choice Question, 0 points, 1 attempt permitted)

Correct Choice

Key Generic Substitution Indicators by Month.

X Large Utilization Count.

Membership by Market.

Utilization by Diagnosis.


You selected the correct response, Large Utilization Count.

All other reports are listed in the job aid.

Feedback when incorrect: You did not select the correct response.

The correct response is Large Utilization Count.

All other reports are listed in the job aid.


2. (Multiple Response Question, 0 points, unlimited attempts permitted)


The correct answer is Confidential and Expanded.

Standard level users only see the reports with an ** after them.

Non Confidential level users see all reports EXCEPT Detail Payment or Large Loss

reports.

Feedback when incorrect: Incorrect.

The correct answer is Confidential and Expanded.


Non Confidential level users see all reports EXCEPT Detail Payment or Large Loss

reports.


3. Use the XYZ Job Aid to answer the following question.

Which category is the Top Ranked by Net Paid report under?

(Word Bank Question, 0 points, unlimited attempts permitted)

Correct Choice

Financial Reports.

Managed Pharmacy Reports.

X Medical Reports.

Membership Reports.


The correct answer is Medical Reports.

It is located toward the bottom of the Medical Reports list.


The correct answer is Medical Reports.

It is located toward the bottom of the Medical Reports list.

4. (Matching Drag and Drop Question, 0 points, unlimited attempts permitted)



The correct order is:

1. Select a Policy Number or Group Segment Filter (GSF).

2. Select a Date Type.

3. Add More Details to the Report.

4. Limit the Report's Results.


The correct order is:









You selected the correct response: Never.

When an external caller is having technical issues with an eCR report, we write the email

on their behalf, but do not provide the email address to them.

When he/she is locked out, we reset the password via the Employer eServices portal.

Feedback when incorrect: You did not select the correct response.

The correct response is: Never.

When external callers are having technical issues with an eCR report, we write the email


When they are locked out, we reset the password via the Employer eServices portal.


2. (Multiple Response Question, 0 points, unlimited attempts permitted)

Feedback when correct: That's right! The correct answers are:

-Phone number.

-User ID.

-Report with parameters.

Mailing address and user date of birth are not necessary.

Feedback when incorrect: Incorrect. The correct answers are:

-Phone number.

-User ID.

-Report with parameters.

Mailing address and user date of birth are not necessary.


3. (Fill in the Blank Question, 0 points, unlimited attempts permitted)

Choice

[email protected]

[email protected].

Feedback when correct: That's right! The correct answer is [email protected].

Feedback when incorrect: Incorrect. The correct answer is [email protected].

4. (Word Bank Question, 0 points, unlimited attempts permitted)



The correct answer is external caller with eCR technical issues.


The correct answer is external caller with eCR technical issues.

Level 2 (Learning) Evaluation - Final Assessment Questions



Feedback when correct: The correct answer is: Ends with a u.

The quickest way to distinguish between an internal and an external caller is by his/her

user ID.

When a user ID ends with a u, this indicates an internal user.

External callers' user IDs do not end with a u.

Feedback when incorrect: The correct answer is: Ends with a u.

The quickest way to distinguish between an internal and an external caller is by his/her

user ID.

When a user ID ends with a u, this indicates an internal user.

External callers' user IDs do not end with a u.



Feedback when correct: You selected the correct link: Users.

After clicking this, a new link populates called:

Search for User to Edit.

Feedback when incorrect: You selected the incorrect link.

The correct link is Users.

After clicking this, a new link populates called:

Search for User to Edit.

3. From this screen, where do you click to advance to the eCR Admin Tool?

(Hotspot Question, 1 points, 1 attempt permitted)


Feedback when correct: You selected the correct location: Continue to this website (not

recommended).


Feedback when incorrect: You selected the incorrect location.

The correct location is: Continue to this website (not recommended).

From there, you log in.

4. Use the XYZ Job Aid to answer the following question.

Which of the following reports are available under eCR?

(Multiple Response Question, 1 points, 1 attempt permitted)

C

orrect

Choice

X Key Generic Substitution Indicators by Month.

Maximum Pharmacy Outputs.


X Membership by Market.

X Utilization by Diagnosis.

Feedback when correct: The correct answers are: Key Generic Substitution Indicators by

Month, Membership by Market, and Utilization by Diagnosis.

Maximum Pharmacy Outputs is fictitious and is not a report available under eCR.

Feedback when incorrect: The correct answers are: Key Generic Substitution Indicators

by Month, Membership by Market, and Utilization by Diagnosis.

Maximum Pharmacy Outputs is fictitious and is not a report available under eCR.



Feedback when correct: The correct answer is: Non Confidential.


Confidential and Expanded level users see ALL reports.

Feedback when incorrect: The correct answer is: Non Confidential.


Confidential and Expanded level users see ALL reports.

6. Use the XYZ job aid to answer the following question.

Which category is the Bill Count by Month report under?

(Word Bank Question, 1 points, 1 attempt permitted)

Correct Choice

Financial Reports.

Managed Pharmacy Reports.

X Medical Reports.

Membership Reports.

Feedback when correct: The correct answer is: Medical Reports.

It is located near the middle of the Medical Reports list.

Feedback when incorrect: The correct answer is: Medical Reports.

It is located near the middle of the Medical Reports list.


7. (Matching Drag and Drop Question, 1 points, 1 attempt permitted)

Feedback when correct: The correct order is:





Feedback when incorrect: The correct order is:







Feedback when correct: The correct answer is: Never.



When an external user is locked out, we reset the password via the eCR website.

Feedback when incorrect: The correct answer is: Never.



When an external user is locked out, we reset the password via the eCR website.

9. (Multiple Response Question, 1 points, 1 attempt permitted)


Feedback when correct: The correct answers are:

Email address, name, and policy number.

Direct supervisor name and fax number are not required or necessary.

Feedback when incorrect: The correct answers are:

Email address, name, and policy number.

Direct supervisor name and fax number are not required or necessary.

11. Which is the correct email address of the eCR Help Desk?

(Multiple Choice Question, 1 points, 1 attempt permitted)

Correct Choice

[email protected].

X [email protected].


[email protected].

[email protected].

Feedback when correct: The correct answer is: [email protected].

This is the email address you give to internal callers when:

-You are unable to reset his/her password.

-He/she is having further issues with eCR.

The other email addresses are fictitious.

Feedback when incorrect: The correct answer is: [email protected].

This is the email address you give to internal callers when:

-You are unable to reset his/her password.

-He/she is having further issues with eCR.

The other email addresses are fictitious.

11. (Word Bank Question, 1 points, 1 attempt permitted)


Feedback when correct: The correct answer is: External caller with eCR technical issues.

Reset an external caller's password via the eCR website.

Reset an internal caller's password via the eCR Admin Site.

Provide the eCR Help Desk email address to internal callers with eCR technical issues

(they can write their own email).

Feedback when incorrect: The correct answer is: External caller with eCR technical

issues.

Reset an external caller's password via the eCR website.

Reset an internal caller's password via the eCR Admin Site.

Provide the eCR Help Desk email address to internal callers with eCR technical issues

(they can write their own email).


Appendix D

The following is the Level 3 (Behavior) evaluation interview form I used with the supervisor of each trainee of the eCR course.

Ericson Employer

and Consumer Services

Company

Level 3 Behavior Interview/Performance Skills Assessment

Review Information

Your Name: [Your Name]

Date: [Date] Review Period: October 1, 2012 to April

1, 2013

Guidelines

Complete this review, using the provided scales.

The middle figures are our standards/goals for this evaluation.

Repeat the review annually and compare your yearly scores.

Quality

(5) = 99% (4) = 97% (3) = 95% (2) = 93% (1) = Below 93%

Processes eCR according to SOP ☐ ☐ ☐ ☐ ☐


Listens effectively to callers ☐ ☐ ☐ ☐ ☐

Communicates effectively (overall) ☐ ☐ ☐ ☐ ☐

Verbal communications ☐ ☐ ☐ ☐ ☐

Written communications such as reports ☐ ☐ ☐ ☐ ☐

Email etiquette ☐ ☐ ☐ ☐ ☐

Telephone etiquette ☐ ☐ ☐ ☐ ☐

Customer Satisfaction

(5) = Exceptional(4) = Exceeds Requirements

(3) = Meets Requirements (2) = Gets By

(1) = Needs Work

Strong customer advocate ☐ ☐ ☐ ☐ ☐

Sets aside personal biases and wants ☐ ☐ ☐ ☐ ☐

Gives good, practical guidance ☐ ☐ ☐ ☐ ☐

Fosters loyalty in customers ☐ ☐ ☐ ☐ ☐

Productivity (compared to standard)

(5) = Exceptional(4) = Exceeds Requirements

(3) = Meets Requirements (2) = Gets By

(1) = Needs Work

Average call handle time ☐ ☐ ☐ ☐ ☐

Average calls/hour ☐ ☐ ☐ ☐ ☐

Project TAT completion rate ☐ ☐ ☐ ☐ ☐


Appendix E

The following are the meeting minutes of the executive summary meeting I held for stakeholders, design and training:

Level 1 evaluation survey results indicated to us that trainees were pleased with the objectives of the course matching the content, the relevancy to their job, and the way the materials were written overall (engaging and grammatically correct). Trainees did however report lower scores for sufficient opportunities for practice. We will look into creating simulations and/or alternatives to enhance practice for this course if our budget will allow.

Level 2 evaluation results were astounding. The average score of the class was a 99%! The knowledge gained due to training was a huge success. We think we attribute such success to multiple knowledge checks along the way and to an excellent design and training staff. Our hiring staff also deserves recognition due to carefully selecting excellent prospects who have delivered excellent results.

Level 3 evaluation results yielded pleasing statistics providing evidence that trainees truly had performed at a level that was trained and applied the objectives well after 6 months. Their ability to follow the standard operating procedures and use eCR efficiently provided excellent evidence that the training was a success.

Level 4 results yielded slightly below the desired results, but still an excellent improvement when compared to the previous mCR statistics. Quality had raised from 88% to 93%, 2% shy of the goal, but still a great improvement. Further planning and investigation will be held in order to continue improvements on this aspect. Customer satisfaction has greatly increased (64% to 88%) primarily (we think) because our callers are having to wait for problem resolution nearly half as long as they previously were with mCR (an average of 50.4 business hours vs. 96 business hours). This is a most pleasing result as it decreases strain, lowers FTE requirements, and keeps us a competitive option for our customers. Lastly our average calls per hour goal was met as our staff now handles an average of 9 calls/hour vs. 6 (mCR).

Based on these results, it has been decided to retire mCR completely in the archives. ECR will be the primary/sole operating system from this point forward. We are able to eliminate the need for half of the contracted temporary staff (for now) which is still a savings of $156,000 year after year. We will carefully consider future decisions as we test the waters and see how we fare. As the new full time staff become seasoned, and our training program gets shaped and molded into a more successful version, we will consider further future business solutions.

evaluation report for electronic customer reporting (ecr) web viewevaluation report for electronic...

Documents