project 4: the evaluation plan rachel murch edci 577 ... · mailchimp provides online marketing...

Running head: PROJECT 4: THE EVALUATION PLAN 1

Project 4: The Evaluation Plan

Rachel Murch

EDCI 577 Section 4

Purdue University

PROJECT 4: THE EVALUATION PLAN 2

Executive Summary

MailChimp is an online email marketing company that can be used by anyone who needs

to send mass emails or email newsletters. With a wide audience of customers who continue to

use the tool and pay for the service, MailChimp is sure to increase profits by increasing its

customer base. To help increase the number of customers, MailChimp needs to evaluate the

support resources that customers use to learn how to upload a list of email addresses and send an

email (or campaign). The Getting Started online training videos will be evaluated to determine

how effective they are in making it easy for customers to use the MailChimp service. The

objective of this evaluation is to increase customer satisfaction and revenue by providing online

training videos that teach customers how to use the MailChimp tool.

MailChimp’s evaluation plan will be executed using Kirkpatrick’s 4-level evaluation

model, which includes measuring customer reaction, learning, behavior, and results. The entire

evaluation process will take 10 weeks, but can be repeated as necessary once the first cycle is

complete. Customer evaluations will be administered over a 4-week time period, with results

being tracked on a daily or weekly basis. Since interaction with customers is virtual, the

evaluations will be delivered and tracked electronically, which will help to keep evaluation costs

down.

The level 1 evaluation will include both a customer reaction questionnaire and a

continuous feedback option. These tools are designed to gather feedback about the training

videos so that adjustments can be made to improve the learning experience, if necessary. Level 2

will be measured using a combination of pre- and post-tests and using a focus and experimental

group of customers. Using a focus group will help to isolate the learning that occurred because

of the training videos. Also, participation will be voluntary but incentives will be used to

increase participation. The third level will be evaluated using Google Analytics to track

customer behavior on the website. By using this tool, the evaluation team will have a good

understanding of when the videos are being accessed and what section of the website the

customer came from. Level 4 will be measured by taking some customer and revenue statistics

before and after the evaluation program is administered. These calculations will help to

determine the results due to the training.

A final report will be developed to show the evaluation results and any

recommendations. Expected results of the evaluations include a significant increase in revenue

and customer use of the MailChimp service. If the results prove to be different, then the training

program will need to be adjusted appropriately.


Introduction

MailChimp provides online marketing tools such as mass email and email newsletters

that are simple, easy to use, and fun! These tools are used by people and businesses all over the

world because MailChimp makes email marketing so simple and attractive. Though the culture

at MailChimp is very casual, don’t be fooled, because the 100 or so employees are focused on

providing “the best email-marketing service in the universe” and providing a “personal

publishing platform” for users.

One obvious advantage that MailChimp has over its competitors is its user interface.

When accessing the home page, it is hard not to feel invited in to explore. Users from the novice

to the most advanced marketer are comfortable diving in to see and love what MailChimp has to

offer.

So what does MailChimp have to offer? This paper will focus on the section of the

MailChimp website that provides user support. This area of the website includes some Getting

Started videos that guide the user through the process of sending emails (or “campaigns” in

MailChimp terminology). Because the target market for MailChimp is so wide, with a variety

of experience levels, it is important to support all users in a way that meets their needs and also

matches the company culture. MailChimp seems to have found a way to do that through fun and

informative online training videos. However, it would be useful to validate MailChimp’s

assumption by conducting a full evaluation program using the 4-level Kirkpatrick evaluation

framework as a guide.

The instructional product being evaluated in this paper is Parts 1 and 3 of MailChimp’s

Getting Started Videos. Part 1 covers Creating and Importing a List of Subscribers, and Part 3

covers Creating and Sending a Campaign. These support videos are available to the public and

located on the Support tab of the MailChimp.com website. This document will outline an

evaluation plan and framework, the tools and procedures for doing such evaluations, and the

timeline for which to do so.

Audience and Purpose

MailChimp is an online company that caters to anyone who uses the Internet and wishes

to send a large number of emails (called campaigns). A tool like this can be very useful because

many email clients have limits on the number and size of emails that can be sent. Even with

these limits, a client can usually send a large list of emails to friends and family for reasons such

as electronic holiday cards, invitations, or announcements. In a business environment, however,

the given limits can be very restricting. One possible way around this is to send emails to

sections or groups at a time, or to use a tool like MailChimp.

Because people and companies have different needs, MailChimp caters to a diverse group

of users. Its target audiences include people who want to send a campaign one time, those who

want to send an occasional campaign, and those who send campaigns regularly. Regular users

will not reference the training videos frequently, but casual users will reference the videos again

and again. The training video modules evaluated in this paper are geared toward those people

who have not used MailChimp before (or who don’t use it often) and who want to understand

exactly how to create an email “campaign” and create or import a list of people to send the email

to. The content for these two modules covers the basic requirements for sending a campaign.


The purposes of this evaluation program are to: 1) gather user reaction and gauge the

level of learning from Parts 1 and 3 of the Getting Started training videos; and 2) determine the

effectiveness of the Getting Started training videos.

Evaluation Framework

As mentioned above, the Kirkpatrick evaluation method is the framework used to

evaluate MailChimp’s Getting Started online training videos in this Evaluation Plan. The

traditional 4 levels for this model include measuring reaction, learning, behavior, and results.

The following sections describe the tools used for each level of evaluation, the procedures with

which to implement those tools, and possible limitations.

Level 1: Evaluating Reaction

The first measurement instrument (Appendix C) is a questionnaire designed to capture

the reaction of customers who use the training videos as a support resource. Because MailChimp

is an online company, it must be able to support its customers virtually. Measuring customer

reaction to the support resources is one way to gauge customer satisfaction with regard to the

training videos (Kirkpatrick & Kirkpatrick, 2006, p. 27). Also, if customers know how to use the

MailChimp tools, then they are more likely to have a good experience with the website and want

to continue using it, which keeps the company in business. For this reason, the training videos

must be a great resource to provide virtual customer service and support.

Measuring customer reaction can be done in many ways, but for this situation the

reaction questionnaire would launch at the end of each training video. Having the questionnaire

linked to the video would help to increase the response rate (with the optimum response rate

being 100%). However, because many customers could view the videos again and again, there

would be an opt-out button so that customers wouldn’t get upset that they are forced to provide

feedback with each viewing. This opt-out option could lead to a lower response rate, but the

benefit of not upsetting customers would outweigh the risk.

The first part of the reaction questionnaire contains multiple choice questions designed to

gather information such as which training modules the users viewed, their overall reaction to the

video, and the quality, timing, and professional design of the videos. These questions can help

MailChimp understand what characteristics of the videos may need to be improved upon. The

second part of the questionnaire asks customers to identify both things that they felt could be

improved, and things they liked best. Allowing some open-text answers will enable MailChimp

to get honest feedback from customers.

Another way to measure reaction in an online environment is to gather feedback as the

learner is doing the training. A continuous feedback icon (Appendix D) would be available on

the screen as the customer is viewing the training videos. It would offer the opportunity for

customers to click and provide feedback at a particular section of the content when the thought is

fresh in their minds.

Implementing the reaction evaluation tools does not have to be contingent on this

evaluation process. Similar feedback is customarily gathered continuously for any and all

learning tools or classes. For this reason, the reaction tools could be implemented as soon as


possible, though the formal process that includes analysis and reporting will not begin until the

beginning of November 2012. However, for the purpose of this plan, all documents and

timelines have been created with the assumption that all evaluations will follow the timeline

created for this evaluation program.

Level 2: Evaluating Learning

The second measurement instrument (Appendix E) is designed to capture if and what

learning occurred from watching the video training modules. Certain skills and knowledge are

needed in order to efficiently and appropriately use MailChimp’s service, which can also impact

levels of customer satisfaction (similar to reaction). The learning instrument consists of multiple

choice, fill-in, and true/false questions that directly relate to the content covered in the training

videos.

Using a focus group and experimental group, and administering pre-and post-tests to

measure learning, are great ways to isolate what learning occurred because of the training.

MailChimp learning can be evaluated using both of these methods, and though participation from

users is critical, this can be successful if the right tools are used. In an online environment, there

is no option to observe customer behavior in person, so it is necessary to use an analytics tool to

track customer behavior on the MailChimp website. See below for the procedures and activities

for the focus group and experimental group:

Focus Group.

The focus group would consist of customers who did not watch the training videos, but

who were able to create a list and send a campaign to subscribers anyway. This group would be

randomly selected to participate in a process to evaluate how user-friendly the website is without

instruction. Though participation would be optional, an incentive of a free campaign would be

offered to help entice customers to participate

This group would first be asked to take a pre-test learning questionnaire (Appendix E).

Next, they would be asked to create a list and send a campaign to that list of subscribers. Then,

once the campaign was sent, this group would be prompted to complete the post-test learning

questionnaire (Appendix E) that is comprised of the same questions from the pre-test.

Experimental Group.

The experimental group would consist of customers who are randomly selected to take

part in a process to evaluate the usefulness of the training videos, with the incentive of getting a

free month of campaigns. First, they would be prompted to take a pre-test learning questionnaire

(same as what the focus group took). After taking the pre-test, they would be asked to watch the

training videos. Once the training videos were viewed, they would be instructed to create a list

and send a campaign to that list of subscribers. The last step would be for them to take the

learning questionnaire as a post-test. See Table 1 for the Evaluation Tools for Levels 1 and 2.

There may be some challenges with this group due to the amount of time and effort it will

take to participate. Though each participant gets a free month of campaigns, it might not be

enough of an incentive. The compensation needs to be worthwhile and should be increased as

necessary to increase participation.


Level 3: Evaluating Behavior

Tracking customer behavior on the web can be done efficiently and accurately by using

Google Analytics. This tool makes it possible to do many things, including tracking specific

events that take place online. This is called Event Tracking, and will be used in this evaluation

to track how many times a user clicks on a particular link, button, or webpage. In addition,

tracking will include the location from which the customer originated. See Example 1 below.

Example 1:

Situation: Customer A is a participant in the Evaluation Program. Customer A and is on

the Campaigns screen and navigates away from it, to the Resources page and clicks

“play” on Video 3. After watching the video, Customer A navigates back to the

Campaigns page and successfully creates a campaign.

Google Analytics Results: Customer A came from the Campaigns page and watched

Video 3, one time.

The assumption, based on Example 1, is that Customer A was unaware of how to create a

campaign and therefore watched a training video to learn. Because Customer A is a participant

in the Evaluation Program, MailChimp also has tracking of whether he/she is a part of the focus

or experimental group. If he/she was in the focus group, then the training hadn’t been viewed

before, and therefore the training was effective in helping create a campaign. If Customer A was


in the experimental group, then some assumptions can be made about the training itself. It is

possible that the video was too complicated to digest in one viewing, or for some other reason

that may have been indicated in the reaction feedback from the level 1 evaluation.

Appendix F shows a tracking table that will be used to capture customer behavior. The

tracking will show which main page the customer came from: Dashboards, Campaigns, Lists, or

Reports, and how many times the “play” button was clicked on which video. This information is

valuable because if a customer is on Page X and goes to the Support page and watches Video X,

then it is understood that the user didn’t know or remember how to do something on that page.

Although online customer behavior can be tracked, it is impossible to understand the

reason behind that behavior. For example, a customer may have accidentally clicked on a link,

page, or button and it was an unintended behavior. The electronic tracking of that behavior only

knows that it happened, not for what reason it happened. For this reason, the evaluation team

will need to look at statistically significant outliers and exclude them from the results as a part of

the analysis process.

Level 4: Evaluating Results

Evaluating the results of the Getting Started training videos will be done electronically,

using a predetermined set metrics that can be tracked and reported on automatically. These

include:

1. Average number of campaigns sent per user

2. Average number of list uploads per user

3. Average number of new customers

4. Average revenue per user account.

The calculations for the above information will be made at the beginning of the

evaluation program, as listed in the Evaluation Plan (Appendix A) and in the Gantt Chart

(Appendix B). To track progress, a pulse will be taken at the end of each week, for four weeks.

At the end of the evaluation program, the final calculations will be made and compared to the

baseline metrics. To do this, a report will be generated for the evaluators to use when writing the

final report. In addition, the evaluators will complete a questionnaire that asks them to identify

potential external reasons why the calculations are what they are. For instance, there may have

been some site downtime that caused customers to not be able to login to their accounts.

Another reason might be that the upload feature of the website was down for a period of time. It

is important to include an option for the evaluators to provide any explanations of situations that

could have impacted the results and would not have been part of the calculations on the report.

See Appendix G for an example of the report and the questions that accompany it for the final

results.

Expected Findings and Reporting

Reaction Evaluation

The reaction questionnaire (Appendix C) results will be gathered and reported in an

electronic format that allows for easy viewing and analysis of the data. Acceptable results for

this evaluation would be an average of 4 (or equivalent) or better. For instance, the first main


question asks how the person would describe the training, and an acceptable average reaction

level would be the “It was super helpful!” or “It was helpful” answers only. If training isn’t

helpful, then it is useless. For the other questions, an average rating of 4 or 5 would be

acceptable. The open-ended questions would be used for analyzing the questions that precede

them.

Using the electronic tool that tracks these responses, the training team must first

determine what the average rating was for each question on a weekly basis. Based on those

ratings, it might then make changes to the training videos, such as timing, narrators, and/or

design. Weekly reports will be created and available for company management as necessary.

See Table 2 for an example of the Level 1 Results Report.

Table 2


The results of the second reaction evaluation (Appendix D) will be monitored on a daily

basis by a help desk team. This continuous feedback would either be acted on immediately, in

the event that there is an issue with the videos, or it would be sent to the training team and added

to the data from the last two questions on the reaction questionnaire.

Learning Evaluation

Pre-test and post-test scores for the focus and experimental groups will be tabulated

(electronically) and compared.

Expected Results:

Pre-test scores for the focus group are expected to be similar (+/- 5%) to those of

the experimental group.

Post-test scores for the experimental group are expected to show an increase

(approximately 20%) higher than those of the focus group (calculation: post-test

score average minus pre-test average score).

Reporting on the learning evaluations will be done based on the number of “hits” to the

training videos. The higher the number of “hits,” the more often the evaluations will be

tabulated, and vice versa. At a minimum, a monthly report will be generated and analyzed by the

training team. The team would need to take into consideration unknown variables such as

customer experience using online training tools, customer comfort with technology, incomplete

evaluations, and the fact that customers may have been rushing through the questions.

Behavior Evaluation

Customer behavior will be tracked and reported through the Google Analytics tool using

Event Tracking. The expected behavior is such that a user who has already viewed the Getting

Started videos (or experimental group) will navigate back to the videos 25% less than the users

who did not watch the videos in the first place (or focus group). See Appendix F for the tracking

that will occur while a user is on the MailChimp website.

Results Evaluation

The results report (Appendix G) will be used to report weekly results to management. At

the end of the evaluation period, the report will also be used to compile the final report.

Automatic processes will provide the statistics, but the some of the analysis will need to be

completed by one of the evaluators.

The objective of the evaluation process is to increase customer satisfaction and revenue

by providing online training videos that teach customers how to use the MailChimp tool.

Ultimately, the management team is looking for an indication that the videos are effective at

increasing the use of the MailChimp tool. An increase in the use of the tool correlates to

increased revenue. The expected results include:

A 25% increase in the average number of campaigns sent per user.


A 10% increase in the average number of list uploads per user.

A 5% increase in the average number of customers/users.

A 20% increase in the average revenue per user account.

Conclusion

MailChimp has a diverse audience of people who want an email marketing tool that is

easy to use. Engaging and informative online training videos can help make the MailChimp

service simple and attractive to a customer. If a customer is happy with the service and the

support offered, he/she is more likely to continue to use the service. For this reason, Parts 1 and

3 (the tool basics) of the Getting Started training videos are critical to the success of the business.

This evaluation will be used to gather information to help make the training videos more

effective, and subsequently, increase revenue. Measuring customer reaction, learning, behavior,

and results will help provide a pulse of how effective the videos are now, and will unveil any

areas needing more attention or improvement. The procedures and tools for each of the levels of

evaluation are described in this document and will serve as a roadmap for executing the

evaluation program.


References

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels

(3rd

ed.). San Francisco: Berrett-Koehler Publishers, Inc.

Part 1: Creating and Importing a List of Subscribers [Video file]. Retrieved from

http://MailChimp.com/support/webinar-recordings/

Part 3: Creating and Sending a Campaign [Video file]. Retrieved from

http://MailChimp.com/support/webinar-recordings/

http://mailchimp.com/support/webinar-recordings/

http://mailchimp.com/support/webinar-recordings/


Project

Scope

Creator

ID

Purpose

Level 1 Reaction

Level 2 Learning

Level 3 Behavior

Level 4 Results

Appendix A

Evaluation Plan

Person 2

Increase customer satisfaction and revenue by providing online training videos

that teach customers how to use the Mailchimp tool.

10/16/2012

Google Analytics: $2500

TOTAL: $26800

Review/present results to Steering

Committee12/14/2012

Eval Team Lead, Tech

Team, and Eval Team

Evaluation BudgetTech Resources: $9300

Evaluation Resources: $15000

Prepare final report 12/13/2012Eval Team Lead and Eval

Team

Collect and Analyze data (weekly) 11/5-12/4/2012 Person 2 and Person 3

Launch Level 1 & 2 Evals in Production: 11/5/2012 Tech Team

Collect and store baseline metrics: 11/2/2012 Evaluation Team

Set up Google Analytics Event Tracking: 11/2/2012 Tech Team

Develop instruments: 10/24/2012 Evaluation Team

10/24/2012 Person 3

10/18/2012

10/23/2012

Person 1

Person 1

ProcessTask Final Date Due Who

Create evaluation plan: 10/19/2012 Eval Team Lead

Levels of

Evaluation 1 Reaction 3 Behavior

2 Learning 4 Results

Objectives

that will be

measured

Evaluation PlanMailchimp Online Training Videos

Evaluation of Parts 1 and 3 of the Getting Started Training Videos

Mailchimp Online Training Evaluation Plan

GoalsDetermine the value of the training videos.

Rachel Murch


Data Item By Whom

Continuous

FeedbackElectronic

Reaction Electronic

Pre-test Electronic

Post-test Electronic

Event

tracking

Google

Analytics/

Electronic

Company

revenue

Finance

and Eval

Team

Weekly avg

of new

customers

Finance

and Eval

Team

Weekly avg

campaigns

sent per

user

Finance

and Eval

Team

Weekly avg

of lists

uploaded

per user

Finance

and Eval

Team

Behavior Electronic trackingGoogle Analytics tracking

specific behaviorPerson 2

Before launching the eval

program and after each of

4 weeks

Internal finance toolsCompany health

report



4 weeks


report



4 weeks


report

1. Transfer of learning metrics

2. Value of training

3. Action plan for modifications

Steering Committee Electronic report and presentation

ResultsRanking and comparison

using output averagesManual explanations

Eval Team Lead

and Person 3

Report of FindingsWhat results are reported To whom How communicated

Learning Average scoresMaterial only covers

what is in training videosPerson 1

Data Analysis

Target MeasuresMethods of performing

calculations

How to control for

other factors

Who will

perform analysis

ReactionAverage scores and

common themesContinuous feedback link Person 1

After watching the training Electronic Survey Customer/User



4 weeks


report

After training, while

customer is using the

Mailchimp tool

Google Analytics Customer/User

During training Electronic Survey Customer/User

Before training Electronic Survey Customer/User

After each training video Electronic Survey Customer/User

Data CollectionWhen How Collected From Whom


Appendix B

Gantt Chart


Appendix C

Level 1 Evaluation

Thank you for using our online training tutorials! So that we can continue to provide awesome

support for you, we ask that you please complete this brief survey.

I completed the following online training module(s):

Part 1: Creating and Importing a List of Subscribers

Part 3: Creating and Sending a Campaign

I would describe the online training to one of my friends like this:

It was super helpful!

It was helpful!

It was fine.

It was not as helpful as I would have liked it to be :\

It was a complete waste of time

Please use the following scale:

5- Strongly agree 4- Agree 3- Neutral 2- Disagree 1- Strongly disagree

The quality of the online tutorials was professional:

5 4 3 2 1

The videos were enjoyable to watch and kept my interest:

5 4 3 2 1

The timing of the videos was appropriate for the topics covered:

5 4 3 2 1

The narration of the video was clear and easy to understand.

5 4 3 2 1

I feel confident that I can create and send a campaign to a list of subscribers.

5 4 3 2 1

The videos could be improved in the following ways:

______________________________________________________________________________

______________________________________________________________________________

What I liked best about the training videos is:

______________________________________________________________________________

______________________________________________________________________________

Thank you for providing honest feedback!


Appendix D

Level 1 Evaluation: Continuous Feedback


Appendix E

Level 2 Evaluation – Pre-test and Post-test

Please select the option that best describes you:

I have NOT watched the MailChimp Getting Started training videos

I HAVE watched the MailChimp Getting Started training videos

MailChimp terminology for email content is a: ________________________________________

MailChimp terminology for a list of email addresses as a: _______________________________

Place a checkmark beside the two requirements, as mandated by law, that any email coming

from MailChimp must have:

An easy way for subscribers to remove themselves from your list

A subject line explaining what the email is about

The business address from the business that is sending the email

A plain-text only version available

To ensure that your email has the highest number of opens and click-throughs, it is a good idea

to allow your subscribers an opportunity to view the email in different formats.

True False

Creating different groups can be useful when: _______________________________________

When importing a list, you can use any kind you want, including 3rd

party lists.

True False

Select all the options available to you when creating your list:

Copy/past from Excel

Upload from a file

Upload list from Constant Contact

Type everything in manually

Import from a number of external services such as Salesforce

All of the above

Select the four types of campaigns that MailChimp offers:

Regular ol’ campaigns

Plain-text campaigns

Fun campaigns

A/B split campaigns

Professional/direct campaigns

RSS-driven campaigns

By selecting a segment of an existing list, you can target subscribers for a particular campaign.

True False


By selecting to use Google Analytics, you can see if your campaign is driving traffic back to

Google.

True False

Choose the template options available to you when designing the content for a campaign:

Create a basic template

Create a designer template

Code your own template

Choose from a previously saved template

Import a previously created template from an external source

All of the above

I can make the campaign look the way I want by uploading pictures and changing the color, font,

and style of it.

True False

Campaigns can be personalized to each recipient by using the __________ function inside the

Style Editor.

There is an option to send a campaign right away or schedule it to be sent on a different day

and/or time.

True False

Thank you so much for participating!

You now have one FREE campaign added to your account!


Appendix F

Level 3 Evaluation


Appendix G

Level 4 Evaluation

Has the average number of campaigns sent per user increased? Yes No

If yes, please describe any events or activities that can explain this (unrelated to training):

______________________________________________________________________________

______________________________________________________________________________

Has the average number of lists uploaded per user increased? Yes No

If yes, please describe any events or activities that can explain this (unrelated to training):

______________________________________________________________________________

______________________________________________________________________________

Has the average new customers per week increased? Yes No

If yes, please explain any events or activities (such as a new promotion campaign) that can be

attributed to this result (unrelated to training):

______________________________________________________________________________

______________________________________________________________________________

Has the revenue per user account increased? Yes No

If yes, please explain anything that can be attributed to the increase in revenue (unrelated to

training):

______________________________________________________________________________

project 4: the evaluation plan rachel murch edci 577 ... · mailchimp provides online marketing...

Documents