LYNDA.COM: EVALUATING REACTIONS AND LEARNING Nicholas R. Rider

Purdue University – EDCI577

Dr. Molly Lane

Lynda.com: Evaluating Reactions and Learning 2

The Instructional Product Lynda.com is an “online learning” company that specializes in video-based training and tutorials to teach a broad variety of skills (“About Us”, n.d., para. 1). A LinkedIn company, they specialize in teaching “business, software, technology and creative skills.” (“About Us”, n.d., para.1). The company was created in 1995 and has customers across many verticals like education, business, government, and even personal learning.

Within Lynda.com’s Library are 1000’s of training sessions available on-demand by subscribers to the Lynda.com service. For this paper, the focus is on a course within the Education & Elearning Tutorial category titled Instructional Design Essentials: Storyboarding by Daniel Brigham (n.d.). The course consists of an introduction, 5 learning modules, and a conclusion. Table 1 outlines the structure of the modules.

MODULE CONTENT NOTES

I. Introduction Welcome What you should know before watching this

course Using the exercise files

Duration: 2 min

II. Overview of Storyboarding What is a storyboard? Benefits of storyboading Core concepts of storyboarding Understand when to start Exploring the course project

Duration: 8 minutes

III. Text-Based Storyboarding Understanding text-based storyboards Storyboarding an opening slide Storyboarding an animated-content slide Storyboarding a scenario Storyboarding scenario feedback

Duration: 18 minutes Asset: Text_Based_Template.docx

IV. Mock-Up Storyboarding Understanding mock-up storyboarding Storyboarding an opening slide Storyboarding an animated-content slide Storyboarding a scenario Storyboarding scenario feedback

Duration: 14 minutes Asset: Visual_Template.pptx

V. Rapid-Prototype Storyboarding

Understanding rapid prototyping Prototyping an opening slide Prototyping an animated-content slide Prototyping a scenario Prototyping scenario feedback

Duration: 25 minutes

VI. Addressing Common Challenges

Publishing your project Sharing your storyboards with others Editing storyboards based on feedback

Duration: 6 minutes

VII. Conclusion Next Steps Duration: 1 minute

Table 1: Module structure within Instructional Design Essentials: Storyboarding (Storyboarding)

Lynda.com: Evaluating Reactions and Learning 3

Training Audience and Context The storyboarding course is marketed within Lynda.com’s Instructional Design Essentials package. As such, the target audience for this course includes instructional designers, education technologists, educators, eLearning developers, and others aspiring to positions like these.

The course is modeled for the novice and includes a lot of introductory content around the value of storyboarding, types of storyboards, brief how-to’s, and challenges when building and sharing storyboards. The course is available online, 24x7 and is self-paced. A course, such as this, might be included in new employee orientation for those in the aforementioned audiences or, just as likely, a personal or professional development endeavor self-initiated by the learner.

Measurement Instruments and Procedures Lynda.com currently collects course evaluations on a voluntary basis at the end of each course. This evaluation instrument is available in Appendix A. This proposal seeks to:

1. Improve upon the current Lynda.com evaluation instrument by focusing on Kirkpatrick’s (2006) first level (reaction) and by applying evaluation best practices Morrell-Samuels (2002).

2. Develop a separate learning evaluation instrument for the purpose of evaluating knowledge, skills, and attitudes (as defined in Kirkpatrick’s (2006) Level 2 - Learning) obtained from the course.

3. Provide details around the reporting process from the evaluations.

Evaluating Reaction (Level 1)

Evaluating learner impressions of the content, instructor, material quality, and usability are all parts of Kirkpatrick’s (2006) evaluation at the Reaction level. Measuring learner reactions will allow Lynda.com to employ better instructors, improve on content in existing courses, increase customer satisfaction, and decrease customer frustrations. To collect such data, it is recommended that Lynda.com utilize an online form engine to design a survey that can be provided to learners at the end of their course.

Qualtrics is one option available to Lynda.com and is a popular platform among education clients. Lynda.com currently uses an online evaluation form (Appendix A), however, several changes (Appendix B) are recommended and have been demonstrated in a proof of concept (Appendix C).

Kirkpatrick (2006) recommends getting a 100% immediate response from learners (p.28). With this in mind, it is recommended that Lynda.com leverage one of three options to achieve this level.

Option 1: Include the survey as part of the course design. Require the final activity be completion of the survey.

Option 2: Lynda.com offers certificates of completion following each course. Provide a link to the survey and require that it be completed prior to issuing of the certificate.

Option 3: Email a link to the survey once the learner completes all videos and/or exercises in the course. An incentive could be included to promote increased responses (such as a gift card or free month’s subscription)

Evaluating Learning (Level 2)

Lynda.com: Evaluating Reactions and Learning 4 Kirkpatrick (2006) states that an evaluation at level 2 consists of determining what knowledge, skills, or attitudes were learned. Further, he holds that “it is important to measure learning because no change in behavior can be expected unless one or more of these learning objectives has been accomplished” (Kirkpatrick, 2006, p.42). Behavior is ultimately what a training program can impact in a learner (trainee). It is expected, then, that promoting specific behaviors in trainees will results in organizational level goals (Kirkpatrick, 2006).

To measure learning for this Lynda.com course, it is recommended that a learning questionnaire (Appendix D) be employed. Again, this questionnaire can be accommodated through an online survey program like Qualtrics. Here, again, Kirkpatrick (2006) recommends obtaining 100% compliance. To reach this goal, it is proposed that Lynda.com offer two certificates. The first certificate, which is already offered, is for completion. A second, new certificate would be offered upon obtaining a passing score on the learning questionnaire. This certificate would indicate that proficiency in the subject matter was validated.

Contrary, perhaps, to the sequence expectation, it is recommended that the learning questionnaire come before the reaction survey and be designed as part of the course. Testing can be an effective means of learning and can be reviewed as a check for understanding (Belluck, 2011, McConnel, St-Onge, & Young, 2015).

Reporting

Data from an online system like Qualtrics is very accessible and can be utilized at predetermined intervals. It is recommended that Lynda.com evaluate results from the course after a 6 month window (or other suitable period to obtain a sufficient sample of responders) while making the results available on-demand for the course author and support personnel. If the course author or support group identify issues that need immediate resolution from any of the evaluation instruments, they can work with a Lynda.com account manager to coordinate a change.

Early results in the training may not be readily interpreted as it will be important to establish a baseline, or standard, from which to compare scores. As more time passes and responses come in, a more reliable baseline will be established. Once results are identified that fall outside of the desired threshholds, an effort should be made to remedy the deficiency and monitor responses after the efforts to remedy are completed. This should reveal whether those efforts are working as expected.

Beyond improving the course, the data should also reveal satisfaction with the various categories being assessed and learner success with the course objectives. If responders include positive remarks, along with their name and permission, testimonies can be included to allow Lynda.com to promote the course further.

Appendices • Appendix A: Lynda.com Course Evaluation Survey (Current Survey) • Appendix B: Recommendations for Lynda.com Course Evaluation Survey (by Question) • Appendix C: Lynda.com Course Evaluation Survey (Modified Survey) • Appendix D: Lynda.com Learning Effectiveness (Learning Survey)

Lynda.com: Evaluating Reactions and Learning 5 Appendix A: Lynda.com Course Evaluation Survey (CURRENT SURVEY)

Lynda.com: Evaluating Reactions and Learning 6

Lynda.com: Evaluating Reactions and Learning 7 Appendix B: Recommendations for Lynda.com Course Evaluation Survey

Question 1: Recommend removing Dissatisfied, Neutral, and Satisfied (Morrell-Samuels, 2002)

Question 3: Recommend a scale to allow participants a range of answers

Question 4: Recommend changing “author” to “instructor”. It is not apparent that the narrator wrote the course. Recommend removing Dissatisfied, Neutral, and Satisfied (Morrell-Samuels, 2002)

Question 5: Recommend changing “author” to “instructor”

Question 6: Recommend changing “author” to “instructor”

Question 7: Question combines two criteria – recommend separating or simplifying

Question 9: Question combines two criteria – recommend separating or simplifying

Question 10: Recommend changing “author” to “instructor”

Question 11: Recommend changing “author” to “instructor”

Question 12: Recommend changing “author” to “instructor”

Question 13: Recommend changing the word “artifacts” to “distortion”. Artifacts in this context could be interpreted to mean examples.

Question 14: Question combines two criteria – recommend separating or simplifying

Question 15: Question combines two criteria – recommend separating or simplifying

GENERAL: Recommend changing questions to LIKERT scale format where possible to increase speed at which form can be completed and improve response to currently “optional” questions.

Lynda.com: Evaluating Reactions and Learning 8 Appendix C: Lynda.com Course Evaluation Survey (Modified) Preview at: https://purdue.qualtrics.com/SE/?SID=SV_1M8W2bDdPPK2ycR

Lynda.com: Evaluating Reactions and Learning 9

Lynda.com: Evaluating Reactions and Learning 10

Lynda.com: Evaluating Reactions and Learning 11 Appendix D: Lynda.com Learning Effectiveness Survey (Proposed) Preview at: https://purdue.qualtrics.com/SE/?SID=SV_9WGO2DBtu6cBVDD

Lynda.com: Evaluating Reactions and Learning 12

Lynda.com: Evaluating Reactions and Learning 13

Lynda.com: Evaluating Reactions and Learning 14

References

Brigham, D. (n.d.). Instructional Design Essentials: Storyboarding | Lynda.com Training. Retrieved September 18, 2015, from http://www.lynda.com/sdk/PowerPoint-tutorials/Instructional-Design-Essentials-Storyboarding/160064-2.html

Lynda.com Course Evaluation. (n.d.) Retrieved September 18, 2015, from http://www.surveygizmo.com/s3/419696/lynda-com-Course-Evaluation

Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006). Evaluating training programs (3rd ed.). San Francisco: Berrett-Koehler Publishers, Inc.

Morrel-Samuels, P. (2002). Getting the truth into workplace surveys. Harvard Business Review, 80(2), 111 – 118.

McConnell, M., St-Onge, C., & Young, M. (2015). The benefits of testing for learning on later performance. Advances in Health Sciences Education, 20(2), 305–320. doi:10.1007/s10459-014-9529-1

Phillips, K. (2007). Eight tips on developing valid level 1 evaluation forms. Training Today, 8, 14 – 20.