Scenario design: In-depth and hands-onModule 1April 2015

Syllabus for the courseDescription of the courseRelevant book chapters for the session: 6, 7, 8, 9

Is this a scenario?

SmartBuilder: Using Computer Ports

Unicorn Training: Risk Management

© 2015 Cathy Moore 1

US CDC: recently removed from site

Scenario; developer's blog post about it

SmartBuilder: Make the Sale

© 2015 Cathy Moore 2

Inspired eLearning: Sexual Harassment: Supervisors

Allen Interactions / NYU Medical Center: ICIS Training

CameraSim

© 2015 Cathy Moore 3

What's a scenario?

3 main ingredients:1. A character (or "you") faces a realistic decision.2. The learner makes the decision.3. Feedback shows the results of the decision.

Not a decision-making scenario by my definition: Watch someone and then evaluate what they did, unless that's actually the

learner's job Click to reveal info, even if the context is realistic Read or watch a case study

How do scenarios help?

Skills built by scenarios: Analyze and respond to situations Correctly apply facts to the job Apply mental models designed by others Recognize own possibly dysfunctional models and adjust them -- this is

rarely done but seems really valuable to me; the Haji Kamal scenario was one attempt at this

Monitor progress Recognize and recover from mistakes

Why use a scenario instead of the traditional info presentation and quiz? Can accelerate learning Can be more memorable Reduce risks & the costs of mistakes Learners like them They can respect learners’ intelligence Can improve transfer to job Can build critical thinking skills

© 2015 Cathy Moore 4

Inspire discussion Standardize training

Best uses for scenarios: Decisions require judgment, not just “Now do step 3” It would take a long time to see results in real life It’s risky to learn on the job Decision is infrequent but important

A scenario alone is probably not enough. At a minimum, need some sort of debrief Ideally, discuss it live, in person

Action mapping

We want to improve a measurable performance goal that matters to the business. We'll do it by changing behavior. Our answer to "Put this content online" is to walk the client through action mapping.

1. Complete the goal and list actions needed to reach the goal.

2. For each action, use the following flowchart to identify why it's not being done and what solution is best. Record the solutions in your action map. You can see a video demo of how to use the flowchart and download a slightly bigger version of the chart here.

© 2015 Cathy Moore 5

© 2015 Cathy Moore 6

Identifying a goalCommon challenges:

Challenge Try this

The client doesn't know the business impact of the problem.

Ask, What do you see that tells you that

there's a problem? What's currently being measured? If you looked at the metrics for your

team a year from now, how could you tell whether the problem was solved?

How is your team members' performance measured and evaluated?

The client just wants people to "be aware" of something, or they want a "welcome to our company" sort of course.

Ask, What will change when they're

aware of it? What will go wrong if they're not

aware of it? How can we tell looking at the

performance of the unit that people are aware of it?

If Bob is aware of the thing and Jane isn't, how can we tell by watching

© 2015 Cathy Moore 7

Challenge Try thisthem do their jobs?

Stakeholders are nervous about saying that the project will improve business performance.

Emphasize that your materials aren't expected to create the change all on their own. They're part of a larger business strategy (or they should be!). If necessary, word the goal as "Contribute to [business goal]."

The client has a goal that they think can't be measured, such as, "Leaders will manage conflict effectively."

Ask, How do you know you have a

conflict-management problem? How will you know the materials

have had an effect? What will you see?

What are you measuring now that you can refer to? (employee evaluation scores? number of people leaving the organization? number of formal complaints about conflict?)

What else is happening in the company to improve conflict management skills, and what are they measuring?

Could you do a survey before the material is developed to capture employees' perception about current conflict management? That could give you some measures that you want to change with the material and related interventions, and you could then do a follow-up survey to see if those measures really did change.

The client says it's a compliance course, so the goal is just to meet a legal requirement.

Ask, What costs might the business face if

people aren't compliant? If you compare two organizations,

how could you tell if one was compliant and the other wasn't?

If you watch a worker do their job, how can you tell that they're compliant?

The client says the training is optional and doesn't relate to a specific business need.

If you're juggling a lot of projects, suggest that you need to give priority to initiatives that support current

© 2015 Cathy Moore 8

Challenge Try thisbusiness strategies. This could inspire the client to justify the existence of their project. It will at least make clear that your job is to improve business performance, not just to produce materials on demand.

If you must design a course, ask several of the questions that appear earlier in this table to focus the materials on performance rather than knowledge.

The client is developing the material to sell it to others, so they think their goal is basically, “Sell this course.”

Encourage the client to view the material from the learners’ perspective. Ask, Why would customers buy this

material? What problem are they hoping to

solve? How are they likely to measure or

imagine their return on investment?

© 2015 Cathy Moore 9

Identifying behaviorsCommon challenges:

Challenge Try this

You have a huge number of actions. Is your audience too diverse, covering too many different jobs? Consider breaking your analysis and training into different modules for different jobs.

Are your actions super-specific, where one practice activity could easily cover several actions? Consider grouping them.

Have your SME identify the most common mistakes or highest-impact behaviors and focus on those.

You have only 3 or 4 actions. Have you broken them down enough? Consider limiting each action to what someone can do in 20 minutes on the job.

Your actions are fuzzy, like “Provide effective feedback.”

Ask, What does this behavior look like? If

we watch Jorge prepare and provide “effective” feedback, what is he doing that other people don’t do?

Is there a model we can use, such as “5 steps to effective feedback”?

Your actions include thinking verbs like “understand” or “appreciate.”

Ask, How can we tell by watching

someone do their job that they understand or appreciate X?

What do they do differently?Make those behaviors your actions.

Your actions include test words like “identify” or “list.”

Ask, Do people really “identify X

correctly” on the job? How can we tell by watching

someone on the job whether they can correctly identify X?

© 2015 Cathy Moore 10

You have an action called “follow the procedure.”

Break it down into several actions, probably one for each step of the procedure.

Your analysis of the problem reveals that training isn’t likely to help, but you’re still required to deliver a course.

Try to identify some behaviors that could have an effect, however small, on the problem, and focus your intervention on those.

For example, if the problem is cultural (environment) more than knowledge and skills, identify some behaviors that could change the culture, and target those.

Brainstorm activities

We're avoiding the traditional approach, which bases the activity on the information that people supposedly need to know:

Instead, we let real-world behaviors inspire our activity ideas.

© 2015 Cathy Moore 11

The activity then becomes much more nuanced.

Base your activity on the real-world job task, not the information you were given. You might just forget about the information for now.

One way to gauge the usefulness of an activity is to ask, "How much discussion will this inspire?"

© 2015 Cathy Moore 12

Cognitive task analysis

Decisions are based on: Rules of thumb Cues Judgment

© 2015 Cathy Moore 13

We can elicit the above info from our SMEs with one of the following techniques: They do the work while thinking aloud We video them working, then show them the video and ask what they were

thinking We ask them to tell us stories about how they made past decisions

Categories of info we're looking for: Rule of thumb: Is it legal here? Cues: Cars coming? Pedestrians? Decision: Turn right? When? How fast? Monitoring: Has anything changed? Should I stop or speed up?

Using stories: "critical decision method," also known as "critical incident technique." First, ask the expert to think of a difficult event during which they made several decisions. Go through the same story three times:

1. Get the gist of the story. What happened? What were the major decisions and when did they take place? Consider creating a timeline showing decision points.

2. Go through the story again and identify the cues, rules of thumb, "gut feelings," and other factors that influenced each decision.

3. Go through it again and ask about other possible outcomes: What might a less experienced person do differently at this point? What would have happened if... Pay special attention to the most common / most expensive mistakes

In general, ask questions like: What rules of thumb did you apply? Would a person with less experience

know those rules? What cues did you consider? Which ones could be missed by a less

experienced person? Which could be misinterpreted? What were you feeling as you made the decision? Did you have any

emotional reactions or thoughts that you had to manage? Why did you make the decision you did? What alternatives did you consider?

What would have led you to decide differently? What did you monitor as you carried out the decision? What confirmed that

you made the right choice? What would make you change course? What might a novice miss?

What information did you use? Would a less experienced person need to memorize this information, or can they refer to a job aid?

Don't say this: "Tell me everything you know about..." -- ask for a story instead "Can you describe your typical procedure?" -- focus on a specific case instead

© 2015 Cathy Moore 14

Ways to fit this into your workflow: Win the participation of the SME from the beginning (setting the goal). Choose the method that's easiest for the SME. If possible, record them

completing the task while they think out loud. Repeat back to the SME what you've interpreted about their decision, for

example, "So you weren't sure if it was legal, but you saw a sign that looked like 'right turn on red' and there was a break in traffic. You checked for police before you turned, which a novice might not have done," etc.

As you collect information about the decision, start to identify what knowledge needs to be in the decision-maker's brain and what could be looked up.

Homework

1. With your SME, take at least three actions through the flowchart to identify solutions to each one, looking for quick wins like job aids or changes to processes. Here’s a video refresher showing how to do it.

2. In your action map, note the solutions you’ve identified for each of the actions.

3. For the actions that require training, identify which ones would best be treated with scenarios and note that in your map.

4. Choose one action or task that will be addressed with a scenario and complete a cognitive task analysis for it, very preferably working with your SME. Identify:

o The most common mistake, why people make it, and the consequenceo Other mistakes and why people make them, and their consequenceso The correct choice and why it's hard to make, and its consequenceo Any principles, rules of thumb, and other "hidden" information

5. Come to the module 2 session ready to write detailed scenario questions about that task.

© 2015 Cathy Moore 15