keeping your moderating skills sharp
DESCRIPTION
TRANSCRIPT
Keeping Your Moderating Skills Sharp Susan Mercer Senior Experience Researcher [email protected] @susanamercer
The Importance of Neutrality
• Clients want objective answers
• Asking biased questions lead to biased answers
• Participants want to please us • Providing types of feedback can subtly
communicate our expectations • They are more likely to answer future questions
that way
2
The Reality of Moderation
• Observed 17 practitioners • Many usability professionals get sloppy:
• Ask leading questions • Participant: “I would share this article with friends.” • Moderator: “By printing it?”
• Ask closed-ended questions • Moderator: “Would you print this?”
• Focus on known problems for “ammunition” • Use biasing response phrases “Good job”, “Yes,
that’s a problem.”
Source: Norgaard & Hornbaek, 2006 3
What does this mean?
• Current practices are not consistent • Moderating is taught through mentorship
• bad habits can easily be perpetuated
• We need to pay attention to our moderation and remain neutral to capture unbiased data.
4
5 Rules of Great Moderating(A Quick Review)
5 Rules of Great Moderating
1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine
6
1. Know Your Goals
• What are you trying to achieve? • Formative • Summative
• What topics are highest priority?
These will influence • What to focus on when time runs short • How you fine-tune your moderating style
7
5 Rules of Great Moderating
1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine
8
2. Stay Neutral
• Be Quiet! • Don’t insinuate they gave a wrong answer
✘ “Why did you do that?” ✘ “What made you think to click that link?”
• Don’t put ideas into their heads ✘ “Does entering your SSN make you feel uncomfortable?” “What do you think about entering your SSN?”
• Use their words • If they call the dashboard a “chart area”, then call it a
“chart area” • Keep your tone naturally curious, not derisive
• “What did you expect here?”
9
2. Stay Neutral
• Don’t start with a verb ✘ Did you see that? ✘ Was that difficult? ✘ Is this frustrating?
• Start with What, When, Where, Why, How What would you change to make this easier? When in this process would you be ready to buy? Where did you expect to find that information? Why did you say this was easy? How does that error message make you feel?
10
2. Stay Neutral
• Use neutral, “continuing” response phrases • Acknowledge that they gave feedback • Don’t imply whether it is good or bad
✘ “Oh”, “Hmm”, “Interesting” - Assessing ✘ “OK”, “Yeah”, “That’s good” - Agreeing “Uh huh”, “Mhmm”, “Tell me more” – Continuing
• Other tips to staying neutral?
Source: Boren & Ramey, 2000 11
5 Rules of Great Moderating
1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine
12
3. Be Responsible
You have responsibilities to:
• Participant • Company • Future Users • Study Integrity
13
3. Be Responsible
Situations happen during testing. You may need to juggle your responsibilities.
What are your priorities?
• Study Integrity • Company • Future Users • Participant 14
What to do?
If… • Participant overshares personal information • The participant rants about the website and
calls it “crap”
15
5 Rules of Great Moderating
1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine
16
4. Be Confident
• Project confidence • If you make a mistake, keep going • Don’t over-apologize • Pause if you need to collect your thoughts • “Fake it until you make it”
17
5 Rules of Great Moderating
1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine
18
5. Be Genuine
• Don’t play the notes on the page…
…play the music
• Don’t just read the questions from the page…
…have a conversation
19
5 Rules of Great Moderating
1. Know your goals 2. Stay neutral 3. Be responsible 4. Be confident 5. Be genuine
6. What else?
20
Keeping sharp
Keep Improving
1. Identify your improvement areas • Watch your videos • Have colleagues give you feedback
22
Keep Improving
2. Learn from others • Watch others moderate • Be a participant • Listen to talk radio interviews
23
Keep Improving
3. Practice skills in everyday life • People watch (observe) in public • Be quiet and listen in everyday conversations
24
Your Thoughts?
25
Thank You! (Full presentation is on Slideshare. Follow me on twitter for link)
Susan Mercer Senior Experience Researcher [email protected] @susanamercer
26
References / Photo Credits
Boren, T. and Ramey, J. (2000) Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication, 43 (3), 261-278.
Dumas, J. and Loring, B. (2008) Moderating usability tests, Morgan Kaufman. Dumas, J. and Redish, J. (1999) A practical guide to usability testing, Intellect Ltd. Ericsson, K. and Simon, H. (1980) Verbal reports as data. Psychological review. 87 (3),
215-251. Krahmer, E. and Ummelen, N. (2004) Thinking about thinking aloud: A comparison of two
verbal protocols for usability testing. IEEE Transactions on Professional Communication, 47 (2), 105-117.
Norgaard, M. and Hornbaek, K. (2006) What do usability evaluators do in practice? An exploratory study of think-aloud testing. DIS 2006, 209-218.
Slide 4: Samantha Louras Photography Slide 5: flickr: torgeaux Slide 14: flickr: mkorcusa, flickr: Calsidyrose, flickr: marioanima, flickr: breity, flickr: Qole Pejorian Slides 27, 29 and 30: flickr: Raphael Quinet, flickr: acme, flickr: betsyweber, flickr: KirkOls Slide 32: flickr: bberburb Slide 34: flickr: Tulane Public Relations Slide 35: flickr: apdk, flickr: marktristan Slide 38: flickr: Raphael Quinet Slide 39: flickr: l-i-n-k Slide 40: flickr: loppear
27