the impact of conversational navigational guides on the learning, use, and perceptions of users of a...
Post on 15-Dec-2015
216 Views
Preview:
TRANSCRIPT
The Impact of Conversational Navigational Guides on the
Learning, Use, and Perceptions of Users of a Web Site
Art Graesser, Tanner Jackson, Matthew Ventura, James Mueller, Xiangen Hu, and Natalie Person
This research was directly supported by contracts from ONR and IDA, and was partially supported from grants by NSF and the DoD.
Outline (The next 20 minutes)
• HURAA
• Experiment
• Results
• Conclusions
What is HURAA?
• HURAA (Human Use Regulatory Affairs Advisor ) is a web-based facility that provides help and training on the ethical use of human subjects in research.
• HURAA is based on documents and regulations from United States Federal agencies.
HURAA (Features)
• Hypertext• Multimedia• Animated navigational guide• Lessons with case-based and explanation-based
reasoning• Help modules• Context-sensitive FAQs (Point & Query)• Glossaries• Archives• Natural language queries
HURAA (Modules)
• Introduction
• Historical Overview
• Lessons
• Explore Issues
• Explore Cases
• Decision Consequences
• Query IRB Documents
Outline• HURAA
• Experiment
• Results
• Conclusions
Experiment• 155 students
• Between subjects:– Navigational Guides
• Full
• Voice
• Text
• None
• Within subjects:– Two Phases
• Acquisition phase
• Test phase
Experiment (Guides)
• Full guide (agent and voice)
Experiment (Guides)
• Full guide (agent and voice)
• Voice guide (synthesized speech only)
Experiment (Guides)
• Full guide (agent and voice)
• Voice guide (synthesized speech only)
• Print guide (text message only)
Experiment (Guides)
• Full guide (agent and voice)
• Voice guide (synthesized speech only)
• Print guide (text message only)
• No guide (no navigational guidance)
Experiment (Predictions)
• If guidance is important:– Full, Print, Voice > None
• Given guidance:– If speech medium is most effective
• Full, Voice > Print
– If text medium is most effective• Print > Full, Voice
– If the agent/persona effect is most effective• Full > Voice
– If the agent/persona effect is distracting• Voice > Full
Experiment (Acquisition phase)
• Introduction - Flash intro used to hook the user and acquaint them with the system.
• Lessons – 4 example cases used to teach/test the seven critical issues.
Lessons 1
Lessons 2
Lessons 3
Experiment (Acquisition phase)
• Introduction – Flash intro used to hook the user and acquaint them with the system.
• Lessons – 4 example cases used to teach/test the seven critical issues.
• Search Task – Users answer 4 specific questions, designed to use different modules in the system (specifically the query documents module).
Search IRB Documents
Experiment (Test phase)
• Memory – Free and cued recall tested core ideas from the Introduction and Lesson material. A Cloze procedure was used to test memory for key words.
• Issue Comprehension – As a transfer test from the Lesson module, users read two sample cases and provided respective ratings of problematic issues.
• Perception Ratings – Users provided ratings on the system (e.g. “You learned a lot about human subjects protections.”).
Outline• HURAA
• Experiment
• Results
• Conclusions
Results (Core ideas)
0
0.2
0.4
0.6
0.8
Free recall Cued recall Cloze recall
FullVoicePrintNone
Table 1. Proportion correct
Results (Problematic issues)
0
0.2
0.4
0.6
0.8
Hits False alarms d’ score
FullVoicePrintNone
Table 2. Proportion correct
Results (Search task)
0
0.2
0.4
0.6
0.8
Full Voice Print None
Table 3. Proportion correct
Table 4. Completion Time (min)
20
22
24
26
28
30
Full Voice Print None
Results (Time on task)
0
5
10
15
20
25
30
Introduction Issues Search
FullVoicePrintNone
Table 5. Mean minutes on task
Results (Perception ratings)
0
1
2
3
4
5
6
Amountlearned
Interest Enjoyment Ease oflearning
FullVoicePrintNone
Table 6. Mean ratings (1 to 6, higher is better)
Outline• HURAA
• Experiment
• Results
• Conclusions
Conclusions
• No significant results from any of the dependent measures.
• This null result is incompatible with any of our previously stated predictions.
• A practical implication of the result is that the animated conversational agent did not facilitate learning, usage, or perceptions of the interface.
Conclusions
• Perhaps, due to the well structured nature of the site, a navigational guide was superfluous.
• An agent could possibly provide help in a more complex environment.
• The value of a navigational guide may increase as a function of the complexity, ambiguity, and perplexity of the system.
Conclusions
• Animated conversational agents have proven to be effective when they deliver information and learning material in monologues and tutorial dialogues.
• Perhaps there are special conditions when a navigational guide of some form will be helpful, whether it be print, voice, or a talking head.
• However, these precise conditions have yet to be discovered and precisely specified in the literature.
The Impact of Conversational Navigational Guides on the
Learning, Use, and Perceptions of Users of a Web Site
Art Graesser, Tanner Jackson, Matthew Ventura, James Mueller, Xiangen Hu, and Natalie Person
This research was directly supported by contracts from ONR and IDA, and was partially supported from grants by NSF and the DoD.
top related