The agenda…
I. Introduction
II. CROSSTECHONR2 Presentation and DemoDiscussion
III. BlueLiteONR1 Presentation and DemoDiscussion
IV. Wrap-Up
How it began…
Technical manuals are oversized, awkward, hard to move
Navigating through these can be disorienting and confusing
The Navy must maintain and repair a large array of complex on-board machinery.
Our endeavor…
The system we are building will negate the need for paper manuals.
It will use speech and touch input on a Fujitsu Tablet PC, providing flexible access to maintenance information.
After Norfolk, the two ONR groups from the data gathered on the naval base. These two agreed to separately tackle two main needs that emerged needs were navigation and collaboration. In the last three months, these two groups have prototyped, researched, and iteratively designed multi-modal IETM systems with specific emphases in these areas.
THE IETM KNOWLEDGE BASE
CROSSTECH
Agenda
1. Why Collaboration2. Collaboration Evolution3. Design Cycle4. CrossTech Demo5. User Studies6. Case Study7. Tech Rationale
Display ConsoleSLQ-32 v3
Equipment Assembly Cabinet
Our Vision
Buiild a portable IETM that supports:
Collaboration with annotation
Pen + speech I/O
Expert Knowledge Sharing
Placeholding
The Collaboration Evolution
KnowledgeSharing Tool
Non-CollaborativePreventative Maintenance
Support System
Same ship / Different deck
Collaboration Tool
Troubleshooting ScenarioNaval Signal Select Test
Norfolk User Research
Knowledge Sharing Tool How is this a Knowledge sharing tool?
Supports two types of tasks:• Troubleshooting: supports investigative nature of
the task. Users may seek outside experts for help. • Still supports the PMS tasks where navigation is
often rigid and sequential
Supports two types of Collaboration:• Collaborating synchronously with a partner
working on the same task• Sharing Knowledge about the SLQ32 with experts
off-shore
Spring Design CycleNine main prototypes before the summer:
Extreme Design Sessions
Norfolk
Spring Finals
Summer User Study/ Design Cycle
In the summer, we began with two different designs :
After Phase 1 of the summer user studies, we consolidated the best of these two prototypes into one as the basis for our current design.
CROSSTECH
A knowledge sharing system that serves as a centralized point where IETM information and expert knowledge is exchanged and stored
CROSS: Derived from “crosshair” and “cross collaboration”
TECH: derived from Tech manuals and “techs” as the jargon for naval mechanics
S Y S T E M D E M O
Login
1. All naval personnel have identification cards (can further support fingerprint ID technology)2. Client Request
RATIONALE
Task Assignment List
1. Cannot find problem in the manual [CI-NAVY1-b2]2. Flip through manuals for a long time[CI-NAVY1-b1]
RATIONALE
N A V I G A T I O N
NAVIGATIONAL ISSUES
Search through the tasks and easily branch off into sub-tasks.
Efficient navigation through steps and diagrams.
Keep track of where they have been and store temporary references.
Keep the workspace manageable
Global Navigation
1. Users use diagrams to remind themselves of the steps taken [CI4-HUNT2-o1]2. Loses place in the instructions [CI10-AR-b1]3. User scans through all of task before actually beginning task [CI10-AR]
1. Users have a hard time clicking on the global navigation and end up not using it (made it bigger) - [USp2-3]2. Users now prefer to use the global nav to navigate locally -P33. Other users still hesitant to use global nav, so added a hover-over effect - P34. Having an outline around the global step they were on did not provide enough feedback (changed it to a solid color) - P25. Navy Users like the overview to see the high-level [USp0-3]6. Users prefer to see icons denoting warnings, steps, cautions and diagrams so they know what step to go back to.
RATIONALE
USER FEEDBACK
Bubble Steps
1. Users use diagrams to remind themselves of the steps taken [CI4-HUNT2-o1]2. Gather all the tools at once [CI9-NAVY1-048]
1. Users do not notice warnings, notes and cautions when they are included in steps (need to add icons in global and colors and icons in bubble) - [USp1-5]2. Users get confused at the beginning and end of the task when there are up and down arrows (Remove arrows at beginning and end) - P23.Underlined Diagram is not seen as clickable to the user. (USp1-13)4 Users have problems navigating through steps(USp1-16)5. Wont do anything until they see a diagram
RATIONALE
USER FEEDBACK
Shared Space / My Space
1. One works on Console and equipment while other works ont he manual [CI1-NAVY-o1] == myspace shared space2. Students try to be on the same page in the manual and flip through it a long itme [CI6-PIA-b1]3. Cannot see what step the other person is on in the MRC Card [ Ci9-NAVY1-b4]4. Collaborators run back and for to see what other sees [CI9-NAVY1-o12] 5. shared view would be helpful [CI9-NAVY1-o13]
1. Users lose a sense of state - they inadvertently move into SharedSpace and do not have feedback to tell them what happened. P1, P22. Changed the colors on tabs - P23. Changed the background texture of the different spaces - P34. Windows-style icons are preferred by the Navy users [USp0-5]5. Having an import arrow is confusing to users when in the help space (so we removed it).6. People do not know how to easily get to and from shared space (USp1-7)
RATIONALE
USER FEEDBACK
Import Step / Diagram
1. Print out Diagrams for reference 2. Cannot find diagram in manual (flipping through manual for 15 minutes) [CI5-PIA1-o5]3. Students rip out pages from manual to use [CI5-PIA1-b7]4. Students use screwdrivers and pens to hold pages in the manual [CI5-PIA-o7]5. Bookmarks needed only for current task(don't need to be saved) [CI9-NAVY1-o62]
1. Links to open Diagrams are not noticeable - users not clicking on them (made them more noticeable) - P22. New thumbnails make the user think that that is the entire diagram they need to reference - P33. Changed the design to a icon and a blue Diagram open button - P3
RATIONALE
USER FEEDBACK
Show / Hide Steps
1. Client Feedback2. Table Students work on is cluttered [CI5-PIA1]
RATIONALE
USER FEEDBACK1. Users have difficult time targeting the function button - Phase 3
Task Branching1. Scenario provided to us from Navy has multiple subtasksRATIONALE
USER FEEDBACK1. Uses say that having only the number on the task branching tab does not provide enough meaning to them - Phase 3
Search
1. The Project team is also working closely with Pearson KAT, so we needed to prototype this feature.
1. Users often default to entering text in the search box when they are lost in the task - P0
RATIONALE
USER FEEDBACK
S P E E C H
SPEECH ISSUES
• Multimodality is useful for eye-busy, hand-busy situation
• Valuable for mutual disambiguation
• Speech acts as an accelerator- makes the task easier
Speech
USER FEEDBACK
1. Hands busy most of the time [CI9-NAVY1-o25]2. Environment is loud, cramped and small. Always Make speech robust [CI1-NAVY1-b4]3. Place Manual on closet table with relevant page left open [CI1-NAVY1-o2]
1. Users have no visual feedback of what they said or what happened (incorporated a text window of what they said) - P22. One word grammars get misrecoed a lot when equipment or parts are being used (switched to multi-word grammars) - P23. Not enough feedback of when they are in the speech system (added orange highlight) USp1-64. Users often forget the active grammars - P2
RATIONALE
C O L L A B O R A T I O N
Display ConsoleSLQ-32 v3
Equipment Assembly Cabinet
COLLABORATION ISSUES2 person collaborative task
Work and communicate with non-collocated users.
Need a private workspace during collaboration session, in case users need to search for information for their own use
A picture is worth a thousand words – good share visual condition of the task
Voice recognition interferes with shared phone capability
Phone
1. "We dont know whats going on" - studnest waiting for students inside the craft [CI6-PIA2-b2]2. Delayed response, cannot hear answers clearly [CI6-PIA2-b6]3. Maintenance sites have sound powered phones that do not always work [CI9-NAVY1-84]4. Don't know what person is doing without direct communication [CI9-NAVY1-o16]
1. Not enough feedback of when phone is on or off (added orange highlight)- USp1-42. Not clear how to active the phone connection (added grammar to label) - P2
RATIONALE
USER FEEDBACK
Annotations
1. Students use color to mark up diagrmas showing the differnet features [CI5-PIA-o5]2. Sharing picture, annotations, video would be helpful for collaborators [CI9-NAVY1-o7]
1. Screen gets cluttered with annotations very quickly, added eraser tool - [USp1-19]2. One point line is not thick enough to annotate (changed to four point line) - P3
RATIONALE
USER FEEDBACK
Camera
1. Sharing picture, annotations, video would be helpful for collaborators [CI9-NAVY1-o7]2. Collaborators run back and for to see what other sees [CI9-NAVY1-o12] 3. Pictures of equipment would be helpful, dont always know what it looks like [CI9-NAVY-o50]
1. All functions and icons that deal with sharing should be semantically grouped together - P12. Camera Icon provided a clear affordance of what the feature does to users - P2
RATIONALE
USER FEEDBACK
K N O W L E D G E S H A R I N G
KNOWLEDGE SHARING ISSUES
Have access to expert information when experts are not readily available
Some maintenance problems need advice from experts who are not located on site
Colleagues might have done a particular task before. The notes they left could be useful for workers doing the same task.
Tips
1. Car Mechanics like to "tear into" system [CI2-CAR-02] -- give tips to give help when needed, and not dictate their terminology in the manual difficult to understand.
2. Novice user receives tips and advice from the expert and tires to learn from the experience [CI4-HUNT2-o2] 3. Updates to manual also come in form of comic book which also includes tips for maintenance [CI4-HUNT2-o4]
4. Students ask instructors for help first [CI5-PIA1-o3]
RATIONALE
Suggested Content
1. Students rely on instructors when faced wiht a problem instead of troubleshooting using the manual [CI6-PIA2-o2]2. User needs additional help [CI-10-AR-b4]3. Most frequently used content should bubble up to the top [Faculty Feedback]
RATIONALE
USER FEEDBACK“Help” tab is confusing. User thinks it is system help- Phase 3
Suggested Contacts
1. Requesting Help via sending messages/recieving messages by sending pictures2. Students ask instructors for help first [CI5-PIA1-o3]3. Pictures of equipment would be helpful, dont always know what it looks like [CI9-NAVY-o50]
RATIONALE
U S E R S T U D I E S
We ran nine Naval technicians and civilian contractors through four paper prototypes -- gathered issues related to usability and usefulness.
We ran eight members of the CMU community through a combined paper-prototype and digital collaborative tools (Microsoft Netmeeting).
PHASE ZERO - close encounters with naval kind
PHASE ONE - what features did we want to keep?
User Study Phases
We ran five users through our entire prototype, and made quick iterative changes and noted larger issues we were going to take of before phase three.
PHASE TWO:Basic interaction/usability issues
PHASE THREE:Validating our design
User Study Phases
We tested another five users on the larger issues we corrected from phase two, and noted outstanding issues.
Task Simulation : Antenna RotatorADVANTAGES
– It matches well to the Navy scenario– Manual Format Similar to IETMs– Scalable task (30m - 1.5h)– It incorporates hands-on mechanical
work– Hands busy task– Could test people without mechanical
repair experience since it was meant to be assembled at home
Implications of the Antenna Rotator Task
GOOD DATA FROM THE TASK
Easy to recognize user errors while doing the task since we knew the correct sequence intimately
Users used speech abundantly as the task required physical manuveuring and they were hands-busy
Simulated 2 person task with one person outside assembling antenna and the person inside wiring the rotator
Base
Antenna
Coaxial Cable
NOT USEFUL DATA
Bad feedback regarding the language in the manual
Difficulty with wiring when testing with less skilled users
Whack-a-Mole: A Case-study in Usability
1. Initially we used a link to denote where the diagram was in the steps. Users were reading through the steps and completely bypassing the links. This led to many errors in their task.
Whack-a-Mole: A Case-study in Usability
2. To provide a better affordance, we changed the link to a button. This worked well on most steps, however when placed next to a ‘Warning’ note, users again did not notice the button, perhaps because of the similar color tones.
Whack-a-Mole: A Case-study in Usability
3. We then decided to put a thumbnail of the diagram in the step, thinking that the users would associate the button with the thumbnail. Instead, they would just try to read from the diagram without clicking on the Diagram button.
Whack-a-Mole: A Case-study in Usability
4. In our final iteration, we removed the thumbnail but added in icons that denoted figures, tables, and other steps. We also changed the color to blue so that it is consistent with the blue headings of the figures and tables.
T E C H R A T I O N A L E
Tech decisionsLimitations of technology also serves as a sanity check.
– Don’t want a design that can’t be built!
JavaScript – Language works with all speech tools– deployable on a browser which is the current
technology used by clients and other collaborators
Microsoft Speech – A robust commercial speech engine– can be integrated easily with JavaScript– Easier development (can add grammars)– Can use it with Internet Explorer
We learned during our trip to Norfolk that the Navy technicians need to bring an Oscilloscope and a Multimeter to the majority of their daily tasks:
Naval technicians confirmed that a Tablet PC would be a good size considering they need to bring these tools with them as well.
Why a Tablet PC?
AcknowledgementsBob KrautCarolyn RoseBonnie John
Jane SiegelAlex RudnickyEllen AyoobBrian GollumYitao SunThe Office of Naval Research
User Feedback: Sarah, Kevin All our other users!
Our lovelies in ONR1:HariJanetteJennJoonPoloSandhya
Peer Crit Group:
JeffJessicaLorriMarina
C O D E W A L K T H R O U G H
The two teams have agreed that the following are the eight most important observations/needs that should be explored in the future:
1. Simulation: Integrating existing tools (o-scope and the multimeter) into the interface
2. Explore and gather user data regarding knowledge sharing in the context of the Navy
3. Speech error input design4. Auto-logging during tasks5. Saving/history of annotation6. Development of user profiles7. How to enforce reading warnings, cautions, and
notes.8. Integrating SuperManual results into suggested
content
Feature StrengthsONR 1• Diagram Zoom/Pan• Drill down diagrams• Grammar • Multimodal input/output flexibilty• Signal Tracing• Managing multiple information sources
ONR2
• Window/space management• Collaborative spaces spaces • Global overview nav bar• Resource/Expert Content• Placeholding• Camera interaction
BOTH GROUPS• The design of steps: how they are handled and displayed• Task branching