Team 12:  meMem

Adam Faeth Dave Watkins Sujoy Kumar Chowdhury

SD4:  It's expensive

Summary We set out to create a prospective memory aid that would help people over the age of 55 stay active by scheduling appointments, and organizing lists of information that they may need to recall. While the users vary in experience and abilities, we noticed similar needs among our users. Among the nine people we interviewed, we found that they all used a paper-based notebook or calendar to keep track of their schedules, appointments, notes, and passwords. Although our users reported that they did not believe they needed a memory aid, several users reported anecdotes of breakdowns in their paper-based solutions. Sometimes an event was recorded on the schedule without any context ("6:00" written on the day without an event), other times they inadvertently scheduled simultaneous meetings.

This report will present the paper prototype that we took to users and the feedback that we received from the users. The first section details the prototype that we created based on the interviews and analysis conducted for previous assignments. After designing the prototype, we collected responses from three users. In the discussion section, we summarize the surprising responses and our reactions to what we learned about user evaluation from the evaluation of this prototype. 

Prototype We created the meMem prototype, as a low-fidelity prototype in order to focus the users responses on the design of the interaction rather than details of the design itself such as color choices. The prototype is paper-based, with an emphasis on hand-drawn elements. The paper prototype presented a small challenge in distributing to team members, so we chose to scan the drawings and clean them up in an image editor for distribution to the rest of the team. This allowed some refinement of the drawings and duplication of similar interface elements while maintaining the overall simplicity of a sketch.

The meMem prototype consists of two devices, the wall calendar and the portable device. The wall calendar is the size of an A3 sheet of paper (approximately 11x17"), and is designed for use on the wall. The portable device is the size of an A6 sheet of paper (approximately 4x6"). We decided to use two devices because it seemed like each device could serve a separate role, and many of the peer evaluations we received liked the alternative that consisted of two devices.

Figure 1: Wall calendar paper prototype

The wall calendar consists of a vertical row of nine buttons on the left side, and a large touch screen area. The prototype also contained a microphone and speaker for speech interaction, and a fingerprint scanner for biometric authentication. 

Figure 2: Touch screen keyboard for the wall calendar prototype

Figure 3: Add appointment screen

The screen to schedule an appointment would appear over the entire touch screen area covering the whole calendar of the previous view. There is also a slip containing the hours of the day that slides through slots in the paper to simulate scrolling. 

Figure 4: Today view on the wall calendar

The today view shows the user upcoming appointments from their day and items on their to-do list. This screen would also appear over the entirety of the touch screen area on the wall calendar prototype when the user presses the Today button.

Figure 5: Paper prototype of the portable (left) and extra cutouts (right)

Most of the portable device is also a touch screen. Five physical buttons on the top and bottom of the screen allow quick access to the functions of the device. The portable also supports speech interaction through a built-in microphone and speaker, or a Bluetooth headset. Some of the other elements that can be cut out and used to highlight changes on the prototypes are also shown on the right. For example, when the user presses one of the buttons at the top of the portable device, or issues the equivalent speech command, a LED on the button would light up giving the user feedback about their action. The portable device also combines the three plus-buttons to add appointments, notes, and to-do items into one plus button.

Figure 6: Screens from the portable device

The prototype has several screens that were cut out and laid over the touch screen area. These screens show the notes storage interface to the portable (upper left and lower right), the screen presented after the user presses the (+) button (upper right), and the today view of the portable device.

Figure 7: Additional screens from the portable memory aid

The next set of screens depicts how the device would prompt the user to authenticate with the fingerprint scanner (top left), view a two-column list (bottom left), and schedule an appointment (right). Once again, the slip of paper with the times on it can slide through the slots in the paper prototype.

Evaluation Methods Approach

To evaluate the meMem paper prototype, we selected five tasks that represented a cross-section of the tasks that the user might perform with the memory aid. In SD2, we outlined a range of three tasks that demonstrated the interaction of a memory aid. The tasks were of scheduling a new event, retrieving a password, and viewing a snapshot of today's appointments and tasks. Since the prototype consists of two devices, we decided to test these tasks on each device. However, we dropped the password retrieval task from the Wall calendar because we thought that the user was much more likely to use the portable for this task, and that the interaction would be largely redundant on the wall calendar. This resulted in a series of five tasks across the two devices. 

For each task, we decided to record the number of steps and the number of errors that occurred while the user completed the task. The measurement of the number of steps was suggested in class by Dr. Gilbert as an alternative to measuring completion time because time measurements are problematic with a paper prototype. We defined an error as any step that did not bring the user closer to accomplishing the goal, and recorded the total number of errors separately from the total number of steps.

In addition to the quantitative measurements, we recorded the qualitative evaluation of the prototype by our users. We decided to use Think-Aloud, post-task discussion, and a post-evaluation interview to gather the qualitative evaluation of our prototype. Before starting the tasks, we asked each participant to tell us what they were thinking as they worked through the tasks. If they paused, we prompted them to keep talking. If there were any interesting comments or interactions, we decided to ask them for clarification after the task. Finally, we created a series of short interview questions to ask after all the tasks were complete to solicit further comments from each participant. We decided to use these three methods because we did not want to trust one method of qualitative data gathering. The Think-Aloud protocol can sometimes break down when the user stops talking. In post-task or post-evaluation discussion, the user might give an incorrect reason for why they did something, because they forgot or did not think about it. We felt that the combination of qualitative methods, combined with the quantitative measurements would provide a more accurate evaluation of the prototype. 

Users Our users have one thing in common, they have a minimum of 55 years of experience; it is a diverse population that continues to grow. They face unique challenges including memory loss, vision changes, hearing loss, and other general health concerns. Due to these health concerns they are encouraged by health professionals to become organized, set goals, eat healthy, exercise regularly, and be active. This group has a variety of skills:  typing, organizing, speaking, leading, dancing, and teaching. They are experienced with devices such as phones and alarm clocks, however they are appreciative of computers. Most have basic computer experience that includes checking email, editing documents, and searching for information.

ScriptBefore meeting with your participant, modify:

The portable "today" view to reflect the date of the interview The wall "today" view to reflect the date of the interview Position the highlight cell on the wall calendar on today's date Change any dates on the task cards

Thank you for agreeing to help us test a prototype of a memory aid.  The prototype consists of two

devices, one that would be hung like a wall calendar, and the other is a portable device that you could take with you. We will present you with a paper prototype of each device and ask you to complete a set of five tasks. Remember that we are testing the prototype, not you, so please point out any difficulties or frustrations with the prototype. We would also like you to think-aloud while you perform the tasks, so that we may gain a better understanding of your interaction with the device. If you have a question at any time, please feel free to ask. You are also free to withdraw at any time.

The wall calendar (meMemCal): Has nine physical buttons on the left side (you can trace the outline of each button, but do not tell

them the function of each) The entire calendar area is a touch-screen The calendar also has the ability to recognize voice commands when you turn on that ability

The portable device (meMemPod): Has five physical buttons The rest of the area is a touch screen Has a fingerprint reader (do not indicate where) The portable also has the ability to recognize voice commands when you turn on that ability

For the researcher: Present each of the task cards one at a time

Read the task scenario Give the user the task card with the goal written on it

DO NOT tell the user how to accomplish the task Prompt:  If the user stops talking, say "keep talking please" or something similar Step:  Record each button press, voice command, and touch screen touch as a step Error:  Record each step that does not bring them closer to achieving the goal as an error Write:  How the user accomplished the task, the number of steps and errors

I would also like to ask you a few questions about the prototype. 

Give interview

Thank you for taking the time to help us to evaluate this prototype today. 

TasksTask 1 User Scenario: You are starting to make breakfast and you are trying to decide if you want to make a pot of coffee to go with it. You have a nagging feeling that you are supposed to meet someone for coffee later, so maybe it's not worth making now. You want to quickly find out what time the meeting is before the toast pops up.  

Goal:  Determine what time you're meeting Adam today using the wall calendar

Task 2 User Scenario: You are sitting in your favorite comfortable spot at home when you receive a call from your good friend Dave. He asks if you would be free to go to dinner this evening at 6:00 p.m. You need to add the

appointment quickly because the call from Dave reminded that you wanted to get to the copy place on the way to the doctor's appointment. Use the wall calendar to schedule this event on your way out the door.

Goal:  Schedule Dinner with Dave at 6:00 p.m. on Friday, November 21st using the wall calendar.  Task 3 User Scenario: The copy shop was just finishing your fliers when you walked in the door, and they are putting them in a box for you. You just have a few minutes before your doctor's appointment, but luckily it is just down the street. You want to cross off that you'd picked up your fliers. 

Goal:  Cross off the reminder to pick up the fliers using the portable device.

Task 4 User Scenario: You just finish a regular doctor's appointment and they want to schedule your next appointment six months from now on May 29th, 2009 at 10 a.m. Rather than wait for their friendly post-card reminder, use the portable calendar to put this appointment on your schedule while you are still at the doctor's offices.

Goal:  Schedule the doctor's appointment for 29 May 2009 at 10:00 AM using the portable device  Task 5 User Scenario: You just got an intriguing email from your friend with a link to an article in the New York Times about how the economy is bouncing back and we'll all be rich soon. This must be good and so you are determined to take a look. However, the New York Times requires that you log in to view the article, and you've forgotten the password to your account. Use the portable device to find your password.

Goal:  Retrieve the password for your New York Times online account using the portable device

Post-evaluation Interview 1.  What are your overall impressions of the prototype?  2. Are there any additional thoughts or suggestions that you would like to share?  3. How did you prefer to interact with the device, through the speech interface or the buttons?  4. Would you prefer a default reminder before each event or to customize the amount of time between each event and its reminder? 5. How long before an appointment do you like to be reminded? 

a. 15 minutes before event b. 1 hour before event c. 1 day before event d. 1 week before event

6. How much are you ready to pay for this device?

Testing Environment The evaluations were conducted in environments that were similar to the environment that we would expect the user to need the device:  in their homes or meeting friends. Two users conducted their evaluations at their homes, where they were comfortable, and free from distractions. A third evaluation was conducted at a bustling, noisy coffee shop, a favorite of the participant. There was an interruption of

that user during the task when his nephew phoned him during the completion of one of the tasks, but since we were not measuring time to completion and he remembered where he left off, we did not have him repeat the task.

Results We asked three users to evaluate our prototype memory aid, and recorded the number of steps they required to complete each task and the number of errors that occurred for each user. If a user's action did not bring them closer to completing the goal of the task, we recorded that action as an error rather than a step.

There was a contrast in the overall technology use of our participants. User 2 and his wife continued a conversation after the evaluation where they discussed how they dislike the effect of computers on society. They brought up two general points of frustration. First, they pointed out was how devices, screen sizes, and the font of instructions that accompany those devices continue to get smaller. User 2 pulled out a piece of paper that explained the function of his remote control, which was about the size of that same remote. He said it was difficult to read, and so he had written notes on top of it for him to reference later. Second, he and his wife also expressed frustration with the negative effects of computers on billing mistakes and transactions.  Referencing a conversation with a friend, they could not believe it when their friend said they could not live without their Palm Pilot. User 3 clearly held the opposite belief, he said he was always open to new technologies. User 1 also has a more positive view of computers, moving several group calendars of organizations he belongs to online, and maintaining seven blogs on a variety of topics.

User 1 User 2 User 3

Task: Steps Errors Steps Errors Steps Errors

Task 1:  View today (wall) 2 0 2 1 1 0

Task 2:  Schedule appointment (wall) 4 0 4 3 6 4

Task 3:  View Today (pod) 3 1 3 1 2 0

Task 4:  Schedule Appointment (portable) 5 1 5 1 7 1

Task 5:  Retrieve password 4 5 4 0 3 0

Table 1:  Qualitative user evaluation of the prototype

In the description of the system, we indicated it was possible to turn on a speech recognition interface. The first user primarily used speech recognition to interact with the prototype. The first button that User 1 pressed enabled the speech interface. After that, he only pressed a button or the touch screen of either device once. The third user alternated between the speech and button interfaces within the tasks. The second user did not use the speech interface for any part of the tasks, however, he commented after the evaluation that he would love to use the speech interface because he had difficulty typing.

Most of the commands spoken by the users brought them one step closer to achieving the goal of the task. However, they did issue some interesting commands that we reprinted in Figure 8 because they give an idea of the kinds of speech interactions the user might like to have when viewing or adding events. 

The most complicated tasks were Tasks 2 and 4, which asked the user to schedule an appointment.  One user required a hint before they were able to complete Task 2. User 3 got stuck and we reminded the user that the speech interface was still on, and listening to them. User 3 also scheduled the event a half-hour early. When we asked why he did this, User 3 responded he wanted to leave a buffer before the event. We did not categorize this as an error because we think this is a matter of individual preference rather than an error in completing the task. Later, when User 3 was working on Task 4, he expected to have to authenticate before adding an event, and after some time passed, he understood that no authentication was required.

In contrast, User 1 did not encounter many errors with the scheduling tasks, but struggled with Task 5. This task asked the user to retrieve a password from the portable device. User 1 issued a sequence of consecutive speech commands that the device would not be able to interpret that is reprinted below. 

After a pause, the investigator asked him what he was looking for. He responded that the device should be smart enough to just input the password when he followed the link to the New York Times article from his email. He then said that since the device was obviously just a dumb calendar, he would have stored his passwords as appointments on the date of his birth. The investigator asked him if he saw another way of storing passwords on the device. User 1 then said, "Oh, there's a notes button." He did not press the button, but instead issued a sequence of voice commands that accomplished the task with no further errors.

Sometimes the design did not match the conceptual model of the user, and this usually resulted in additional or redundant button presses. After we presented User 2 with Task 1, he pressed the calendar button on the wall calendar and expected it to do something, but since the device started with the calendar view, the button did not do anything. Similarly, User 3 issued the voice command, "Notes," while working on Task 3 on the portable device. The device presented him with the notes stored on the device, and illuminated the physical Notes button at the top of the device. He then pressed the physical Notes button, which resulted in no effect.

The button labels also confused our users when they tried to accomplish the tasks. User 2 did not recognize the 'add appointment' button on the wall calendar prototype, so he tried several other buttons before he found it and was able to complete Task 2. The same user also did not recognize the 'plus' button

Viewing an event (Task 1): "Computer, show me coffee w/Adam""Computer, show me where""Computer, who is Sujoy?""What am I'm calling Sujoy about?"

Adding an event (Task 4):"Appointment please""Make an appointment""Do appointment""Remind me at 4:00""Remind me 24 hours before the appointment" "List doctor's phone number" (add the phone number of the doctor to the event details)"Email him one week in advance to confirm"

1. "Retrieve password for New York Times."2. "Computer, open New York Times."3. "Computer, open Passwords."4. "Computer show me my <expletive> password!"5. "Computer, why don't you understand me?"

Figure 8: Sample of Speech interactions that the prototype does not yet support

Figure 9: String of consecutive speech commands that resulted in errors for User 1

on the prototype as what he needed to use to add appointments, and tried many of the buttons labeled with words before discovering the plus button. User 3 also commented after the evaluation that he did not notice the 'plus' buttons on either device and did not understand the difference between the Notes and To-do buttons on the wall calendar.

Post-Evaluation Interview Results The post-evaluation interview asked for overall impressions of the prototype, how the users would like to interact with the device, how they like to be reminded of appointments, and how much users would be willing to pay for the device.   All three users believe the overall concept was good.  They felt the speech interface was easier than dealing with buttons.  User 3 suggested the Notes and To-Do functions should be merged together.  User 2 was interested in making sure button sizes and text was large enough.  In addition to these suggestions, one major issue brought out by User 2 and 3 were the plus signs on the buttons.  They both did not equate the symbol with the function it performed.   For the alerts, User 3 wanted to make each field for the date editable while the other two users did not give any real response.  User 3 and User 1 expressed a preference to be alerted 15 minutes prior and one week prior to an event, respectively.  In the same manner that all the users agreed this was a good idea, they all agreed the device should be low in cost.  How they determined their buying-price varied.  User 1 picked $100.  User 2 tried to think of a comparable device (palm-pilot), and User 3 reasoned things out in terms of cost of the product after it's initial introduction into the market where he decided the device should be about $25 to $75.   

DiscussionThe largest source of errors appeared to be a mismatch between the conceptual model of the user and the model of the device that we created. When User 1 struggled with the sequence of commands to display the password in Figure 9, he was either expecting the device to be too smart or too dumb. He expected it to be smart enough to recognize that he was asking for a list of passwords, and that it would realize that it had a list of passwords stored as a note. Or he expected it to be too dumb, that the list of passwords was hard-wired as a specific command. In reality the device was designed to be flexible, and the designers had created a list of passwords as a note. The device was listening for the word 'note' or a derivative before it could recognize that one of the notes was called "Passwords." It is interesting that this hierarchy was created to provide flexibility and resulted in a model that was difficult for the user to create because their expectations were both that it was either smarter than that or hard-wired for specific commands.

Other mismatches occurred when the User 2 and User 3 pressed the buttons corresponding to the mode that they were already in or did not recognize the function of a button from its label. It is interesting that the redundant button presses also occurred on the portable device, which gave the user feedback by illuminating a light on the 'Notes' button. While the users struggled to recognize the button labels, a real device might have a diagram or manual explaining the different parts of the device.  It was interesting to test the device without this kind of prior information, and this gave us an idea of how easy the device would be to pick up and use, rather than how well the user could learn to use the device. In order to better improve the experience of first time users, we might want to include even more feedback from the interface to help the user understand whether their action was valid or invalid. 

For the users that found the speech interface, it appeared that they learned how to use it. User 3 saw the speech button illuminate when he pressed it, and remarked that it was similar to an on-off light switch. User 3 also changed the pattern of their speech commands; his early commands gave the impression that he was talking to an imagined third person in the room, and the later commands were direct commands to the device. In contrast, User 2 said that he would love to use the speech interface instead of typing; this suggests that he did not understand that the device could respond to his speech commands. This is interesting because the users were presented with the same description of the device, where possible. 

One difference in the description of the device occurred when the investigator let it slip not only that the device included the ability to recognize the users fingerprint, but also indicated the location of the fingerprint scanner. This may have caused User 2 to attempt a fingerprint scan before starting several of their tasks. However, User 3 also tried to find a method to authenticate before starting some of their tasks, without this additional information. This makes it unclear whether the users had a preconception that they had to authenticate before using the device, or that the indication of the fingerprint scanner caused them to try to use it before using the device.

A surprising result was that the paper prototype appeared to provide structure to the speech interaction. We did not provide the users with a list of speech commands that they could use, yet the users largely seemed to grasp the concepts that the prototype would recognize from the tasks or the depiction of the device itself. While it is possible that the users picked up some of the concepts from the task cards, we did try to avoid writing anything that would give away the commands required to perform the task, like avoiding the word "Today" on the tasks that would require the user to press the Today button. After making a number of errors on Task 5, User 1 took another look at the device and recognized that he needed to talk to it about 'notes' to reach the password list. This suggests that the depiction of the device had some influence on the speech commands issued by the users.

The paper prototype method did work to focus the evaluation from our users on the interaction itself rather than arbitrary design choices like color palette. However, the paper prototype did also have drawbacks. It was difficult to convey feedback to the user when the device would use animation, sounds, and vibrations to provide feedback. Sometimes we forgot to provide the feedback after an action, especially if it was the result of a sequence of actions that we had not anticipated.

It was also a small challenge to use the Think-Aloud protocol with a prototype that contained a speech interface. We came up with a couple of workarounds like only enabling the speech interface with a button. Another method of resolving issues with the speech interface and Think-Aloud was to stop the user, reach for the appropriate piece of paper, and repeat their command, "When you said November 24th, the device changed to show you this."

Another surprise was that most users completed the tasks in fewer steps than the number of steps indicated in the Hierarchal Task Analysis performed for SD2. While the sample size is very small, this does point to some accuracy in both the HTA from SD2. It also suggests that the prototype did not introduce many additional steps into the sequence required to complete the tasks. While it was not something we intended to use to evaluate our prototype, it was interesting to be able to compare the results to our design. It appears to be a useful method of evaluating whether a prototype introduces unnecessary steps in the completion of tasks.

The main changes that the evaluation suggested for the design is to try another set of button labels. It appeared that users had difficulty in recognizing the purpose of each of the buttons, except for the speech

interface button. We could also use a better method to convey the purpose of the device for list storage. Most users seemed to regard it as a calendar, and had trouble understanding the list-storage function of the memory aid. It also seems that none of our users would buy the device as prototyped, unless they could obtain both devices for under $100. A redesigned prototype might try to make use of existing devices that are approximately the size of the wall calendar and the portable, such as a television or an iPod touch to save money. It also might be possible to use either the wall calendar or the portable, but it is not a question that we asked our users. 

The project was an interesting exercise in obtaining user evaluations. The paper prototype was successful in testing the interaction rather than specific design choices, however we did have the feeling that some users were still holding back criticism or trying to be polite. User 1 remarked, "Oh, you must have spent a lot of time on this," when he saw the pieces of paper arrayed on the table. The evaluations were still useful in pointing out areas that need improvement, as there were plenty of statements that we did not expect. We learned a lot about the users reaction to the design with just the minimal investment in the paper prototype. Paper prototyping and user evaluation seem like an effective combination for obtaining early feedback on risky or new ventures. This is the first venture of our own education in obtaining honest and useful feedback from user evaluations, and it is an area where there is a lot of nuance to master.