The Panlingual The Panlingual Mobile CameraMobile Camera

CSE 490fCSE 490fJonathan Pool, Neb Tadesse,Jonathan Pool, Neb Tadesse,

Tim Wong, Luke WoodsTim Wong, Luke Woods

OverviewOverview

►Overall problem & solutionOverall problem & solution►Contextual inquiry description & Contextual inquiry description &

resultsresults►ESM description & resultsESM description & results►Representative tasks Representative tasks ►Task Scenarios Task Scenarios ►SummarySummary►Q & AQ & A

The Overall ProblemThe Overall Problem

►Have you ever been in a situation Have you ever been in a situation where you can’t understand some where you can’t understand some text?text?

►Language barriersLanguage barriers► Illiteracy Illiteracy ►Vision disabilityVision disability

A SolutionA Solution

►A camera phone that:A camera phone that: Allows the customer to photograph textAllows the customer to photograph text Efficiently receives a fast explanation of Efficiently receives a fast explanation of

the textthe text

Contextual Inquiry Contextual Inquiry DescriptionDescription

► Three subjects in different situations:Three subjects in different situations: (1) subject wearing blurring goggles to hinder (1) subject wearing blurring goggles to hinder

visionvision

(2) subject amidst many non-English signs + (2) subject amidst many non-English signs + labelslabels

(3) subject with low English fluency(3) subject with low English fluency

► In various contexts: locations + activitiesIn various contexts: locations + activities

Contextual Inquiry Contextual Inquiry DescriptionDescription

►Goals:Goals: (1) subjects’ purpose(1) subjects’ purpose (2) kinds of text subjects wanted to (2) kinds of text subjects wanted to

understandunderstand (3) urgency of understanding(3) urgency of understanding (4) alternative sources of understanding(4) alternative sources of understanding (5) means of coping without (5) means of coping without

understandingunderstanding (6) subjects’ text-selection strategies(6) subjects’ text-selection strategies

Contextual Inquiry ResultsContextual Inquiry Results

Contextual Inquiry Results Contextual Inquiry Results

ESM DescriptionESM Description

►Goals:Goals:

Determine how well users can take photographsDetermine how well users can take photographs►Clear or blurry?Clear or blurry?►Size vs distance?Size vs distance?

Reveal how users might indicate the specific text Reveal how users might indicate the specific text to be understoodto be understood

Determine what sort of environments people are Determine what sort of environments people are encountering throughout the dayencountering throughout the day

ESM DescriptionESM Description

►Design:Design: Prompt approx every hour between 8am and Prompt approx every hour between 8am and

9:30pm9:30pm►Asked to take one or more photographs of textAsked to take one or more photographs of text►Encouraged to take diverse photosEncouraged to take diverse photos

Subjects were also allowed to take any additional Subjects were also allowed to take any additional photographs at any timephotographs at any time

Subjects could record any audio-visual commentsSubjects could record any audio-visual comments

ESM ResultsESM Results

►Not everyone takes good photosNot everyone takes good photos

ESM ResultsESM Results

►Environment doesn’t always allow for Environment doesn’t always allow for good photosgood photos

ESM ResultsESM Results

Some photos Some photos contained both legible contained both legible and illegible textand illegible text

Some users pointed Some users pointed towards texttowards text

► Cannot rely on the user to indicate Cannot rely on the user to indicate area area

Representative Tasks Representative Tasks

► (Easy) Customer looks at a road sign in a (Easy) Customer looks at a road sign in a foreign language and tries to infer whether foreign language and tries to infer whether it defines a speed limit. it defines a speed limit.

► (Moderate) The customer wanders around a (Moderate) The customer wanders around a foreign bookstore looking for the philosophy foreign bookstore looking for the philosophy section. section.

► (Hard) The customer approaches an (Hard) The customer approaches an apartment building and wants to know apartment building and wants to know whether a person is listed in the building's whether a person is listed in the building's directory, which is written in a script that directory, which is written in a script that the customer doesn't know. the customer doesn't know.

Task Scenarios Task Scenarios

► (1) A traveler in (1) A traveler in Germany rents a car. Germany rents a car.

He comes across this He comes across this sign and wants to know sign and wants to know if he can park here if he can park here legally.legally.

He wants to take a He wants to take a photo and annotate it, photo and annotate it, describing the info he describing the info he wants to understand.wants to understand.

Task ScenariosTask Scenarios

► (2) A person travels to (2) A person travels to China, and wants to go China, and wants to go to a Chinese to a Chinese restaurant. However, restaurant. However, the person does not the person does not understand the menu understand the menu in Chinese.in Chinese.She takes a photo. She She takes a photo. She wants to circle several wants to circle several random items of her random items of her choosing for choosing for translation.translation.

Task ScenariosTask Scenarios

► (3) A person (3) A person understands spoken understands spoken Japanese, however he Japanese, however he cannot read it.cannot read it.

He wishes to hear an He wishes to hear an audio clip of this road audio clip of this road sign.sign.

He takes a photo and He takes a photo and requests to receive an requests to receive an audio translation.audio translation.

Task ScenarioTask Scenario

►UI Design ideas:UI Design ideas:

SummarySummary

►A device that can allow you to translate A device that can allow you to translate and/or understand text on the flyand/or understand text on the fly

►Supports multiple delivery mediums:Supports multiple delivery mediums: AudioAudio TextText

►Must be fast, efficient, and easy to use.Must be fast, efficient, and easy to use.

Q & AQ & A

►Questions????Questions????