discovering real-world usage for a multimodal math search interface

Discovering Real World Usage for a

Multimodal Math Search InterfaceThesis Defense | Keita Del Valle Wangari

Rochester Institute of Technology

December 16, 2013

THE PROBLEMThe Intention Gap.

How do you “Google it”?

Query options

• n choose k (verbal expression)

• binomial coefficient (related term/concept)

• {{n}choose{k}} (LaTeX)

• …(template editor)

Target Audience

Non-experts in the math domain:

• less likely to know math expression names

• less likely to know a math encoding language

• less likely to be familiar with math template editors or the

sites that use them

Intention Gap

Intention Gap

(Zha et al, 2010)

Intention Gap

To use math expressions in search, current search engines

require knowing expression names or using a structure

editor or encoding language (e.g., LaTeX) to enter

expressions. For people who are not math experts, this can

lead to an “intention gap” between the math query they

wish to express, and what the interface will allow.

Problem Statement

THE STUDY GOALS

min

(Sasarak et al, 2012)

Demo 1: drawn input

Demo 2: typed input

Demo 3: search

Study goals

1. To observe whether min changes user search behavior

2. Discover real-world scenarios for math search interfaces

What do we know about...

• …the visual aspects of math?

• …inputting math on a computer?

• …math search?

PREVIOUS WORKPerceiving, inputting, & searching for math.

Perceiving math

• Appearance affects reasoning – cognitive illusion (Landy &

Goldstone, 2007)

• Grounded in visual structure (Landy & Goldstone, 2007)

• Designed to encourage us to think visually (Anthony, Yang &

Koedinger, 2005)

• Diagrammatic – derive meaning from layout (Landy &

Goldstone, 2007)

Inputting math

• Typed input not optimal (Anthony, Yang & Koedinger, 2005)

• Handwriting most natural and satisfactory (Anthony, Yang &

Koedinger, 2005)

• Equation editors tedious (Smithies, Novins & Arvo, 2001)

Math search interfaces

• Text keywords

• Expressions coded in LaTeX, TeX, MathML

• Expressions built using a template-based editor

LaTeX input in Wolfram Alpha

www.wolframalpha.com

http://www.wolframalpha.com/

Editor input in MathFind(Munavalli & Miner, 2006)

Zhao et al study

• written math expression

not useful as a search term

• doubt value of query-by-

expression capability

• prefer inputting LaTeX

• text most viable form of

searching

• specialized input

modalities unwieldy (Zhao, Kan, & Theng, 2008)

min



THE STUDYDesign & execution.

Design considerations

• Observational – min is in prototype phase

• Peer-assist style – reduce math anxiety

• Math professor input – ensure tasks are level-appropriate

• Pilot – test and refine the protocol

Participants

• 16 participants

• 18 or older

• first- or second-year college math course @ RIT

• Beginner or Intermediate level in math knowledge

• Comfortable or Very Comfortable using the internet

• Recruited via email

The test session

• In the Usability Lab,

Golisano Hall @ RIT

• 1 hour duration

• 1 moderator & 1

observer

• Recorded

• $20 compensation

Artifacts

• Screener Survey (online)

• Orientation Script (printout)

• Consent Form (printout)

• Background Survey (online)

• Task Sheets (printouts)

• Pre-demo Questionnaire (printout)

• Post-Study Questionnaire (online)

• Post-Study Interview Sheet (printout)

• Project Information Sheet (printout)

Task Expressions

•

Task Keywords

• polynomials

• Pascal‟s triangle

• binomial coefficients

• prime counting function

Tasks 1 & 2

•

Tasks 3 & 4

•

Counterbalanced

Group Task Order

1 1 2 3 4

2 4 1 2 3

3 3 4 1 2

4 2 3 4 1

Search Conditions

• Text books, notes, websites, and/or online search

• Online search only without the min interface

• min interface only

• Online search only with the option of using the min

interface

Introducing min

• In between search condition 2 & 3

• Participant impression noted first

• Keyboard & mouse-drawn modalities demoed

• Upload modality described

• All tools and search function demoed

min – hands-on use

• Supports diagrammatic aspects of math notation

• Affords preferred handwritten method of math input

1. Does min change behavior?

Metrics

• Expression use in search query

• Query length

• # of query reformulations

• Task time

2. Real-world use?

Metrics

• Input modality used

• Self-rated task success

• Satisfaction ratings

• Participant ideas

What else can we observe?

THE RESULTSPrimary findings.

Goal 1


• Yes.

Expression Use

2

0

16

10

0

12 16

0

2

4

0

2

4

6

8

10

12

14

16

18

Cond. 1: no

min, open resources

Cond. 2: no

min, search

Cond. 3: min req. Cond. 4: min (min

opt.)

Cond. 4: no min (min

opt)

n=14 n=16 n=16 n=12 n=4

To

tal

Nu

mb

er o

f In

itia

l Q

uer

ies

Without Expression

With Expression

Average Time by Condition

112

209

133 132133

96

136

108

343

289

517

112

158173

405

164

79

50

0

29

0

180

360

540

720

900

prime counting function polynomial pascals triangle binomial coefficient

Sec

on

ds

Condition 1 - no min, open resources

Condition 2 - no min, online search

Condition 3 - min, min req.

Condition 4 - min canvas, min opt.

Condition 4 - no min canvas, min opt.

n=3n=3

n=3n=1

n=3n=1

n=3n=1

n=4 n=0

Task Success

0

4

8

12

16

20

24

28

32

Successful Somewhat successful Unsuccessful

Without min

With min

Finding a resource

Without min

• submit search query then “cherry pick” from search

results

With min

• submit search query to multiple databases

Reformulating

Without min

• when reformulating a search query containing an

expression, modifications made to the expression, as well

as any text keywords

With min

• when reformulating a search query containing an

expression, modifications made only to the keywords.

Goal 2


• Yes.

Real-world use

• 12 out of 16 participants (75%) identified

scenarios where they would use min or could have used

min in the past.

Participant Comment

“when they start to get nasty and use a lot of Greek

letters, it's hard to search a Greek letter”

Participant Comment

“for more complex problems…even over wolfram

alpha, a lot easier to put problems in and can still search

wolfram”

Participant Comment

“if searching for something with a radical or some weird

symbol that‟s really hard to enter that in… I like that you

can draw it knows what you're talking about and can

detect it and you can search it right there … I don't have

to Google and type in the term in place of the symbol”

Participant Comment

“it would be really nice when you have a really long

equation … like when using wolfram alpha a lot of the

equations I put in there you have to put like 10

parentheses in it just to get it to work and it ends up taking

at least 10 minutes to make sure you have it right so this

would be nice to be able to actually just draw it out and

have it recognize what you draw”

Participant Comment

“thought it was neat you can just write it in because it's

hard to Google or wolfram alpha equations”

CONCLUSIONDiscussing the results.

Increased Expression Use

• The affordance of the interface

• The novelty of the interface

• The ability for the interface to bridge the “intention gap”

Intention Gap

Intention Gap – bridged?

“I was so surprised when it picked up on 4 choose 2.”

“Like 4 choose 2 – that‟s really hard to „write‟ but it knew what I

meant and it accurately translated what I was trying to say to it.”

122.75152.13

134.86 121.57

0

60

120

180

240

300

360

420

480

540

600

660

720

780

Sec

on

ds

263.71242.00

461.00

134.29

0

60

120

180

240

300

360

420

480

540

600

660

720

780

Sec

on

ds

Task Time by Expression

Without min With min

FUTURE WORKmin development & areas to explore.

min improvements

• Typed and recognized expressions are now rendered

using the online MathJax service

• Handwritten strokes are now hidden after recognition

• Now a button on toolbar brings up correction menu

• Now allows operator shorthand in text input such as „x^2‟

min improvements – demo 1

min improvements – demo 2

Ideas for future work

• testing the improved min interface

• expressions with complex structure and notation

• experimental comparisons tests

• different populations (e.g., age, education, income)

• usage patterns over time

Ideas for future work

• field use rather than lab use

• actual success rather than perceived success

• other domains that use diagrammatic terms and non-

keyboard characters

AcknowledgementsFor expertise, guidance and support.

Dr. Zanibbi | Primary Advisor

Drs. Yacci & Rozanski | Advisors

Dr. Agarwal | Task design

Awelemdy Orakue | Assistant

DPRL members | Technical Support

AAUW | 2009 CD Grant

Thank you.