detecting problematic interactions on the web. the cope project: coping strategies on the web

33
{ Problematic Interactions on the Web The COPE Project: Coping strategies on the Web markel.vigo@{ehu.es | manchester.ac.uk} @markelvigo Donostia, 18/11/2013

Upload: markel-vigo

Post on 20-Dec-2014

300 views

Category:

Technology


1 download

DESCRIPTION

These slides depict the work carried out on the COPE project from 2011 to 2013. We propose a method to identify problems and coping tactics in populations that are constrained. Then we generalise the problems encountered and the strategies employed to broader audiences. We conclude that there is an overlap between the problems and strategies of blind and sighted users. This provides more support to the hypothesis of the accessibility continuum.

TRANSCRIPT

Page 1: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

{

Detecting Problematic Interactions on the Web

The COPE Project: Coping strategies on the Web

markel.vigo@{ehu.es | manchester.ac.uk}

@markelvigoDonostia, 18/11/2013

Page 2: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

The Web is a complex environment

Motivation

Page 3: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Motivation

Ceaparu,I., Lazar,J., Bessiere,K., Robinson,J., and Shneiderman,B. (2004) Determining causes and severity of end-user frustration. International Journal of Human-Computer Interaction 17(3), 333–356.

Page 4: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Design Unfamiliar conventions Lack of skills

Problems

Page 5: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Post-hoc analysis: 87% detection accuracy (Feild, 2010)- Self-reported frustration- Usage metrics: query length, task time.

Feild, H.A., Allan, J. and Jones, R. (2010) Predicting searcher frustration. ACM SIGIR ’10, 34–41.

Detecting problems

Page 6: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Task-driven vs. emergent tasks

Top-down vs. bottom up Constrained vs. ecological SERP vs. navigation and more

Detecting problems

Page 7: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Reactions are cognitive markers that indicate problems

Ask for help (Novick, 2007) Other means (Ceaparu, 2004)

- "I found an alternative solution"- "I figure out how to fix myself"

Novick, D. G., Elizalde, E., and Bean, N. (2007) Toward a more accurate view of when and how people seek help with computer applications. ACM SIGDOC '07, 95–102. Ceaparu, I., Lazar, J., Bessiere, K., Robinson, J., and Shneiderman, B. (2004) Determining causes and severity of end-user frustration. International Journal of Human-Computer Interaction 17(3), 333–356.

Reactions to problems

Page 8: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Reactions are cognitive markers that indicate problems

If we are able to automatically detect coping we can provide the means to overcome the situation

Identifying strategies

Page 9: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Challenges- Low frequency- Non-overtly manifested- Ecological setting (get out of the lab!)

Consequences- Significant amount of observations into the wild

- Time and personnel++

Identifying strategies

Page 10: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

We can transfer strategies from populations who cope more frequent and overtly to general audiences

Hypothesis

Page 11: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Overlap of problems between:- The constrains imposed by reduced screens and keyboards (situational impairments)

- Problems encountered by motor impaired desktop users

Same scope, different magnitude

Evidence

Yesilada,Y., Harper,S., Chen,T., Trewin,S. (2010) Small-device users situationally impaired by input. Computers in Human Behavior 26 (3), 427–435. Simon Harper, Yeliz Yesilada, Tianyi Chen (2011) Mobile device impairment...similar problems, similar solutions? Behaviour & IT 30(5),673-690

Page 12: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

MAGNITUDE

There is an accessibility continuum with overlapping- Problems- Strategies- Solutions

Hypothesis

SCOPE

lower access

end

higher access end

people with disabilities

able bodied users

situational impairment

Page 13: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

HypothesisMAGNITUDE

SCOPE

lower access

end

higher access end

Our goal is more ambitious: explore the commonalities of distant populations.- Visually disabled users who cope- Able bodied users encountering difficulties

physically impaired

mobile users

Yesilada et al. (2010)Harper et al. (2011)

visually disabled

able bodied users

Page 14: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Our proposal4 step method:

1. Observation and identification of strategies

2. Implementation of algorithms to detect strategies

3. Deployment into the wild4. Run user studies

Go to step 2 until obtaining reliable

results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 15: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 1. Observationand identification of strategies

2 independent ethnographic studies and user tests

24 screen reader and screen magnifier users

8 coping strategies (17 implementations) were identifiedVigo, M and Harper, S. (2013) Coping tactics employed by visually disabled users on the web.

International Journal of Human-Computer Studies 71(11), 1013–1025.

Page 16: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 1. Observationand identification of strategiesT1. Asking for assistanceT2. Impulsive clicking

- I21. Deliberately clicking on low-scented links

- I22. Clicking on any link- I23. Clicking on any link as long as

it is accessible

T3. Exploration tactics- I31. Persevering- I32. Escaping from content by

moving down- I33. Move down without listening

to content- I34. Move around

T4. Narrowing down search

Vigo, M and Harper, S. (2013) Coping tactics employed by visually disabled users on the web. International Journal of Human-Computer Studies 71(11), 1013–1025.

T5. Gaining orientation- I51. Gaining local orientation- I52. Gaining global orientation

T6. Redoing- I61. Re-checking- I62. Re-typing- I63. Re-tracing- I64. Re-starting

T7. Not operating- I71. Delegating on assistive

technology- I72. Swapping assistive

technology- I73. Waiting

T8. Giving up

Page 17: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 1. Further analysis I We analyse the skill acquisition process of visually disabled users over time

A decrease in last resort tactics is observed across sessions

In parallel, users exhibit more exploration tactics and explore more websites

Vigo, M and Harper, S. (under review) A snapshot of the first encounters of visually disabled users with the Web.

Page 18: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 1. Further analysis I While effectiveness increases, coping success does not

Misuse of tactics may be indicative of a larger repertoire or increased confidence.

Vigo, M and Harper, S. (under review) A snapshot of the first encounters of visually disabled users with the Web.

Page 19: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 1. Further analysis II Users are not always driven by information scent

Navigation is driven by the need to overcome an interaction problem

Users escape from the current web patch- Users click on misleading links if by doing this problems are averted.

- Low satisfacing levels: in extraordinary situations any web patch is “good enough”.

Vigo, M and Harper, S. (2013) Challenging information foraging theory: screen reader users are not always driven by information scent. ACM Conference on Hypertext and Social Media, 60-68

Page 20: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Our proposal4 step method:

1. Observation and identification of strategies ✔

2. Implementation of algorithms to detect strategies

3. Deployment into the wild4. Run user studies

Go to step 2 until obtaining reliable

results

Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 21: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 2. Implementation ofalgorithms to detect strategies

I61. Re-checking: quick revisitationswebpagei→wpj→wpiwpj

I63. Re-tracing: users retrace the steps in a sequence of pages. wpj→wpk→wpl→wpm→wpl→wpk→wpj→wpk→wpl→wpm

Page 22: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Our proposal4 step method:

1. Observation and identification of strategies ✔

2. Implementation of algorithms to detect strategies ✔

3. Deployment into the wild4. Run user studies

Go to step 2 until obtaining reliable

results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 23: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 3. Deployment intothe wild

WebTactics

t1(){..}

DB

User Website Manager

/Researcher

1.1 Detection algorithms are injected onto web pages.

1.2 Each algorithm keeps track of determined sequences of events and actions.

3. This event is asynchronously sent to a remote location.

2. When the use of a tactic is detected a notification is triggered.

4. Reports from users are viewed by interested parties.

ti(){..}

tn(){..}

id timestamp URL tactic

ccgu1331569030153 1333922552190 http://www.bbc.co.uk/sport/ t5

ccgu1331569030153 1333922556391 http://www.bbc.co.uk/sport/ t1

Vigo, M. and Harper, S. (2013) Evaluating Accessibility-in-use. W4A '13, article 7.

Page 24: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 3. Deployment intothe wild

Vigo, M. and Harper, S. (2013) Evaluating Accessibility-in-use. W4A '13, article 7.

Page 25: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Our proposal4 step method:

1. Observation and identification of strategies ✔

2. Implementation of algorithms to detect strategies ✔

3. Deployment into the wild ✔4. Run user studies

Go to step 2 until obtaining reliable

results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 26: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 4. Run user studies 18 participants, 10 days 126 retraces and 67 rechecks Tabbed browsing was interfering

Feedback on false positives:- “I’m browsing across tabs”- “I’m comparing different web pages”- “I’m navigating through different tabs”

Vigo, M. and Harper, S. (in progress) Detecting navigation tactics to cope on the World 'Wild' Web. Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 27: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Our proposal4 step method:

1. Observation and identification of strategies ✔

2. Implementation of algorithms to detect strategies ✔

3. Deployment into the wild ✔4. Run user studies ✔

Go to step 2 until obtaining reliable

results

Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 28: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 2. Implementation ofalgorithms to detect strategies

I61. Re-checking: quick revisitationswebpagei→wpj→wpiwpj

I63. Re-tracing: users retrace the steps in a sequence of pages. wpj→wpk→wpl→wpm→wpl→wpk→wpj→wpk→wpl→wpm

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

NON-TABBED

Page 29: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Our proposal4 step method:

1. Observation and identification of strategies ✔

2. Implementation of algorithms to detect strategies ✔✔

3. Deployment into the wild ✔✔4. Run user studies ✔

Go to step 2 until obtaining reliable

results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

Page 30: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Step 4. Run user studies 20 participants, 10 days 24 retraces and 16 rechecks

Vigo, M. and Harper, S. (in progress) Detecting navigation tactics to cope on the World 'Wild' Web. Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.

I63. Retracing I61. Rechecking

Page 31: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Conclusions

There is an overlap on problems and strategies

More evidence supporting the accessibility continuum

Implications for accessibility-in-use

Page 32: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

Future work Ongoing work

- 4th iteration with users- Testing with visually disabled users

2nd stage of the COPE project- Intervening in the interface- Long term objective- Triangulation

Page 33: Detecting Problematic Interactions on the Web. The COPE Project: Coping strategies on the Web

{

Detecting Problematic Interactions on the Web

markel.vigo@{ehu.es | manchester.ac.uk}

@markelvigoDonostia, 18/11/2013

questions?