detecting problematic interactions on the web. the cope project: coping strategies on the web
DESCRIPTION
These slides depict the work carried out on the COPE project from 2011 to 2013. We propose a method to identify problems and coping tactics in populations that are constrained. Then we generalise the problems encountered and the strategies employed to broader audiences. We conclude that there is an overlap between the problems and strategies of blind and sighted users. This provides more support to the hypothesis of the accessibility continuum.TRANSCRIPT
{
Detecting Problematic Interactions on the Web
The COPE Project: Coping strategies on the Web
markel.vigo@{ehu.es | manchester.ac.uk}
@markelvigoDonostia, 18/11/2013
The Web is a complex environment
Motivation
Motivation
Ceaparu,I., Lazar,J., Bessiere,K., Robinson,J., and Shneiderman,B. (2004) Determining causes and severity of end-user frustration. International Journal of Human-Computer Interaction 17(3), 333–356.
Design Unfamiliar conventions Lack of skills
Problems
Post-hoc analysis: 87% detection accuracy (Feild, 2010)- Self-reported frustration- Usage metrics: query length, task time.
Feild, H.A., Allan, J. and Jones, R. (2010) Predicting searcher frustration. ACM SIGIR ’10, 34–41.
Detecting problems
Task-driven vs. emergent tasks
Top-down vs. bottom up Constrained vs. ecological SERP vs. navigation and more
Detecting problems
Reactions are cognitive markers that indicate problems
Ask for help (Novick, 2007) Other means (Ceaparu, 2004)
- "I found an alternative solution"- "I figure out how to fix myself"
Novick, D. G., Elizalde, E., and Bean, N. (2007) Toward a more accurate view of when and how people seek help with computer applications. ACM SIGDOC '07, 95–102. Ceaparu, I., Lazar, J., Bessiere, K., Robinson, J., and Shneiderman, B. (2004) Determining causes and severity of end-user frustration. International Journal of Human-Computer Interaction 17(3), 333–356.
Reactions to problems
Reactions are cognitive markers that indicate problems
If we are able to automatically detect coping we can provide the means to overcome the situation
Identifying strategies
Challenges- Low frequency- Non-overtly manifested- Ecological setting (get out of the lab!)
Consequences- Significant amount of observations into the wild
- Time and personnel++
Identifying strategies
We can transfer strategies from populations who cope more frequent and overtly to general audiences
Hypothesis
Overlap of problems between:- The constrains imposed by reduced screens and keyboards (situational impairments)
- Problems encountered by motor impaired desktop users
Same scope, different magnitude
Evidence
Yesilada,Y., Harper,S., Chen,T., Trewin,S. (2010) Small-device users situationally impaired by input. Computers in Human Behavior 26 (3), 427–435. Simon Harper, Yeliz Yesilada, Tianyi Chen (2011) Mobile device impairment...similar problems, similar solutions? Behaviour & IT 30(5),673-690
MAGNITUDE
There is an accessibility continuum with overlapping- Problems- Strategies- Solutions
Hypothesis
SCOPE
lower access
end
higher access end
people with disabilities
able bodied users
situational impairment
HypothesisMAGNITUDE
SCOPE
lower access
end
higher access end
Our goal is more ambitious: explore the commonalities of distant populations.- Visually disabled users who cope- Able bodied users encountering difficulties
physically impaired
mobile users
Yesilada et al. (2010)Harper et al. (2011)
visually disabled
able bodied users
Our proposal4 step method:
1. Observation and identification of strategies
2. Implementation of algorithms to detect strategies
3. Deployment into the wild4. Run user studies
Go to step 2 until obtaining reliable
results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Step 1. Observationand identification of strategies
2 independent ethnographic studies and user tests
24 screen reader and screen magnifier users
8 coping strategies (17 implementations) were identifiedVigo, M and Harper, S. (2013) Coping tactics employed by visually disabled users on the web.
International Journal of Human-Computer Studies 71(11), 1013–1025.
Step 1. Observationand identification of strategiesT1. Asking for assistanceT2. Impulsive clicking
- I21. Deliberately clicking on low-scented links
- I22. Clicking on any link- I23. Clicking on any link as long as
it is accessible
T3. Exploration tactics- I31. Persevering- I32. Escaping from content by
moving down- I33. Move down without listening
to content- I34. Move around
T4. Narrowing down search
Vigo, M and Harper, S. (2013) Coping tactics employed by visually disabled users on the web. International Journal of Human-Computer Studies 71(11), 1013–1025.
T5. Gaining orientation- I51. Gaining local orientation- I52. Gaining global orientation
T6. Redoing- I61. Re-checking- I62. Re-typing- I63. Re-tracing- I64. Re-starting
T7. Not operating- I71. Delegating on assistive
technology- I72. Swapping assistive
technology- I73. Waiting
T8. Giving up
Step 1. Further analysis I We analyse the skill acquisition process of visually disabled users over time
A decrease in last resort tactics is observed across sessions
In parallel, users exhibit more exploration tactics and explore more websites
Vigo, M and Harper, S. (under review) A snapshot of the first encounters of visually disabled users with the Web.
Step 1. Further analysis I While effectiveness increases, coping success does not
Misuse of tactics may be indicative of a larger repertoire or increased confidence.
Vigo, M and Harper, S. (under review) A snapshot of the first encounters of visually disabled users with the Web.
Step 1. Further analysis II Users are not always driven by information scent
Navigation is driven by the need to overcome an interaction problem
Users escape from the current web patch- Users click on misleading links if by doing this problems are averted.
- Low satisfacing levels: in extraordinary situations any web patch is “good enough”.
Vigo, M and Harper, S. (2013) Challenging information foraging theory: screen reader users are not always driven by information scent. ACM Conference on Hypertext and Social Media, 60-68
Our proposal4 step method:
1. Observation and identification of strategies ✔
2. Implementation of algorithms to detect strategies
3. Deployment into the wild4. Run user studies
Go to step 2 until obtaining reliable
results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Step 2. Implementation ofalgorithms to detect strategies
I61. Re-checking: quick revisitationswebpagei→wpj→wpiwpj
I63. Re-tracing: users retrace the steps in a sequence of pages. wpj→wpk→wpl→wpm→wpl→wpk→wpj→wpk→wpl→wpm
Our proposal4 step method:
1. Observation and identification of strategies ✔
2. Implementation of algorithms to detect strategies ✔
3. Deployment into the wild4. Run user studies
Go to step 2 until obtaining reliable
results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Step 3. Deployment intothe wild
WebTactics
t1(){..}
DB
User Website Manager
/Researcher
1.1 Detection algorithms are injected onto web pages.
1.2 Each algorithm keeps track of determined sequences of events and actions.
3. This event is asynchronously sent to a remote location.
2. When the use of a tactic is detected a notification is triggered.
4. Reports from users are viewed by interested parties.
ti(){..}
tn(){..}
id timestamp URL tactic
ccgu1331569030153 1333922552190 http://www.bbc.co.uk/sport/ t5
ccgu1331569030153 1333922556391 http://www.bbc.co.uk/sport/ t1
Vigo, M. and Harper, S. (2013) Evaluating Accessibility-in-use. W4A '13, article 7.
Step 3. Deployment intothe wild
Vigo, M. and Harper, S. (2013) Evaluating Accessibility-in-use. W4A '13, article 7.
Our proposal4 step method:
1. Observation and identification of strategies ✔
2. Implementation of algorithms to detect strategies ✔
3. Deployment into the wild ✔4. Run user studies
Go to step 2 until obtaining reliable
results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Step 4. Run user studies 18 participants, 10 days 126 retraces and 67 rechecks Tabbed browsing was interfering
Feedback on false positives:- “I’m browsing across tabs”- “I’m comparing different web pages”- “I’m navigating through different tabs”
Vigo, M. and Harper, S. (in progress) Detecting navigation tactics to cope on the World 'Wild' Web. Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Our proposal4 step method:
1. Observation and identification of strategies ✔
2. Implementation of algorithms to detect strategies ✔
3. Deployment into the wild ✔4. Run user studies ✔
Go to step 2 until obtaining reliable
results
Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Step 2. Implementation ofalgorithms to detect strategies
I61. Re-checking: quick revisitationswebpagei→wpj→wpiwpj
I63. Re-tracing: users retrace the steps in a sequence of pages. wpj→wpk→wpl→wpm→wpl→wpk→wpj→wpk→wpl→wpm
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
NON-TABBED
Our proposal4 step method:
1. Observation and identification of strategies ✔
2. Implementation of algorithms to detect strategies ✔✔
3. Deployment into the wild ✔✔4. Run user studies ✔
Go to step 2 until obtaining reliable
results Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
Step 4. Run user studies 20 participants, 10 days 24 retraces and 16 rechecks
Vigo, M. and Harper, S. (in progress) Detecting navigation tactics to cope on the World 'Wild' Web. Vigo, M. and Harper, S. (2013) Considering people with disabilities as überusers for eliciting generalisable coping strategies on the web. ACM Web Science, 441-444.
I63. Retracing I61. Rechecking
Conclusions
There is an overlap on problems and strategies
More evidence supporting the accessibility continuum
Implications for accessibility-in-use
Future work Ongoing work
- 4th iteration with users- Testing with visually disabled users
2nd stage of the COPE project- Intervening in the interface- Long term objective- Triangulation
{
Detecting Problematic Interactions on the Web
markel.vigo@{ehu.es | manchester.ac.uk}
@markelvigoDonostia, 18/11/2013
questions?