Caught in the Web- How young people use the Internet to

resolve civil justice problems

Dr Catrina DenvirUniversity College London

2011 LSRC research looking at use of the Internet for civil justice problems

• Findings were counterintuitive (for some groups, i.e. young people) – High levels of Internet access but low levels of use for problems– Lesser success when used– Even worse for disadvantaged young people

• Fit with existing research: – Resource or ‘toy’ (Nicholas et al., 2003; EdComms 2007) – Willingness correlated with level of educational attainment (Iske, Klein

and Kutscher, 2005) – Difficulty searching/obtaining information related to use of key words

(Landauer 1992)

Research Context

Aims of this Study

What is going wrong?

Finding appropriate information Getting to the right websites? Using appropriate search terms? Assessing trustworthiness of information / jurisdictional relevance? Cross-checking information from a number of sources?

Search behaviour Getting frustrated or giving up easily?

What the Internet was delivering Could the information found replace the need for advice? Was the internet acting as a standalone self-help resource?

Limitations of survey approaches

People may think they remember:

• How long they spent online• The websites they used (although most don’t remember this either)

BUT….not reliable

• Detailed search history / order of search impossible• Not possible to identify errors being made (e.g. inappropriate sources)

Online experiments

An experiment was devised in which young people were presented with legal problems and asked to use the Internet to resolve them

Able to track exactly what they did, the errors they made and the impact of the Internet on knowledge and problem solving behaviour

Structure of Presentation

1. Introduce Experiment

2. How Young People searched online

3. Website behaviour

4. Impact of Internet use on knowledge of rights/strategy

5. What this means for policy

6. Moving Forward – Exploring the content/capability dynamic

Internet Experiment

Two groups targeted

• School aged participants undertaking GCSE’s/A Levels (15-18 years old)• University Students (from a range of disciplines)

208 participants in total

Procedure

Individuals were supplied with a username and password and assigned a specific ‘virtual desktop’ number

Required to read instructions, log into the virtual desktop page, select and run their desktop, and open up the surveyNO time limit

Treat the hypothetical as if it was a problem they were facing themselves

Questions - HousingAlisha rents a house for one year. Discovers the bath has been leaking, causing damp. Landlord is asked to repair. Without providing any notice, the landlord visits the house one afternoon and lets himself in to inspect the leak.

Q1: Is the landlord entitled to enter the house in this way?Q2: Is the landlord obliged to repair the leak?

The landlord refuses to repair the leak. So, after three months ALISHA pays for the repairs and deducts the cost from the next rent payment. ALISHA encloses a note with the rent payment explaining what she has done. After the next rent becomes due, the landlord calls ALISHA and says that she must leave the house in 28 days time. He says she is in breach of the tenancy agreement by not paying the rent in full.

Q3: Has ALISHA breached her tenancy agreement by not paying the rent in full? Q4: If ALISHA refuses to leave, will the landlord be able to evict ALISHA without first obtaining a Court

Order saying that Alisha must leave?

After 28 days, two employees of the landlord arrive at the house to help ALISHA move out.

Q5: Do the two employees have the legal right to enter the property to remove ALISHA’S belongings?

Suppose that before the 28th day, the landlord had obtained a Court Order stating that ALISHA must leave the house by the 28th day

Q6: Would the two employees now have the legal right to enter the property to remove ALISHA’S belongings after 28 days have passed?

Questions - EmploymentALISHA is 19 years old. She has been working 48 hours per week as an employee ZAP COMPUTERS for 1 year. She earns £5.50 per hour. Her manager, PAUL, says he needs her to increase her hours to 50 hours per week. ALISHA does not want to work the extra hours. PAUL shows her a part of her contract which says she can be asked to work up to 50 hours per week.

Q1: Does ALISHA have to work 50 hours per week? Q2: Is ALISHA’S salary, £5.50 per hour, above, below or the same as the National Minimum Wage?

Q3: Does the National Minimum Wage vary according to how old you are?

ALISHA has been asking to see details of the main terms of her contract of employment since she started at ZAP

Q4: Does ALISHA have a legal right to see the main terms of her contract of employment?

One month later – when ALISHA has been working at ZAP for just over a year – PAUL tells her she is going to lose her job

Q5: Is ALISHA covered by the full range of unfair dismissal laws?

Zap’s personnel manager explains that ZAP is reducing the number of technicians it employs, and that ALISHA is going to be made redundant. The personnel manager tells her it is only fair that ‘the younger staff go first’.

Q6: Is ZAP allowed to consider ALISHA’S age in deciding who is to be made redundant?

Internet Experiment- Virtual Desktop

Internet Experiment - stages

Internet use and access

Socio-demographics

Housing scenario

Questions on rights (1)

Hint website

Questions on rights (2)

Advice preferences

No hint

Questions on rights (2)

Advice preferences

Employment scenario

Questions on rights (1)

Hint website

Questions on rights (2)

Advice preferences

No hint

Questions on rights (2)

Advice preferences

Tracked Activity

OverviewTime Spent

• < 1 minute to < 43 minutes• Mean = 11:46 (SD = 7.56)• Only small differences (<1 minute) between groups on the basis of group

assignment

Searches

• Minimum = 0 : Max: 14• Mean 4 (SD = 2.9)• 19 individuals performed no searches (going directly to the hint website, only

one of these individuals did not receive the hint)• Small differences between searching for those in hint/no-hint groups with the

hint group searching less• Number of searches did not differ between housing/employment groups

Simple• Two to three words, lack of familiarity with the subject-area, or likely to yield

very broad results, e.g. ‘average working week’• Commonly used by those in the housing group

Specific• Familiarity either with the subject area or with the way in which search

engines handled search terms. Typically four terms long, ‘specific’ search terms were distinguished by their sophistication, e.g. ‘national minimum wage UK’, or ‘EU Working Time Directive’

• More often used by those studying law at university

Question• Question format, closed-ended questions aimed at answering a specific

element of the hypothetical problem the protagonist faced, e.g. if my contract says I can work up to 50 hours a week do I have a choice

• More common among those with housing questions, those in no-hint group and school students

Types of Searches

• 54% refined search terms one or more times (on initial failure of original search term)

• No participant went beyond the first page of search results• 85.6% did not use search boxes within websites• Respondents search terms belied a lack of understanding of the nature of the

problem• Law Students were often hindered by their knowledge • Trial and error with little thought given to search terms that might yield the

best results

Search Behaviour

If you are not first, you don’t feature……

• Average of 9 webpages visited• Employment group visited (slightly) more pages than the housing group• Hint group visited a greater number of pages than the no-hint group• University (law) students visited the most number of pages

Websites Used

51.5% of individuals went to commercial

websites

29.5% went to direct.gov.uk

www.gov.uk which 22.5% of

participants visited

which transitioned to

halfway through fieldwork

39% went to AdviceGuide.org.uk

27.5% visited the

Shelter website

30% ‘other government website’ 30% ‘other

third sector website’

16.5% Yahoo Answers/ ‘e-How’/ Discussion Boards

Smaller numbers (<7%) visiting online forums, news sites, blogs and

acas.gov.uk

School students more often used ‘mostly

commercial’ websites

• 35.5 per cent of participants were ‘browsers’• 44.2 per cent of participants were ‘dippers’• 8.6 per cent ‘mixed Users’• School students more commonly engaged in ‘dipping’ behaviour – inability to

specify informational needs?• Where individuals returned to the same website at a latter point in their

browsing, this was often because it had shown up in search results again - brand loyalty?

Website Interaction

Use of the HintEmploymentHint Group - 66.7% of those supplied with the hint used itNo-Hint Group - 44% went on to find the hint website during their searchingHousing Hint Group - 70% of those supplied with the hint used itNo-Hint Group - 40% went on to find the hint website during their searching

• 19.6% visited a website with irrelevant content• 41.5% visited a website with legal content intended for another jurisdiction• Some realised their mistake and added ‘UK’ to the end of search terms• Errors more common among school students

age/inexperience?• School students significantly more likely to arrive at websites with foreign

content. • 53 % of school aged respondents would seek information from one or more

webpages outside of the relevant jurisdiction, v 29.7% of university (law) students and 36.3% of university (other) students

Errors

Impact of the Internet on KnowledgeHousing

• Before - Knowledge poor in regards to eviction without a court order, and whether the landlord’s employees can remove you from the property

• Uncertainty as to what constituted a breach of the lease• After - improvement across the board, but uncertainty about who can evict a

tenant

Employment

• Before - Lack of knowledge in relation to maximum working hours, the minimum wage and who is covered by unfair dismissal

• After - Improvements in knowledge after use of the Internet, yet still unsure as to the rules surrounding unfair dismissal

Improvements, but question dependent

Knowledge of rights and impact of the Internet

Employment

Housing

0 1 2 3 4 5 6

3.3

3.4

4.8

4.6

After InternetBefore Internet

Mean Score Correct out of 6

Impact of the hint

• Hint did not make a difference to score improvement• Housing - hint group achieved lower mean scores after using the Internet than

no-hint group• Hint group viewed more webpages and went to a site where all the

information was contained yet did not improve their scores as much no-hint group.

How can we explain this?• Those with the hint often did not go to other websites• Searching for information in a direct fashion may be beneficial – assuming the

right kind of websites appear in search results• Those using the hint website sometimes used discussion boards – content

issue?

What is the purpose of directing people to a site?

Avoid the pitfalls they would otherwise encounter if left to search alone – assuming that content on the site is delivered in an accessible format

Predictors of improvement

Predicting score improvement on the basis of participant characteristics, search behaviour and website visitation found that:

• School students least likely to improve their scores• University ‘other’ students most likely to improve their scores

• While not statistically significant, those adding UK to the end of search terms were associated with an increase in score, although its contribution was small.

• Other characteristics of searching were not statistically significant (e.g. hint/search type (simple, specific, question), search engine and interaction type (dipper/browser)

• Employment hypothetical, visiting direct.gov.uk was associated with score improvement, gov.uk was not.

• Housing hypothetical, visiting shelter.org.uk was associated with score improvement, other websites were not.

Where should Alisha go (if seeking advice)?

Housing

Broad range of sources of advice

CAB

Lawyer/Solicitor

Free Legal Advice

Union

Local Authority/Council

Friends/Family

Internet

Other

Shelter

ACAS

Don’t know

0 5 10 15 20 25 30 35

After Internet %Before Internet %

% of Respondents

Advi

ce P

rovi

der

Where should Alisha go (if seeking advice)?

Employment

Some changes, though no more coherent

CABLawyer/Solicitor

Free Legal AdviceUnion

Local Authority/CouncilFriends/Family

InternetOther

ShelterACAS

Don’t know

0 5 10 15 20 25 30 35

After Internet %Before Internet %

% of respondents

Advi

ce P

rovi

der

Appropriate action

Before using the Internet

Good knowledge of CAB and the need to get legal advice

But….

Expectation that there are other agencies/ombudsmen/courts that handle problems within an appropriate time frame

Employment• ‘Report it to an authority (home office?)’, • ‘Talk to a government agency that deals with issues of workers' rights’

Landlord• ‘Contact either the police or the local court and explain her reasons for not

moving out or letting the employees in’• ‘Contact a solicitor or ombudsmen, keep any records or proof they tried to

enter’• ‘Get ready to move out and find a landlord who isn't an [expletive]’

Appropriate actionAfter using the Internet

Greater emphasis on informal resolution in respect of employmentRespondents fail to understand legal processes (particularly for housing)Answers do not reflect urgency (interpret with caution)

Employment• ‘Ask her trade union for help’• ‘She should ask a tribunal to decide if she has been discriminated against

because of her age’

Landlord• ‘Get an injunction from the court to stop the eviction notice’• ‘Contact Savilles’ (the estate agents)• ’Leasehold Advisory Service’• ‘Consult the Rent Act’• ‘Get help from your tenants' association…bring up situation at next meeting.’• ‘…..She should contact a property lawyer….’

Increased knowledge of rights not translating to appropriate action

Confidence dealing with problems

Internet may increase knowledge of rights, but does not increase confidenceKnowledge does not equate to confidence/ability to handle alone…

05

101520253035404550

Confidence Handling the Protagonist's Problem Alone

% o

f Res

pond

ents

Problem Handling PreferencesHow would you handle the problem?

Retain preference for family/friends and advisers

Little appetite for handling problems alone

62% would speak to one or both of their parents about the problem88.5% said they would use the Internet to obtain information about rights

57.2

20.7

13.9

5.8

Family/friends AND an advisor/representativeAdviser/representativeFamily/friendsOn my ownNothingI'm unsure

Summary

Internet increased knowledge of rights• Question dependent• Many still got answers wrong, despite availability of resources

Searches were structures around search engines• Not as much searching within sites as might be expected • Errors stemming from search terms• Some lack of awareness regarding jurisdiction• Little attempt to discriminate between sources of information• Rigid adherence to order of search results

In addition...

Knowledge does not equate to appropriate action• Little change in knowledge of relevant processes / appropriate action• Internet did not increase understanding of advice sources

Knowledge does not equate to confidence• Continue to rely on family / friends / advisers, valuing their experience• Seemingly little appetite for self-help

Implications

For public legal education and self-help (in light of policy changes)• Internet blunt instrument for PLE in the context of this experiment• Knowledge does not equal confidence or competence with regard to action

For design of websites• The best designed websites will receive little attention if they are not first in

search results (and young people did not take direction well)• Future developments and website functionality• Phrasing of search terms has implications for website designers

Policy findings • Problems likely to be exacerbated for vulnerable young people• 1st and 2nd digital divide• Less reliable support structures

Internet appears to be being used as a diagnostic tool to confirm the nature of a problem before advice is sought from another source (parent/advisor).

We do not know if this is to do with the content available online or the way it is perceived by audiences

Capability or Content – Search Experiment

• Final part of study looked at whether varying search terms (in relation to the hypothetical questions) led to different websites and websites of different quality.

• The results emerging from the internet experiment offer less comfort than might be hoped in respect of the capacity of a technologically proficient group of young people to devise search terminology.

• A lack of familiarity with the context of the hypothetical problem rendered participants predisposed to relying on cues drawn from the rights-based questions they were asked.

• Where such cues are not available (as may well be the case in everyday problem solving) young people may struggle to conceive of search keywords.

• We know that ‘simple’, ‘specific’ and ‘question’ based search terms did not have a (statistically) significant influence on whether or not participants’ improved their knowledge following Internet use.

• But, do the different search terms lead to search results of a different nature or quality?

• In the search experiment study, 580 web pages examined – some of which were duplicates

• ‘Simple’, ‘Specific’ and ‘Question’ based search terms were devised for each one of the hypothetical questions

• Top ten Google and Yahoo search results for each search term/phrase were assessed against an evaluation framework

• Framework based on similar framework used by Advicenow in 2006 and more general indicators of quality/credibility drawn from the literature

Search Experiment

Additionally Classified According to whether:Government/Third Sector/Commercial/Union/News Site/Discussion Board/Blog/Other

Mean problem support, skills support and credibility scores were typically higher for government and third sector websites than for other website types.

Credibility and problem support scores were far higher for union websites than for any other type of website in relation to the employment hypothetical.

In respect of the housing hypothetical, commercial websites tended to score far lower on problem support than commercial websites evaluated for the purposes of the employment hypothetical.

Overall, irrespective of the type of website, all fared quite poorly in terms of skills support. Only a small number of websites (20.1%) met the criteria for ‘providing information specific to young people’, most often, these websites were provided by the government (58.1%).

Search Results- Quality

Compared to use of a ‘simple’ search term, a ‘specific’ search related to a significant increase in the likelihood of a relevant result.

There was also an increase in the likelihood of a relevant result for question based searches (when compared to ‘simple’ searches) though the difference fell well short of statistical significance.

Compared to Google, Yahoo was significantly less likely to yield relevant results. A Google result could be expected to be relevant around 70 per cent of the time compared to 59 per cent for Yahoo.

Google/Yahoo relevance is likely tied to the fact that the presence of commercial websites in the Yahoo result listings was in some cases more than double that of the Google listings for both the employment and housing searches.

Question based search terms more commonly led to discussion board, ‘communities of knowledge’ type information.

Search Results- Relevance

• Search Engines are absorbing some of the ‘bad habits’ of young people when it comes to searching. Ten years ago a question based search would have been rejected, now search algorithms are being designed around these behaviours

• Although search behaviour does not have as much of an influence as expected in this field, this might be due to the relatively small number of websites operating in the legal advice space.

• Nevertheless, search behaviour can lead to less desirable websites and if individuals lack the capacity to distinguish between what is and what is not a reputable site, then this is not ideal.

• If it is expected that more young people will be using the Internet to self-help, then PLE and technological capability go hand in hand

• The idea that technology will compensate for any lack of capacity on the part of the user is appealing in its simplicity, but this simplicity ultimately underestimates the centrality of the user to the process of civil justice self-help.

What can we conclude?