Demonstrating outcomes in youth work: challenges and possibilities

Dr John Bamber

© The Centre for Effective Services 2011

2

Aims for the input

1. To explain the concept of evidence in a youth work context

2. To distinguish between the terms ‘evidence-based’ and ‘evidence-informed ‘

3. To discuss the possibilities afforded by an evidence informed approach to youth work, involving a theory of change, and a logic model

4. To set out the challenges involved for different actors

3

Why evidence?

The Age of RESULTS and ACCOUNTABILITY

Key question: What difference does youth work make?

4

1. Systematic reviews Objective?2. Randomised control trials3. Quasi-experimental study4. ‘Before’ and ‘after’ designs5. Retrospective designs (no baseline)6. Independent large scale evaluations7. Traditional literature review8. Independent small scale evaluations9. Independent single studies10. Opinion pieces – peer reviewed journals11. Expert panel/committee12. Internal evaluation – large or small scale13. Practice wisdom – policies, standards, manuals14. Personal experience – reflective practice Subjective?

Is there a hierarchy of evidence?

1-4 ‘evidence based’ = proof of ‘what works’?

http://www.campbellcollaboration.org/

5

Evidence based programmes

http://www.promisingpractices.net/

http://whatworks.uwex.edu/Pages/2evidenceregistries.html

http://www.colorado.edu/cspv/index.html

6

An evidence informed approach

Professor Sandra Nutley, University of Edinburgh

CES Practice Seminar Series, October 2010

Keynote address

Evidence-informed practice: Using research to improve services for children and young people

http://www.effectiveservices.org/ces-projects/p012-seminar-series-practice-issues

7

• Know-about (problems): e.g. the nature, formation, and interrelations of social problems.

• Know-why (requirements of action): explaining the relationship between values and policy and practice.

• Know-how (to put into practice): e.g. pragmatic knowledge about service and programme implementation.

• Know-who (to involve): e.g. care teams; building alliances for action. Sandra Nutley (2010)

Knowledge required is more than ‘what works’

8

An approach that helps people and organisations make well-informed decisions by putting the best available

evidence at the heart of practice development and service delivery.

Sandra Nutley (2010)

Evidence informed is...

9

Good outcomes depend on effective action

Local decision-making, and the harnessing of local initiative, imagination and adaptation can be pre-eminent, but are much more likely to lead to effective action when combined with a broad collection of accessible knowledge that is drawn from practice and theory as well as research (Schorr, L., 2003: 21-21).

http://www.lisbethschorr.org/4.html

10

Constructing a theory of change

Consultation

Practice Policy

Research and

Theory

http://www.theoryofchange.org/about/what-is-theory-of-change/

11

• Value led and educative• Dialogical and empowering• Deliberative and

responsive• Critical and reflective• Process and results

focussed

Evidence from practice

http://www.indefenceofyouthwork.org.uk/wordpress/wpcontent/uploads/2010/11/20252-Youth-stories-report-2011_4th-1.pdf

12

Psychology:EriksonPsychosocial development

Education:PiagetSchemas, stage theory

Multiple intelligencesGardner

Experiential Learning:KolbLearning cycleLearning styles

Communication:BerneTransactional analysis

ConstructivismBanduraReciprocal determinism

Action learningRevansL=P+Q

Political theory:MarxClass struggle

Theory of knowledge:HabermasCommunicative action

Social change:FreirePraxis

Evidence from theory – a selection

13

Evidence from research…a selection

McKee, V., Oldfield, C. and Poultney, J. (2010) The Benefits of Youth Work. Lifelong Learning UK and Unite.

Wiggins, M., Bonell, C., Burchett, H., Sawtell, M., Austerberry, H., Allen,E. and Strange, V. (2008) Young People’s Development Programme Evaluation: Final Report. Social Science Research Unit, Institute of Education and University of London with the London School of Hygiene and Tropical Medicine.

Bielby, G., Purdon, S., Agur, M., Gardiner, C., George, N., Golden, S. and Taylor, E. (2009) Empowering Young People Pilots (EYPP) Evaluation, Final Report. National Foundation for Educational Research.

Lewis-Charp, H. Hanh C.Y., Sengouvanh S. and Lacoe, J. (2003) Extending the Reach of Youth Development Through Civic Activism: Outcomes of the Youth Leadership for Development Initiative. Innovation Center for Community and Youth Development.

Eccles, J. and Appleton Gootman, J. Editors (2002) Community Programs to Promote Youth Development Committee on Community-Level Programs for Youth, National Research Council and Institute of Medicine. Washington, DC: National Academy Press.

Fullerton, D. And Burtney, E. (2010) Young People’s Health Community Based Approaches, A Rapid Review. Report to the Sexual Health Centre Cork. Insights Health and Social Research.

Institute of Education, London. (March, 2012) International review of research literature on youth development . 2012. Commissioned by Centre for Effective Services on behalf of Department of Children and Youth Affairs, Dublin.

14

Evidence from policy

15

“I think you should be more explicit here in Step Two.”

A common problem is that activities and strategies often do not lead to the desired outcomes.

This is where evidence from consultation comes in…

16

Outcomes

GoalsValues

Needs and issues

Consultation: participants peers

experts

Evidence from consultation

What are the needs, issues and interests?What changes would people like to see?What difference have we made, and has anybody benefitted?

17

Proximal outcomes

Outcome Area Sample outcome statementsAttitudes, beliefs Participants are more ready to take on new and more diverse experiences

Enhanced confidence, self-esteem, awareness (personal and social) More open to people from diverse backgrounds

Knowledge Increased knowledge of local area, of societyIncreased knowledge of rights, social issues, politicsMore informed about health, sexuality, the law, careers and formal education

Skills Enhanced capacity regarding: public speaking , problem solving, self-efficacy, making decisions, critical thinkingEnhanced interpersonal abilities in relation to: teamwork, group work, communicationsIn creased abilities in arts and creativity regarding: music, dance, drama, writingIncrease in physical competence in relation to: sports, games, outdoor activities

Behavior More engagement in structured and constructive activitiesIncreased involvement in decision-makingEnhanced positive and pro-social behaviour and diminishing negative and anti-social behaviour

Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press

18

Distal outcomes

Outcome Area Sample outcome statementsEducation Deeper engagement with formal education and alternative routes to

qualifications

Relationships with adults

Improved communications and relationships between young people and adults within communities

Health Safer and more enjoyable sex, reduction in alcohol and substance use

Social conditions Increased community cohesion, improvement in local conditions and amenities, developing local leadership

Economic conditions Increased self-sufficiency

Safety Reduction in violence, vandalism, offending rates

Service provision Increased support for homeless young people, Improved cooperation and coordination between local services

Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press

19

Short Medium Long-TermYoung people recognise the benefits of engaging in informal learning, and spend more time in structured developmental activities

Enhanced sense of capabilities regarding community involvement, education, training

Learning skills acquired and transferred to other areas of life eg work, family life, politics

Young men more able to articulate their needs and wants

Enhanced political skills through their involvement in decision making in youth work settings and the wider community

Young people friendly environment , and safer community for residents

Chain of outcomes - example

20

Outcomes and indicators - examples

Well defined information which shows whether something is happening.

21

Outcomes and indicators - sources

http://www.urban.org/center/cnp/Projects/outcomeindicators.cfm

http://www.ces-vol.org.uk/

http://www.ceni.org/publications/research.asp

22

Monitoring and evaluation

Strategies InputsProcess

Outputs Outcomes

Evidence

GoalsValues

Needs and issues INDICATORS

23

Logical Chain of Connections Showing What the Program is to Accomplish

23

Situation Analysis

Goals derive from Aim, mission, vision, values, ideology

Goals inform analysis Populations and target groupsProblems and causes, Situations and issuesCurrent provision: strengths, weaknesses, gaps

Socio-economic and organisational context affect outcomesPolicy, legislation, funding, politics, economy, culture, local conditions

Outputs are key areas of work that enable desired outcomes

Who we will reachWhat we will do Where, when and how it will happenTargets and numbers to be reachedFrequency of activitiesStandards to be achievedBenchmarks

Outcomes are cumulative changes

Short termGains in knowledge, skills, awarenessChanges in attitudesChanges in individual and group behaviour – social action

Medium TermOrganisational developmentChange in local situation and circumstancesChange in policy, decision-making, practice

Long termSocial change – health, education, civic, environment...

Implementation Resources Activities

Strategies distil and give focus to goals

Outline of main ways to achieve outcomes

INDICATORS

Monitoring and EvaluationCollecting and interpreting information about implementation and results, and reporting findingsThese processes take external factors into account and inform analysis, planning and implementationTargets and Indicators that can provide signs of progress or achievement, may derive from standards and benchmarks

Evidence informs all aspects of the logic modelTaking ideas from social scientific research, literature, practice wisdom, policy and consultation processes, to inform understandings of problems, situations and issues, as well as ideas about work that can enable desired outcomes and ways of monitoring and evaluating the work

Inputs (matched to strategies)

FundingBuildingsTechnologyPartnersStaff and volunteers

ProcessValue led and educativeDialogical and empoweringDeliberative and responsiveCritical and reflectiveProcess and results focussed

Sample logic model

24

IntentionsOrganisational purposeTo develop young people personally, socially, educationally and recreationally in a high-quality, safe environment that respects its users.

Analysis (includes consultation)

Issues to be addressed:

1. Young people’s psycho-social development2. Social change 3. Sexual health

Inputs (matched to strategies)

StaffVolunteersFundingBuildingsTechnologyPartners

ProcessValue led and educativeDialogical and empoweringDeliberative and responsiveCritical and reflectiveProcess and results focussed

Outputs

Adventure:8 linked weekend challenge courses over 18 months (includes personal and group learning inventory) for 20 participants

Social action project:Over 12 months on an issue of importance to young peopleSeries of monthly films, talks and discussions on social issues

Information, advice, training:Trial (x2) Teen Talk programme (2 evenings per week for 2-3 hours, over 6 weeks)Plan for further work

Outcomes

•Participants are more ready to take on new and more diverse experiences (S)•Enhanced interpersonal skills (M)•Learning skills acquired and transferred to other areas of life: work, family life, politics (LT)

•Increased involvement in decision-making processes (S)•More informed decision-making (M)•Local services for young people are more user friendly (LT)

•More aware of harmful behaviours to self and others (S)•Risk factors reduced (M)•Safer sexual practices (LT)

Implementation Resources Activities

Results Short Medium Long term

Monitoring and Evaluation

Evidence

Strategies1. Adventure activities2. Social action project3. Information, advice, training

25

Challenges of an evidence informed approach

•Youth workers•Managers•Policy makers•Researchers

26

Challenge for youth workers

• Assumptions underlie much of what we do

• Faulty assumptions are often the reason for poor results – e.g. ‘process is all there is’

• Evidence helps us make our assumptions explicit so that we can check and test them: espoused versus theory-in-use

• Reflective practice (Schön, 1983)http://www.infed.org/thinkers/et-schon.htm

27

Challenge for line managers

• Trust versus command and control• Monitoring and reporting versus support, training

and development• Creating spaces for reflection• Promoting a culture of evidence• Underpinning own practice with evidence

28

The challenge for policy makers

Performance Assessment

Coaching

Training

Selection

Systems Intervention

FacilitativeAdministration

Decision SupportData System

AdaptiveTechnical

Leadership

Core Implementation

Components

http://www.implementationconference.org/materials/plenary

Fixsen, D. et al (2011)

29

Challenge for researchers

• Providing policy makers with usable ideas and information that addresses current imperatives

• Developing user friendly frameworks and tools to evaluate the process and the outcomes

• Collaborating with practitioners in addressing pressing issues and problems

• Offering a critical and objective view that addresses the realities of practice

30

Centre for Effective Services

W.K. Kellog Foundation

Together 4 all

Wisconsin online

Information about logic models

Information about theories of change

Wisconsin templates

Blueprint model programmes

Information about evidence informed approaches

CyferNet Search University of Arizona

ActKnowledge

AECF Community Change

AECF Guide

Learning for sustainability

EPPI Centre

Campbell Collaboration

PROSPER Partnerships

youngballymun

Information about evidence-based and informed programmes

Archways

Information about programmes in Ireland and Northern Ireland

Lifestart Foundation

Evidence Network

Centre for Evidence-Based Intervention

CDI TallaghtPreparing for Life

SAMSHA resources

Routes to evidence in youth work?

31

John BamberPrincipal Fellow – Community Development Centre for Effective Services

jbamber@effectiveservices.org0035314160513www.effectiveservices.org