regeneration project evaluation framework v3

7/31/2019 Regeneration Project Evaluation Framework V3

1/180

Marie Broadhurst Evaluation Co-ordinator

Regeneration North Staffordshire

Evaluation Framework for Regeneration PractitionersJuly 2004 - V3

1

ENGAGE

STAKE

HOLDERS

ENSURE

USE &

SHARE

LESSONS

LEARNT

DESCRIBE

THE

PROGRAM

GATHER

CREDIBLE

EVIDENCE

JUSTIFY

CONCLUSIONS

FOCUS THE

EVALUATION

DESIGN

RegenerationProject

Evaluation

Framework


2/180


3/180

CONTENTS1. INITIATING

Barriers to doing evaluation

Creating an Evaluation Strategy

The Evaluation Framework figure 1Whos Who in Regeneration

Neighbourhood maps identifying the need

Costs

Raising Awareness/Communication

Roles and Responsibilities

Linkage Nationally

Linkage Regionally

Ethical Considerations

Internal/External Evaluation

Terms of Reference

Bid Writing

2. PLANNING/DELIVERING

Aims

Objectives

Plausibility

Baseline Setting/Updating

Data Collection

Data Protection

Analysing Information

Outputs

Milestones/Targets

MonitoringOutcomes

Evaluating Processes

Risk Assessment

Assessing Impact

Benchmarking

Interim/Rolling/Final Evaluations

3. CLOSURE/FOLLOWUP

Strategic Alignment

Sustainability

Mainstreaming

Forward Strategy

Project closure

Disseminating Results

Influencing Policy and Practice

Action Learning




3


4/180

4. INVOLVING OTHERS

Linkage with other Projects/Initiatives

Linkage with Private Sector

Linkage with Public Sector

Involving the Wider Community

Involving Young People

Involving the BME Community

Involving the Elderly Community

Evaluating Partnerships

Capacity Building Creating an Evaluation Culture!

5. SUPPORT

Support and Guidance Available

Learning/Training Opportunities

Relevant Contacts




4


5/180

Section 1.

INITIATING




5


6/180

BARRIERS TO DOING EVALUATION FROM THE PERSPECTIVE OF ..

Below are some of the possible reasons for why organisations, services and individuals

do not use evaluation to its full potential. These have been collated from members

views at a North West Evaluation Society network meeting, views from individual

regeneration projects throughout North Staffordshire and views from the ProgrammeCo-ordination Unit based at Trentham, Stoke-on-Trent.

The Participant The Evaluation Funders The EvaluatorAwareness of how to become involved Monitoring vs evaluation distinction

between needs to be clear

Multi-funded work: complex

evaluation requirements

Practical barriers eg transport Level of detail / ethics Anecdotal discredited

Power differential Timing of submissions not in line with

monitoring systems

Unachievable targets lack SMART

objectives etc.

Whats in it for the participants? Worth / importance of evaluation is low Needs more understanding by others

Lack of feedback to participants Guidance is lacking and commonguidance is even more lacking!

Involving the wider community is hard

Confidence to become involved Co-ordination in joint projects is requiredfor outcomes

Involving private businesses is hard

Are known as the worker in theevaluation

Financial constraints Identifying experienced people isdifficult

Ethics Timing for appropriate feedback Linkage with partner organisations isdifficult

Design not good enough to engageparticipants effectively

That results need to be disseminated Monitoring & reporting systems arerequired and are resource intensive

Fear - both services users and staff Funders have different views ofparticipation

Lack common guidance andunderstanding

Cynicism raising expectations Control an issue funders trying tocontrol results of research

Plain English language is never used!

Can merely tell what they think you

want

Too much jargon language!

Methods can be barrier forming

Language &Literacy barriers

Access difficulties, lack of support tobecome involved

Equality in voice (confident articulateare more vocal that yourself)

Disabilities less able not included

Over-consultation of a problem




6


7/180

CREATING AN EVALUATION STRATEGY

The 5WH principle

What

Who

When Where

Why

How

So what is evaluation?

You Own Evaluation! It Is Not Done To You!

Is a project review to gather evidence on whether a project is meeting its outputs,

outcomes & milestone targets? Project evaluations look at the basic aims & objectivesof the project and assess whether they are being met effectively by the project, thereby

ensuring that the overall programme will achieve its strategic aims and objectives.

(Villages Initiative project evaluation guidance notes)

The primary purpose of evaluation is to learn lessons, both positive and negative.

(New Deal for Communities Monitoring, Review & Evaluation 2000)

It is a way of assessing whether objectives have been achieved.

It is a way of assessing the outcomes of projects for people who use them.

It is a way of learning from successes and mistakes.

Key questions that evaluation aims to address:

1) Initial condition of the area: baseline info and other evidence.

2) The scheme: objectives & strategy targets adopted and their rationale,

activities & expenditure, explaining significant amounts.

3) Outputs: recorded outputs by strategic objective, especially most important,

compare outputs with targets, explaining significant differences.

4) Outcomes: changes in area in respect of local conditions, how far they reflect

the scheme, other internal / external factors, improved conditions permanently,

initiated improvement that will continue after the scheme.

5) Process: eg. Introducing better forms of partnerships working, communityinvolvement, likely to be sustainable.

6) Conclusions: Scheme achieved, what cost, if failed why, especially successful,

why, performance against initial key objectives.

7) Lessons & recommendations: methods & approaches especially effective

effective models, anything done differently, anything still needs doing.

(ODPM guidance 1999)

Why evaluate?




7


8/180

Specific objectives vary between schemes.

Evaluation shows how far the scheme is succeeding.

Monitoring = ensure the approved schemes are making satisfactory progress towards

the targets set out in partnership delivery plans.Evaluation goes further than this = by assessing impact against a schemes objectives

and testing that those objectives remain valid.

Evaluation may suggest ways to manage the scheme more effectively, it may identify

noteworthy practice for wider dissemination, reporting back accurately to partners,

local people and Government Offices & Regional Development Agencys.

(ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide, March

1999)

How much evaluation?

Depends on

1) Size of scheme

2) Duration of scheme

3) Complexity of scheme

4) Innovation of scheme

How much evaluation is appropriate for the scheme? Depends of the following:

1) Info needed to account to the partners and the people who the scheme is

intended to help community sufficiently informed?

2) Info needed to manage the scheme effectively identify projects where change

is required?

3) Info needed to identify any important good practice lessons? Why some projects

are more successful that others?

(ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide (March

1999)

When evaluate?

The accountable body, in conjunction with the partnership should, from the outset of a

scheme, draw up an evaluation plan ensuring that evaluation throughout the lifetime ofa scheme is built into their structures and processes. In addition a specific project

should be set up for post scheme evaluation studies.

(SRB Guidance manual, revised March 2003)

3 timings for evaluation:




8


9/180

Final, interim and rolling

Final:

At or near the end of scheme. The main purpose is to discharge the duty of

accountability, showing how far achievements of the programme justify the public

expenditure and other resources it has absorbed.

Interim:

In the mid-point of the scheme. The main function is to help partnership boards,

executive teams etc to improve performance. This also contributes to a forward

strategy.

Rolling eval:

These are performance assessments carried out quarterly through key indicators and

annually through periodic review. This leaves less to the discretion of partnerships than

final and interim evaluations. Procedures are prescribed by government guidance and

involved Government Office /Regional Development Agency supervisory because it isdirectly linked to continuation of funding and rollover of delivery plans.

Minimum evaluation requirements

Some partnerships require each project leader to provide a brief annual, six-monthly or

even quarterly evaluation report, covering evidence of outcomes as well as the output

statistics that are always required.

Agreement must be made within the partnership.

(ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide March

1999)

Who evaluates?

These are the possible options for who will undertake your evaluation:

Local Universities

Local private consultants

Joint evaluations with other local services or organisations (piggy-back

research)

National Universities and consultants

Build a team of community researchers

Build a team of employees as a resource in your organisation or service

How to evaluate.introduction to the framework




9


10/180

SCHEME/PROGRAMME DEVELOPMENT

RAISING AWARENESS project level strategic level Roles &Responsibilities partnership level

CLEAR BID WRITING

INTERNAL /EXTERNAL EVALUATION

AIMS / OBJECTIVES

BASELINE / TARGETS

MILESTONES/TARGETS

UPDATED BASELINE

OUTPUTS

OUTCOMES

ASSESS IMPACT

Meaningful Info out

B e n c h m a r k i n g (with self and others)

Interim Rolling Final

Eval Eval Eval

STRATEGIC ALIGNMENT (to related schemes)

MAINSTREAMING / SUSTAINABILITY

FORWARD STRATEGY

NOTEWORTHY DISSEMINATE Partnership

PRACTICE RESULTS Partners GO/RDA Community

Other schemes




Benchmarking

Added value

DeadweightDouble counting

DisplacementSustainable benefits

TIME,PEO

PLE,R

ESOU

RCES,,RISKASSES

S

MENT

ON-GOINGSUPPORT&GUIDANCE,ACTIONLEARNING,

INVOLVEME

NTOFOTHERPROJECTS,T

HE

WIDERCOMMUNITY,P

UBLIC&PRIVATESECTOR-MAINSTR

EAMING

Linkage to National

/ Regional initiatives

Sources ofbaseline data

existing /original research

Measurement &Data collection

Fi ure 1 The Evaluation Framework for Pro ect Leaders

CHANGING

POLICY &

PRACTICE

FeedbackIntoProjectStrate

gy

10


11/180

WHOS WHO IN REGENERATION

National level

Regional level

Local level




ODPM

REGENWM

NRU

GOWM

North Staffordshire

Regeneration Zone

S-o-T LSP

(Including NRF and

SRB)

Programme Co-ordination Unit

Area Implementation

Teams / Community

Facilitation Service

AWM

Housing Market

Renewal / RENEW

North Staffs

Housing

Corporation

English

Partnerships

Regional Co-

ordination

Unit

11


12/180

TERMS OF REFERENCE FOR BODIES IN REGENERATION

National level

Office of the Deputy Prime Minister (ODPM)

The aim of the ODPM is:

Thriving, inclusive and sustainable communities in all regions

The objectives of the ODPM are:

Work with the full range of Government Departments and policies to raise

levels of social inclusion, neighbourhood renewal and regional prosperity

Provide for effective devolved decision making within a framework of national

targets and policies

Deliver effective programmes to help raise the quality of life for all in urban

areas and other communities

(www.odpm.gov.uk)

English Partnerships

English Partnerships is the national regeneration agency, helping the Government to

support high quality sustainable growth in England. They have 5 core business areas

and deliver objectives through:

Developing a portfolio of strategic sites

Acting as the Governments specialist advisor on Brownfield land

Ensuring that surplus Government land is used to support Government

objectives, especially the Sustainable Community agenda Helping to create communities where people can afford to live

Supporting the urban renaissance by improving the quality of towns and cities

Their work is characterised by:

The early and active involvement of local communities

Close and productive relationships with the private sector and through joint

adventures

The innovative use of our own sites to promote new standards of sustainable

development and advance best practice

The highest standards of design, environmental sustainability and construction The recognition that lasting development unites the local, regional and national

dimensions

(www.englishpartnerships.co.uk)




12
http://www.odpm.gov.uk/http://www.englishpartnerships.co.uk/http://www.odpm.gov.uk/http://www.englishpartnerships.co.uk/


13/180


14/180

Regional level

Government Office of the West Midlands (GOWM)

Will be working closely with partners in the region and with colleagues in central

Government, on a wide range of policies and programmes in 2003/04, making a strongcontribution to:

Improving the economic performance of the region and promoting a sustainable

approach to economic development

Promoting the development of a skills and adaptable workforce in the region

and providing opportunities for people to develop their potential

Improving quality of life through supporting the development of partnerships

and providing support to communities

Providing good quality intelligence to central governmental departments and

communicating government policies effectively in the region

(www.go-wm.gov.uk)

Advantage West Midlands (AWM)

AWM is one of nine Regional Development Agencies established by the Government

in 1999 to transform Englands regions through sustainable economic development.

Their role is to help transform the West Midlands regional economy by connecting need

and opportunity to create a better place in which to invest, work, learn, visit and live.

They will do this primarily by leading on the development and delivery of the West

Midlands Economic Strategy that is the framework for all of this partnerships activity

up to 2010.

The four objectives/pillars that will drive the transformation are as follows:

1. Developing a diverse and dynamic business base supporting enterprise and

innovation and securing inward investment to establish, retain and grow more

businesses in the West Midlands

2. Promoting a learning and skilful region raising skill levels to ensure our

workforce meets the needs of present and potential employers in the region.

3. Creating the conditions for growth securing improvement to the regions

transport, communication and property infrastructure to support the

development of a diverse and dynamic business base.

4. Regenerating communities in the West Midlands focusing resources in places of

greatest need to ensure that all our communities enjoy a better quality of life.These are supported by the need for a:

Powerful voice for the region communicating the regions strengths, opportunities and

needs, ensuring that all these are clearly articulated and fully understood both within

and outside of the region.

(www.advantagewm.co.uk)

Regen West Midlands (RegenWM)




14
http://www.go-wm.gov.uk/http://www.advantagewm.co.uk/http://www.go-wm.gov.uk/http://www.advantagewm.co.uk/


15/180

RegenWM is one of the regional centres being set up to promote good practice and

better skills in regeneration. They respond to local need and to the Governments

Sustainable Communities agenda.

The specific role and remit of RegenWM is still under debate but most members of the

regeneration community have said that they want RegenWM as a networking, events

and learning body.(www.regenwm.org)

Regional Co-ordination Unit (RCU)

The RCU acts as the corporate centre of the Government Offices throughout the

regions, representing them in Whitehall, raising their profile and ensuring that their

experience and expertise is taken into account in policy formulation.

The Government Offices and RCU are at the heart of the delivery of policies and

programme that reduce social exclusion.

The RCUs 4 key themes of work are:

1. Acting as the corporate centre of the Government Offices2. Better co-ordination of Area Based Initiatives

3. Involving Government Offices in the policy making process

4. Joining-up in the regions

(www.rcu.gov.uk)




15
http://www.regenwm.org/http://www.rcu.gov.uk/http://www.regenwm.org/http://www.rcu.gov.uk/


16/180

Local level

Housing Market Renewal / RENEW North Staffs

RENEW is one of nine HMR Pathfinders throughout England, funded by ODPM,

aimed to:Create better future for communities hit by poor demand for property and poor

housing quality

Objectives are as follows:

1. Balance the supply and demand for housing by removing surplus properties and

providing a better choice of homes in appropriate locations

2. Provide sustainable neighbourhoods through better management and increasing

population by building at higher densities

3. Reduce outward migration from Stoke-on-Trent and retain and attract new

population to the pathfinder area

4. Improve the environment radically by removing housing from heavily pollutedroads, from areas of polluting industry and from areas with no long-term

residential future. Quality open space will be provided on former housing land

that is no longer needed.

5. Promote social cohesion by ensuring ethnic minorities can access new build

housing and be reducing overcrowding

6. Link to wealth creation by supporting the new commercial core, town centres

and by providing construction training to local people

7. Reduce crime and fear of crime in order to promote safe neighbourhoods

(www.renewnorthstaffs.gov.uk)

Stoke-on-Trent Local Strategic Partnership (LSP)

This Partnership reflects the merging of the Single Regeneration Budget 2, 5 and 6 in

the City, better alignment of Partnerships, more coherent and co-ordinated programmes

under the emerging Community Strategy, North Staffordshire Regeneration Zone and

Advantage West Midlands agenda.

There are five pillars of activity under the LSP:

1. Housing and Development

2. Enterprise and Innovation

3. Education and Skills

4. Health and Wellbeing5. Community Safety

There are six cross cutting groups throughout the LSP activity:

Image

Community Development

Learning

Young People

Environment and Sustainable Development

Community Cohesion




16
http://www.renewnorthstaffs.gov.uk/http://www.renewnorthstaffs.gov.uk/


17/180

North Staffordshire Regeneration Zone (NSRZ)

The NSRZ aims to ensure that:

North Staffordshire is a successful and rewarding place in which to work and invest,

am exciting place to live with an increased quality of life for all.

The core enabling activities are:

The development of an integrated economic and spatial strategy to identify the

forward vision for the region and to clearly articulate how opportunities can be

secured and threats mitigated.

The establishment of an integrated transport strategy and advocacy case to

ensure major transport infrastructure can be made.

Support for the further development of the planning framework for the sub-

region to ensure that the needs of both the major activities proposed within the

Regeneration Zone area and the Housing Market are met.

Contributing to the Housing Market Renewal Programme, led by OPDM,creating a dynamic new housing market that helps people into jobs, tackles

social exclusion, and helps to create stronger, diverse communities.

Activity will be delivered through a priority portfolio on activity grouped under four

pillars linking closely with the West Midlands Economic Strategy:

1. Creating a diverse and dynamic business base

2. Promoting a learning and skilful region

3. Creating the conditions for growth

4. Regenerating communities in the West Midlands

(www.nsrz.org)

Programme Co-ordination Unit (PCU)

To be agreed

Area Implementation Teams (AITs) / Community Facilitation Service

To be agreed




17
http://www.nsrz.org/http://www.nsrz.org/


18/180

NEIGHBOURHOOD MAPS Identifying the need?!

As part of the Neighbourhood Renewal Strategy, the Knowledge Management Team at

Stoke-on-Trent City Council has divided the six key wards of the City into 49 smaller

neighbourhood areas. This has gone through thorough consultation with key partners of

the City and all have signed up to agree to participate in developing research for theseneighbourhood areas.

The Knowledge Management Section is headed by Ray Garside and includes a small

team dedicated to the mapping of neighbourhood data. The Team has produced

neighbourhood maps for Stoke-on-Trent, showing where the deprivation hot spots or

priority areas are in relation to themes such as crime, health, education and skills and

also where there is generic low deprivation. It is expected that regeneration initiatives

will focus on these key areas and if your initiative does not include these key areas then

reasons must be given for choosing the location of your initiative. The maps can be

accessed through either Ray Garside or Steve Johnson (01782 232695) or via the City

Council website in the near future.

All regeneration research activity in the near future must contribute to the wealth of

information that the Knowledge Management Team has. In return the Team can assist

regeneration initiatives in providing both qualitative and quantitative evidence.




18


19/180

COSTS

The project lead should cost out how much money will be required to conduct the

whole of the initiative, including service delivery and also, the evaluation element,

which is forgot in the majority of cases. However, it may be that you will be allocated a

certain amount of money and you must cost the project around that amount.

The Childrens Fund and Sure Start require that roughly 3% of your total running costs

be allocated and used to conduct local evaluation. This amount is fluid depending on

whether the initiative requires tailor-made methods and primary data collection, such

as group consultation and surveys, and therefore more money should be reserved for

evaluation.

Initiatives must consider pooling resources together with other stakeholders or similar

initiatives to undertake co-ordinated research that will ultimately costs less than single

ventures (see also the section that follows on raising awareness, regarding the Aplaws

system).




19


20/180

RAISING AWARENESS / COMMUNICATION

What do we mean by Raising Awareness

Telling everyone possible within your target audience that your initiative exists and

informing them of what is can do in relation to them on an ongoing basis.

Why is it important

It is vital that once the project has a solid structure that it is publicised to all potential

partners and service users. This will enable partners to gain interest in working with the

initiative and will assist in attracting the right service users. This will ultimately inform

the roles and responsibilities between partners.

How to

Stoke-on-Trent City Council has an internal system for raising the awareness and co-ordinating activity throughout the City. The system is termed Aplaws which merely

means the process of getting information onto the system. When projects are planning

specific research activity and want to raise awareness of this to others, also possibly

inviting stakeholders to contribute to costs etc, they should make contact with the

relevant person(s) who will put the brief details onto the City Council Community

section of the website.

The relevant contact for project leaders to initially pass details of research activity onto

is initially the Evaluation Co-ordinator for Regeneration North Staffordshire Marie

Broadhurst, contactable through the Programme Co-ordination Unit.

Information of research activity being undertaken can be accessed via the Regeneration

Zone and LSP websites, alternatively contact the PCU.




20


21/180

ROLES AND RESPONSIBILITIES

Why is it important

It is vital to identify ALL stakeholders.

To ensure that all roles and responsibilities within the initiative are defined for all

stakeholders involved in the initiative, and that these are allocated to relevant

individuals both within the project, throughout the partnership and the community, that

these individuals understand what their role is and what will be expected from them by

when. This must be agreed between all stakeholders involved in the initiative.

If roles and responsibilities are not defined and assigned to people then there is no

certainty of the actions being undertaken at all or to what quality these will be.

How to

The latter chapters on Involving Others will give further information and advice on

how to involve and agree roles and responsibilities.

The following checklist will also help.

To identify all stakeholders that will benefit from the initiative (primary

stakeholders) and all stakeholders that are involved in the delivery

(secondary stakeholders).

To ascertain how you will add value to each group.

Plan the evaluation down to detailed actions of what needs to be done.

For example, questionnaire development, distribution, analysis and

feedback to partners.

Plan timescales of when the evaluation needs to be done by. For

example, the questionnaire must be developed by March, will be

distributed after every session, results will be analysed every quarter and

results distributed every quarter.

Allocate evaluation to the relevant individuals who will be responsiblefor conducting evaluation at all of the stages planned.

Ensure that the individuals allocated actually understandwhat is

expected of them, by when, to whom and in what format. Ensuring that

there is clear agreement between all parties of expectations.

Ensure that all roles and responsibilities are documented and circulated

to all relevant parties involved.




21


22/180

Ensure that all communication that takes place which makes

amendments to previous agreements is documented and circulated to all

parties involved, to ensure that understanding is maintained.

Have roles and responsibilities been identified? Including roles of

partner representatives.

Have roles and responsibilities been clearly specified? Is there a shared

understanding? Documenting agreed roles and responsibilities might

help to raise awareness.

Are there structures in place to remind partners of their responsibilities?

Or to induct new partners?

Is there ownership and commitment amongst partner representatives and

their organisations to fulfil the roles and responsibilities set out?

Have management responsibilities been agreed and defined?

Types of stakeholders

The community whose situation the programme seeks to change

Project Field Staff who implement activities

Programme Managers who oversee programme implementation

Funders and other decision-makers who decide the course of action related to

the programme

Supporters, critics and other stakeholders who influence the programme

environment

(Monitoring and Evaluation Toolkit Stakeholder Participation May 2001)




22


23/180

LINKAGE NATIONALLY

What do we mean

All major regeneration programmes throughout the country will have some form of

national evaluation being undertaken, many are contracted through a major University.The national evaluation will involve making contact with a sample of locally delivered

projects under the programme and collecting evidence of what is working and what is

not.

Why is it important

Any national evaluation should work alongside and support what regional and local

partnerships are already doing and what they plan to do in the future about their own

monitoring and evaluation work. A key aspect of national evaluation should be action

research this aims to provide useful feedback to the partnerships and others within

the lifetime of their scheme.Much can also be learnt from observing the results and conclusions from national

evaluation undertaken, for example, research methods used, what worked, what did not

work, types of regeneration initiatives.

How to link with National Evaluations

For this to happen, the national evaluation should fully engage with partnerships,

communities and the Government Offices. It is essential that the main evaluation is a

genuine partnership between all those who have an interest in the success of the

programme. Data available will be used to show how projects stand in relation to

district and national averages. Details of & guidance from the national evaluation

should be circulated as soon as possible to partnerships and also project level.

Examples of National Regeneration schemes to learn from

Information and reports regarding the following national schemes can be found via the

internet:

National Evaluation of Single Regeneration Budget DTLR Research report

Urban Regeneration Companies Learning the lessons DTLR Research report

Neighbourhood Regeneration lessons and evaluation evidence from SRB case studies

DTLR Research reportsPartnerships in Community Safety, an Evaluation of Phase 2 of the Safer Cities

Programme DTLR Research report

New Deal for Communities National Evaluation

Neighbourhood Management National Evaluation

Neighbourhood Renewal National Evaluation

Housing Pathfinder National Evaluation




23


24/180

LINKAGE REGIONALLY

What do we mean

Ensuring that all regeneration initiatives occurring within a region are aware of each

others activity and link activity undertaken where applicable. That regions are aware ofregeneration initiatives occurring within other regions are link where appropriately and

learn from noteworthy practice.

How to link with regional initiatives

RegenWM

RegenWM has an initiative termed Strengthening Regeneration Networks. The purpose

of the project is to encourage and support the development of high quality networks

within the region that enable regeneration practitioners (professionals, volunteers and

residents within communities) to access:

Good quality information including information about learning and fundingopportunities, policy developments, good practice etc.

To lobby and to advocate up the system as well as down it.

Counties that the initiative will cover include West Midlands, Warwickshire,

Worcestershire, Herefordshire, Shropshire and Staffordshire.

The work is being carried out by Fran Fahy who has been seconded to GOWM and

AWM, the RDA to work on this development (0121 202 3260).




24


25/180

ETHICAL CONSIDERATIONS

What do we mean

Anyone who carries out research needs to make sure that activities are ethnical.

Why is it important to consider

Depending on the nature of your project, its participants and the sensitivity of the issues

being dealt with, ethnical consideration may lead you to revise your research questions.

How to be ethnical

The main factors to consider are as follows:

1. What sort of information will you be collecting and from whom?

2. What will you need to do to make sure that you are getting fully informedconsent?

3. How will you make sure that information is collected and used sensitively?

4. How will you fully inform service users about this evaluation?

5. How will you make sure that no harm is done through this evaluation?

The majority of people who are asked to provide information for an evaluation do so

willingly. However, the responsibility lies with the evaluator to ensure that:

1. There is no coercion

2. That people are treated with respect when information is collected and used

3. That people know how the information is to be used and they are happy for it to

be used in that way

4. That the report does not identify individuals




25


26/180

INTERNAL / EXTERNAL EVALUATION

What is internal / external evaluation

Internal evaluation is where evaluation skills and expertise is planned and undertaken

from persons within the initiative.

External evaluation is where evaluation skills and expertise is sought from persons

outside of the initiative and there is usually a large cost implication.

Why it is important to consider this issue

It is important to state how the evaluation of your initiative will be planned, undertaken

and managed at the development stage. Careful consideration needs to be made to what

the initiative aims to achieve, how evidence can be collected throughout the delivery

phase to prove that a difference has been made to the wider community, how this

evidence will be collected, considering timescales and other resources and who will beresponsible for undertaking this. Resources and expertise available to undertake

evaluation will be a major influence in whether internal or external evaluation will be

undertaken.

How to

Local evaluators should be independent of both partnership organisations and service

providers (for example, independent consultants, members of a local research

organisation group). To maintain this independence, partnerships should agree relevant

and ethical protocols, for example, to access to partners, staff, users and other

stakeholders, and on confidentiality, reporting and publication. Conversely,

partnerships and providers of services must also have confidence in the expertise and

knowledge of the local evaluations themselves.

(Childrens Fund Guidance on objectives, target setting,

local monitoring and evaluation Nov 2001)

Internal evaluation

If evaluation is to be undertaken internally, for example as self-evaluation, then the

principles throughout the Evaluation Framework and the remainder of this guidance

document, especially regarding undertaking primary research, should be followed

where applicable.

External evaluation

Where evaluation is to be undertaken externally, then the principles throughout the

Evaluation Framework should be considered as minimum.

Possibilities of people who can undertake evaluation follows:

Local Universities Staffordshire University & Keele University

Local private consultants

Other public agencies (for example Audit Commission and IDeA)

Peer evaluation

Local Research Groups




26


27/180

Tender

Where an invitation to tender must be written to invite possible external evaluators, the

following important points should be covered in the invitation:

Background Purpose

Objectives

Outline of possible methods

Timescales

Budget

Deliverables

Obtaining further information

Supporting documents

Allow at least one month for submissions from tenders.

A decision must be made as to whether the invitation will be:

1. An open tender to all public

2. A closed tender to a selected list of reputable consultants

3. A single action tender to one consultant

(Evaluation UK A Guide to Programme Evaluation December 1997)




27


28/180

TERMS of REFERENCE (ToR)

What do we mean

This lists the evaluation background as well as services and products that the evaluator

is expected to deliver and should be drafted.

Why it is important

There is a mandatory requirement from the Single Programme Appraisal Guidance

(Department for Work and Pensions 2003) to:

Summarise the proposed arrangements for evaluating the results of the project

during and after its delivery and show that the arrangements will be in place to

ensure relevant data is made available. Explain how any lessons learnt will be

disseminated and fed into future projects.

How to develop ToR

The ToR should:

Describe the purpose of the evaluation and who the stakeholders of the

evaluation are, specifying why the evaluation is being requested and what the

information will be used for.

In consultation with stakeholders focus on the key issues and concerns to be

addressed and questions to be answered, for example handling sensitive issues.

Specifying the methodology, describing data gathering instruments and methods

of analysis. May be developed with the assistance of an expert. List background documentation that will be provided to be the evaluator.

Specify the composition of the evaluation team profile etc. The evaluation

methods as well as availability of funds will determine the evaluation team

composition.

Describe the evaluation work plan. Specify the responsibilities of the evaluator

and the team leader, detail specific tasks to be undertaken as well as the time

lines involved, Indicate the audiences are to receive which information at what

times, what nature and schedule of written reports and oral briefings will be,

how results will be disseminated.

(Monitoring and Evaluation toolkit Planning and Managing an Evaluation May

2001)




28


29/180

BID WRITING

Why bid submissions are important

It is important to state as clearly and concisely as possible what your initiative is, how

this will be delivered, what money is required to deliver this, how this will be evaluatedand how it will contribute to strategic links and other strategies.

How to submit a bid

The Programme Co-ordination Unit has produced an updated version of the

Neighbourhood Renewal Programme initiative submission form, which asks questions

more specifically and clearly to assist in the planning, delivery and evaluation of

initiatives. This common submission form will form the basis of all bids that are written

and cover the same wide-ranging issues that the Evaluation Framework for

Regeneration North Staffordshire specifies.

A standard application form similar to this will be produced in time for all other

programmes such as Regeneration Zone and Housing Market Renewal.

The Neighbourhood Renewal Fund Application Fund

NRF APPLICATION

FORM.doc




29


30/180

Section 2.

PLANNING / DELIVERING




30


31/180


32/180


33/180

PLAUSIBILITY

What is it

Based on the methodology stated, initiatives must consider how possible their actions

are in achieving their aim(s) and objectives.

Why is it important

If actions to achieve objectives are not possible then objectives will not be achieved.

How to

In essence, activities must be:

1. Relevant to the baseline position

2. Supported by evidence from elsewhere (case studies or general evidence that theintervention can be applied elsewhere)

3. Have plausible theory of change some logic which links them to the

outcome (eg. Improving locks on doors will help decrease burglaries)

4. Are not necessarily funded projects - They might be activities which involve

mainstreaming

5. Are not simply off the shelf unless they pass all of the above tests

(LSP Toolkit renewal.net December 2003)

Initiatives must describe the methodology behind achieving the aims and objectives

stated, how will they be achieved in detail of what will be done, by whom, by when,how.

Initiatives should then score how plausible their actions would be by using the

following table. This score will be recorded at the Programme Co-ordination Unit along

with other evaluation detail to support programme and project evaluation:




33


34/180


35/180

Evaluation Methodology

It is vital that thought must be given at this stage of setting aims and objectives that

consideration should be given to how evaluation will occur and in what ways it should

occur. This is where project developers can be innovative in the methods that they

choose to conduct evaluation and this will make it easier when the time comes to collectevidence of whether aims and objectives have been achieved.

The list below gives some examples of methods to choose from:

Types of evaluation Methodology both quantitative and

qualitative

Evaluate project interventions Questionnaires

Evaluate overall project achievement of

outcomes

Comparisons, focus groups, observations

Evaluate overall project processes Comparisons, focus groups, observations

Evaluate partner / stakeholder perspective Interviews,

Evaluate community perspective Questionnaires, focus groups, interviews

Evaluate overall programme Questionnaires, focus groups, interviews

Other methods:

Evaluate in tandem with other projects

Video diaries

Having students to evaluate the effect of

something on something else eg. The

effect of bonding and attachment

Tracking of beneficiariesAnalysis of professionals understanding of

the project

Satisfaction surveys

This information is not exhaustive and innovative tried and tested methods are

contained throughout this guidance, for further information see the data collection

section.




35


36/180

BASELINE SETTING / BASELINE UPDATING

What is a baseline and why is it important

A baseline is a bulk of information, both quantitative and/or qualitative, that describes

a certain geographical location and/or its characteristics, which can be used to describechange over time.

The purpose of requiring partnerships to describe progress against baselines is to

identify problems and priorities in certain neighbourhoods, to then use this information

to decide on outcomes that are required to be delivered and also choose indicators to

help to measure progress towards those outcomes.

Indicators will show how we have achieved an outcome some examples of indicators

showing achievement of outcomes are given in the table below.

Baselining is related to benchmarking (future section) where initiatives compareinformation they have obtained over time. Benchmarking can also be done where

information obtained is compared to similar organisations, to compare both outputs and

outcomes achieved and processes undertaken to achieve them.

How to set a baseline

For many strategic objectives baseline data is already available and should be used.

Where numerical data and/or targets are available, they should be used to describe the

intended changes. Where such data are not available partnerships should be as precise

as possible in describing their starting point. partnerships should take steps to fill gaps

in baseline data. Some data that is available may only be so in the form of national data,

so projects will need to establish whether it is feasible to ask a relevant agency to

perform a specific computer run on the data to obtain local data.

(SRB Guidance manual, revised May 2003)

The baseline variables need to measure the essence of the local economic and social

conditions prevailing in the location both at the start of the programme and at the end of

the programme, relative to the national average. The baseline also needs to throw light

on whether any improvements in baseline conditions recorded is likely to be sustainable

when the programme ends, or whether any improvements made could only be sustained

if the regeneration effort continued.(Dept of Land Economy Discussion Paper An examination of baseline issues -

Essential Features of a Baseline 1999)

Projects need to identify at least 1 baseline indicator per key strategic objective OR

Select baseline indicators when drawing up the delivery plan, providing the opportunity

to ensure output measures among key indicators are as closely linked as possible to

baseline indicators.

Baseline Indicators can be modified after the interim evaluation process

this is why it is important to review against them on an ongoing basis!




36


37/180

Projects need to ensure that baselines are:

Appropriate (how do they relate to the strategic objective?)

Measurable

In geographical boundaries of the scheme area (Never easy to meet data

always regional or national never less than Local Authority level) The data be easily obtained

The data be easily updated

It is important to remember that the purpose of evaluation is to assess the

improvements that the scheme brings to the lives of people living and working in

the area.

Where to get a baseline from

Contact Ray Garside at the City Council for information on existing data that is

readily available regarding your target population and area.

Find out what information your own organisation has in relation to your target

population and area.

Use the following web links and contacts to find what information is readily

available regarding your target population and area.

You will need to collect new information specifically for your purposes

especially if the information you require is so specific that is has not been

collected (For example, data regarding local transport facilities, shops, local

services & community facilities) see sections on data collection and co-

ordinating research activity, as referred to in the initiating section.




37


38/180

TIPS:

Make a start in making a baseline as early as possible.

Split it into small and manageable tasks.

Take advice from other with expertise.

Try and present as much information as is possible, quantitative & qualitative,

which build up a complete picture of the area.

But do not collect information for the sake of it.

You should use the most recent information that is available to show the current

position of the neighbourhood.

You must take into account the time gap between any data sets you compare

bearing this in mind when interpreting the information. It would be easier to get

comparable data from the similar time period.

If there is more than one source for the same data please present all! This will

be a useful cross-reference.

Use at least the local district and England whenever possible, however, in

addition you may wish to show County or regional comparisons, to make more

sense of the conditions.

You should use the smallest geographical boundaries possible for your area depending on the information you are seeking and the boundaries you have

drawn up. Eg. If your neighbourhood is made up of a number of wards, you will

need to present each separately & calculate and average.

Make a note of post codes for the areas and if boundaries for a particular

indicator do not match the area note which post codes are covered to get hold

of certain data sets at a later date to allow comparison.

For each indicator you consider, check if there is information available from any

previous years to get a reasonable time series (eg. Five years if availableannually) and see what the trend is.

Explore if any other data exists to plug any gaps in data you have collected

paint as fuller picture as possible.

Make a note of when the information you are collecting will be updated, to

update your baseline.

When benchmarking, take care when interpreting the information if different

sources are used because the information will be collected in different ways.




38


39/180

Data that is available

First point of contact for gathering baseline information -

Stoke-on-Trent City Council Knowledge Management Team

Stoke-on-Trent City Council has a small resource that is collating baseline data for theCity, but is also beginning to encompass data for the whole of Stoke-on-Trent and

potentially North Staffordshire. The data can be supplied on a range of geographical

levels, including residential areas, wards, forum areas, the City level, Regeneration

Zone boundaries & can provide combined authority data. The data consists of the 2001

Census with additional supporting data for areas such as health, education,

demographics, social & economic. The resource has the capacity to supply baseline

data to individual projects on an initial basis, the projects then having the potential to

update their own baseline data, which the City Council would like returned to them in

order to update their baselines. This service can assist in identifying where data gaps

exist and where projects must focus further research activity.

The initial point of contact is Ray Garside, City Council.

All projects must make initial contact with the Knowledge Management

Team when obtaining baseline data or collecting baseline data before they begin

All projects must have their project submissions agreed by the Knowledge

Management Team before they are submitted to the Programme Co-ordination

Unit




39


40/180

Other Points Of Contact

CLIP Central and Local Government Information Partnership

This is the main method of consultation between central government and local

government on non-financial data and statistical issues. The purpose of CLIP is toenable central and local government to work together to develop an efficient and

effective information infrastructure for policy development, implementation,

monitoring and reporting. CLIP operates a number of topic-based groups which meet or

three times a year to discuss statistical issues. The subgroups include: Census,

deprivation statistics, education, housing, labour market, population, planning, social

services and transport. These groups offer opportunities for people to get involved in

the work on specific themes. Further information can be found on www.clip.gov.uk

Key indicators for Economic Regeneration

In response to the Spending Review 2002, an inter-agency working group wasestablished for roughly one year to develop a set of indicators for economic

regeneration. These indicators link to Public Service Agreements, floor targets and

RDA targets and other government targets to try and rationalise the number of

indicators that local authorities have to collect for different purposes, they are also a

part of the Local Library of Performance Indicators, Quality of Life Indicators, which

can be used in a wider regeneration context social and environmental as well as

economic. The contextual indicators are of more general use in understanding the local

socio-economy and setting a baseline, subject to the data reliability and availability. To

ensure the indicators meet local need, firstly select the indicators that reflect the local

economic regeneration priorities and then decide on further breakdowns and additional

local indicators.




40
http://www.clip.gov.uk/http://www.clip.gov.uk/


41/180

West Midlands Data & Intelligence Observatory

Formally launched in April 2002. The aim of the Observatory is to provide accurate &

comprehensive intelligence to improve the quality of strategy formulation and policy

making in the West Midlands, by enhancing and sharing understanding amongst

partners of the opportunities and threats facing the region and its sub-regions.The objectives include reviewing the state of the region, engaging partners in

determining information and intelligence priorities and mobilising relevant research

work, provide effective access to information and intelligence global through to local,

ensuring consistency and best practice.

The usefulness of this service requires further research with relevant others.

Boundaries and Statistics for Town Centres

The DTLR report Producing Boundaries and Statistics for Town Centres (2002)

produces interactive maps and tables giving turnover, employment and floor space

statistics for town centres across London. This is released free of charge on the internetand it is planned to extend across all Town Centres across the whole of England during

the next twelve months.

(Business Improvement Districts Guidance working draft August 2003)

Data trends online

There are a number of data sets online at the IDeA website which shows key socio-

economic measures and trends up dated to show trend for 2001 & 2002. The trends

include data for education & training, including foundation & advanced modern

apprenticeship data, crime data, health data, etc.




41


42/180

When data is NOT available

DATA COLLECTION

Local audit / primary research collection

It may be necessary to conduct a local audit, to go out with a number of local residents

and measure directly what the situation is on the ground, recording carefully what you

find. Afterwards you might want to provide some assessment of the situation and

discuss more widely before putting this information into a baseline. You will need

expert assistance when conducting your own research. If you are able to access a

Geographical Information System (GIS) you may find this helps to show what is

actually contained in your neighbourhood.

Examples of local research

Interviews, surveys, factual surveys

(into employment and economic activity levels etc)

Opinion surveys

(ask residents/local businesses for their views on the area they live in)

Focus groups

(exploring residents perception more thoroughly, also exploring process issues)

Observation

(looking through the community centres bookings and observing the state the place is

kept in)

All projects must make contact with Ray Garside and inform him of the data youare collecting and the area you are collecting data on both before you begin

collecting and after you have collected this is to ensure co-ordinated research

activity throughout the City

Need for more local comparative area data!

There has been over reliance on Census of Population data and comparative area data.

There needs to be more co-ordinated local attitudinal surveys, data on ethnic

communities, measurement of baseline conditions needs to be defined and linked to

community development. The baselines variables selected should be desirable on

conceptual grounds but also need, from a practical perspective, to be measurable atdifferent spatial levels, including small local areas. Achieving this balance is a tall

order.

(Dept of Land Economy Discussion Paper An examination of baseline issues -

Essential Features of a Baseline 1999)




42


43/180

General TIPS for collecting own data:

The people interviewed reflect the overall characteristics of the population.

The questions asked match equivalent questions from national surveys to enable

comparisons to be drawn.

The questionnaire is set up and conducted in a way that minimises bias.

A member of your partnership has specific responsibility to liaise with

contractors, community groups and residents.

Trawl for what other surveys have been done in your area to ensure different

residents are used etc.

Agree with contractors the questions to be asked need a wide range ofquestions to address the wide-ranging issues.

That you test/pilot the questionnaire out with a number of residents (20 or more)

to see how well it works in practice (these residents should not be asked to

participate in the main survey). Make alterations as necessary.

That the survey is publicised well, to ensure a high response rate, for example, a

letter from the partnership telling residents about the planned survey, an article

in the newsletter and talking to local groups. Below 65% there may be problems

around the reliability of the information you obtain.

Analyse results obtained from the questionnaire pilot and get a good feel for

what the results will tell you. Then suggest any further information that you

would like out of the data and reorganise questions with the contractors

accordingly.

That the results you obtain from the contractors are in a readable and

understandable format, to enable you to do further analysis in the future if

required.

Feedback the results of you baseline survey to residents, for example throughpamphlets, newspaper articles, presentations etc. This will promote ownership

of the results to the community.




43


44/180

Using residents as interviewers

You could use local residents as interviewers ensuring they have been trained in

interviewing techniques:

Advantages Factors that need consideringIt increases employment opportunities for

local people through training

Selecting residents for training can be

difficult and lead to disappointment

It is a good way to build the capacity of

people to go on and do other things in the

community and develop personally

There is no guarantee that the training and

employment can be applied to other local

opportunities once the survey is completed

It is useful in letting people know about

NDC

The survey process will take longer and

you will need to check the quality more

Residents may possibly open up more or

be more honest when answering questions

from a local resident

The training will need to come from

people who are both technical experts and

good at working with inexperienced

residents

In communities will a diverse, multi0racial

population it is essential that the interview

team includes people from different

sections of the community who speak the

languages required to get a good response

rate

Be careful about ethical problems some

people do not want neighbours to know

about income, health and other sensitive

matters




44


45/180


46/180

Methods of data collection

The research question or questions will determine the most appropriate methodology to

be employed.

Ask yourself the types of questions that you may want to ask about a particular

issue/problem, for example: Who / what / where / why / when / how.for example

How many people are in this situation.. are affected by this problem.. have

been helped by this initiative?

How prevalentis this problem?

Which groups are most affected by these issues .are most at risk?

How much of a differencedoes the initiative/programme make to the

prevalence of these problems?

Why are people affected by the problem?

How do people end up in this situation?

How does the initiative/programme work?

Why does it work/not work?

Whatdo people think about the intervention?

To answer the above, then a combination of both quantitative and qualitative

information would be required, to gather information of both the topic and an

understanding of it.

The strengths and weaknesses related to both methods can be seen below:

QUALitative QUANTitativeStrengths Flexible Produces exact statistical

data

Enables exploration of the

meaning of concepts and

events

Where random probability

samples are used, survey

estimates can be defined

within specific bounds of

precision

Produces valid data if

issues are explored in

sufficient depth to provide

clear understanding

Can measure the extent

prevalence, size and

strength of observed

characteristicEnables study of

motivations and patterns of

association

Can determine the

importance of factors in

influencing outcomes

Provides a detailed

understanding of how

individuals interact with

their environment, etc

Uses standardised

procedures and

questioning, enabling

reproducible results

Limitations Sample sizes are often

small as the research is so

time consuming

Can be costly and time

consuming, particularly if

population rare or hard to

reach




46


47/180


48/180

Method Strengths Weaknesses

Postal Relatively cheap (but

questionnaires must be kept

short and simple)

Can target large sample

Reply rate is usually quite

low

Few means of controlling

how the questionnaires are

completedWeak quality response

from those with poor

literacy and motivation

Worst for missing out data,

having errors etc

Usually lowest response

rate

Web-based Easily target a large

amount of the population

(but they must be signed-

posted where to go)

Only a certain percentage

of the population are techy

Written questionnaire

Focus groups

Observation

Video diaries

Personal written diaries

Students to evaluate

Tracking beneficiaries

Combining a number of methods used

Advantages Disadvantages

Save money Development time may be extended

Improve geographical coverage Costs will be increased

Overcome sample frame bias Keeping track of the outcomes for each

case will be required

Improve data quality, such as response

rate or item non-response

Need ensure that data is not lost and

ensure data can be stitched together to

make sense

Speed up data collection Impact on reliability and validity of data

obtained can be hindered

Overcome resource problems, such as lack

of face-to-face interviewers




48


49/180

Sampling

Sampling is used to make inferences about a population from a relatively small number

of observations, that are assumed to be representative of the population.

In a social researchers ideal world, all data would be collected by census. However, inpractice this approach is not always practical because of time and cost constraints. As

an alternative, sampling methods are used to provide estimates based on data from a

very small percentage of the population.

Sampling works because:

It is not necessary to collect data on ALL people in order to generate statistics

about that population.

After a certain sample size, there is no need to collect more data. The extra data

does not improve the accuracy of the estimate to any great extent.

Sample bias:

Population estimates based on survey data will be inaccurate if the sample is biased.

Sample bias will occur because of the following:

Sample bias occurs when the selected sample is systematically different to the

population. The sample must be a fair representation of the population we are

interested in.

The sample size may be too small to produce a reliable estimate.

There may be variability in the population. If you want your sample to give anestimate that is close to the population value then you need to take into account

how much variability there is in trying to measure. All else being equal, the

greater the variability the larger the sample size needed.

Sampling frames:

The ideal sampling frame is a straight forward list of the elements you are trying to

sample. So for a population sample, a comprehensive list of the population would be

ideal. In practice, the ideal sampling frame hardly ever exists. The following are

examples of possible sampling frames that social researchers can use:

The Post-Code address file: This is the Post Office list of all addresses in the

UK, which receive less than 25 items of post per day. The list is primarily

residential addresses (about 94%) and the list covers approx 99% of all

residential addresses. The list closely approximates to a sampling frame of

households which makes it ideal for surveys of households.

Random Digit Dialling: This involves selecting 11 digit numbers at random, the

first 7 digits being randomly selected from the published list of prefix numbers

and the last 4 being generated at random.

Sample size:




49


50/180

The main decision needed in deciding on a sample design is sample size. The following

questions will help in deciding on this:

What are the key estimates for the study?

How precise do those estimates need to be?

Are there key sub-groups fro which separate estimates will be needed?

Does the survey need to be large enough to detect change over time between

surveys, or differences between key sub-groups?

Participatory evaluation

Participatory methods can be used in place of, or alongside, written feedback. They can

be a rich source of feedback that can provide detailed information, for example on how

the quality of the group event can be improved.

The disadvantages are that participatory methods take longer to carry out than written

evaluation methods and the results are more difficult to analyse and compare acrossdifferent group events.

Interviews

Interviews can be conducted over the telephone or face-to-face. You will need

to decide which method is most suitable for you and your service users.

Decide how many interviewers to carry out and with whom. Decide who will

carry out the interviews, how many each interviewer will carry out and over

what period of time.

Review the information you already have and decide what specific information

you need from interviewers. Decide how to phrase questions to get theinformation you want. Focus on questions on the information you need. You

may need alternative ways of asking the same question in case respondents ask

you to re-phrase it. This is particularly important if several different

interviewers carry out the interviews. It will ensure that interviewers continue to

ask questions in a similar way, thereby ensuring that the responses are

comparable.

Construct an interview schedule detailing the question and the order on which

they should be asked. The schedule should also detail the information given to

the respondent. For example, it will tell them what the interview is for, how the

information will be used, assure them of confidentiality and ask for explicit

permission to carry out the interview.

You may want to record responses in writing or you can use a tape recorder. Get

explicit permission from respondents to use the tape recorder. Bear in mind that

although this records all of the information, it means extra work in terms of

transcribing and extracting the information you need.

Keep your interviews fairly short. Most interviews can be conducted relatively

easily in 15-30 minutes.

Ask potential respondents whether they are willing to participate and whether it

is convenient to carry out the interview or not.




50


51/180


52/180


53/180

Adaptability

Allow data content and/or reporting to be readily modified to address changing needs.

Small-area analysis

Take into account the limitations of applying national data to the state, local and

community levels. This pertains to using small numbers in ones statistics.

Is the data simple to understand?

Accessibility

Provide clear reports to a non-technical audience; make available diverse reports to

information tailored to different needs or users, and provide access to public-use data

sets at a reasonable cost so they can be independently analysed.

Translation & policy applicability

Effectively translate technical data to policy-relevant information.

Disseminations

Accurately and fully inform potential users or decision-makers about the resources and

how to access it effectively.




53


54/180

DATA PROTECTION

What is it and why is it important

This is a summary of the definitive Data Protection Act 1998.

These guidelines apply to anyone involved in the collection, processing and use ofmarket research data and all methodologies quantitative and qualitative and sample

sources which are potentially covered by the 1998 Act.

How to comply with data protection

The guiding principles of the Act are:

Transparency ensuring individuals have a very clear and unambiguous

understanding of the purpose(s) for collecting the data and how it will be used.

Consent at the time that the data is collected, individuals must give their

consent to their data being collected and also at this time have the opportunity to

opt out of any subsequent uses of the data.

Key data protection principles:

1. Personal data shall be processed fairly and lawfully.

2. Personal data shall be obtained only for one or more specified and lawful

purposes and shall not be further processed in any manner incompatible with

that purpose or other purposes.

3. Personal data shall be adequate, relevant and not excessive in relation to the

purpose or purposes for which they are processed.

4. Personal data shall be accurate and where necessary kept up to date (with every

reasonable step being taken place to ensure that data that is inaccurate or

incomplete, having regard to the purpose(s) for which they were collected or for

which they are being further processed, are erased or rectified).

5. Personal data processed for any other purpose or purposes shall not be kept

longer than is necessary for the purpose or those purposes.

6. Personal data shall be processed in accordance with the rights of data subjects

under this Act.

7. Appropriate technical and organisational measures shall be taken against

unauthorised or unlawful processing of personal data and against accidental loss

or destruction of or damage to personal data.




54


55/180

Sensitive data:

Explicit consent is required for processing sensitive data this means that the consent

must be absolutely clear and based on a detailed explanation of how the data will be

used. This is defined as personal information covering:

Race or ethnicity Political opinions

Religious beliefs

Trade union membership

Physical or mental health

Sexual life

The commission or alleged commission of an offence or any proceedings for an

offence committed and the outcome

(Market Research Society Basic guide to the data protection Act 1998)




55


56/180

ANALYSING INFORMATION

What do we mean and why is it important

Interpreting the information that you have collected into conclusions and findings,

drawing out the key messages from all information and making recommendations.

How to analyse information

Interpreting evidence can be a difficult task and enough time should be allowed for this

phase. For example, reading 20 questionnaires and comparing and summarising the

answers to 6 questions can take several hours.

Quantitative information

This type of information can be summarised in numerical forms. For example, How many families use a service per day/week

How many children under 5 attend a play group

The average number of people using a drop in advice service/week

Qualitative information

This type of information can be difficult to unpick. You need to sift through the

findings and explore implications. The following points may help you:

In order to get the best use out of qualitative data, it may be useful to sort thedata into some order, positive/negative and advantages/disadvantages etc

Summarise the material into a series of points

Look for reoccurring points and note how many times they appear and with

what qualification and differences if any

Aggregate the information by noting the most frequently occurring points and

any other points which seem to be of particular importance, bearing in mind the

context

Look back at your original measure of success and outcomes to help pick out

what you need

Avoid being unduly influenced by a tiny number of either very positive or very

critical comments

Try to establish the majority view

Check that your judgements are based on evidence from difference categories of

people involved with the project (eg. Participants, carers, group leaders)

Look for unexpected outcomes

Using quotes can give a flavour of the project and can often sum up unexpected

outcomes indicate the type of person who gave the response but do not give

the actual source

Some quantitative analysis of the information that you have found can support

your findings




56


57/180

Keep it simple!

Evaluations are neither scientific experiments nor social science surveys and analysis

should not be complex. It is for example, rarely necessary to conduct sophisticatedmultivariate analyses or modelling. The rule is Keep it simple!

Most evaluations will only require frequency counts, averages and totals, illustrated by

quotes and qualitative data from questionnaires and interviews.

(Evaluation UK- A guide to programme evaluation - Dec 1997)

Analysing questionnaires

There is a range of database packages that you can use for storing and analysing data,

the most commonly used are Microsoft Excel and SPSS. These packages enable you to

generate statistics, tables and charts readily. For more sophisticated analysis SPSSshould be suitable.

Preparing questionnaires for entry into a database

Familiarise yourself with the database package

Collect all questionnaires and number them in numerical order. Ensure that each

record on the database matched the questionnaire number. This makes it easy to

check that a questionnaire has been entered correctly and can be tracked back to

the original paper copy.

Using a blank questionnaire, decide which answers need to be entered asnumerical and which need to be ent

regeneration project evaluation framework v3

Documents