the results: qualitative evaluation · qualitative evaluation - results • preparation for the...
TRANSCRIPT
![Page 1: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/1.jpg)
Ten D by Night conference Turin,
19 February 2010
THE RESULTS:
QUALITATIVE EVALUATION
Anita Villerusa, Associate Professor
Dean, Faculty of Public health
Riga Stradins University
![Page 2: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/2.jpg)
Ten D by Night conference Turin,
19 February 2010
![Page 3: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/3.jpg)
Ten D by Night conference Turin,
19 February 2010
What is evaluation?
• Evaluation is collecting data about an intervention and its effects, in order to judge the value of the intervention, through making a comparison, so as to make more informed decisions.
Øvretveit, 1998.
• Program evaluations aim to provide convincing evidence that a program is effective. The standards are specific criteria by which effectiveness is measured.
Fink, 1993.
![Page 4: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/4.jpg)
Ten D by Night conference Turin,
19 February 2010
Essentials of evaluation
Evaluation is : to collect data,
to judge the value,
to make a comparison,
to make more informed decisions.
Audit evaluation – comparison what was intended (objectives) with what happened
![Page 5: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/5.jpg)
Ten D by Night conference Turin,
19 February 2010
Project evaluation framework
Project start
Project processes
Project end
Goals/aims/objectives
Planning
Communication
Management
Training
Piloting
Field work
Outputs
Outcomes:
-New preventive approach
-Changes in alcohol/drug related accident rate
![Page 6: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/6.jpg)
Ten D by Night conference Turin,
19 February 2010
The criteria for evaluating a health
programmeCorrespondence - if the action corresponded to objectives and
procedures?
Outcomes (Short term and long term)
Objective measures
Subjective (retrospective perception)
Efficiency - resources consumed and outputs produced
Acceptability
to assess target population and providers willingness to use and
apply the action
Sustainability
the continuation of the programme, if finance continues, or if
withdrawn
![Page 7: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/7.jpg)
Ten D by Night conference Turin,
19 February 2010
Six standard criteria for programme
evaluations
1. What was done? (description)
2. Did all activities follow the plan and achieve the
objectives? (correspondence to plan and objectives)
3. What were the outcomes? (Did they meet needs of
target group?)
4. Was there any waste of resources, or could have used
resources more efficiently? (efficiency)
5. Did people participate in the project? (acceptability)
6. What threatens the continuation – with and without
continued funding? (sustainability)
![Page 8: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/8.jpg)
Ten D by Night conference Turin,
19 February 2010
Methods for qualitative evaluation
• focus group discussion
• In- depth interviews
• questionnaire also with open- ended
questions for managers and for volunteers
• analyses of different documents
(Analyses of written minutes of meetings;
E- mail Correspondence; reporting tables;
project time table and financial plan, press
releseas, publications etc.
![Page 9: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/9.jpg)
Ten D by Night conference Turin,
19 February 2010
Participants in the evaluation
Participants in the focus group discussion- 10 respondents of the
Ten D by Night Project participants from Italy took part in the focus
group; 3 were project managers and 7 were volunteers
In- depth interviews - Project technical coordinators- S&T Italy
Questionnaires – one questionnaire for managers and one for
volunteers (7 project managers and 22 volunteers completed the
questionnaires from al countries except Spain)
![Page 10: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/10.jpg)
Ten D by Night conference Turin,
19 February 2010
Qualitative evaluation - Results
• Preparation for the work
Training of project workers/volunteers.
The equipment - alcohol tests, drug tests, console for measurement of the reaction time.
Project tools - questionnaires, leaflets, information/consent forms, posters advertising the project and action in particular places and times.
• Sampling recreation places (Selection of night clubs, their representativeness and how they correspond to the target audience)
• Field work
• Advertising the project and rising awareness among the local population.
![Page 11: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/11.jpg)
Ten D by Night conference Turin,
19 February 2010
Involvement of volunteers in the
project
• Poland, Bulgaria, the Netherlands and
Belgium where the project partners were
NGOs they used people who they had
collaborated with in the previously
executed projects as volunteers
• Students
• the number of field workers
![Page 12: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/12.jpg)
Ten D by Night conference Turin,
19 February 2010
Information about the project, training
and preparation for the field work
Project managers
• ~ one hour training;
• Some theoretical part
of training;
• Training on the
practical
implementation of the
tests
Volunteers
• A majority assessed
the level of
information as
sufficient
![Page 13: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/13.jpg)
Ten D by Night conference Turin,
19 February 2010
Evaluation of information volunteers
received about the project, its mission,
objectives and tasks before field work
9
8
8
4
6
5
4
2
2
5
6
7
0% 20% 40% 60% 80% 100%
amount
type
quality
sufficient suff>insuff insuff>suff insufficient
![Page 14: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/14.jpg)
Ten D by Night conference Turin,
19 February 2010
The level of skills regarding the
practicalities of the field work
16
8
10
11
11
1
4
3
5
6
6
2
2
2
5
4
3
4
3
0% 20% 40% 60% 80% 100%
alcohol t.
drug t.
console
communic.
questionn.
sufficient suff>insuff insuff>suff insufficient
![Page 15: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/15.jpg)
Ten D by Night conference Turin,
19 February 2010
Selection of night clubs
• Different opinions
Some difficulties
Great number of rejections
in Italy
Less problems
Poland, Bulgaria and Belgium,
The Netherlands
![Page 16: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/16.jpg)
Ten D by Night conference Turin,
19 February 2010
• The main problem was to choose the clubs . When they were chosen, there was no problem in gaining admission. In other words, we faced no problems”. (Poland, Bulgaria and Belgium )
• “Making appointments was difficult/the clubs and discos were poorly accessible/ it was sometimes difficult to start a discussion about drugs”
![Page 17: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/17.jpg)
Ten D by Night conference Turin,
19 February 2010
Reasons for refusal
• luck of interest;
• unwillingness to collaborate;
• fear about control checks of people;
• possible negative effect on the number of
visitors.
![Page 18: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/18.jpg)
Ten D by Night conference Turin,
19 February 2010
Representativeness and the
target audience
• Difficulties to reach representativeness
• the project succeeded in reaching the maximum number of participants who correspond to the target audience- young people aged from 18 to 24;
• attitudes of the customers during the field work - “Overall good even if some of them found that it took too much time to answer the questionnaire and to do the tests. Most of them liked it and found it interesting to do the tests; they cooperated well”.
![Page 19: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/19.jpg)
Ten D by Night conference Turin,
19 February 2010
Field work
![Page 20: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/20.jpg)
Ten D by Night conference Turin,
19 February 2010
The design and methodology of
the project
• The experts
acknowledged that
the design of the
project was
appropriate and
adequate.
• Volunteers-
“How well did you
understand your role
in the field work – the
tasks and duties to
undertake”. 73% said
very well and 27%
quite well.
![Page 21: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/21.jpg)
Ten D by Night conference Turin,
19 February 2010
Evaluation of the suitability of
the tools
5
16
4
8
7
4
6
6
5
5
1
2
5
1
2
1
1
1
0% 20% 40% 60% 80% 100%
questionn.
alcohol t.
drug t.
console
very good good average bad very bad
![Page 22: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/22.jpg)
Ten D by Night conference Turin,
19 February 2010
Field work
Positives• both the managers and volunteers were satisfied with
the process of the field work
• Good collaboration was established with night clubs
• the level of awareness of the necessity of preventive activities increased within the night clubs
• The informative materials provided proved to be useful and small incentives/presents helped to attract participants
• New and expensive equipment was bought
• It was possible to implement the project in several countries at the same time and compare the results
![Page 23: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/23.jpg)
Ten D by Night conference Turin,
19 February 2010
Field work
Negatives
• In some clubs there was no space where to place questionnaires or it was not convenient to complete them
• There were queues to pass tests and some customers left because they did not want to queue.
• The drug tests took too much time
• the questionnaire was too long and some questions were misleading;
• In some cases there was not enough experience among the field workers how to deal with drunken people
![Page 24: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/24.jpg)
Ten D by Night conference Turin,
19 February 2010
Ethical aspects
The anonymity
The system of anonymity worked very well and was
adequate for the needs of project.
no threats to volunteers
![Page 25: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/25.jpg)
Ten D by Night conference Turin,
19 February 2010
Cultural and religious obstacles
and barriers
• No barriers were noticed.
• men showed a higher level of interest
regarding the intervention.
• there were many cases when women
refused to take part in intervention.
• although there were no restrictions on
gender, gender inequality was observed in
some countries.
![Page 26: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/26.jpg)
Ten D by Night conference Turin,
19 February 2010
Publicity of the project
• radio interviews;
• publications in newspapers;
• the information in web-sitewww.tendbynight.com
• posters and brochures
• The experts interviewed gave 3,7 marks out of 5 for the publicity at national level and 3,5 marks at the international level.
![Page 27: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/27.jpg)
Ten D by Night conference Turin,
19 February 2010
Evaluation of the suitability of the
project materials for field work
1
4
1
4
0
6
8
8
9
8
7
6
8
6
8
6
11
5
3
2
1
2
2
3
1
0% 20% 40% 60% 80% 100%
brochures
logo
posters
webpage
leaflets
gadgets
very good good average bad very bad
![Page 28: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/28.jpg)
Ten D by Night conference Turin,
19 February 2010
Use of results
• for practical purposes
• for scientific and research purposes
• for constructing educational programmes
• The further development of the project
idea
![Page 29: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/29.jpg)
Ten D by Night conference Turin,
19 February 2010
Meetings and collaboration
among project countries
• the meetings were useful and the amount
of them as optimal for the project
• Good collaboration
• Good communication via e-mail
![Page 30: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/30.jpg)
Ten D by Night conference Turin,
19 February 2010
“This was a very good project. It
gives understanding to young people
regarding those things which are
important to their health and safety.
The project gave us the opportunity
to talk about these things … and also
opportunity to talk to each other.”
![Page 31: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/31.jpg)
Ten D by Night conference Turin,
19 February 2010
Some final points
• Some changes in the staff of the project. .
• The decrease of the initial budget by 50
%...
• Some corrections in time table...
All the activities were implemented on time.
![Page 32: THE RESULTS: QUALITATIVE EVALUATION · Qualitative evaluation - Results • Preparation for the work Training of project workers/volunteers. The equipment - alcohol tests, drug tests,](https://reader034.vdocuments.site/reader034/viewer/2022050523/5fa6ec4ecb2f1802bd28ea47/html5/thumbnails/32.jpg)
Ten D by Night conference Turin,
19 February 2010
Goodbye Turin!