evaluating communication programmes, products and campaigns: training workshop
DESCRIPTION
A one day workshop on evaluating communication programmes, products and campaigns. The main steps and methods are covered with real life examples given. This workshop was originally conducted by Glenn O'Neil of Owl RE for Gellis Communications in Brussels in OctoberTRANSCRIPT
![Page 1: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/1.jpg)
Evaluating Communication
Programmes, Products and
Campaigns:
1 day training workshop for
communication professionals
Glenn O’[email protected]
Workshop originally conducted for Gellis Communications (www.gellis.com) in Brussels on 30 October 2009
![Page 2: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/2.jpg)
2
1. Introduction & definitions
2. Five steps of evaluation
3. Campaign evaluation methodology
4. Programme evaluation methodology
5. Product evaluation methodology
6. Reporting on communications evaluation
Schedule
![Page 3: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/3.jpg)
3
Communication professionals understand the
key concepts of communications evaluation
and thus increase their effectiveness in
managing evaluation aspects of their projects!
Training objective
![Page 4: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/4.jpg)
4
“Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy compared to a set of explicit or implicit standards, as a means to contributing to the improvement of the program or policy”source: “Evaluation”, by Susan Weiss, (1998)
“A form of research that determines the relative effectiveness ofa public relations campaign or program by measuring program outcomes (changes in the levels of awareness, understanding, attitudes, opinions, and/or behaviours of a targeted audience orpublic) against a predetermined set of objectives that initiallyestablished the level or degree of change desired”source: “Stacks, D. (2006). Dictionary of Public Relations Measurement and Research. Institute for Public Relations.”
What is evaluation?
![Page 5: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/5.jpg)
5
Programmes, projects, campaigns and activities that are dedicated to the management of communications between an organisation and its publicsSource: Grunig, J. (ed.) (1992). Excellence in Public Relations and Communications
What is communications?
![Page 6: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/6.jpg)
6
– A programme is an organised set of communication
activities based on target audiences, themes or functions running continuously or for long periods
– A campaign is an organised set of communications activities, directed at a particular audience usually
within a specified period of time to achieve specific
outcomes
– A product is an individual object, such as a
publication, website or video created to support a communication activity
What is communications?
![Page 7: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/7.jpg)
7
Aim of communications?
![Page 8: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/8.jpg)
8
– What is communications attempting to
change? What are the effects desired?
– How can these effects be categorised?
– How can these effects be measured?
What can communications change?
![Page 9: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/9.jpg)
9
When to evaluate?
Monitoring (not evaluation!)
Summative evaluationOutcome evaluationEx-post evaluation
Formative evaluationProcess evaluationIntermediate evaluationMid-term evaluation
Formative evaluation
Baseline evaluationEx-ante evaluationAppraisal
Impact evaluation
Activity
Different terms that mean the same thing!
![Page 10: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/10.jpg)
10
EPost only
TEET
TE
E
E
E
ETE EE
ETE
EE
Pre-post
True & constructed
cohort studies
Time series
Field
experiments
Meta-analyses TE
E
T E
TE T EE
E
Communications evaluation design
![Page 11: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/11.jpg)
Evaluating Communication
Programmes, Products and
Campaigns
2. Five steps of evaluation
Glenn O’Neil
![Page 12: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/12.jpg)
12
All evaluations, communications or otherwise,
follow similar steps in how they are carried
out:
Five steps of an evaluation
Planning Creation Management Analysis Presentation
![Page 13: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/13.jpg)
13
Five steps of an evaluation
Planning
Creation
Management
Analysis
Presentation
• Creating inception report
• Creating the evaluation framework
• Selecting the evaluation methods
& sample
• Consulting with client
• Establishing the team
• Determining the timeframe
• Determining the deliverables
• Creating the workplan
• Preliminary desk review
• Designing the evaluation methods
• Designing the templates and tools
• Designing final report outline• Refining inception report
• Testing the templates and tools
• Collecting the data
• Managing the data
• Analysing and interpreting the results
• Formulating conclusions
• Creating recommendations
• Writing the final report
• Submitting the final report• Presenting the final report• Disseminating the final report
• Promoting the final report
• Creating follow-up mechanisms (e.g. steering group)
![Page 14: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/14.jpg)
14
– Which steps are usually done well?
– Which steps are usually done less well or
skipped over?
Five steps of an evaluation
![Page 15: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/15.jpg)
Evaluating Communication
Programmes, Products and
Campaigns
3. Campaign evaluation
methodology
Glenn O’Neil
![Page 16: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/16.jpg)
16
Public information campaign
Advocacy campaign
Two types of campaigns
![Page 17: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/17.jpg)
17
– In theory, campaigns are easier to evaluate
than programmes
– Where do you start if you have been asked to
evaluate a campaign?
Evaluating campaigns
![Page 18: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/18.jpg)
18
– A good starting point is to map out the “theory
of change”
– The theory of change shows the pathway
from inputs to impact
What was this campaign trying to achieve?
Theory of change
Activities OutcomesInputs Impact
![Page 19: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/19.jpg)
19
Partners-led
Training programme
Example: Theory of change
ActivitiesInputs
Planning of campaign goals and activities
Preparation of campaign materials
Consultation and briefing of campaign partners
Special events & conferences
Grassroots mobilisation
Media campaign
Web campaign
Adaptation & production of campaign material
Activities
Organisation-led
Special events & conferences
Global day
Web campaign
Artistic projects (film, cartoon, book, poster)
Communication tools
Media campaign
![Page 20: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/20.jpg)
20
Outcomes Impact
Alliances
Network actively participated in the campaign
Engaged multiple stakeholders in the campaign at the country level & globally
Awareness
Increased awareness of human rights in general amongst rights holders
Action
Stimulated debate, spurred action and reaffirmed commitment of governments, civil society, educational, cultural and human rights institutions
Helped bridge gaps in HR implementation at the national level
Garnered further support for the organization
Outcomes Outcomes
People are
protected and
empowered to
realise their
rights
Example: Theory of change
![Page 21: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/21.jpg)
21
– The “theory of change” assists in clarifying
the objectives of the campaign
– The next step would be to create the
evaluation framework – the link from
objectives/outcomes to indicators to
evaluation methods
Evaluation framework
![Page 22: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/22.jpg)
22
Evaluation framework - exampleCampaignOutcomes
Proposed Indicators
Means of verification(evaluation tools)
Selection frame
6. Increased association of organisation as key actor for today’s ecological challenges
- Change to level of association of individuals
-Change to level of visibility in the media of the organisation
-Online panel study of individuals to assess changes of association
- Street polls in major cities(voxpop)
- Event attendance statistics and feedback
- Number of mentions of organisation and other key words in the media and online
- Number of visitors to campaign portal
-individuals recruited online
- urban population
- key events
- Selection of print and online media
- Campaign portal
![Page 23: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/23.jpg)
23
– Standard evaluation methods are used in
campaign evaluation…in combination with
methods particularly adapted for campaigns
and communications programmes
– A combination of qualitative and quantitative
methods is recommended
Evaluation methods
![Page 24: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/24.jpg)
24
Standard
Surveys
Interviews
Panels
Focus groups
Case studies
Observation studies
Evaluation methodsAdapted
Expert reviews
Content analysis
Media monitoring
Web metrics
Tracking mechanisms
Network mapping
We focus on these methods as they are special to communications!
![Page 25: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/25.jpg)
25
– A specialist examines a communication
activity or product and provides an
assessment
– Assessment is made often against best
practices or standards
A brochure is compared to the corporate
identity guidelines of an organisation
A website is measured against usability
standards
Expert review
![Page 26: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/26.jpg)
26
Expert review - exampleYour organisation
Comparative organisation A
Comparative organisation B
Comparative organisation C
High-level design decisions and strategy
57% 71% 71% 71%
Content design62% 85% 73% 93%
Navigation and search59% 87% 84% 94%
Content presentation59% 86% 86% 93%
General design aspects45% 95% 90% 92%
Overall compatibility58% 86% 83% 92%
Compatibility of organisation web tool and comparative tools with ISO usability standard
![Page 27: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/27.jpg)
27
– Media reports, documents or other
sources are analysed and categorised to
identify trends and patterns
– Content analysis assists in identifying
preference, priorities, trends, etc.
Content analysis
![Page 28: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/28.jpg)
28
Content analysis - example
Theme (Question) Category No. of posts KeywordsPolicy in the Middle East research 110 Middle East, Jordan, Iran, participation
Policy in Rwanda research 89Rwanda, research, quotas, gender, leadership
Policy and gender research 88 gender, legislation
Careers in policy work advise 55 consultancy, careersHeads of state and policy research 22 research, leadership
Upcoming elections & policy advise 18 Ecuador, elections, quotas
Indicators for measuring policy advise 13 policy, research, evaluation, PeruPolicy and representation advise 5 Congo, representation
Policy development advise 1 New Zealand, development
Content analysis of postings in online forum
![Page 29: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/29.jpg)
29
Content analysis - example
Comparison of media releases & updates by crises
Crisis one
Crisis two
Crisis three
Day of crisis 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
1 square = 1 media release or update 2 squares = 2 media releases or updates
issued on that day issued on that day
![Page 30: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/30.jpg)
30
– Media monitoring measures visibility of an
issue or organisation in the media
– Most monitoring counts mentions of
keywords in a pre-selected group of media
using automated software
– Media monitoring can be an indication of
levels of awareness amongst publics – but
it is not a replacement!
Media monitoring
![Page 31: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/31.jpg)
31
Media monitoring - example% of coverage
![Page 32: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/32.jpg)
32
Media monitoring - exampleNo. of articles
0
500
1000
1500
2000
2500
nov
dec jan
feb
mar
april
may
june
july
aug
sept
octnov
dec jan
Campaign 2008/9 Campaign 2005/6
![Page 33: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/33.jpg)
33
– Web metrics is data collected by
automated software on visits and other
actions on web sites
– This can be both for an organisation’s
website or a sector
– Web metrics can measure different
variables including interests, preferences,
interaction and online behaviors
Web metrics
![Page 34: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/34.jpg)
34
Web metrics - example
Language – no. articles (%) Language of visitors (%)
English 90 69
Chinese 6 15
Russian 3 4
French 0.5 3
German -- 2
Spanish
--
2
Other -- 5
Combination of content analysis (language content) with web metrics (language of visitors per computer settings) for online portal
![Page 35: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/35.jpg)
35
– Tracking mechanisms record actions
taken on issues, policies, legislation, etc.
– Tracking mechanisms are usually
manually tracked on standard forms in a
systematic manner
Recording how many partners join a
campaign
Tracking and recording the number of
business leaders that speak out on an
issue
Tracking mechanisms
![Page 36: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/36.jpg)
36
Tracking mechanisms - exampleCampaign year
![Page 37: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/37.jpg)
37
– Network mapping measures the relations
and flow between people, ideas and
organisations
– Network mapping is useful in measuring
growth of networks and interconnectivity
between publics and issues
Network mapping
![Page 38: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/38.jpg)
Network mapping - example
Before After
Conference participants – networks
![Page 39: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/39.jpg)
Network mapping - example
No content
Legend
Mostly out-of-date Mostly up-to-date
Size of square indicates number of visits Connecting lines indicate users have visited both directories
Thickness of connecting line indicate number of users that have visited both directories
Network map of directories of online portal combining web metrics (number of visits per directory), content analysis (data updated or not) and user survey data (visits to which
directories)
![Page 40: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/40.jpg)
Evaluating Communication
Programmes, Products and
Campaigns
4. Programme evaluation
methodology
Glenn O’Neil
![Page 41: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/41.jpg)
41
– This type of evaluation examines a series
of communication activities grouped under
a programme, for example:
By target audience: communication
programme aimed at young people
By function: online communication
programme
By theme: communication programme on
health policy
Programme evaluation
![Page 42: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/42.jpg)
42
– The evaluation steps are the same as for
campaign evaluation, but will normally be over
a longer period (2 – 6 months)
– More consultation and interim meetings with
the client are usually built into the evaluation
planning
Evaluation steps
Planning Creation Management Analysis Presentation
![Page 43: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/43.jpg)
43
Programmes are typically more difficult
to evaluate than campaigns because:
– They are often on rolling timeframes with
no clear end
– They often have unclear or very broad
objectives
– They often lack an institutional memory on
past activities and achievements
However, organisations increasingly need to
evaluate such programmes!
Programme evaluation
![Page 44: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/44.jpg)
44
– A similar methodology can be applied as
to campaigns
– Determining the programme’s objectives
and defining the evaluation framework are
key
Programme evaluation
![Page 45: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/45.jpg)
45
Programme evaluation can focus on three
distinct areas:
– Process: how has the programme been
managed?
– Outcomes: what has the programme
achieved?
– Impact: what has the programme contributed
to overall?
Programme evaluations can combine
elements of all three!
Programme evaluation
![Page 46: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/46.jpg)
46
Questions for completing the evaluation
framework:
– Are programme objectives documented?
– Does any baseline data exist?
– Has any programme monitoring being done?
– What is the balance between “outcome”,
“process” and “impact” evaluation questions?
Evaluation Framework
Indicators ToolsObjectives /Questions Source
![Page 47: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/47.jpg)
47
Standard
Surveys
Interviews
Panels
Focus groups
Case studies
Observation studies
Onsite visits
Evaluation methodsAdapted
Expert reviews
Content analysis
Media monitoring
Web metrics
Tracking mechanisms
Network mapping
![Page 48: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/48.jpg)
48
– In ongoing programmes operating across
multiple countries, onsite visits are often
included as an evaluation method
– Onsite visits involve a combination of
observation, interviews and discussions
– An evaluator can observe directly a
programme’s activities, discuss with its
implementers and gain in-depth knowledge
– Onsite visits add credibility to the evaluation
findings
Onsite visits
![Page 49: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/49.jpg)
Evaluating Communication
Programmes, Products and
Campaigns
5. Product evaluation methodology
Glenn O’Neil
![Page 50: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/50.jpg)
50
– Product evaluation is a more narrower
approach focusing on an individual item
(or series of items)
– This evaluation provides feedback on a
product’s use and its contribution to a
communication programme (or other type
of programme)
Product evaluation
![Page 51: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/51.jpg)
51
– Different types of products can be
considered including:
• Promotional videos
• Publications
• Websites and online tools
Product evaluation
![Page 52: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/52.jpg)
52
– Evaluation questions often include:
• Is the product considered to be of high quality in
terms of design, usability and content?
• Is the product targeted to the right audiences?
• Is the product available, accessible and
distributed to the intended target audiences?
Criteria for evaluation
![Page 53: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/53.jpg)
53
– Evaluation questions often include (cont.):
• Is the product used in the manner for which it
was intended - and for what other unintended
purposes?
• What has the product contributed to broader
communication and organizational goals?
• What lessons can be learnt for improving future
editions of the product and design, distribution and promotion in general?
Criteria for evaluation
![Page 54: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/54.jpg)
54
Surveys
Interviews
Focus groups
Case studies
Observation studies
Evaluation methods
Expert reviews
Content analysis
Web metrics
Tracking mechanisms
Distribution statistics
The evaluation methods have to be adapted to
the type of product and can include:
![Page 55: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/55.jpg)
55
Distribution statistics - example
Student / teacher
Fax
orders
5%
National
offices
30%
Promotional
distribution
40%
Web
orders
25%
Local partners
NGOsMedia
Student / teacher
Fax
orders
5%
National
offices
30%
Promotional
distribution
40%
Web
orders
25%
Local partners
NGOsMedia
![Page 56: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/56.jpg)
56
Mapping use - example
Training
Resource Working tool
Policy support
Used for training of
national partners
Used for staff training
Develop teaching
materialsCreate presentations for clients
Charts & tables used in
production
Used as guidelines
for product design
Used by NGOs to
influence debates on regulations
Used by authorities
to revise guidelines
Product
![Page 57: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/57.jpg)
57
Mini case study - exampleCapacity building for women in Uzbekistan, Central Asia
Nargiz, portal member, Uzbekistan
In Uzbekistan, Nargiz, a portal member is part of a group of 50 women who were preparing to run in parliamentary elections. For her, the portal has been a valuable source of support and information.
“In the e-discussions I got important feedback on fundraising strategies and financing of campaigns. This information will be used!”
Nargiz especially mentions an interesting experience from Mauritius, shared on the portal, which she can apply in her daily work at the Women’s NGO Forum of her country
“The material on capacity building is also very useful for us. I hope that in the future, we can share more of our own resources with the network.”
![Page 58: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/58.jpg)
58
Devising precise questions
• In all communications evaluations, if an evaluation
framework exists, it should be relatively easy moving from indicators to questions or criteria for collecting data
• But these questions and criteria must be created, documented and shared with the persons undertaking
the evaluation
• Questions and criteria would normally be documented in
templates and guides
![Page 59: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/59.jpg)
59
Portal facilitatesa global
exchange ofresources
Are resources
being exchanged?
Objectives Evaluation questions Indicators Precise questions
Devising precise questions
By whom, what type,
within what regions and at which frequency?
- Level of usage of resource areas of website
- Frequency and type of
resources exchanged
- Instances of
uses of resources
How often do you use the
resource section?
Have you contributed
resources?How have you used the
resources found on the
Portal?
- etc.
Extending the Evaluation Framework
We are here!
![Page 60: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/60.jpg)
Evaluating Communication
Programmes, Products and
Campaigns
6. Reporting on communications
evaluation
Glenn O’Neil
![Page 61: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/61.jpg)
61
– We have now jumped to the final phase of the
evaluation
– The presentation phase is often the most
neglected of all the phases
– Evaluation regularly fails in ensuring that
people know of the findings and take action
Reporting and presenting
Planning Creation Management Analysis Presentation
![Page 62: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/62.jpg)
62
Different presentation formats
– Written report
– Written summary (Word or PPT)
– Scorecard or findings table
– Video report
![Page 63: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/63.jpg)
63
Creating readable reports
A Good Evaluation Report is… A Weak Evaluation Report is…
• Impartial • Credible • Balanced • Clear and easy to understand • Information rich • Action oriented and crisp • Focused on evidence that supports conclusions
• Repetitious • too long • Unclear and unreadable • Insufficiently action oriented • Lacking hard data and relying on opinion • Poorly structured and lacking focus on key findings• Lacking comprehension of the local context • Negative or vague in its findings
![Page 64: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/64.jpg)
64
Summary sheet: an example
![Page 65: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/65.jpg)
65
Scorecard:an example
![Page 66: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/66.jpg)
66
Findings table:an example
Summary of the review’s key findings
Expected Results Rating
Outputs
Eight directories of the CR established and
accessible to potential users from the disaster
management community worldwide.
Largely achieved
Eight directories of the CR stocked with relevant,
appropriate and up-to-date information on disaster
management capacities.
Only partially achieved
Outcomes
Potential users from the disaster management
community worldwide learned of the CR.
Very limited achievement
Potential users from the disaster management
community worldwide visited the CR and registered
Very limited achievement
Users obtained information of use to them in one or
more of the eight directories of the CR.
Only partially achieved
Users contributed information from their
organisations to one or more of the eight directories
of the CR.
Very limited achievement
Information found on the CR facilitated the rapid
identification of appropriate disaster management
services.
Very limited achievement
Information found on the CR contributed to the rapid
delivery of humanitarian emergency assistance.
Not achieved
Impact
Delivery of humanitarian emergency assistance
improved. Not measured in this review
![Page 67: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/67.jpg)
67
Video report: an example
http://www.youtube.com/watch?v=q6nKXcUrNXA
![Page 68: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/68.jpg)
68
Follow-up mechanisms
Evaluations may require follow-up
mechanisms to ensure that the findings are disseminated and acted upon, including:
– Workshops with staff and donors to discuss findings
– Steering committees to discuss findings and implementation
– Plans of action based on findings and recommendations of the evaluation
![Page 69: Evaluating Communication Programmes, Products and Campaigns: Training workshop](https://reader033.vdocuments.site/reader033/viewer/2022052821/55492d0ab4c905b44c8c06ac/html5/thumbnails/69.jpg)
69
A parting quote
Scientific quality is not the principle standard; an
evaluation should aim to be comprehensible, correct and complete, and credible to partisans on all sides
Professor Lee Cronbach
www.owlre.com