lunch & learns for evaluators in ottawa - … · lunch & learns for evaluators in ottawa:...
TRANSCRIPT
Lunch & Learns for Evaluators in Ottawa:
A promising model for professional development
and networking
CES Conference 2009 June 3
Anna EngmanNatalya KuziakKate Powadiuk
Ken StephensonLynda Weaver(Jane Whynot)
Discussant: Steve Montague/Charles Lufthaus1
Presentation Outline
• Ottawa L&L: History, Structure & Past Sessions
• Why Do the L&Ls Work: Theory of Learning
• Ottawa L&L: Survey Findings & Lessons Learned
• Toronto L&L: History, Structure, Past Sessions and Lessons Learned
• Considerations When Starting Your Own L&L
2
Ottawa Lunch and Learns
• Initiation: three evaluators met at 2007 CES conference
• Motivation: free, informal professional development and networking
• Resources: in-kind contributions (leading, hosting and participating)
• Growth: currently 100 participants, website
History and Structure
4
Past Sessions
• Reflective practice and evaluation
• Managing evaluation contracts
• Evaluability assessment• Vested interests in
evaluation• Attribution vs. contribution• Planning for gathering
performance results evidence for improved measurement, monitoring, evaluation and reporting of achievements
• Cultural competence in evaluation
• Feminist evaluation• The case study approach• Systematic lit reviews• Theory driven evaluation
and theories of change• Efficiency analysis• Appreciative inquiry in
evaluation I & II• Evaluation mistakes• Evaluation of networks and
partnerships
5
• Nutrition definitely improves learning!– Studies on children
• Eating with colleagues probably improves nutrition!– I’m not going to eat fries in front of you!
• We have different learning styles– E.g., visual, auditory, experiential, mixed
• We come with baggage!– So do the ‘teachers’
• Our attention span is about 11 minutes!
Theory for Continuing Education
6
• We decide what is important, what we will do with the info– Relevant, immediately applicable
• We learn best when info is for solving a problem
• We validate info against our beliefs, experiences– We may disagree
• We expect a collaborative learning environment
• We learn best when we participate actively
How Adults Learn
7Meloche, Hélène, www.hmeloche.com
Didactic lectures have little effect on changing behaviour
– Not to mention boring and not nutritious
Interactive, challenging, sequenced
– Enhance activity
– Opportunities to practice skills
– Learn…Work…Learn
Proof, Meta-Analysis
8
Davis, D., O’Brien, M. A. T., Freemantle, N., Wolf, F. M., Mazmarin, P. & Taylor-Vaisey, A. (1999). Impact of formal and continuing medical education. JAMA, (282(9): Internet document retrieved November 3, 1999 from http://jama.ama-assn.org/issues/v282n9/full/jrv90027.html
• Non-threatening, non-judgmental
• Community of Practice develops
…groups of people who share a passion for something that they know how to do and who interact regularly to learn how to do it better.
Peer Learning
9Wenger, 2002
1. Tell stories2. Play games3. Explore and experiment4. Use pictures5. Have a coach6. Learn with others7. Focus on what is important8. Take recess9. Have lunch10. Be passionate11. Keep learning
Eleven Common Sense Learning Principles
10Forman, D. C. (2003). Eleven common-sense learning principles. Training and Development , September, pp. 39-44
☻Lunch
☻Buddies
☻Active
Therefore..L&L best practices are:
11
• Online survey sent to 97 L&L participants in April 2009
• 54% response rate
• Of those who didn’t respond:– 42% had never attended a session
– 4% were too busy
– 53% unknown
Survey Methods
12
• Broad range of age and experience
• Occupation:
– 94% working; 24% studying; 20% doing both
• Majority of respondents are:
– Highly educated (70% Masters level; 14% PhD)
– Senior level at work (66%)
– In federal public sector (67%)
– CES members (85%)
– Female (76%)
Who are the L&L Participants?
13
Gender
* “2005 national survey” refers to: Borys, S., Gauthier, B., Kishchuk, N., Roy, S.N. (2005) Survey of evaluation practice and issues in Canada. Paper presented at the Joint CES/AEA Conference, Toronto, October 26, 2005. 14
0%
10%
20%
30%
40%
50%
60%
70%
80%
Male Female
Lunch & Learn survey (n=46) 2005 national survey (n=634)
Education
15
0%
10%
20%
30%
40%
50%
60%
70%
80%
Other Bachelors Masters or
post-grad
PhD
Highest Degree Completed
Lunch & Learn survey (n=49) 2005 national survey (n=638)
Employment Sector
16
0%
10%
20%
30%
40%
50%
60%
70%
Federa
l publ
ic
Not fo
r pro
fit
Private
Provinc
ial p
ublic
Mun
icipal p
ublic
Reg
iona
l pub
lic
College
/ uni
vers
ity
Lunch & Learn survey (n=48) 2005 national survey (n=647)
Experience (n=50)
17
Job Position (n=49)
“The ability to actually attend the sessions is the biggest obstacle for our
team at the moment... if messaging comes from the director level across...
that personnel should attend, and at least one staff member can be
released to attend on the team's behalf, we're all the better for it.”
* Officer / consultant
What is Appealing About the L&L?Interest in Topics
• 94% agreed that the topics are of interest to them
• 92% agreed that the topics are relevant to their work/studies
• Most well attended L&L sessions:– Evaluability Assessment
– Vested Interests in Evaluation
– Conducting Case Studies
– Cultural Competency in Conducting Evaluations
– Cost-efficiency Analysis
– Feminist Evaluation
– Appreciative Inquiry
19
What is Appealing about the L&L?Interest in Topics
“I always circulate the topics in advance to
members of my evaluation team, and encourage
representation if I think a topic is particularly
relevant for our work here. Anyone who attends
is then asked to present back to our whole
group, so we can all benefit from the learning.
We really appreciate this!”
20
What is Appealing about the L&L? Value for Participants
• 55% have benefited from the opportunity to network with other evaluators
– 22% were neutral, 10% disagreed
• 51% have applied the techniques or knowledge they gained from a L&L in their work or studies
– 18% were neutral, 16% disagreed
21
What is Appealing About the L&Ls?Applying What They Learned
22
L&L Topic How participants applied the knowledge or technique they learned
Evaluability Assessment, Case Studies, Cost-Effectiveness, Program Theory
Used to design or guide evaluation studies; used the methodology
Vested Interests in Evaluation
Dealing with requests to modify evaluation findings; addressing ethical issues; dealing with influence of difficult clients
Reference Materials Used the resources and reference materials; referred colleagues to them
Appreciative Inquiry Used in focus groups; incorporated into evaluation and planning practices
Systematic Lit Reviews Used the methods suggested, e.g., how to write literature search questions
What Other Value or Benefits Do You Get from Attending L&Ls? (n=20)
23
35% Networking professionally and socially; Staying connected; Being part of a community
35% The opportunity to share information with other evaluators, Ask them questions and learn from their experience
15% Learning how other evaluation firms or government departments conduct evaluations; Hearing about what is going on in evaluation in other shops
10% The opportunity to learn new strategies, tools, techniques and approaches in evaluation
10% It’s fun and motivational
10% Learning about good articles, publications and studies related to evaluations.
“A good way to hear what's going on in the field of evaluation. Good networking possibilities. Good to
feel part of a community.”
“By speaking with evaluators who completed similar projects, I avoided making mistakes and did not
have to reinvent the wheel.”
Other Value or Benefits of L&Ls
24
• The majority of participants (80-82%) are satisfied with the readings, the presentations and the format
• Slightly lower level of satisfaction (68%) with the locations of the L&Ls
• Factors that kept people from attending: too busy (86%), date not convenient (31%), locations not convenient (29%), time not convenient (20%)
• Only 31% of participants have led or co-led a L&L
What’s Working Well & What Could Be Better
25
• For the most part, continue!
• Challenge: Travel and time required
– Occasional evening sessions?
– Target evaluation managers and directors?
– CES accreditation?
• Challenge: Need more session leads
– Mentoring or teaming up to lead sessions?
– Session on facilitation?
– Promote more participatory formats?
– Promote the L&Ls as a spin-off opportunity?
Where do Ottawa L&Ls go from here?
26
• Challenge: Communications & awareness
– Made the Google group website public!
– Funding for a website?
– Collaborate with CES regarding promotions to NCC’s members?
• Challenge: Increasing interest in sessions
– Regular polls on what topics interest participants?
– Encourage participants to bring in experts occasionally?
• Challenge: Continuous improvement
– Evaluation forms after each session?
– Guest book to keep track of attendance?
Where do Ottawa L&Ls go from here?
27
The Toronto Lunch & Learns
28
• Started in 2006, ended in 2008
• Brought together a variety of individuals
• Originally held at Cathexis, then shift to different locations
• Variety of topics including:
– Building excitement about using data
– Capacity building
– Using existing data sets
– Innovative data collection methods
History of the Toronto L&Ls
29
What happened…
30
What worked well:
• Excitement about learning
• Diversity of experience and opinion
• Variety of presentation methods
• Collaborative planning
Challenges:
• Travel time
• Lack of varied attendees
• Loss of interest over time
• Competing deadlines
• CES Ontario conference – discussion session planned– Different format?
– CES involvement in organizing?
• Continuing to find new ways to promote professional development at Cathexis internally
What’s next for Toronto L&L??
31
• Where to find participants?
– Personal network, PD events, CES local chapter, university programs, managers, chain e-mails
• Where to find a facility?
– Multiple facilities (participants’ workplaces, community centers, libraries)
• How to increase likelihood of sustainability?
– A coordinator, a website, foster participant ownership, recognize contributions, participatory format, encourage spin-offs, make it a priority
Considerations when Starting Your Own L&L
32
• Where do you think the L&Ls in Ottawa & Toronto should go from here? Which suggestions do you agree with?
• How does the L&L as a learning model fit into your professional development?
Questions for Discussion
33
• Ottawa L&L:
– [email protected] http://groups.google.com/group/lunch-and-learn-for-evaluators-in-ottawa
• Toronto L&L:
• Victoria L&L (starting up):
For Additional Information
34