measuring the effectiveness of an ... - science...
TRANSCRIPT
Kobe, Japan
February 2018
S h e l l i H e n d r i c k s , F i e l d i n g G r a d u a t e U n i v e r s i t y
K a t h e r i n e C o m e r S a n t o s , T h e S c i e n c e E x c h a n g e I n t e r n a t i o n a l S e a T u r t l e I n t e r n s h i p P r o g r a m
W a l l a c e J . N i c h o l s , S e n i o r F e l l o w , C e n t e r f o r t h e B l u e E c o n o m y , M i d d l e b u r y I n s t i t u t e o f I n t e r n a t i o n a l S t u d i e s , a n d R e s e a r c h A s s o c i a t e , C a l i f o r n i a A c a d e m y o f S c i e n c e s
E l i z a b e t h W h i t m a n , F l o r i d a I n t e r n a t i o n a l U n i v e r s i t y a n d T h e S c i e n c e E x c h a n g e
L i z e t t e G u z m a n - Z a r a g o z a , T h e S c i e n c e E x c h a n g e I n t e r n a t i o n a l S e a T u r t l e I n t e r n s h i p P r o g r a m
E l i z a b e t h K o c e k M a r q u e z , T h e S c i e n c e E x c h a n g e I n t e r n a t i o n a l S e a T u r t l e I n t e r n s h i p P r o g r a m
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
Measuring the Effectiveness of an International Sea Turtle Internship Program in Developing Future Leaders
Thank You, Sponsors
Background:Overview of The Science Exchange
Components:
▪Work with sea turtle experts in the field of applied conservation
▪Share the culture of the hosts, living simply outside of their home country
▪Complete their own STEM research project
▪Earn university credit
“Our mission is to train the next generation to become scientifically literate, international team players through our two-month STEM research
internships in Latin America and the Caribbean.”
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
Research Goals
Conservation attitude
Cultural/ Soft skills
Research skills
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
Interns often describe their experiences as “life-changing”
What is the most reliable, inexpensive way to measure learning effectiveness of this program?
Role of Water
Methods
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
1. Analyzed quantitative and qualitative short-term and long-term learning outcomes over 9 years and 51 interns
2. Compared 4 evaluation methods in a case study using a mixed-methods approach:
Short term evaluation methods:• Pre and post program self-assessment
surveys of skills*
• Intern Field Journals - coded*
• Supervisor evaluations*
Long-term methods:• Mix of 3-year “where are you now?”
surveys, social media or direct communication* *Data sources as of 15 Aug 2017.
Short-term Outcomes
Intern Monica in 2009 Akumal, Mexico
Intern Lizette in 2012 Tortuguero, Costa Rica
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
Cognitive Evaluations: Self-rated ∆ in Skills – pre & post
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Research Skills Soft Skills
Source = Pre- and Post-program Self-assessment Surveys
22 Student Self-ratings
Self
-rat
ed A
vg. ∆
in S
kills
(1
-10
)
n=22
Behavior Evaluation:Supervisor Observations – post
1
1.5
2
2.5
3
3.5
4
4.5
5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
average research average soft skills
Source = Post-program Supervisor Evaluation
Supervisors of 29 Students
Avg
. Ski
ll R
atin
gs b
y Su
pe
rvis
or
n=29
Impact Evaluation:What did I learn about myself?
Source = Open Text in Exit & Follow-up Surveys
n=12
1. “I learned to be more flexible”
2. “I learned how to persevere”
3. “I enjoy conducting field research”
4. “I can build relationships with anyone”
5. “I have already become a conservationist”
Impact Evaluation:How did your views of Conservation change?
Source = Open Text in Exit & Follow-up Surveys
n=12
1. “My commitment and passion have increased”
2. “I have become more aware of conservation issues”
3. “My eyes have been opened to other cultural views of conservation”
4. “I learned how efforts may vary by location”
5. “I pick up trash at the beach”
Impact Evaluation:Field Journal Analysis
n=1549 excerpts from 9 journals
Research Skills16%
Soft Skills45%
Water8%
Conservation20%
Emotion11%
Source = Participant Field Journals
Long-term Outcomes
Monica in 2016 Lizette in 2016
65%
Working in STEM
41%
Graduate School in STEM
Source: Direct communication, 3 year survey, or social media
n=39 n=43
Long-term Impact:Where are they now?(2008 – Aug 2017)
84%
Leader in STEM*
n=44
*Science, Technology, Engineering, Math (STEM)
85%
Active in Conservation
n=39
76%
International Experiences
n=29
Source: Direct communication, 3 year survey, or social media
Long-term Impact:Where are they now?(2008 – Aug 2017)
Case Study: Comparison of 4 MethodsResearch Skills
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6 7 8 9
super post research skills/5 Self post research skills/10 Journal Research Skills/skills
= career in STEM
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
100.0%
1 2 3 4 5 6 7 8 9
Super post soft skills/5 Self post soft skills/10 Journal Soft Skills/skills
International Conservation
= either active in conservation, international experiences, or both
Case Study: Comparison of 4 MethodsSoft Skills
DiscussionLearning
ObjectivesRecommendations Timing
Sho
rt-T
erm
Cognitive• Knowledge assessment OR• Continue self-assessments of change in knowledge
• Pre & Post-program
Behaviorincl. Leadership
• Continue Supervisor evaluations• Online leadership assessment, incl. feedback from
others• Online cultural assessments
• Post-program
Impactincl. Conservation
• Open text via journals or post surveys • Post-program
Lon
g-Te
rm ImpactCareer choices
volunteer activities
• Social media search or Qualified 3-party interviews
• Annually
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
ReferencesBogart, K.S. et al (2016). Sustainability leadership: Integrating values, meaning and action. Fielding Monograph
Series: 5(1): 7-307.
Bowers, C. (2016). A critical examination of STEM: Issues and challenges. New York (NY): Routledge.
Carpi, A. Ronan, D.M., Falconer, H.M., Lents, N.H. (2017). Cultivating minority scientists: Undergraduate research increases self-efficacy and career ambitions for underrepresented students in STEM. Journal of Research in Science Teaching 54(2):169–194.
Clarke, I., Flaherty, T.B., Wright, N.D., McMillen, R. M. (2009). Student intercultural proficiency from study abroad programs. Journal of Marketing Education. 31(2):173 – 181. First published date: May-01-2009 10.1177/0273475309335583
Frazier, J. G. (n.d.). The turtles' tale: Flagships and instruments for marine research, education, and conservation. Smithsonian Contribution to the Marine Sciences 38
Gilmore, J., Vieyra, M., Timmerman, B., Feldon, D. & Maher, M. (2015). The Relationship between Undergraduate Research Participation and Subsequent Research Performance of Early Career STEM Graduate Students. The Journal of Higher Education 86(6):834-863.
Kirkpatrick, D. (1995). Techniques for evaluating training programs. Journal of the American Society for Training and Development, 13, 3-9.
Nichols, W.J. (2014). Blue mind: The surprising science that shows how being near, in, on, or under water can make you happier, healthier, more connected and better at what you do. New York, NY: Little, Brown & Company.
Toomey, A.H. and Domroese, M.C. (2013). Can citizen science lead to positive conservation attitudes and behaviors? Human Ecology Review 20(1):50-62.
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
Thank you!! Gracias!Doumo Arigatouどうもありがとう
Education | Outreach & Advocacy2018 International Sea Turtle Symposium
Contact:[email protected]
andhttp://thescienceexchange.org/