measuring the effectiveness of an ... - science...

Kobe, Japan

February 2018

S h e l l i H e n d r i c k s , F i e l d i n g G r a d u a t e U n i v e r s i t y

K a t h e r i n e C o m e r S a n t o s , T h e S c i e n c e E x c h a n g e I n t e r n a t i o n a l S e a T u r t l e I n t e r n s h i p P r o g r a m

W a l l a c e J . N i c h o l s , S e n i o r F e l l o w , C e n t e r f o r t h e B l u e E c o n o m y , M i d d l e b u r y I n s t i t u t e o f I n t e r n a t i o n a l S t u d i e s , a n d R e s e a r c h A s s o c i a t e , C a l i f o r n i a A c a d e m y o f S c i e n c e s

E l i z a b e t h W h i t m a n , F l o r i d a I n t e r n a t i o n a l U n i v e r s i t y a n d T h e S c i e n c e E x c h a n g e

L i z e t t e G u z m a n - Z a r a g o z a , T h e S c i e n c e E x c h a n g e I n t e r n a t i o n a l S e a T u r t l e I n t e r n s h i p P r o g r a m

E l i z a b e t h K o c e k M a r q u e z , T h e S c i e n c e E x c h a n g e I n t e r n a t i o n a l S e a T u r t l e I n t e r n s h i p P r o g r a m

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

Measuring the Effectiveness of an International Sea Turtle Internship Program in Developing Future Leaders

Thank You, Sponsors

Background:Overview of The Science Exchange

Components:

▪Work with sea turtle experts in the field of applied conservation

▪Share the culture of the hosts, living simply outside of their home country

▪Complete their own STEM research project

▪Earn university credit

“Our mission is to train the next generation to become scientifically literate, international team players through our two-month STEM research

internships in Latin America and the Caribbean.”

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

Research Goals

Conservation attitude

Cultural/ Soft skills

Research skills

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

Interns often describe their experiences as “life-changing”

What is the most reliable, inexpensive way to measure learning effectiveness of this program?

Role of Water

Methods

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

1. Analyzed quantitative and qualitative short-term and long-term learning outcomes over 9 years and 51 interns

2. Compared 4 evaluation methods in a case study using a mixed-methods approach:

Short term evaluation methods:• Pre and post program self-assessment

surveys of skills*

• Intern Field Journals - coded*

• Supervisor evaluations*

Long-term methods:• Mix of 3-year “where are you now?”

surveys, social media or direct communication* *Data sources as of 15 Aug 2017.

Short-term Outcomes

Intern Monica in 2009 Akumal, Mexico

Intern Lizette in 2012 Tortuguero, Costa Rica

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

Cognitive Evaluations: Self-rated ∆ in Skills – pre & post

-1.5

-0.5

0.5

1.5

2.5

3.5

4.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Research Skills Soft Skills

Source = Pre- and Post-program Self-assessment Surveys

22 Student Self-ratings

Self

-rat

ed A

vg. ∆

in S

kills

(1

-10

)

n=22

Behavior Evaluation:Supervisor Observations – post

1

1.5

2

2.5

3

3.5

4

4.5

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

average research average soft skills

Source = Post-program Supervisor Evaluation

Supervisors of 29 Students

Avg

. Ski

ll R

atin

gs b

y Su

pe

rvis

or

n=29

Impact Evaluation:What did I learn about myself?

Source = Open Text in Exit & Follow-up Surveys

n=12

1. “I learned to be more flexible”

2. “I learned how to persevere”

3. “I enjoy conducting field research”

4. “I can build relationships with anyone”

5. “I have already become a conservationist”

Impact Evaluation:How did your views of Conservation change?

Source = Open Text in Exit & Follow-up Surveys

n=12

1. “My commitment and passion have increased”

2. “I have become more aware of conservation issues”

3. “My eyes have been opened to other cultural views of conservation”

4. “I learned how efforts may vary by location”

5. “I pick up trash at the beach”

Impact Evaluation:Field Journal Analysis

n=1549 excerpts from 9 journals

Research Skills16%

Soft Skills45%

Water8%

Conservation20%

Emotion11%

Source = Participant Field Journals

Long-term Outcomes

Monica in 2016 Lizette in 2016

65%

Working in STEM

41%

Graduate School in STEM

Source: Direct communication, 3 year survey, or social media

n=39 n=43

Long-term Impact:Where are they now?(2008 – Aug 2017)

84%

Leader in STEM*

n=44

*Science, Technology, Engineering, Math (STEM)

85%

Active in Conservation

n=39

76%

International Experiences

n=29

Source: Direct communication, 3 year survey, or social media

Long-term Impact:Where are they now?(2008 – Aug 2017)

Case Study: Comparison of 4 MethodsResearch Skills

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9

super post research skills/5 Self post research skills/10 Journal Research Skills/skills

= career in STEM

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

1 2 3 4 5 6 7 8 9

Super post soft skills/5 Self post soft skills/10 Journal Soft Skills/skills

International Conservation

= either active in conservation, international experiences, or both

Case Study: Comparison of 4 MethodsSoft Skills

DiscussionLearning

ObjectivesRecommendations Timing

Sho

rt-T

erm

Cognitive• Knowledge assessment OR• Continue self-assessments of change in knowledge

• Pre & Post-program

Behaviorincl. Leadership

• Continue Supervisor evaluations• Online leadership assessment, incl. feedback from

others• Online cultural assessments

• Post-program

Impactincl. Conservation

• Open text via journals or post surveys • Post-program

Lon

g-Te

rm ImpactCareer choices

volunteer activities

• Social media search or Qualified 3-party interviews

• Annually

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

ReferencesBogart, K.S. et al (2016). Sustainability leadership: Integrating values, meaning and action. Fielding Monograph

Series: 5(1): 7-307.

Bowers, C. (2016). A critical examination of STEM: Issues and challenges. New York (NY): Routledge.

Carpi, A. Ronan, D.M., Falconer, H.M., Lents, N.H. (2017). Cultivating minority scientists: Undergraduate research increases self-efficacy and career ambitions for underrepresented students in STEM. Journal of Research in Science Teaching 54(2):169–194.

Clarke, I., Flaherty, T.B., Wright, N.D., McMillen, R. M. (2009). Student intercultural proficiency from study abroad programs. Journal of Marketing Education. 31(2):173 – 181. First published date: May-01-2009 10.1177/0273475309335583

Frazier, J. G. (n.d.). The turtles' tale: Flagships and instruments for marine research, education, and conservation. Smithsonian Contribution to the Marine Sciences 38

Gilmore, J., Vieyra, M., Timmerman, B., Feldon, D. & Maher, M. (2015). The Relationship between Undergraduate Research Participation and Subsequent Research Performance of Early Career STEM Graduate Students. The Journal of Higher Education 86(6):834-863.

Kirkpatrick, D. (1995). Techniques for evaluating training programs. Journal of the American Society for Training and Development, 13, 3-9.

Nichols, W.J. (2014). Blue mind: The surprising science that shows how being near, in, on, or under water can make you happier, healthier, more connected and better at what you do. New York, NY: Little, Brown & Company.

Toomey, A.H. and Domroese, M.C. (2013). Can citizen science lead to positive conservation attitudes and behaviors? Human Ecology Review 20(1):50-62.

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

Thank you!! Gracias!Doumo Arigatouどうもありがとう

Education | Outreach & Advocacy2018 International Sea Turtle Symposium

Contact:shendricks@email.fielding.edu

andhttp://thescienceexchange.org/