Assessing Impact: Study Assessing Impact: Study Abroad at Two InstitutionsAbroad at Two Institutions
Elaine Meyer-LeeElaine Meyer-Lee Saint Mary’s CollegeSaint Mary’s College
Carol WarfieldCarol WarfieldAuburn University Auburn University
2006 AIEA Annual Conference2006 AIEA Annual Conference February, 2006February, 2006
Why Assess Outcomes?Why Assess Outcomes?
1,022 US institutions reported information 1,022 US institutions reported information on their study abroad programson their study abroad programs
174,629 U.S. students studied abroad174,629 U.S. students studied abroad
Data from Open Doors Report on International Educational Exchange 2003-2004Data from Open Doors Report on International Educational Exchange 2003-2004
Inform continuous quality Inform continuous quality improvementimprovement
Advocate for international educationAdvocate for international education
Satisfy regional accrediting bodiesSatisfy regional accrediting bodies
Add to basic understanding of Add to basic understanding of student growth in this areastudent growth in this area
Young, wide-open fieldYoung, wide-open field
Much need for better methods and more Much need for better methods and more data on many different dimensionsdata on many different dimensions
This presentation is not a step-by-step This presentation is not a step-by-step overview, but two examples overview, but two examples
(See Forum’s upcoming Guide to (See Forum’s upcoming Guide to Outcomes Assessment in Education Outcomes Assessment in Education Abroad)Abroad)
Multi-method Longitudinal Assessment of Multi-method Longitudinal Assessment of Intercultural Engagement, IC Sensitivity, Intercultural Engagement, IC Sensitivity, Identity, and Goals: Preliminary ResultsIdentity, and Goals: Preliminary Results
Elaine Meyer-LeeElaine Meyer-Lee Director of International & Intercultural LearningDirector of International & Intercultural Learning
Joy EvansJoy EvansResearch Analyst-Center for Women’s InterCultural LeadershipResearch Analyst-Center for Women’s InterCultural Leadership
Saint Mary’s CollegeSaint Mary’s CollegeNotre Dame, Indiana Notre Dame, Indiana
206 AIEA Annual Conference206 AIEA Annual Conference February 25, 2006February 25, 2006
Presentation OverviewPresentation Overview
ContextContext
Assessment Methods Assessment Methods
Review of Instruments Review of Instruments
Preliminary ResultsPreliminary Results
Contributions to Study Abroad AssessmentContributions to Study Abroad Assessment
Saint Mary’s CollegeSaint Mary’s College
Catholic Women’s Liberal ArtCatholic Women’s Liberal Art Established 1844Established 1844
About 1600 enrollmentAbout 1600 enrollment• 98% full-time98% full-time• 82% on-campus82% on-campus• 85% Catholic85% Catholic• 9% students of color9% students of color
20% of students study abroad20% of students study abroad
Mission of the Mission of the Center for Women’s InterCultural LeadershipCenter for Women’s InterCultural Leadership
Saint Mary's College's mission includesSaint Mary's College's mission includespreparing women to make a differencepreparing women to make a difference
in an increasingly complex andin an increasingly complex andinterdependent world. The role of theinterdependent world. The role of the
Center for Women's InterCulturalCenter for Women's InterCulturalLeadership (CWIL) within this missionLeadership (CWIL) within this mission
is to foster the intercultural knowledgeis to foster the intercultural knowledgeand competence critical to educatingand competence critical to educating
the next generation of women leaders.the next generation of women leaders.
CWIL at Saint Mary’sCWIL at Saint Mary’s
Encompasses much more than study Encompasses much more than study abroad (see brochures)abroad (see brochures)
Is adopting Byram’s definition of Is adopting Byram’s definition of intercultural competenceintercultural competence
Is launching a portfolio-based Is launching a portfolio-based certificate program in Women’s certificate program in Women’s Intercultural LeadershipIntercultural Leadership
Study Abroad ProgramsStudy Abroad Programs
Year/ Semester• Ireland (Year only)• India (Semester only)• Rome• Spain• France• Australia• Austria• South Africa
4-2 Weeks• European Studies (4)• Internships in Europe (4+)• Honduras (3)• Ecuador (3-2) • Greece (2 ½) • S. Korea (2)
1 Week• Prague/Poland• Jamaica• Mexico
Study Abroad ProgramsStudy Abroad Programs
Assessing Study Abroad… Assessing Study Abroad… Going Beyond Going Beyond
Student SatisfactionStudent Satisfaction
Evaluation & Assessment GoalsEvaluation & Assessment Goals
Evaluate programs’ effectivenessEvaluate programs’ effectiveness Measure Students’:Measure Students’:
• Changes in sensitivity to cultural differences Changes in sensitivity to cultural differences • American identity developmentAmerican identity development• Own sense of growth toward goalsOwn sense of growth toward goals
Explore correlations of change with:Explore correlations of change with:• Intercultural engagement while abroad Intercultural engagement while abroad
(interaction and reflection)(interaction and reflection)• Program or demographic characteristicsProgram or demographic characteristics
Assessment Methods/DesignAssessment Methods/Design
Longitudinal Research DesignLongitudinal Research Design• Pre-test/Post-test (some grad follow-up)Pre-test/Post-test (some grad follow-up)• Procedures – 70% response rate so farProcedures – 70% response rate so far
Control Group Control Group • 30 First-year students30 First-year students
Focus GroupFocus Group• Study Abroad Re-entry CourseStudy Abroad Re-entry Course• Written reflection and Kolb’s LSI Written reflection and Kolb’s LSI
Assessment Instruments: Pre-TestAssessment Instruments: Pre-Test
Study Abroad ParticipantsStudy Abroad Participants
Intercultural Development Intercultural Development Inventory (IDI)Inventory (IDI)
American Identity American Identity QuestionnaireQuestionnaire
Goals/motivations from Goals/motivations from Saint Mary’s College Study Saint Mary’s College Study Abroad SurveyAbroad Survey
Control GroupControl Group
Intercultural Intercultural Development Inventory Development Inventory (IDI)(IDI)
American Identity American Identity QuestionnaireQuestionnaire
Assessment Instruments: Post-TestAssessment Instruments: Post-Test
Study Abroad ParticipantsStudy Abroad Participants
Intercultural Development Intercultural Development Inventory (IDI)Inventory (IDI)
American Identity American Identity Questionnaire (AIQ)Questionnaire (AIQ)
SMC Study Abroad Survey SMC Study Abroad Survey Program Impact & Program Impact & Measure of Intercultural Measure of Intercultural Engagement (MIE)Engagement (MIE)
Control GroupControl Group
Intercultural Intercultural Development Development Inventory (IDI)Inventory (IDI)
American Identity American Identity QuestionnaireQuestionnaire
Data Collection TimelineData Collection Timeline2004-20052004-2005 2005-20062005-2006 2006-20072006-2007
Fall Fall SpringSpring FallFall SpringSpring FallFall SpringSpring
Pre-testPre-test
N=16----N=16----
Pre-testPre-test
N=44----N=44----
--------------------
--------------------
Pre-testPre-test
N=83----N=83----
Control Control
N=32----N=32----
Post-testPost-test
N=(16)N=(16)
Post-testPost-test
N=(44)N=(44)
Post-testPost-test
N=(6)---------N=(6)---------
--------------------------------
Pre-testPre-test
N=(?)---------N=(?)---------Spring SemesterSpring Semester
Post-testPost-test
N=(48?)--N=(48?)--Fall Fall SemesterSemester
----------------------
----------------------
Pre-test?Pre-test?
N=(?)----N=(?)----SummerSummer
Post-testPost-test
N=(29?)N=(29?)
YearYear
ControlControl
N=(32)N=(32)
Post-testPost-test
N=(?)N=(?)
Post-test?Post-test?
N=(?)N=(?)
Review of InstrumentsReview of Instruments
Intercultural Development Inventory Intercultural Development Inventory (IDI)(IDI)
50-item Self-Assessment50-item Self-Assessment
Valid & Reliable Psychometric Measure Valid & Reliable Psychometric Measure
Culture general in focusCulture general in focus
Quantifies the subjective experience of cultural differenceQuantifies the subjective experience of cultural difference
Most widely used currentlyMost widely used currently
Based on Milton Bennett’s Developmental Model of Based on Milton Bennett’s Developmental Model of Intercultural Sensitivity (constructivist stage model)Intercultural Sensitivity (constructivist stage model)
Slide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Web-pageSlide Available Online:http://www.coe.int/T/E/North-South_Centre/Programmes/3_Global_Education/c_GENE/GENE%20Web-page%20-%20(3)%20Hesse%20article.asp%20-%20(3)%20Hesse%20article.asp
IDI Measurement ScalesIDI Measurement Scales
Denial/DefenseDenial/Defense
ReversalReversal
MinimizationMinimization
Acceptance/AdaptationAcceptance/Adaptation
Encapsulated MarginalityEncapsulated Marginality
Overall ProfileOverall Profile
American Identity QuestionnaireAmerican Identity Questionnaire 10-item Likert-style Self Assessment10-item Likert-style Self Assessment
Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record)track record)
Based on Eriksonian identity development theory. Measures 2 Factors: Based on Eriksonian identity development theory. Measures 2 Factors:
• identity search (e.g. I have often talked to other people about what it means to identity search (e.g. I have often talked to other people about what it means to be an American.)be an American.)
• affirmation, belonging, and commitment (e.g. affirmation, belonging, and commitment (e.g. Being American plays an Being American plays an important part in my life.)important part in my life.)
One goal of study abroad is more complexity in understanding own culture, One goal of study abroad is more complexity in understanding own culture, and students in pilot focus group felt it captured an important dimensionand students in pilot focus group felt it captured an important dimension
Currently refining adaptation’s psychometric propertiesCurrently refining adaptation’s psychometric properties
Saint Mary’s College Saint Mary’s College Study Abroad SurveyStudy Abroad Survey
Semester/Year-Long Semester/Year-Long ProgramsPrograms
Core SurveyCore Survey• 111 Items111 Items• 4 open-ended 4 open-ended
questions questions
Customized AddendumCustomized Addendum
Summer/Short-term Summer/Short-term ProgramsPrograms
Core SurveyCore Survey• 58 Items58 Items• 1 open-ended 1 open-ended
questions questions
Customized AddendumCustomized Addendum
Saint Mary’s College Saint Mary’s College Study Abroad SurveyStudy Abroad Survey
Semester/Year-Long ProgramsSemester/Year-Long Programs
Question Sections Include:Question Sections Include:• Demographics/BackgroundDemographics/Background• GoalsGoals• Pre-DeparturePre-Departure• Academic ProgramAcademic Program• Cultural Tours & ExcursionsCultural Tours & Excursions• On-Site Student ServicesOn-Site Student Services• Intercultural EngagementIntercultural Engagement• Overview of ProgramOverview of Program• Program Impact Program Impact (Goals)(Goals)• Volunteer InterestsVolunteer Interests
Summer/Short-term Summer/Short-term ProgramsPrograms
Question Sections Question Sections Include:Include:• DemographicsDemographics• Pre-DeparturePre-Departure• Living ArrangementsLiving Arrangements• Intercultural Intercultural
EngagementEngagement• Overview of ProgramOverview of Program• Volunteer InterestsVolunteer Interests
Self-reported Growth in:Self-reported Growth in: Understanding of different culturesUnderstanding of different cultures
Different perspective on American cultureDifferent perspective on American culture
Greater understanding of the US in world affairs and historyGreater understanding of the US in world affairs and history
Different perspective on your ethnic heritageDifferent perspective on your ethnic heritage
Understanding of my faithUnderstanding of my faith
Self-awarenessSelf-awareness
Maturity and self-confidenceMaturity and self-confidence
Interpersonal skills and the ability to adaptInterpersonal skills and the ability to adapt
Intellectual development in the liberal artsIntellectual development in the liberal arts
Ability to communicate in a foreign language Ability to communicate in a foreign language
Measure of Intercultural Measure of Intercultural EngagementEngagement
Asks students to quantify the Asks students to quantify the frequency of their engagement frequency of their engagement through:through:
Interaction with cultural resourcesInteraction with cultural resources Interaction with local peopleInteraction with local people Explicit reflection on these Explicit reflection on these
interactionsinteractions
Interaction with cultural resourcesInteraction with cultural resources
Theater/opera/symphony/concerts Theater/opera/symphony/concerts
Museums/ historical sitesMuseums/ historical sites
Forests/parks/beaches Forests/parks/beaches
Traveling outside of the host country Traveling outside of the host country
Traveling within the host country Traveling within the host country
Interaction with local peopleInteraction with local people Shopping/visiting local marketsShopping/visiting local markets Movies/entertainment catering to localsMovies/entertainment catering to locals Restaurants/pubs/cafes catering to localsRestaurants/pubs/cafes catering to locals Visiting local families/friends in their homesVisiting local families/friends in their homes Participating in worship servicesParticipating in worship services On-campus informal conversationsOn-campus informal conversations Events sponsored by the study abroad programEvents sponsored by the study abroad program Extra-curricular activities (student clubs, Extra-curricular activities (student clubs,
intramurals, etc.) intramurals, etc.) Volunteering, service-learning, internships, Volunteering, service-learning, internships,
shadowingshadowing Reading or listening to local news Reading or listening to local news
Explicit reflection on these Explicit reflection on these interactionsinteractions
Journaling/BloggingJournaling/Blogging
Photography Photography
Other artistic expression (painting, drawing, collages, etc.) Other artistic expression (painting, drawing, collages, etc.)
Class discussionsClass discussions
Informal conversations with localsInformal conversations with locals
Informal conversations with students studying abroadInformal conversations with students studying abroad
Calling/e-mailing family/friends/faculty in America about Calling/e-mailing family/friends/faculty in America about experience experience
Pre-test ResultsPre-test Results IDI Worldview Profile: Pre-AbroadIDI Worldview Profile: Pre-Abroad
IDI Worldview Profile “Pre”: IDI Worldview Profile “Pre”: ControlsControls
Pre-test Comparison to Control: Pre-test Comparison to Control: American Identity Questionnaire American Identity Questionnaire
Affirmation, belonging and Affirmation, belonging and commitment commitment
no significant differenceno significant difference
Identity searchIdentity search
statistical significance statistical significance (at the .05 level) (at the .05 level)
Summary of Pre-test FindingsSummary of Pre-test Findings
IDI: Students’ scores do not support IDI: Students’ scores do not support developmental modeldevelopmental model
IDI: Low Minimization Score IDI: Low Minimization Score • Catholic beliefs ….We’re all God’s childrenCatholic beliefs ….We’re all God’s children
Control on both: Heritage Seeker FactorControl on both: Heritage Seeker Factor• Are study abroad participants a different Are study abroad participants a different
population than other SMC students?population than other SMC students?
Preliminary Analysis of Change Preliminary Analysis of Change Data for Matched SetData for Matched Set
No significant change in means for No significant change in means for IDI or AIQ yet (on paired T-test on IDI or AIQ yet (on paired T-test on pre-post mean scores, n=42 and 27)pre-post mean scores, n=42 and 27)
BUT, high self-reported growth (1.7 BUT, high self-reported growth (1.7 when 1 is high) and wide range in when 1 is high) and wide range in change scores and program meanschange scores and program means
What about those who do change?What about those who do change?
Correlation of AIQ with IDI scales (bivariate Correlation of AIQ with IDI scales (bivariate non-parametric correlations)non-parametric correlations)
Correlations with Intercultural Correlations with Intercultural Engagement:Engagement:• Cultural resource scale with AIQ changeCultural resource scale with AIQ change• Overall and reflection with program impact Overall and reflection with program impact
(especially interpersonal items)(especially interpersonal items) 50% did grow on at least one scale 50% did grow on at least one scale
(defined as 2 standard deviations of (defined as 2 standard deviations of positive change, n=21)positive change, n=21)
How do the “growers” compare?How do the “growers” compare?
CharacteristicCharacteristic GrowersGrowers Non-GrowersNon-GrowersFrom N.America From N.America 91%91% 9595%%WhiteWhite 67%67% 8686%%LatinaLatina 2424%% 4.5%4.5%Never been abroadNever been abroad 53%53% 63%63%Abroad < 3 monthsAbroad < 3 months 24%24% 26%26%Whole year programWhole year program 10%10% 46%46%Semester programSemester program 76%76% 46%46%Summer programSummer program 14%14% 8%8%Rome programRome program 24%24% 3232%%Seville programSeville program 2424%% 5%5%MexicoMexico 1414%% 9%9%France France 1414%% 9%9%IrelandIreland 10%10% 4646%%
Methodological FindingsMethodological Findings
Importance and Challenges of Importance and Challenges of Control GroupControl Group
Procedural effects (e.g. on-line vs. Procedural effects (e.g. on-line vs. paper)paper)
Signs of internal consistency in Signs of internal consistency in measures (inter-item correlations)measures (inter-item correlations)
Intercultural Intercultural Development Development Inventory:Inventory: Standardized Standardized
instrumentinstrument
Ability to compare Ability to compare study abroad program study abroad program outcomes with outcomes with domestic intercultural domestic intercultural programsprograms
? Intercultural Intercultural Development Development Inventory:Inventory:? Requires qualifying Requires qualifying
training to administertraining to administer
? Highly rigorous and Highly rigorous and proprietaryproprietary
? Labor- and cost-Labor- and cost-intensive for large intensive for large scale usescale use
Benefits to Benefits to
Study Abroad AssessmentStudy Abroad Assessment & Challenges to
& Challenges to
Institutional Research Institutional Research
Program Impact & Program Impact & MEIMMEIM Comparable & program Comparable & program
specific data on students’ specific data on students’ intercultural interactionsintercultural interactions
AIQAIQ Examine identity Examine identity
dimension, connect to dimension, connect to literature on thisliterature on this
? Program Impact & Program Impact & MEIMMEIM? Labor-intensive Labor-intensive
development and development and administrationadministration
? AIQAIQ ? Reliability and Validity of Reliability and Validity of
adaptationadaptation? Needs refiningNeeds refining
Benefits to Benefits to
Study Abroad AssessmentStudy Abroad Assessment & Challenges to
& Challenges to
Institutional Research Institutional Research
Next StepsNext Steps AnalysisAnalysis
• More on change and changers with full datasetMore on change and changers with full dataset• More on correlations with intercultural engagement, More on correlations with intercultural engagement,
program variables (including Engle & Engle’s category), program variables (including Engle & Engle’s category), and student variables (including original motivation)and student variables (including original motivation)
• QualitativeQualitative• Psychometrics on AIQ, Program Impact, MIEPsychometrics on AIQ, Program Impact, MIE
Data collectionData collection• Add 1 year follow-up Add 1 year follow-up • Draw on Women’s Intercultural Leadership Portfolios?Draw on Women’s Intercultural Leadership Portfolios?
ApplicationsApplications• Many improvements to (and rationales for) programs Many improvements to (and rationales for) programs
already. Overhauling re-entry course. already. Overhauling re-entry course.
For more handouts or bibliography on For more handouts or bibliography on assessing intercultural outcomes of assessing intercultural outcomes of study abroad programs, go tostudy abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/IIhttp://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.phpLOutcomes.php
or email me:or email me: [email protected] [email protected]