assessment toolbox for international educators

Assessment Toolbox for Assessment Toolbox for International EducatorsInternational Educators

Presenters: Presenters: David Comp, Darla K. Deardorff, David Comp, Darla K. Deardorff,

Elaine Meyer-Lee, Elaine Meyer-Lee, Lee Sternberg, Victor SavickiLee Sternberg, Victor Savicki

Session OverviewSession Overview

Assessment Overview and ContextAssessment Overview and Context

Highlights of Specific Assessment Highlights of Specific Assessment Tools/MethodsTools/Methods

Tools/Methods Exploration through Tools/Methods Exploration through “Roundtable” discussions“Roundtable” discussions

Assessing International Assessing International Education: An OverviewEducation: An Overview

Dr. Darla K. DeardorffDr. Darla K. Deardorff

Duke UniversityDuke University

[email protected]@duke.edu

Why assess international Why assess international education?education?

Inform continuous quality improvementInform continuous quality improvement

Advocate for international educationAdvocate for international education

Satisfy regional accrediting bodiesSatisfy regional accrediting bodies

Add to basic understanding of student growth in Add to basic understanding of student growth in this areathis area

Provide feedback to students on their personal Provide feedback to students on their personal growthgrowth

Why Assess? Measure successWhy Assess? Measure success

Moving beyond Moving beyond

OUTPUTS to OUTCOMESOUTPUTS to OUTCOMES

What are meaningful outcomes What are meaningful outcomes (results) of internationalization (results) of internationalization

efforts?efforts?

__________________________________________________________________________

INTERNATIONALIZATIONat institutions of higher education

Inputs/Resources

needed for implementation of components of internationalization|

Activities/Components of Internationalization(college leadership, faculty involvement, curriculum, study abroad, international

students/scholars/faculty, international co-curricular units) |

Outputs of Internationalization(i.e., number of international students, number of study abroad programs, number of students

studying foreign languages, etc.)|

Outcomes of InternationalizationIntercultural competence – what is it?

How do higher education administrators define it? intercultural experts?How can it be assessed?

= Long-Term Impact of Internationalization

_______________________________________________________________Figure 2. General program logic model applied to internationalization. (Deardorff, 2004, Page 58)

Ways of approaching Ways of approaching assessmentassessment

Some key points…Some key points…

ASSESSING OUTCOMESASSESSING OUTCOMES

Starting point:Starting point:

Mission StatementMission Statement intointo

Goals Goals into into

Measurable ObjectivesMeasurable Objectives

Assessment-ContextsAssessment-Contexts

““At Home”At Home” – – curriculum, extra-curriculum, extra-curricular activities, community curricular activities, community involvement/impact, domestic/internat’l involvement/impact, domestic/internat’l student interaction, policy, financial, student interaction, policy, financial, campus climatecampus climate

AbroadAbroad (Cross-border)(Cross-border) – – involves involves cross-border delivery of education through cross-border delivery of education through exchange, distance and e-learning, branch exchange, distance and e-learning, branch campuses, partnerships, host community campuses, partnerships, host community impactimpact

Assessment CycleAssessment Cycle

Define outcomes (based on Define outcomes (based on mission/goals) and establish measurable mission/goals) and establish measurable criteriacriteria

Identify appropriate assessment Identify appropriate assessment methodsmethods

Collect dataCollect data Analyze dataAnalyze data Reflect on needed changesReflect on needed changes Design and apply changesDesign and apply changes

Assessment – Key PointsAssessment – Key Points

Multi-method, multi-perspectiveMulti-method, multi-perspective

OngoingOngoing

IntegratedIntegrated

Student Involvement Student Involvement in Assessmentin Assessment

Student outcome assessment is not a Student outcome assessment is not a measurement of learning but an measurement of learning but an integral integral part of learning. Can include part of learning. Can include language/disciplinary assessment, personal language/disciplinary assessment, personal growth, intercultural competence, etc. growth, intercultural competence, etc.

Involve students in negotiating Involve students in negotiating outcomesoutcomes

Assessment MethodsAssessment Methods Methods determined by clearly Methods determined by clearly

articulated assessment questions – articulated assessment questions – what do we want to measure?what do we want to measure?

Include:Include: DIRECT METHODS – authentic assessment, DIRECT METHODS – authentic assessment,

portfolios, observationportfolios, observation

INDIRECT METHODS – surveys, focus groups, INDIRECT METHODS – surveys, focus groups, interviewsinterviews

COMBINATION OF QUANTITATIVE AND QUALITATIVECOMBINATION OF QUANTITATIVE AND QUALITATIVE

DIRECT METHODSDIRECT METHODS

Embedded course assessmentEmbedded course assessment PortfoliosPortfolios PerformancePerformance TestingTesting Papers/projectsPapers/projects CapstoneCapstone

Indirect MethodsIndirect Methods

Surveys (inc. self-assessments)Surveys (inc. self-assessments) InterviewsInterviews Focus groupsFocus groups Curriculum/transcript analysisCurriculum/transcript analysis Documented dataDocumented data

ICC Assessment Tools (85+!)ICC Assessment Tools (85+!)

Intercultural Development InventoryIntercultural Development Inventory Cross Cultural Adaptability InventoryCross Cultural Adaptability Inventory Intercultural Conflict Styles InventoryIntercultural Conflict Styles Inventory Language Strategies SurveyLanguage Strategies Survey Strategies Inventory for Culture LearningStrategies Inventory for Culture Learning Beliefs, Events, Values InventoryBeliefs, Events, Values Inventory GAP TestGAP Test Assessment of Intercultural CompetenceAssessment of Intercultural Competence

Questions to consider…Questions to consider…

What are the specific goals and What are the specific goals and objectives?objectives?

What does the tool/method What does the tool/method specifically measure?specifically measure?

Limitations and cultural biases of Limitations and cultural biases of tools/methods?tools/methods?

More questions to consider…More questions to consider…

How will assessment data be utilized? How How will assessment data be utilized? How will data be used to provide feedback to will data be used to provide feedback to students? To improve the students? To improve the program/curriculum, etc?program/curriculum, etc?

Is assessment multi-method, multi-Is assessment multi-method, multi-perspective, ongoing, intentional, perspective, ongoing, intentional, integrated, part of strategic plan?integrated, part of strategic plan?

OVERVIEW OF OVERVIEW OF ASSESSMENT TOOLSASSESSMENT TOOLS

GAP, American Identity, GAP, American Identity,

ICAPS, BEVIICAPS, BEVI

Assessment Toolbox for Assessment Toolbox for International Educators: International Educators:

GAPGAP

David CompDavid CompThe University of ChicagoThe University of Chicago

[email protected]@uchicago.eduNAFSA: Association of International EducatorsNAFSA: Association of International Educators

Annual ConferenceAnnual ConferenceMinneapolis, Minnesota ~ June 1, 2007Minneapolis, Minnesota ~ June 1, 2007

Global Awareness Profile (GAP)Global Awareness Profile (GAP)

Developed by J. Nathan Corbitt in 1998.Developed by J. Nathan Corbitt in 1998.• J. Nathan Corbitt is Professor of Cross-J. Nathan Corbitt is Professor of Cross-

Cultural Studies at Eastern University in Cultural Studies at Eastern University in St. Davids, Pennsylvania and President of St. Davids, Pennsylvania and President of BuildaBridge International, a cross-cultural BuildaBridge International, a cross-cultural non-profit arts education organization.non-profit arts education organization.

J.N. Corbitt, Global Awareness J.N. Corbitt, Global Awareness Profile. Profile. (Yarmouth, ME: Intercultural (Yarmouth, ME: Intercultural Press, 1998).Press, 1998).


Definition of Global AwarenessDefinition of Global Awareness• Global Awareness involves a recognition Global Awareness involves a recognition

and appreciation of the size, complexity, and appreciation of the size, complexity, and diversity of the earth conceived as a and diversity of the earth conceived as a single entity. It is literally the single entity. It is literally the recognition of a worldview. It enables us recognition of a worldview. It enables us to perceive the vastness of the world, its to perceive the vastness of the world, its dynamic complexity, and the diversity of dynamic complexity, and the diversity of its peoples, cultures, and environment.its peoples, cultures, and environment.

- J.N. Corbitt, - J.N. Corbitt, http://www.globalawarenessprofile.com/http://www.globalawarenessprofile.com/

Global Awareness Profile (GAP)Global Awareness Profile (GAP) Structure of the Global Awareness Profile (GAP)Structure of the Global Awareness Profile (GAP)

• The GAP test is a self-scoring inventory that gives The GAP test is a self-scoring inventory that gives participants a graphic representation of their global participants a graphic representation of their global awareness. It presents 120 questions based on awareness. It presents 120 questions based on common knowledge in six geographic regions (Asia, common knowledge in six geographic regions (Asia, Africa, North America, South America, the Middle Africa, North America, South America, the Middle East and Europe) and six subject areas East and Europe) and six subject areas (environment, politics, geography, religion, (environment, politics, geography, religion, socioeconomics and culture), along with twelve socioeconomics and culture), along with twelve questions about broad global issues. After scoring questions about broad global issues. After scoring their tests, participants transfer their scores to their tests, participants transfer their scores to profiles that allow them to see the areas in which profiles that allow them to see the areas in which they are strongest and weakest.they are strongest and weakest.

http://www.globalawarenessprofile.com/http://www.globalawarenessprofile.com/


Who uses the GAP test?Who uses the GAP test?• UniversitiesUniversities measuring global awareness measuring global awareness

outcomes outcomes • Cross-cultural trainersCross-cultural trainers preparing trainees for preparing trainees for

overseas service of multi-cultural contexts overseas service of multi-cultural contexts • EducatorsEducators teaching complex thinking about teaching complex thinking about

world issues and general wariness of the world world issues and general wariness of the world • Business personsBusiness persons who wish to develop their who wish to develop their

global competence or intelligence global competence or intelligence • IndividualsIndividuals for personal self-awareness for personal self-awareness

http://http://www.globalawarenessprofile.comwww.globalawarenessprofile.com//

Global Awareness Profile (GAP)Global Awareness Profile (GAP) Validity and ReliabilityValidity and Reliability

• University professors and international students were used in University professors and international students were used in researching the GAP test by asking a single question: What researching the GAP test by asking a single question: What should a person in your discipline or country know as common should a person in your discipline or country know as common knowledge. Designed from hundreds of questions, and updated knowledge. Designed from hundreds of questions, and updated in 2005 by graduate students from St. Joseph’s University, the in 2005 by graduate students from St. Joseph’s University, the inventory, was tested using face and content evaluation, and inventory, was tested using face and content evaluation, and retesting to validate the instrument and check for reliability.retesting to validate the instrument and check for reliability.

• Testing is ongoing in a number of high schools and universities Testing is ongoing in a number of high schools and universities with both undergraduate and graduate students to establish with both undergraduate and graduate students to establish median scores. median scores.

http://http://www.globalawarenessprofile.comwww.globalawarenessprofile.com//

• According to Paige & Stallman, “one study by Corbit reports a According to Paige & Stallman, “one study by Corbit reports a satisfactory test-retest reliability coefficient (.83). This study satisfactory test-retest reliability coefficient (.83). This study provides evidence of the instrument’s validity by showing that provides evidence of the instrument’s validity by showing that the GAP was able to discriminate between individuals who had the GAP was able to discriminate between individuals who had and had not studied abroad.”and had not studied abroad.”

A Guide to Outcomes Assessment in Education AbroadA Guide to Outcomes Assessment in Education Abroad. . (2007). Edited by (2007). Edited by Mell Bolen. The Forum on Education AbroadMell Bolen. The Forum on Education Abroad

New Tools for New Tools for Intercultural Learning Intercultural Learning

Outcomes AssessmentOutcomes AssessmentElaine Meyer-LeeElaine Meyer-Lee

DirectorDirector

Joy EvansJoy EvansAssistant Director for Research and ScholarshipAssistant Director for Research and Scholarship

Saint Mary’s College Saint Mary’s College Center for Women’s InterCultural LeadershipCenter for Women’s InterCultural Leadership

Notre Dame, Indiana Notre Dame, Indiana

For five years we conducted a For five years we conducted a comprehensive study of our comprehensive study of our

intercultural and international intercultural and international learning on and off campus. learning on and off campus.

For study abroad we used a multi-For study abroad we used a multi-method longitudinal assessment of method longitudinal assessment of

intercultural engagement, IC intercultural engagement, IC sensitivity, identity, and goalssensitivity, identity, and goals

Evaluation & Assessment GoalsEvaluation & Assessment Goals

Evaluate programs’ effectivenessEvaluate programs’ effectiveness Measure Students’:Measure Students’:

• Changes in sensitivity to cultural differences Changes in sensitivity to cultural differences • American identity developmentAmerican identity development• Own sense of growth toward goalsOwn sense of growth toward goals

Explore correlations of change with:Explore correlations of change with:• Intercultural engagement while abroad Intercultural engagement while abroad

(interaction and reflection)(interaction and reflection)• Program or demographic characteristicsProgram or demographic characteristics

American Identity MeasureAmerican Identity Measure

Adaptation of Jean Phinney’s Multigroup Ethnic Adaptation of Jean Phinney’s Multigroup Ethnic Identity Measure (good track record), and her Identity Measure (good track record), and her much newer American Identity Questionnaire much newer American Identity Questionnaire developed for underrepresented groupsdeveloped for underrepresented groups

Based on Eriksonian identity development theory. Based on Eriksonian identity development theory. Measures 2 Factors: Measures 2 Factors:

• identity search (e.g. I have often talked to other people identity search (e.g. I have often talked to other people about what it means to be an American.)about what it means to be an American.)

• affirmation, belonging, and commitment (e.g. Being affirmation, belonging, and commitment (e.g. Being American plays an important part in my lifeAmerican plays an important part in my life.).)

American Identity MeasureAmerican Identity Measure Adapted it (with Phinney’s blessing) for study Adapted it (with Phinney’s blessing) for study

abroad students because one goal is more abroad students because one goal is more complexity in understanding one’s own culturecomplexity in understanding one’s own culture

Students in returnee courses felt this national Students in returnee courses felt this national identity version captured an important dimensionidentity version captured an important dimension

10-item Likert-style Self Assessment (5 for each)10-item Likert-style Self Assessment (5 for each)

Currently refining adaptation’s psychometric Currently refining adaptation’s psychometric properties, but alpha coefficients good so far properties, but alpha coefficients good so far (.79+)(.79+)

SourcesSources

Phinney, J. (1992). The Multigroup Ethnic Phinney, J. (1992). The Multigroup Ethnic Identity Measure: A new scale for use with Identity Measure: A new scale for use with adolescents and young adults from diverse adolescents and young adults from diverse groups. groups. Journal of Adolescent ResearchJournal of Adolescent Research, , 77, , 156-176156-176..

Phinney, J., & Devich-Navarro, M. (1997). Phinney, J., & Devich-Navarro, M. (1997). Variations in bicultural identification Variations in bicultural identification among African American and Mexican among African American and Mexican American adolescents. American adolescents. Journal of Journal of Research on AdolescenceResearch on Adolescence, , 77, 3-32., 3-32.

Measure of Intercultural Measure of Intercultural EngagementEngagement

Asks students to quantify the Asks students to quantify the frequency of their engagement frequency of their engagement through:through:

Interaction with cultural resourcesInteraction with cultural resources Interaction with local peopleInteraction with local people Explicit reflection on these Explicit reflection on these

interactionsinteractions

MEIMEI Comparable & Comparable &

program specific data program specific data on students’ on students’ intercultural intercultural interactionsinteractions

AIQAIQ Examine identity Examine identity

dimension, connect to dimension, connect to literature on thisliterature on this

? MEIMEI? Labor-intensive Labor-intensive

development and development and administrationadministration

? AIQAIQ ? Reliability and Validity of Reliability and Validity of

adaptation still being adaptation still being establishedestablished

? Needs refiningNeeds refining

Benefits to Benefits to

Study Abroad AssessmentStudy Abroad Assessment & Challenges to

& Challenges to

Institutional Research Institutional Research

Intercultural Leadership Intercultural Leadership Certificate ProgramCertificate Program

Capture Saint Mary’s strengths and CWIL’s unique location at the Capture Saint Mary’s strengths and CWIL’s unique location at the intersection of intercultural (local and global) and leadership intersection of intercultural (local and global) and leadership education in the context of a women’s collegeeducation in the context of a women’s college

Guide students through a developmental process into a deeper Guide students through a developmental process into a deeper capacity for intercultural leadership capacity for intercultural leadership

Bring coherence and provide an organizing framework for every Bring coherence and provide an organizing framework for every effort (curricular and co-curricular) being undertaken at Saint effort (curricular and co-curricular) being undertaken at Saint Mary’s College to encourage intercultural leadershipMary’s College to encourage intercultural leadership

Portfolio assessment of students’ proficiency as an intercultural Portfolio assessment of students’ proficiency as an intercultural leader in leader in 6 proficiency areas6 proficiency areas

Recognize the Leader WithinRecognize the Leader Within Articulate Your Ethical/Spiritual CenterArticulate Your Ethical/Spiritual Center Engage With & Value DiversityEngage With & Value Diversity Dialogue on Power & PrivilegeDialogue on Power & Privilege Create Inclusive & Equitable CommunityCreate Inclusive & Equitable Community Make Your Difference in the WorldMake Your Difference in the World

For more handouts or bibliography on For more handouts or bibliography on assessing intercultural outcomes of assessing intercultural outcomes of study abroad programs, go tostudy abroad programs, go to http://www.saintmarys.edu/~cwil/php/intercultural.learning/IIhttp://www.saintmarys.edu/~cwil/php/intercultural.learning/IILOutcomes.phpLOutcomes.php

or email me:or email me: [email protected] [email protected]

Assessment Toolbox for Assessment Toolbox for International Educators: International Educators:

ICAPSICAPS

Victor SavickiVictor Savicki

Western Oregon UniversityWestern Oregon University

Intercultural Adjustment Intercultural Adjustment Potential Scale (ICAPS)Potential Scale (ICAPS)

Developed by David Matsumoto and Developed by David Matsumoto and associates to predict adjustment to a associates to predict adjustment to a foreign cultureforeign culture

Matsumoto, D., Leroux, J. A., Ratzlaff, C., Matsumoto, D., Leroux, J. A., Ratzlaff, C., Tatani, H., Uchida, H., Kim, C., & Araki, S. Tatani, H., Uchida, H., Kim, C., & Araki, S. (2001). Development and validation of a (2001). Development and validation of a measure of intercultural adjustment measure of intercultural adjustment potential in Japanese sojourners: The potential in Japanese sojourners: The Intercultural Adjustment Potential Scale Intercultural Adjustment Potential Scale (ICAPS).(ICAPS). International Journal of Intercultural International Journal of Intercultural Relations,Relations, 1-28. 1-28.

Many subsequent supporting studiesMany subsequent supporting studies

Intercultural adjustment Intercultural adjustment potential definitionpotential definition

““the ability to adapt successfully to life in the ability to adapt successfully to life in a cultural environment different than that a cultural environment different than that which one is accustomed to”which one is accustomed to”

““The ICAPS-55 is the only scale available The ICAPS-55 is the only scale available today that can reliably and validly predict today that can reliably and validly predict the degree to which a person will the degree to which a person will successfully adjust to living, working, and successfully adjust to living, working, and playing effectively in a new and different playing effectively in a new and different cultural environment.”cultural environment.”

Applied to both foreign nationals coming Applied to both foreign nationals coming to the U.S and U.S. nationals going abroadto the U.S and U.S. nationals going abroad

ICAPS ScalesICAPS Scales

ICAPS TotalICAPS Total: overall potential for successful : overall potential for successful intercultural adjustmentintercultural adjustment

Emotion RegulationEmotion Regulation: The ability to monitor and : The ability to monitor and manage one’s emotional experiences and expressions, manage one’s emotional experiences and expressions, and to channel their energies in constructive waysand to channel their energies in constructive ways

OpennessOpenness: The ability to encounter new experiences, : The ability to encounter new experiences, emotions emotions

FlexibilityFlexibility: The ability to assimilate new experiences, : The ability to assimilate new experiences, schemas, and ways of thinking into one’s ownschemas, and ways of thinking into one’s own

Critical ThinkingCritical Thinking: The ability to think outside the box in : The ability to think outside the box in creative and autonomous wayscreative and autonomous ways

ICAPS StructureICAPS Structure

55 items 55 items Responses given on a scale ranging from 1 Responses given on a scale ranging from 1

(Strongly Disagree) to 7 (Strongly Agree)(Strongly Disagree) to 7 (Strongly Agree) Takes about 10-15 minutes to completeTakes about 10-15 minutes to complete Scores reported as T-scores (mean=50, SD=10) Scores reported as T-scores (mean=50, SD=10) Scale and scoring through David MatsumotoScale and scoring through David Matsumoto

• www.davidmatsumoto.info/Research_Tools/www.davidmatsumoto.info/Research_Tools/icaps.htmicaps.htm

Advantages and DisadvantagesAdvantages and Disadvantages

Predicts intercultural adjustment over Predicts intercultural adjustment over monthsmonths

Gives several scores to interpretGives several scores to interpret Highlights individual’s strengths and Highlights individual’s strengths and

weaknessesweaknesses Is sensitive to training and educationIs sensitive to training and education Negligible face validity (I like haiku poems.)Negligible face validity (I like haiku poems.) Is sensitive to timing of administrationIs sensitive to timing of administration No self-scoring yet..No self-scoring yet..

The Beliefs, Events, and Values InventoryThe Beliefs, Events, and Values Inventory(BEVI)(BEVI)

Developed by Craig N. Shealy, Ph.D.Developed by Craig N. Shealy, Ph.D.James Madison UniversityJames Madison University

[email protected]@jmu.edu

http://www.jmu.edu/ibavihttp://www.jmu.edu/ibavi

The Beliefs, Events, and Values The Beliefs, Events, and Values Inventory Inventory

(BEVI)(BEVI)

What does the BEVI measure? What does the BEVI measure? The BEVI…The BEVI…is designed to assess a number of is designed to assess a number of

processes relevant to international experiences processes relevant to international experiences including (but not limited to): basic openness; including (but not limited to): basic openness; receptivity to different cultures, religions, and receptivity to different cultures, religions, and social practices; the tendency (or not) to social practices; the tendency (or not) to stereotype in particular ways; self and emotional stereotype in particular ways; self and emotional awareness; and preferred but implicit strategies awareness; and preferred but implicit strategies for making sense of why ‘other’ people and for making sense of why ‘other’ people and cultures ‘do what they do’ (from Shealy, 2005)cultures ‘do what they do’ (from Shealy, 2005)

The BEVI:The BEVI:

Has been in development since the 1990sHas been in development since the 1990s Is derived from belief-value statementsIs derived from belief-value statements Is comprised of three validity and ten Is comprised of three validity and ten

process scalesprocess scales Does not appraise “correctness” of Does not appraise “correctness” of

individual items but an overall response individual items but an overall response pattern pattern

Is valid and reliable over timeIs valid and reliable over time

The BEVI:

• Has 54 demographic items including age, gender, ethnic background, educational attainment, political orientation, and religious affiliation

• Has 391 items across a wide set of beliefs and values rated on a four-point Likert scale

• Takes 45-55 minutes to complete

• Is available online

BEVI items include:

I am never sad.I have felt jealousy toward someone I loved.

There is no such thing as destiny. You can't argue with fate.

We should do more to help minority groups in our culture.Big corporations often harm the earth.

I know that evil people go to hell when they die.Hinduism and Buddhism have much to teach the modern world.

The BEVI:

• Is effective in showing how people’s beliefs and values change as a result of having an international experience, including the way people feel about their own cultures, the value of learning about other

cultures, the importance of learning a second language, the value in learning about world events, and other attitudes.

• Measures complex phenomena and is not face-valid

• Administration time can be long for some

Your Turn…!Your Turn…!

Join us for “Roundtable Discussions” Join us for “Roundtable Discussions” on the specific tools you’ve just on the specific tools you’ve just heard about… ask more questions, heard about… ask more questions, learn more in detail about how and learn more in detail about how and when to use these instrumentswhen to use these instruments

assessment toolbox for international educators

Documents