Developing Meaningful Learning Experiences:

Developing Meaningful Learning Experiences:

Research & Assessment in Leadership Education

© A. Vorreyer & D. Covey

Research & Assessment in Leadership Education

© A. Vorreyer & D. Covey

Introductions!Introductions!

Dr. Anita L. Vorreyer University of North Florida

Dr. Douglass F. Covey Georgia State University

Dr. Anita L. Vorreyer University of North Florida

Dr. Douglass F. Covey Georgia State University

OverviewOverview

Questionnaires Definitions Theoretical (Research) Bases The Intersection of Student Development

and Leadership Education Assessment—Complete with STEPS! Examples at the micro and macro level

Questionnaires Definitions Theoretical (Research) Bases The Intersection of Student Development

and Leadership Education Assessment—Complete with STEPS! Examples at the micro and macro level

Pre-Session QuestionnairePre-Session Questionnaire

1. Student Learning Outcomes: A. Are measures of content knowledge (attainment)

B. Are intentional learning objectives

C. Are the stated objectives used by Student Affairs units

1. Student Learning Outcomes: A. Are measures of content knowledge (attainment)

B. Are intentional learning objectives

C. Are the stated objectives used by Student Affairs units

2. You can measure SLO’s by: A. Student satisfaction surveys

B. Tracking student participation/usage in programs and services

C. Observing and documenting changes in behavior

2. You can measure SLO’s by: A. Student satisfaction surveys

B. Tracking student participation/usage in programs and services

C. Observing and documenting changes in behavior

3. Institutional Effectiveness Indicators are measurements of:

A. Continuous improvement

B. Student learning

C. A measure of adherence to CAS Standards

3. Institutional Effectiveness Indicators are measurements of:

A. Continuous improvement

B. Student learning

C. A measure of adherence to CAS Standards

4. How would you rate your confidence in your ability in designing a program using Student Learning Outcomes:

A. Unable B. Uneasy C. Somewhat confident D. Confident E. Extremely confident

4. How would you rate your confidence in your ability in designing a program using Student Learning Outcomes:

A. Unable B. Uneasy C. Somewhat confident D. Confident E. Extremely confident

5. How familiar are you with CAS Standards:

A. Unfamiliar B. Somewhat familiar C. Familiar with Standards D. Familiar and have used to a limited degree E. Very familiar and have regularly used in program

assessment

5. How familiar are you with CAS Standards:

A. Unfamiliar B. Somewhat familiar C. Familiar with Standards D. Familiar and have used to a limited degree E. Very familiar and have regularly used in program

assessment

Learning ObjectivesLearning Objectives

Participants will be able to Define the terms “assessment”, “evaluation”, and

“student learning outcomes (SLOs)” Understand the differences conceptually between

Outcomes Assessment Institutional Effectiveness (continuous improvement)

Design programs driven by SLOs Recognize principles of good SLO assessment Understand the steps in outcomes assessment Design assessment strategies re: SLO chosen

Participants will be able to Define the terms “assessment”, “evaluation”, and

“student learning outcomes (SLOs)” Understand the differences conceptually between

Outcomes Assessment Institutional Effectiveness (continuous improvement)

Design programs driven by SLOs Recognize principles of good SLO assessment Understand the steps in outcomes assessment Design assessment strategies re: SLO chosen

DefinitionsDefinitions

Research: a methodical investigation into a subject; guides theory development and tests concepts

Assessment: Any effort to gather, analyze, and interpret evidence which describes institutional, divisional, or agency effectiveness. (Schuh, Upcraft, & Associates, 2001)

Research: a methodical investigation into a subject; guides theory development and tests concepts

Assessment: Any effort to gather, analyze, and interpret evidence which describes institutional, divisional, or agency effectiveness. (Schuh, Upcraft, & Associates, 2001)

Evaluation: any effort to use assessment evidence to improve institutional, departmental, divisional effectiveness.

Assessment describes effectiveness. It guides good practice. Evaluation uses these descriptions in order to improve effectiveness.

Evaluation: any effort to use assessment evidence to improve institutional, departmental, divisional effectiveness.

Assessment describes effectiveness. It guides good practice. Evaluation uses these descriptions in order to improve effectiveness.

Two additional termsTwo additional terms

Institutional Effectiveness Plan: Systematic efforts of the university to engage in ongoing institution-wide review of all of its academic, administrative, and educational support units and programs.

Student Learning Outcome: That which you want the student in your course/program to learn as a result of your time with them. It may be cognitive and or behavioral.

Institutional Effectiveness Plan: Systematic efforts of the university to engage in ongoing institution-wide review of all of its academic, administrative, and educational support units and programs.

Student Learning Outcome: That which you want the student in your course/program to learn as a result of your time with them. It may be cognitive and or behavioral.

Why should Student Affairs focus be on SLOs?

Why should Student Affairs focus be on SLOs?

Student learning and development are inherent in our missions as divisions of Student Affairs.

The formal education of students consists of the curriculum and the co-curriculum, and must promote student learning and development that is purposeful and holistic. (CAS Standards)

Outcomes assessment helps us improve the learning of students rather than simply documenting participation or satisfaction.

Student learning and development are inherent in our missions as divisions of Student Affairs.

The formal education of students consists of the curriculum and the co-curriculum, and must promote student learning and development that is purposeful and holistic. (CAS Standards)

Outcomes assessment helps us improve the learning of students rather than simply documenting participation or satisfaction.

CAS StandardsCAS Standards

What is CAS: The Council for the Advancement of Standards in Higher Education.

Since 1979 CAS has written and promulgated standards for programs and services in higher education.

Complying with national standards is one of the means by which educators can assure high quality educational practices and subsequently student learning.

Provides learning domains and sample achievement indicators

What is CAS: The Council for the Advancement of Standards in Higher Education.

Since 1979 CAS has written and promulgated standards for programs and services in higher education.

Complying with national standards is one of the means by which educators can assure high quality educational practices and subsequently student learning.

Provides learning domains and sample achievement indicators

Theoretical BaseTheoretical Base

Student Development

Leadership

Student Development

Leadership

Theoretical FamiliesTheoretical Families

Psychosocial / Identity Theories

Cognitive Development Theories

Typology Theories

Environmental Impact Theories

Psychosocial / Identity Theories

Cognitive Development Theories

Typology Theories

Environmental Impact Theories

Psychosocial / Identity TheoriesPsychosocial / Identity Theories

Focus on definition of self and relationship to others

Erickson Chickering

Focus on definition of self and relationship to others

Erickson Chickering

Cognitive Development TheoriesCognitive Development Theories

Focus on development of critical thinking / reasoning

Perry Kohlberg Gilligan

Focus on development of critical thinking / reasoning

Perry Kohlberg Gilligan

Typology TheoriesTypology Theories

Focus less on change process than upon persistent characteristics (learning styles, personality type)

Meyers Briggs Holland

Focus less on change process than upon persistent characteristics (learning styles, personality type)

Meyers Briggs Holland

Environmental Impact TheoriesEnvironmental Impact Theories

Focus on interaction of individual with organization and culture of college life

Astin Pace Tinto

Focus on interaction of individual with organization and culture of college life

Astin Pace Tinto

Common Assertions Common Assertions

Development is a continuous process Development is cumulative in nature Development progresses along a continuum Development is fairly orderly, stage

oriented Development is reflected in behavior

Miller and Winston 1990

Development is a continuous process Development is cumulative in nature Development progresses along a continuum Development is fairly orderly, stage

oriented Development is reflected in behavior

Miller and Winston 1990

Developmental Tasks of Particular Interest to Leadership

Educators

Developmental Tasks of Particular Interest to Leadership

Educators The ability to appreciate multiple viewpoints The ability to understand pluralistic values

systems The ability to resolve conflicts The ability to be genuine The capacity for empathy The ability to be an agent of change

The ability to appreciate multiple viewpoints The ability to understand pluralistic values

systems The ability to resolve conflicts The ability to be genuine The capacity for empathy The ability to be an agent of change

Leadership Theoretical BaseLeadership Theoretical Base Stogdill: “There are as many definitions of

leadership as there are people who have tried to define it!”

4 common components of leadership definitions (Northouse):

Leadership is a process Leadership is relational Leadership is based on influence Leadership is goal oriented

Stogdill: “There are as many definitions of leadership as there are people who have tried to define it!”

4 common components of leadership definitions (Northouse):

Leadership is a process Leadership is relational Leadership is based on influence Leadership is goal oriented

We assist our students in learning leadership in many ways

We assist our students in learning leadership in many ways

Through the curriculum; in coursework

Through Co-curricular efforts: Workshops Retreats Advising student groups Working with special student communities (Greeks,

Student Government, Residential students, etc.)

Through the curriculum; in coursework

Through Co-curricular efforts: Workshops Retreats Advising student groups Working with special student communities (Greeks,

Student Government, Residential students, etc.)

We may use different theoriesWe may use different theories

Situational Leadership (Hersey & Blanchard)

Servant Leadership (Greenleaf)

Transformational Leadership (Burns)

Relational Leadership Model (Komives, et al)

Social Change Model (HERI)

Situational Leadership (Hersey & Blanchard)

Servant Leadership (Greenleaf)

Transformational Leadership (Burns)

Relational Leadership Model (Komives, et al)

Social Change Model (HERI)

From a curricular perspective…From a curricular perspective…

We usually discuss the particular theories of leadership, and examine various components of the leadership process:

Change Ethics Teams Organizations Emotional Intelligence

We usually discuss the particular theories of leadership, and examine various components of the leadership process:

Change Ethics Teams Organizations Emotional Intelligence

Within Student Affairs…and from a co-curricular perspective

Within Student Affairs…and from a co-curricular perspective

Team Building Other Key Components:

Motivation Conflict Resolution/Management Communication Decision-Making Ethical Behavior

Team Building Other Key Components:

Motivation Conflict Resolution/Management Communication Decision-Making Ethical Behavior

Whether curricular or co-curricular, our aim is to help them as individuals and as positional leaders to: 1) become aware of the different components of

leadership, 2) engage in reflective practice, and 3) develop key skills in leading themselves and

in leading others.

Whether curricular or co-curricular, our aim is to help them as individuals and as positional leaders to: 1) become aware of the different components of

leadership, 2) engage in reflective practice, and 3) develop key skills in leading themselves and

in leading others.

CAS Standards for Leadership Education:

CAS Standards for Leadership Education:

Leadership is an inherently relational process of working with others to accomplish a goal or to promote change. Most leadership programs seek to empower students to:

Enhance their self-efficacy as leaders Understand how they can make a difference whether

as positional leaders or active participants in a group or community process.

Leadership is an inherently relational process of working with others to accomplish a goal or to promote change. Most leadership programs seek to empower students to:

Enhance their self-efficacy as leaders Understand how they can make a difference whether

as positional leaders or active participants in a group or community process.

Research (theory) guides us in developing meaningful SLO’s.

We must make, and help our students to make, the theory to practice connections. We must build a “bridge”…………

Research (theory) guides us in developing meaningful SLO’s.

We must make, and help our students to make, the theory to practice connections. We must build a “bridge”…………

The “Bridge”The “Bridge”

Leadership development involves:

Self awareness Understanding of others Values Diverse perspectives Organizations

Leadership development involves:

Self awareness Understanding of others Values Diverse perspectives Organizations

Change Competence in

establishing purpose Working

collaboratively Managing conflict

Change Competence in

establishing purpose Working

collaboratively Managing conflict

Meaningful Leadership Learning Experiences (SLO)

Meaningful Leadership Learning Experiences (SLO)

Examples from the CAS Standards for LeadershipExamples from the CAS Standards for Leadership

Learning Domain: Intellectual Growth Achievement Indicator: Uses critical thinking in

problem solving

Effective Communication Able to influence others through speaking/writing

Leadership Development Comprehends the dynamics of a group

Learning Domain: Intellectual Growth Achievement Indicator: Uses critical thinking in

problem solving

Effective Communication Able to influence others through speaking/writing

Leadership Development Comprehends the dynamics of a group

Assessment & EvaluationAssessment & Evaluation

Different Types of Assessment Principles of Good SLO Assessment Summative and Formative Direct and Indirect Qualitative or Quantitative Barriers and Resources

Different Types of Assessment Principles of Good SLO Assessment Summative and Formative Direct and Indirect Qualitative or Quantitative Barriers and Resources

Different Types of Assessment (Upcraft & Schuh)

Different Types of Assessment (Upcraft & Schuh)

Tracking Needs assessment Satisfaction assessment Campus environment/student culture Cost effectiveness Comparable institution National standards Outcomes assessment

Tracking Needs assessment Satisfaction assessment Campus environment/student culture Cost effectiveness Comparable institution National standards Outcomes assessment

Principles of Good SLO Assessment

Principles of Good SLO Assessment

1. Assessment should always be linked to course/program objectives

2. Assessment should be regular and ongoing

3. Assessment should be designed to facilitate learning (yours and your students)

1. Assessment should always be linked to course/program objectives

2. Assessment should be regular and ongoing

3. Assessment should be designed to facilitate learning (yours and your students)

Summative & FormativeSummative & Formative

Summative assessment: tests, papers, assignments; grades over the length of the course.

Formative assessment: gives students and leadership educators feedback on the nature and progress of student learning; mastery, not grades.

Summative assessment: tests, papers, assignments; grades over the length of the course.

Formative assessment: gives students and leadership educators feedback on the nature and progress of student learning; mastery, not grades.

Direct & IndirectDirect & Indirect

Direct: scores, writing samples, portfolio artifacts, projects and presentations, oral exams, internships

Indirect: course grades, surveys, questionnaires, focus groups, interviews (dealing with students’ perception of their own learning)

Direct: scores, writing samples, portfolio artifacts, projects and presentations, oral exams, internships

Indirect: course grades, surveys, questionnaires, focus groups, interviews (dealing with students’ perception of their own learning)

Qualitative & QuantitativeQualitative & Quantitative

Qualitative methodologies: detailed descriptions of situations, events, people, interactions and observed behaviors

Quantitative methodologies: assign numbers to objects, events, or observations according to some rule. Instruments used to collect data; statistical methods used to analyze data.

Qualitative methodologies: detailed descriptions of situations, events, people, interactions and observed behaviors

Quantitative methodologies: assign numbers to objects, events, or observations according to some rule. Instruments used to collect data; statistical methods used to analyze data.

Barriers & ResourcesBarriers & Resources

Barriers: Lack of time Lack of money Fear of results

Resources: Colleagues Institutional Research departments Graduate students

Barriers: Lack of time Lack of money Fear of results

Resources: Colleagues Institutional Research departments Graduate students

The StepsThe Steps

1. Define/Identify the SLO (cognitive, behavioral, or affective

2. Align the curriculum/content 3. Identify/Determine the assessment

strategy 4. Engage and Assess 5. Evaluate

1. Define/Identify the SLO (cognitive, behavioral, or affective

2. Align the curriculum/content 3. Identify/Determine the assessment

strategy 4. Engage and Assess 5. Evaluate

1. Define the SLO What is it, exactly, that you want students to be able

to know or do at the end of your program/course?

2. Align the curriculum/content of the program

Is what you are doing/covering in the program/course going to lead to what you want your students to know or do at the end?

1. Define the SLO What is it, exactly, that you want students to be able

to know or do at the end of your program/course?

2. Align the curriculum/content of the program

Is what you are doing/covering in the program/course going to lead to what you want your students to know or do at the end?

3. Determine the assessment strategy: What is it that you are going to measure? What techniques will you use to obtain this?

Interviews Focus groups Observations Performance/demonstration Portfolios Surveys/questionnaires Pre-test & post-test

Confirm the “do-ability” of your chosen technique

3. Determine the assessment strategy: What is it that you are going to measure? What techniques will you use to obtain this?

Interviews Focus groups Observations Performance/demonstration Portfolios Surveys/questionnaires Pre-test & post-test

Confirm the “do-ability” of your chosen technique

4. Engage and Assess

5. Evaluate: use your evidence to improve your program (IEP).

4. Engage and Assess

5. Evaluate: use your evidence to improve your program (IEP).

ExamplesExamples

From the micro-level

From the macro-level

From the micro-level

From the macro-level

MacroviewMacroview

Georgia State’s Division Wide SLO Domains

Cognitive Maturity Effective Citizenship and Intercultural Maturity Mature Relationships Integrated Identity and Personal Maturity

Georgia State’s Division Wide SLO Domains

Cognitive Maturity Effective Citizenship and Intercultural Maturity Mature Relationships Integrated Identity and Personal Maturity

Cognitive MaturityCognitive Maturity

Sub Domain: Problem Solving in Context

Student Judicial Board SLO:

Demonstrate ability to effectively evaluate evidence

Sub Domain: Problem Solving in Context

Student Judicial Board SLO:

Demonstrate ability to effectively evaluate evidence

Effective Citizenship and Intercultural Maturity

Effective Citizenship and Intercultural Maturity

Sub Domain: Acknowledging, respecting and responding to difference

Safe Zone Committee (LGBTQ) SLO:

Be able to identify two listening skills that will assist in interacting with someone expressing a different perspective from own

Sub Domain: Acknowledging, respecting and responding to difference

Safe Zone Committee (LGBTQ) SLO:

Be able to identify two listening skills that will assist in interacting with someone expressing a different perspective from own

Mature RelationshipsMature Relationships

Sub Domain: Conflict Resolution

Recreational Services Student Employee SLO Will be able to describe and demonstrate two

methods to manage or resolve conflict with participants in the Student Rec Center

Sub Domain: Conflict Resolution

Recreational Services Student Employee SLO Will be able to describe and demonstrate two

methods to manage or resolve conflict with participants in the Student Rec Center

Meaningful Leadership Learning Experiences (SLO)

Meaningful Leadership Learning Experiences (SLO)

Designing SLOsDesigning SLOs

What group of students Who participate in What activity/event/course/program Will be able to do, know, value what At what level of accomplishment As determined by what means

Keeling, Wall, Underhile, & Dungy

What group of students Who participate in What activity/event/course/program Will be able to do, know, value what At what level of accomplishment As determined by what means

Keeling, Wall, Underhile, & Dungy

Post-Session QuestionnairePost-Session Questionnaire

1. Which of the following is NOT an indicator of Institutional Effectiveness

A. Student usage/participation data for a facility or program

B. Student satisfaction surveys

C. SLO assessment

1. Which of the following is NOT an indicator of Institutional Effectiveness

A. Student usage/participation data for a facility or program

B. Student satisfaction surveys

C. SLO assessment

2. Meaningful use of SLO requires A. That learning must be measured B. The learning being measured must be the

learning that was intended C. That the results of assessment must be used to

refined programs/experiences D. All of the above

2. Meaningful use of SLO requires A. That learning must be measured B. The learning being measured must be the

learning that was intended C. That the results of assessment must be used to

refined programs/experiences D. All of the above

3. If your intended SLO is for participants to be able to demonstrate acquisition of conflict resolution skills, the most relevant assessment measure would be:

A. Questionnaire of understanding of conflict resolution techniques

B. Professional observation of student performance C. Peer perception of skills

3. If your intended SLO is for participants to be able to demonstrate acquisition of conflict resolution skills, the most relevant assessment measure would be:

A. Questionnaire of understanding of conflict resolution techniques

B. Professional observation of student performance C. Peer perception of skills

4. As a result of attending this session, my confidence in my ability to apply SLOs to the development of leadership programs and experiences is:

A. No better than before

B. Somewhat better than before

C. Significantly better than before

4. As a result of attending this session, my confidence in my ability to apply SLOs to the development of leadership programs and experiences is:

A. No better than before

B. Somewhat better than before

C. Significantly better than before

ExplanationExplanation

From the Student Learning Imperative: Student Affairs must model what we wish

for our students: an ever increasing capacity for learning and self-reflection.

From the Student Learning Imperative: Student Affairs must model what we wish

for our students: an ever increasing capacity for learning and self-reflection.

THANK YOU FOR YOUR PARTICIPATION!!!

Dr. Douglass F. Covey, dcovey@gsu.edu

Dr. Anita L. Vorreyer, avorreye@unf.edu

THANK YOU FOR YOUR PARTICIPATION!!!

Dr. Douglass F. Covey, dcovey@gsu.edu

Dr. Anita L. Vorreyer, avorreye@unf.edu