Capstones and Outcomes: Using Capstone Events to Assess Student Achievement

• Tony V. Klucking, Ph.D. (tony.klucking.1@us.af.mil)Director, Program Assessment Education Support SquadronAir University

• Ithuriel Arden Gale, Ed.D. (ithuriel.gale@us.af.mil)Deputy Director, Program AssessmentEducation Support SquadronAir University 1 April 2019

Disclaimer

The views presented are those of the speakers and do not necessarily represent the views of the Department of Defense or its components.

2

5 CFR § 3601.108

Capstones and Outcomes

Capstone events, if well designed, provide experiences for students to demonstrate comprehensively the knowledge and skills attained across their education program. As culminating events, capstones offer opportunities to capture evidence of how effective the program has been in producing students who have fully achieved the student learning outcomes for the program. Assessing students’ achievement of program-level student learning outcomes via capstone events can provide many advantages but also entails specific challenges that assessment experts must address for successful execution.

3

Agenda

• Background– Air University– Air War College– Global Challenge

• Methodology– Creating the Capstone / Capstone Rubric– Sampling: opportunities for observations– Recruiting and training evaluators

• Assessment Challenges– Strategic, Operational, Tactical– Moving forward

4

Air University (AU)

5

• AU Overview video from AU Portal– https://www.airuniversity.af.edu/

– Select video #2 from lower right of opening page graphic

Air University and Air War College

• Air University– Continuum of Learning– Professional development throughout every member’s career

• Air War College (AWC)– Air Force’s senior professional military education (PME)

institution– AWC Mission: Educate senior military and civilian teammates to

serve as critical and strategic thinkers able to serve as national security senior leaders.

6

Air War College (AWC)

• AWC Vision: The foremost college for air, space, and cyberspace education and thought – preparing the world’s best joint senior leaders.

• AWC Key Tasks– Prepare officers to lead at the strategic level across the range of military

operations, in joint, interagency, and multinational environments.

– Develop cross-domain mastery of air, space, and cyberspace domains and their strategic contributions to national security.

– Advance innovative thought on national security issues.

7

Air War College (AWC)

8

• AWC Learning Outcomes. Graduates are able to… – lead successfully as senior leaders in joint, coalition, and interagency

environments, exhibiting the traits essential to the profession of arms and understand the proper role and employment of airpower capabilities;

– critically analyze complex political-military issues and clearly articulate through written and oral methods solutions to influence senior level decisions;

– develop and shape military strategies, which, in concert with other instruments of national power, achieve national security strategic objectives; and

– capitalize upon diverse personal and professional relationships forged from the broader AWC educational experience.

Air War College – Academic Year

JUL AUG SEP OCT NOV DEC JAN FEB MAR APR MAY JUN

Inte

rnat

ion

al O

ffic

er

Sch

oo

lIn

-Pro

cess

ing

and

Ori

en

tati

on

Aca

de

mic

Pre

par

atio

n

Foundations of Strategy

Glo

bal

Ch

alle

nge

Nat

ion

al S

ecu

rity

Fo

rum

Gra

du

atio

n

Inte

rnat

ion

al O

ffic

er

Sch

oo

l (N

ext

AY

)Strategic Leadership & Profession of Arms

National Security & Decision Making

Regional Security Studies

JLASS

Global Security

Elective 1 Elective 2 Elective 3

Professional Studies Paper (PSP) Research

Commandant’s Lecture Series

AirpowerTheater Strategy & Campaign Planning

Language Electives (Optional)

Physical Fitness

Global Challenge

• Global Challenge: Capstone event that focuses on the use of national instruments of power—political, military, economic, and cultural—to address threats to U.S. national security

• Capstone event for resident master’s program – Entire student body (less Grand Strategy Seminar students)– Five groups (3-4 seminars each)– One scenario– Three moves

• Move 1: National Priorities• Move 2: National Strategy• Move 3: Operational Approach

• Unit of analysis: Seminar (~14 students)

10

Global Challenge

• Framework– Requires integration of concepts from all academic departments

• International Security• Strategy• Leadership and Warfighting

– No defined roles per se; players function as• National Security Council-level personnel (Move 1 and Move 2)• Combatant Command Staff-level personnel (Move 3)

– Demonstrate mastery of concepts taught by each department over the course of the academic year• The aim is to present students with a complex and evolving global crisis, and

require seminars to make sound, reasoned recommendations and • to defend those recommendations during rigorous questioning by the faculty.

11

Global Challenge

• Key Points– Senior faculty leadership and facilitation– Highly qualified observers– Inputs to student teams are verbal only– Student deliverables

• develop national priorities, national strategy, and operational approach• brief and defend these products to senior faculty & mentors

• Senior Mentors (AY19): includes five retired 3-star and 4-star general officers—experts from the field who have led missions to ensure national security

12

Global Challenge

• Objective: Critically analyze and evaluate crisis situations while functioning as a senior-level planning team to develop strategy at the strategic and operational levels of war.– [NSDM, GS] Prioritize nation security challenges and identify opportunities in a future

operating environment. – [FS, NSDM] Craft a strategy that employs elements of national power to achieve strategic

objectives. – [TSC] Design an operational approach that responds to a strategic problem. – [FS, AP] Assess the capability of military power, including airpower, to achieve strategic

objectives.– [SLPoA] Demonstrate appropriate leadership skills to accomplish assigned tasks and

communicate effectively to positively influence senior strategic leaders.– [TSC] Apply design methodology to formulate and make recommendations for senior

strategic leaders in a time-constrained environment.

13

Airpower (AP); Foundations of Strategy (FS); Global Security (GS); National Security and Decision Making (NSDM); Senior Leader Profession of Arms (SLPoA); Theater Strategy and Campaigning (TSC)

Methodology

14

Methodology

• Creating the Capstone & Capstone Rubric

• Sampling: Opportunities for observations

• Recruiting and training observers

15

Methodology

• Creating the Capstone & Capstone Rubric– Faculty owned event; faculty owned rubric

• Draft rubric by expert faculty

• Socialize rubric across department heads/course directors for inputs

– Adapted across years• Primarily qualitative data collection – what can students do?

• Allows for “not observed” option

• Quantitative scale reduced to 5 points (no ½ points)– 4 (Outstanding/Excellent) to 0 (No Opportunity to Observe)

– simplicity of use & data compilation

16

Methodology: AY17 RubricCriterion and Top Performance Standards

• Leadership– Both formal and informal leaders worked seamlessly as situations dictated the need for particular skills and knowledge.– Leader(s), whether formal or informal, established an atmosphere where creativity and innovation flourish. – Implications at strategic level were the primary consideration in deliberations. When discussion drifted to operational or tactical, action was

taken to return to strategic level.

• Informed Global Perspective– Analysis was concise yet comprehensive, particularly well developed and clearly supported by appropriate theoretical concepts and historical

examples.– Recommendations were particularly clever or unique and considered 2nd and 3rd order effects/implications. Attuned to the synchronized

understanding of the environment/situation to include political systems, geography, culture, and economics. – Identified and prioritized the key U.S. interests in each region of the world.

• Strategic Thinking– Strategies creatively and innovatively related ends, ways, means, and risks. – Strategies creatively and innovatively applied other instruments of power to the problem at hand.– Strategies promoted joint perspective.– Strategies promoted innovative aspects of the application of air, space, or cyberspace power; demonstrate exemplary air mindedness.

• Teamwork– Group made an extra effort to involve all members in decision making; incorporated divergent ideas, and logically determined the best course of

action based on research. – Relationships reflected an exceptional level of respect and depth of understanding of members’ strengths. Group was able to fully exploit the

diverse capabilities of multi-service, civilian, and international fellow members.

17

18

Outstanding/Excellent (4) Good/Satisfactory (3) Marginal (2) Unsatisfactory (1) (0)L

ead

ersh

ip

1. Both formal and informal leaders worked seamlessly

as situations dictated the need for particular skills and

knowledge.

2. Leader(s), whether formal or informal, established an

atmosphere where creativity and innovation flourish.

3. Implications at strategic level were the primary

consideration in deliberations. When discussion

drifted to operational or tactical, action was taken to

return to strategic level.

1. Leaders incorporated particular knowledge and skills

of group members in problem solving.

2. Leaders encouraged creative and innovative thinking.

3. Majority of discussions occurred at strategic level.

Tactical and operational-level discussions tied to

strategic implications.

1. Leaders captured by the first (or their own) ideas; were

slow to incorporate particular knowledge and skills of

group members when they were offered.

2. Leaders did not solicit creative or innovative thinking,

but supported discussion of creative/innovative ideas

as they emerge from the group.

3. Majority of discussion focused on operational level,

though eventually tied to strategic implications.

1. Leadership was inadequate to the task.

2. Leaders discouraged creative or innovative thinking in

pursuit of "standard" solutions to problems posed.

3. Discussion centered primarily on tactical

considerations with only occasional reference to

operational or strategic level.

No

Op

po

rtun

ity t

o O

bse

rve

Info

rmed

Glo

bal

Per

spec

tive

1. Analysis was concise yet comprehensive, particularly

well developed and clearly supported by appropriate

theoretical concepts and historical examples.

2. Recommendations were particularly clever or unique

and considered 2nd and 3rd order effects/implications.

Attuned to the synchronized understanding of the

environment/situation to include political systems,

geography, culture, and economics.

3. Identified and prioritized the key U.S. interests in each

region of the world.

1. Analysis was generally effective and supported by

appropriate theoretical concepts and historical

examples

2. Recommendations were supported by analysis of

national security interests/goals, power relationships of

relevant actors, and other important aspects of the

environment/situation which included political

systems, geography, culture, and economics.

3. Identified most key U.S. interests in each region of the

world.

1. Analysis of national security interests/goals, power

relationships of relevant actors, and other important

aspects of the environment/situation (to include

political systems, geography, culture, and economics)

was employed but incomplete.

2. Recommendations were only somewhat supported by

analysis; connections between analysis

3. Identified some key U.S. interests in each region of the

world.

1. Little to no analysis observed. Actions recommended

based on rote application of standard responses with

little regard to the specific environment/situation to

include political systems, geography, culture, and

economics.

2. Recommendations were based on unsupported

assertions instead of logic and evidence.

3. Unable to properly identify or prioritize U.S. interests

in each region of the world.

No O

pp

ort

un

ity t

o O

bse

rve

Str

ate

gic

Th

ink

ing

1. Strategies creatively and innovatively related ends,

ways, means, and risks.

2. Strategies creatively and innovatively applied other

instruments of power to the problem at hand.

3. Strategies promoted joint perspective.

4. Strategies promoted innovative aspects of the

application of air, space, or cyberspace power;

demonstrate exemplary air mindedness.

1. Military strategies effectively related ends, ways,

means, and risk.

2. Strategies considered other instruments of power,

displaying a solid understanding of the contribution

they bring.

3. Strategic thought reflected joint perspectives.

4. Strategic aspects of air, space, and cyberspace power

were considered appropriately; demonstrated proficient

air mindedness.

1. Ends, ways, means, and risk were considered

incompletely or with minor errors.

2. Strategies considered other instruments of power, but

lacked in-depth understanding or appreciation of their

contribution.

3. Strategic thought incorporated more than one Service,

but failed to consider entirety of joint team.

4. Lacked in employment of airpower, either through

ignoring one of air/space/cyberspace or by focusing on

operational or tactical airpower employment;

demonstrated minimal air mindedness.

1. Failed to consider one (or more) of ends, ways, means,

and risk.

2. Strategies failed to consider other instruments of

power.

3. Strategic thought was framed around perspective of a

single service.

4. Air/space/cyber power was misunderstood or

misapplied.

No

Opport

unit

y t

o O

bse

rve

Tea

mw

ork

1. Group made an extra effort to involve all members in

decision making; incorporated divergent ideas, and

logically determined the best course of action based on

research.

2. Relationships reflected an exceptional level of respect

and depth of understanding of members’ strengths.

Group was able to fully exploit the diverse capabilities

of multi-service, civilian, and international fellow

members.

1. Majority of the members of the team engaged and

contributed to problem solving. Group members

discussed ideas and there was evidence of shared

decision making.

2. Relationships were strong with mutual trust and

understanding clearly evident.

1. Parts of the group engaged and contributed but others

were divorced from the discussions or deliberations.

The group discussed ideas, but one or more dictated

actions and/or not everyone was given equal

consideration in making final decisions.

2. Relationships are evident, but guarded. Trust was

weak and mutual respect appeared cursory.

1. No sense of teamwork; individuals pursued their own

tasks without coordination to a common purpose.

Decisions were made by only a few members without

listening to others' ideas.

2. Relationships were strained. Fractures were clearly

visible between members with no effort observed to

bridge gaps of understanding.

No O

ppo

rtunit

y t

o O

bse

rve

Methodology

• Unit of analysis: The seminar (14 students; not individual students)

• Sampling: opportunities for observations– Time of observation

• Reasonable amount in which to see expected behaviors• Reduced from ~30 minutes to ~15 (observer feedback)

– Observers identify “prime time” for expected behaviors– Number of opportunities for observation varied with overall schedule– Determine number of opportunities/number of evaluators/and number

of seminars to maximize coverage of seminars by different evaluators– Number of expert observers is key

19

Methodology

• Recruiting and training evaluators– Recruiting

• Start early: 3-4 months prior to event (“save the date” email)• Target personnel with practical and academic expertise (Ph.D., AWC-level graduate, O-6)• They must be able to commit to time as an observer (5 days)

– Training• Structure of the Capstone• Interpretation of the rubric• For AY19, small group (GSS students) pilot test 2 weeks prior to full student body• Inter-rater reliability

• Recruiting for next year– Make the experience interesting for observers

• Dynamic movement in/out of scenarios• Opportunities to mingle and share experiences with other observers

– Acknowledgement of service from senior program official (AWC Commandant)

20

Methodology

• Capstone Challenges– Continuity & Change in course, program, and leadership– Value proposition for faculty and observers– Ownership – faculty/leadership commitment to sustain the effort & “get it right”– Not an evaluation of the faculty

• Standard of Success: – Well-considered, thoughtful, comprehensive use of all courses’ key concepts– Clearly communicate national priorities, national strategy, and operational approach to senior

mentors

• Learning lessons and closing the loop– Demonstrated performance by the seminars– Opportunities to improve the experience for the next year

• Post-Capstone “Hotwash”• Capstone Steering Committee (Department Chairs and Course Directors)

21

Assessment Challenges

22

Assessment Challenges

• Strategic– Intuition vs. data-driven decisions– Data-driven decisions vs. omniscience

• Operational– Stability vs. improvement– Improvement vs. wholesale change

• Tactical– Standards for success vs. inter-rater reliability– Comprehensive criterion vs. identifiable knowledge, skills, abilities

23

Assessment Challenges

• Lessons Learned– Faculty and Leadership ownership– Engagement of entire student body– Future-focused scenarios

• Unclassified• Inclusive of international officers

– Clear standards of success• Balance between number of criterion and performance levels• Inter-rater reliability

– Moving forward…always forward

24

Summary

• Background– Air University– Air War College– Global Challenge

• Methodology– Creating the Capstone / Capstone Rubric– Sampling: opportunities for observations– Recruiting and training evaluators

• Assessment Challenges– Strategic, Operational, Tactical– Moving forward

25

Questions?

26