slide 1 assessments and culminating experiences institutional workshop mr. bill kohlruss-dr. mark...
TRANSCRIPT
Slide 1
www.aabi.aerowww.aabi.aero
ASSESSMENTS ANDCULMINATING EXPERIENCES
INSTITUTIONAL WORKSHOP
Mr. Bill Kohlruss-Dr. Mark ShermanCO-CHAIRS, GUIDANCE COMMITTEE
FEBRUARY 21, 2013
ASSESSMENTS ANDCULMINATING EXPERIENCES
Where Have We Gone Astray?
Based on Feedback From Accreditation Visits…
ASSESSMENTS ANDCULMINATING EXPERIENCES
COMPILATION OF RECOMMENDATIONSFebruary 2009 to July 2012
Recommendations related to:• Program Educational Objectives – 48• Culminating Experience – 23• Assessment – 19• Outcomes – 14• Safety – 11• Industry Advisory Boards/Committees – 10• Curriculum – 8• Faculty – 6• Facilities, Equipment & Services – 4
Thanks
Ceci &
Vic
ASSESSMENTS ANDCULMINATING EXPERIENCES
Number ONE on the list (48 Hits) is Program Educational Objectives !
Program Education Objectives was addressed in depth at the Workshop last February and we are addressing that weak area through a change in wording in the Criteria…
Now called “Program Mission and Educational Goals”.
New terminology, perhaps easier to understand ?
ASSESSMENTS ANDCULMINATING EXPERIENCES
So… Number 2 and 3 are on the agenda for today.
#2 Culminating Experiences.
#3 Assessments.
ASSESSMENTS ANDCULMINATING EXPERIENCES
To put a POSITIVE SPIN on today’s workshop…
We have asked Dr. Beth Bjerke and Dr. Mark Dusenburyto do a presentation on “How They Assessed the Programs atUND”.
This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.
Beth Bjerke and Mark Dusenbury
ASSESSMENT
“Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development” (Palomba and Banta, 1999, pg. 4)
Assessment Process
Establishing your Program Mission Defining your Program Educational Goals Defining your Student Learning Outcomes (AABI
General, Core and Program) Identifying your Techniques and Target Groups Data Collection ‘Closing the Loop’ process Implementing changes to improve student learning Assessing the Assessment Process!
Techniques and Target Groups Techniques
Direct Assessment of Student Learning Written Exams Performance Evaluations Assignments
Target Group – Students Indirect Assessment of Student Learning and Program Quality
Surveys Target Group - Students, Alumni, Industry
Interviews/Focus Groups Target Group - Students, Alumni, Industry
Survey Data
Designing Surveys
• Online Survey Tools• Easy to use and abuse• Set up a timeline and stick to it• Have a purpose for the survey• Share results with faculty and students• Use to improve the learning experience
Gather Indirect Perception Data(AABI a-k from Students)
Gather Indirect Perception Data(AABI a-k from Alumni)
AABI General Outcomes2007 2009 2011
An ability to apply knowledge of mathematics, science and applied sciences 2.86 2.94 3.05An ability to analyze and interpret data 3.15 3.18 3.27An ability to function on multi-disciplinary teams 3.04 3.09 3.21An understanding of professional and ethical responsibilities 3.17 3.25 3.35An ability to communicate effectively, including both written and verbal communication skills 3.09 3.14 3.24A recognition of the need for, and an ability to engage in, life long learning 3.10 3.18 3.33A knowledge of contemporary issues 2.87 2.99 3.12An ability to use the techniques, skills and modern technology necessary for professional practice 3.39 3.35 3.42An understanding of the national and international aviation environment 3.08 3.06 3.23An ability to apply pertinent knowledge in identifying and solving problems. 3.17 3.22 3.35An ability to apply knowledge of business sustainability to aviation 3.10
Closing the Loop with Survey Data Take the time to have the conversation with faculty
Make recommendations and action items Document the conversation!
Interviews and Focus Groups
Students Have a lot to say, and many great ideas Focus Group Discussions Graduating Senior Interviews
Interviews and Focus Groups
Industry/Alumni Use of Advisory Board Other Sources
Campus Visitors Document!
Closing the Loop with Interviews/Focus Groups
Take the time to have the conversation with faculty Make recommendations and action items Document the conversation!
Direct Learning
Course Prime Reports Stage Check Analysis Capstone
Course Mapping
Course Prime Reports
Stage Check Analysis
Capstone
Writing analysis between majors in capstone courses Infant stages – small sample size – will get better with time Food for thought at this time
Techniques Timeline
Faculty Outcome Discussions
MOST IMPORTANT ASPECT TO ASSESSMENT…
Using what is learned through the process to improve the student
learning experience in your program!
ASSESSMENTS ANDCULMINATING EXPERIENCES
To address our second topic we have Ms. Andrea Georgiou and Mr. Paul Carlson doing a presentation on “Our Capstone/Culmination Experience” at MTSU.
This area was reported as a STRENGTH in the Visiting Team Report during the reaffirmation of accreditation by AABI.
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
Middle Tennessee State UniversityMiddle Tennessee State University
Andrea M. GeorgiouAndrea M. Georgiou
Paul R. CarlsonPaul R. Carlson
FOCUS Lab ProjectFOCUS Lab Project
Middle Tennessee State UniversityMiddle Tennessee State University
Andrea M. GeorgiouAndrea M. Georgiou
Paul R. CarlsonPaul R. Carlson
FOCUS Lab ProjectFOCUS Lab Project
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
• FOCUS concept first proposed in 2009 by Dr. Paul A. CraigFOCUS concept first proposed in 2009 by Dr. Paul A. Craig
• Received NASA grant in 2010 to study team decision-makingReceived NASA grant in 2010 to study team decision-making
• Project would address best practices for communication, Project would address best practices for communication, disruption management, teamwork, and group situational-disruption management, teamwork, and group situational-awarenessawareness
• Conceptualization and construction began in 2010 with a small Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate studentsresearch team comprised of faculty and graduate students
• Students began with tabletop discussions and now has evolved to Students began with tabletop discussions and now has evolved to include:include:
• Complex and real-world scenariosComplex and real-world scenarios
• Immediate feedback and ramifications to decisionsImmediate feedback and ramifications to decisions
• Real-time performance analysisReal-time performance analysis
• In-depth after action reviewsIn-depth after action reviews
• FOCUS concept first proposed in 2009 by Dr. Paul A. CraigFOCUS concept first proposed in 2009 by Dr. Paul A. Craig
• Received NASA grant in 2010 to study team decision-makingReceived NASA grant in 2010 to study team decision-making
• Project would address best practices for communication, Project would address best practices for communication, disruption management, teamwork, and group situational-disruption management, teamwork, and group situational-awarenessawareness
• Conceptualization and construction began in 2010 with a small Conceptualization and construction began in 2010 with a small research team comprised of faculty and graduate studentsresearch team comprised of faculty and graduate students
• Students began with tabletop discussions and now has evolved to Students began with tabletop discussions and now has evolved to include:include:
• Complex and real-world scenariosComplex and real-world scenarios
• Immediate feedback and ramifications to decisionsImmediate feedback and ramifications to decisions
• Real-time performance analysisReal-time performance analysis
• In-depth after action reviewsIn-depth after action reviews
PROJECT PROJECT BACKGROUNDBACKGROUND
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
Aerospace CoreAerospace CoreAerospace CoreAerospace Core
TechnologyTechnologyTechnologyTechnologyFlight Flight DispatchDispatch
Flight Flight DispatchDispatch
Pro PilotPro PilotPro PilotPro PilotMaintenance Maintenance ManagementManagementMaintenance Maintenance ManagementManagement
AdminAdminAdminAdmin
Seminar + SimulationSeminar + SimulationSeminar + SimulationSeminar + Simulation• MTSU AEROSPACE program MTSU AEROSPACE program comprised of 5 concentrations:comprised of 5 concentrations:
• PROFESSIONAL PILOTPROFESSIONAL PILOT
• MAINTENANCE MANAGEMENTMAINTENANCE MANAGEMENT
• AEROSPACE ADMINISTRATIONAEROSPACE ADMINISTRATION
• FLIGHT DISPATCHFLIGHT DISPATCH
• TECHNOLOGYTECHNOLOGY
• MTSU AEROSPACE program MTSU AEROSPACE program comprised of 5 concentrations:comprised of 5 concentrations:
• PROFESSIONAL PILOTPROFESSIONAL PILOT
• MAINTENANCE MANAGEMENTMAINTENANCE MANAGEMENT
• AEROSPACE ADMINISTRATIONAEROSPACE ADMINISTRATION
• FLIGHT DISPATCHFLIGHT DISPATCH
• TECHNOLOGYTECHNOLOGY
•All concentrations take aerospace All concentrations take aerospace core classescore classes•All concentrations take aerospace All concentrations take aerospace core classescore classes
•After specializing in their chosen After specializing in their chosen concentrations, students return to concentrations, students return to take a capstone coursetake a capstone course
• Prior to FOCUS lab, capstone course Prior to FOCUS lab, capstone course only addressed industry topics, only addressed industry topics, résumé, and interview preparationrésumé, and interview preparation
•After specializing in their chosen After specializing in their chosen concentrations, students return to concentrations, students return to take a capstone coursetake a capstone course
• Prior to FOCUS lab, capstone course Prior to FOCUS lab, capstone course only addressed industry topics, only addressed industry topics, résumé, and interview preparationrésumé, and interview preparation
PROJECT CONCEPTPROJECT CONCEPT
PROJECT CONCEPTPROJECT CONCEPTCULMINATING EXPERIENCESCULMINATING EXPERIENCES
• Surveys of industry experts revealed that newly-minted Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand aviation professionals required 5-10 years to fully understand their role and impact within an organization.their role and impact within an organization.
• Surveys of industry experts revealed that newly-minted Surveys of industry experts revealed that newly-minted aviation professionals required 5-10 years to fully understand aviation professionals required 5-10 years to fully understand their role and impact within an organization.their role and impact within an organization.
•Research revealed that new aviation professionals lacked:Research revealed that new aviation professionals lacked:
• operational understanding of others’ roles in the operational understanding of others’ roles in the organizationorganization
• basic understanding of how their decisions and job basic understanding of how their decisions and job performance impacts othersperformance impacts others
• the ability to communicate and coordinate effectively with the ability to communicate and coordinate effectively with others to meet organizational objectivesothers to meet organizational objectives
• the use of foresight, planning, and time management in the use of foresight, planning, and time management in making decisions and performing tasksmaking decisions and performing tasks
•Research revealed that new aviation professionals lacked:Research revealed that new aviation professionals lacked:
• operational understanding of others’ roles in the operational understanding of others’ roles in the organizationorganization
• basic understanding of how their decisions and job basic understanding of how their decisions and job performance impacts othersperformance impacts others
• the ability to communicate and coordinate effectively with the ability to communicate and coordinate effectively with others to meet organizational objectivesothers to meet organizational objectives
• the use of foresight, planning, and time management in the use of foresight, planning, and time management in making decisions and performing tasksmaking decisions and performing tasks
““On average, it usually takes our employees 10 years On average, it usually takes our employees 10 years to fully understand the big picture; how the decisions to fully understand the big picture; how the decisions they make impact others in the company.”they make impact others in the company.”
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
• The FOCUS project is specifically designed to address The FOCUS project is specifically designed to address
teamwork deficiencies of new employees in the aviation teamwork deficiencies of new employees in the aviation
industryindustry
• Students are placed in a realistic simulated airline flight Students are placed in a realistic simulated airline flight
operations control centeroperations control center
• Here, students from all aviation disciplines are placed in Here, students from all aviation disciplines are placed in
teams to practice working with othersteams to practice working with others
• The FOCUS project is specifically designed to address The FOCUS project is specifically designed to address
teamwork deficiencies of new employees in the aviation teamwork deficiencies of new employees in the aviation
industryindustry
• Students are placed in a realistic simulated airline flight Students are placed in a realistic simulated airline flight
operations control centeroperations control center
• Here, students from all aviation disciplines are placed in Here, students from all aviation disciplines are placed in
teams to practice working with othersteams to practice working with others
PROJECT CONCEPTPROJECT CONCEPT
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
•Teams work to meet organizational goals like safety, on-time Teams work to meet organizational goals like safety, on-time
performance, customer satisfaction, and disruption performance, customer satisfaction, and disruption
managementmanagement
• Teams are given various scenarios (or Teams are given various scenarios (or triggerstriggers) that require ) that require
the students to work together to resolve issues quickly and the students to work together to resolve issues quickly and
effectivelyeffectively
• Teams are monitored and evaluated by graduate student and Teams are monitored and evaluated by graduate student and
faculty observersfaculty observers
•Teams work to meet organizational goals like safety, on-time Teams work to meet organizational goals like safety, on-time
performance, customer satisfaction, and disruption performance, customer satisfaction, and disruption
managementmanagement
• Teams are given various scenarios (or Teams are given various scenarios (or triggerstriggers) that require ) that require
the students to work together to resolve issues quickly and the students to work together to resolve issues quickly and
effectivelyeffectively
• Teams are monitored and evaluated by graduate student and Teams are monitored and evaluated by graduate student and
faculty observersfaculty observers
PROJECT CONCEPTPROJECT CONCEPT
FF OO CC UU SSlightlight
perationsperations
enterenter
nifiednified
imulationimulation
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
•FOCUS is a high-fidelity replication of a FOCUS is a high-fidelity replication of a part 121 regional airline operations part 121 regional airline operations control centercontrol center
• Students participate in a 3 hour Students participate in a 3 hour simulated work shift working for a virtual simulated work shift working for a virtual airline we call “Universal E-Lines”airline we call “Universal E-Lines”
• Universal E-Lines operates 30 CRJ-200 Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout aircraft on 16 flight routes throughout the southeastern United States.the southeastern United States.
• Utilizing a hub and spoke system, Utilizing a hub and spoke system, Nashville and Jacksonville act as the Nashville and Jacksonville act as the airline’s hubsairline’s hubs
• Students manage up to 80 flight events Students manage up to 80 flight events in a 3 hour simulation sessionin a 3 hour simulation session
•FOCUS is a high-fidelity replication of a FOCUS is a high-fidelity replication of a part 121 regional airline operations part 121 regional airline operations control centercontrol center
• Students participate in a 3 hour Students participate in a 3 hour simulated work shift working for a virtual simulated work shift working for a virtual airline we call “Universal E-Lines”airline we call “Universal E-Lines”
• Universal E-Lines operates 30 CRJ-200 Universal E-Lines operates 30 CRJ-200 aircraft on 16 flight routes throughout aircraft on 16 flight routes throughout the southeastern United States.the southeastern United States.
• Utilizing a hub and spoke system, Utilizing a hub and spoke system, Nashville and Jacksonville act as the Nashville and Jacksonville act as the airline’s hubsairline’s hubs
• Students manage up to 80 flight events Students manage up to 80 flight events in a 3 hour simulation sessionin a 3 hour simulation session
WHAT IS FOCUS?WHAT IS FOCUS?
CULMINATING EXPERIENCESCULMINATING EXPERIENCESPROJECT CONCEPTPROJECT CONCEPT
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
CULMINATING EXPERIENCESCULMINATING EXPERIENCESPHOTOSPHOTOS
CULMINATING EXPERIENCESCULMINATING EXPERIENCESThere are 10 distinct There are 10 distinct positions in the FOCUS Lab. positions in the FOCUS Lab. They are:They are:
• Flight Ops. CoordinatorFlight Ops. Coordinator
• Weather & ForecastingWeather & Forecasting
• Crew SchedulingCrew Scheduling
• Fuel/Cargo Management*Fuel/Cargo Management*
• Flight Tracking & Schedule Flight Tracking & Schedule
Management* Management*
• Maintenance PlanningMaintenance Planning
• Maintenance ControlMaintenance Control
• Ramp TowerRamp Tower
• Pseudo PilotPseudo Pilot
• CRJ Simulator CrewCRJ Simulator Crew
WX & WX & ForecastingForecasting
Crew Crew SchedulingScheduling
MX MX ControlControl
MX MX Planning & Planning & SchedulingScheduling
PROJECT CONCEPTPROJECT CONCEPT
CULMINATING EXPERIENCESCULMINATING EXPERIENCESPHOTOSPHOTOS
CULMINATING EXPERIENCESCULMINATING EXPERIENCES
The MTSU and its The MTSU and its industry partners industry partners pioneered the ability to pioneered the ability to electronically link to a electronically link to a CRJ-200 simulator CRJ-200 simulator located at the flight located at the flight school. Pilots in this school. Pilots in this simulator can simulator can communicate with communicate with anyone in the operations anyone in the operations center. The CRJ simulator center. The CRJ simulator even appears on flight even appears on flight tracking monitors and is tracking monitors and is displayed in the ramp displayed in the ramp tower as a photo realistic tower as a photo realistic aircraft moving about the aircraft moving about the airport environment. airport environment.
PHOTOSPHOTOS
CULMINATING EXPERIENCESCULMINATING EXPERIENCESPHOTOSPHOTOS
CULMINATING EXPERIENCESCULMINATING EXPERIENCESPHOTOSPHOTOS
CULMINATING EXPERIENCESCULMINATING EXPERIENCESSCENARIO DESIGN & CONTROLSCENARIO DESIGN & CONTROL
Scenario Design & Control:Scenario Design & Control:
• Graduate StudentsGraduate Students
• Monitoring team performance Monitoring team performance and status of the operationand status of the operation
• Provide real-world feedback Provide real-world feedback and consequencesand consequences
• Issue disruptive ‘triggers’Issue disruptive ‘triggers’
• LibraryLibrary
• TaxonomyTaxonomy
• Disruptive potential scaleDisruptive potential scale
• Monitor response to triggersMonitor response to triggers
• Evaluate effectivenessEvaluate effectiveness
CULMINATING EXPERIENCESCULMINATING EXPERIENCESAFTER-ACTION AFTER-ACTION
REVIEWREVIEWThe Aerospace Department The Aerospace Department collaborates with collaborates with the Psychology Departmentthe Psychology Department on the FOCUS on the FOCUS project. Studies include:project. Studies include:
• Occupational stereotypes & attitudesOccupational stereotypes & attitudes
• Group dynamicsGroup dynamics
• Individual & team cognitionIndividual & team cognition
After- Action Review:After- Action Review:
• Engages students to discuss events of Engages students to discuss events of simulationsimulation
• Feedback from team members and observersFeedback from team members and observers
• Discussion of unfavorable outcomesDiscussion of unfavorable outcomes
• How to prevent in the future?How to prevent in the future?
• Anticipating problemsAnticipating problems
• Communication & time managementCommunication & time management
• Reinforces positive behaviors and solutionsReinforces positive behaviors and solutions
• Teams establish goals and new strategiesTeams establish goals and new strategies
CULMINATING EXPERIENCESCULMINATING EXPERIENCESADVANTAGESADVANTAGES
• Active learning experienceActive learning experience
• Students provide performance feedback to peersStudents provide performance feedback to peers
• Has the potential to reveal holes in curriculum Has the potential to reveal holes in curriculum
• Encourages faculty to incorporate a greater ‘real-world’ Encourages faculty to incorporate a greater ‘real-world’ perspective in the classroomperspective in the classroom
• Allows students to use knowledge gained from their Allows students to use knowledge gained from their coursework to creatively solve problemscoursework to creatively solve problems
• High motivation to succeed and learn from High motivation to succeed and learn from mistakesmistakes
• Students learn to consider multiple solutions Students learn to consider multiple solutions to a problem, weighing likelihood, impact, and to a problem, weighing likelihood, impact, and other downstream implicationsother downstream implications
• Gives students a sense of time sensitivity Gives students a sense of time sensitivity while problem solvingwhile problem solving
• Internship & job opportunitiesInternship & job opportunities
• Students enjoy it!Students enjoy it!
CULMINATING EXPERIENCESCULMINATING EXPERIENCESQUESTIONSQUESTIONS
t h a n k y o ut h a n k y o u
Dr. Paul A. CraigDr. Paul A. Craig
Ms. Andrea GeorgiouMs. Andrea Georgiou
Mr. Paul R. CarlsonMr. Paul R. Carlson
withwith
Dr. Glenn LittlepageDr. Glenn Littlepage
Dr. Mike HeinDr. Mike Hein
Dr. Richard MoffettDr. Richard Moffett
Mr. Gerald HillMr. Gerald Hill
[email protected]@mtsu.edu
[email protected]@mtsu.edu
[email protected]@mtmail.mtsu.ed
uu
[email protected]@mtsu.ed
uu
[email protected]@mtsu.edu
[email protected]@mtsu.edu
[email protected]@mtsu.edu
ASSESSMENTS ANDCULMINATING EXPERIENCES
Thank You UND & MTSU Culminating Experience Assessment
Guidance Workshop July 2013 Kansas City