front-end analysis methods for noncommissioned officer education system (ncoes) final program brief...
TRANSCRIPT
Front-End Analysis Methods for Noncommissioned Officer Education
System (NCOES)
Final Program Brief
Gus Finerson Dynamics Research Corporation
Jennifer K. PhillipsCognitive Performance Group
Marisa L. MillerU.S. Army Research Institute
27 Jun 2012
2
• Requirement
• Program Overview
• Process
• Products/Outcomes
• Questions
Agenda
3
Statement of the Need
• Training approaches are differentially suited to developing certain types of skills.
– The Army needs to rigorously and systematically optimize its use of institutional, unit, and self-study training by improving its training site selection process.
• Newly identified operational training requirements have been addressed in institutional training.– There are costs associated with continuously modifying
institutional POIs to account for rapid changes in the COE. – There is a need to distinguish between enduring tasks and
non-enduring tasks, as a potential discriminator between ideal training sites.
4
Program Description
• Front-End Analysis Methods for NCOES project conducted by the Army Research Institute (ARI) in support of the Institute for NCO Professional Development (INCOPD)
– Contractor: Dynamics Research Corporation
– Subcontractor: Cognitive Performance Group
• Period of Performance: August 2011 – July 2012
• Primary Objective: Design an analysis tool that yields optimal
alignment of tasks to institutional training, operational unit training
or self-development/dL
Program Overview
5
Des
ign Identify best
practices and critical site selection factors for placing training
Dev
elop
Describe front-end analysis (FEA) method and instantiate in Site Selection Tool for CTSSBs
Ass
ess Incorporate
user feedback to increase effectiveness of Site Selection Tool
Goal: Improve effectiveness and efficiency of CTSSB process to recommend placement of training.
Phase 1: Design
Objectives• Determine what makes a task appropriate for institutional instruction,
unit training, or self-study• Identify factors for site selection considered by “experts”• Define enduring vs non-enduring tasks, and identify their distinguishing
characteristics
Activities• Literature Review• Data Collection
– Observed CTSSBs• Ft. Benning (11A) and Ft. Bliss (WLC)• Interviewed CTSSB members (4)
– Interviewed SMEs (31)• NCOA Commandants, course managers, instructors, training
developers, etc.• Ft. Benning, Ft. Bliss, Ft. Huachuca, U.S. Coast Guard
6
Phase 2: Develop
• Objectives – Provide a structured and systematic process for considering the
most critical factors for site selection, as identified by SMEs– Assist CTTSB members who have strong operational and
garrison experience to bring to bear, but are novices at site selection
– Maintain the role of human judgment and group discussion, but increase the efficiency and rigor of the process
• Activities– Iteratively developed a logic chain for applying discriminating
factors to site selection recommendation– Identified perceived user requirements for integrating the tool into
CTSSB process– Embedded logic chain in Excel-based Site Selection Tool (SST)
7
Phase 3: Assess
• Objectives – Ensure the logic chain is sound– Maximize usability of the tool within the CTSSB process to ensure
value added to the users• Activities
– Feedback via telephonic demonstrations with 5 SMEs– User Testing at Ft Benning 19A CTSSB– Testing of paper-based SST at Ft Belvoir 35G CTSSB– User Testing at Ft Huachuca 35T CTSSB– Iterations of SST at each juncture
8
Phase 1 Findings
• Concept of enduring vs. non-enduring task is theoretically sound...– Often equated with Doctrine vs. TTP, or general vs. theater-specific– May also relate to systems and equipment that change
9
• … but in practice, task endurability is not considered as a factor for site selection– Some MOSs making efforts to remove specific systems or equipment
from task descriptions in favor of function/outcome– Institution allows for broad and rapid delivery of key training, and
therefore may be appropriate even if the task is non-enduring– Existing tasks can become relevant to new MOSs– Leadership mandates can render the point moot
• Concept of emerging tasks – those that become relevant as adversaries, technologies, and the global environment evolve– Sensitive site exploitation and forensic analysis– Operating against and within networks, both adversary and civilian
• Key leader engagements / tactical questioning • CoIST tasks
Site Selection Factors(SME n=32)
10
• Nature of the Knowledge– Hands-on/Motor skills (17)– Basic facts and procedures (15)– Task complexity (7)– Conceptual and critical thinking (11)– Detail (3)– Segmentability (5)– Difficulty to retain (8)
• Enduring Army Standards– Universality (11)– Core task/Foundational (12)– Need for standardization (9)– Need for control (7)– Leadership (7)– Doctrinal (18)– Task Criticality (8)
• Instructional Affordances– Peer learning benefit (7)– Question & answer (5)– Practical application (17)– Residence of expertise (7)– Need for observation-based assessment (6)
• Task Performance– Safety/Risk (6)– Doctrinal Basis (13)– Frequency of performance (4)– Current force proficiency (3)– Unit-specificity/MOS-specificity (14)
• Site Affordances– Need for equipment (12)– Time available to train (8)– Time required to train (5)– Annual training or qualification (3)– Integration ability (4)– Training updates (5)
Feasible via Self Study?
High = Institution
Institution or Unit?
Resource availability?
Discriminating Factors
11
Goal: Identify a manageable set of factors that will most often produce a sound site selection recommendation.
Factor Short Question Long Question
Safety; Risk Safety concern?DURING TRAINING, what is the risk of injury to personnel or damage to equipment when training this task?
Need for Equipment; Hands-on Motor Skills
Equipment and hands-on training required?
Is actual equipment – a weapon, vehicle, or system – and hands-on training, instead of an equipment substitute, required to train this task?
Task Complexity Difficult to learn? How complex or difficult to learn is this task?
Need for Observation-Based Assessment
Assess performance by watching face-to-face?
Does a trainer need to watch performance face-to-face to assess it?
Training Updates Updates needed more than yearly?
Will the training change more than once a year?
Residence of Expertise SMEs available or accessible at unit?
Are SMEs readily available or accessible to train this task at the unit?
Annual training or qualification
Annual qual./cert. at unit? Is recurring qualification or certification conducted at unit?
Universality; Need for Standardization
Universal task, uniform training required?
Is the task universal and requiring uniformity of training across the MOS?
Need for Control Reliable and controlled assessment?
Is a reliable and controlled assessment highly important?
Peer Learning Benefit Peer-to-peer learning critical and required?
Is sharing experiences with peers from other units critical and required to maximize learning?
Leadership Criticality to leadership? How critical is the task to the leadership role?
Access to Equipment Safety/equipment available at unit?
Do most units have the safety measures and equipment readily available to perform training?
12
Logic Tree
Guiding Principles for Development
• Ease of use for administrators and users• Integrates conceptually with critical task identification process• Enables CTSSB members to indicate disagreement or special
circumstances• Compiles data across CTSSB members• Flags tasks for discussion when consensus is low• Enables real-time additions or modifications to the task list• Can be flexibly applied across range of CTSSB circumstances
13
Site Selection ToolOverview
• User Group 1: Administrators– E.g., Task analysis branch – He/she initiates the process and receives the final results
• User Group 2: CTSSB Members– Soldiers from across the force/MOS
• Excel spreadsheet with incorporated responsive logic
• For each critical task, logic tree of 3-12 questions about nature of the
task and its training
• As function of logic branching, SST recommends training site:
– Institutional, Unit, Self-development/dL
14
How SST Supports CTSSB
15
Critical Task Selection Process
Critical Task Selection Process
Site Selection via
SST
Site Selection via
SST
CTLCTL
CTL Site Recommendations
CTL Site Recommendations
Introductory Briefing
Introductory Briefing
CTSSB
CTL Site Recs Compiled via SST
and Discussed
CTL Site Recs Compiled via SST
and Discussed
Master CTL Site Recommendations
Master CTL Site Recommendations
Post-CTSSB
Consider Institution Time and Resource ConstraintsConsider Institution Time
and Resource ConstraintsJustified
RecommendationsJustified
Recommendations
Board Admin Activities
Board Admin Activities
CTSSB Member Activities
CTSSB Member Activities
ProductsProducts
Key
Sample User SST
16
Task ListShort Version of
Questions
Red Caret for Long Version of
Questions
SST Recommendation
User Opinion, If Different from SST
Task Number Greyed Out Cells Reset ButtonsQuestion Definitions
Demonstration
17
User SST
Sample Administrator SST
18
Set % Consensus Button
Votes and Consensus on User Opinion
All User Comments
Task ListVotes and Consensus on SST Recommendation
Tasks for Group Discussion Highlighted Amber
Individual User ResponsesTask Number
Outcomes of User Assessment
• Three rounds of SST functionality modifications based on user recommendations
• Board members considered a broader range of site selection factors when using the SST– SST resulted in change to site recommendation in 3 cases out of 11
possible 19A LT tasks• Differences in board member opinions were apparent as a result of the
SST, in contrast to the current board process• Board members appreciated the thought priming that results from the
SST• Consensus thresholds must vary, since CTSSB circumstances vary• CTSSB administrators had more favorable reactions to the SST than
the board members; the research team relied more heavily on their comments and suggestions for SST modifications than board members
19
Final SST Functionality
• Issue: Some tasks are outliers, and human judgment trumps SST recommendation– Board member inputs personal site recommendation when he/she
disagrees with SST• Issue: Institutional course, but not the course under review
– “Institution-Functional” is among the board member opinion choices• Issue: Different CTSSBs require different agreement thresholds
– Administrator sets consensus threshold at 50%, 60%, 70%, or 80%. Single highlighting threshold active (amber)
• Issue: Board member may be have limited exposure to task and wish to abstain from a vote– Consensus calculated based on number of respondents vice total board
members; therefore, board member can leave row blank when unfamiliar with the task
20
Potential Future Modifications
• Ability for Administrator to customize the “Institution” recommendation as the specific course under review– E.g., SST Recommendation produced would be “Armor ALC” instead of
“Institution”• SST Recommendations populate only after board member has
answered all questions for all tasks, and “Submits” responses – Pro: Defeats against gaming the system– Con: Adds time requirement; user must re-view each task to select
personal opinion• Ability for Board Members to answer all 12 questions for every task
– Pro: Sense that important questions are always asked and answered– Con: Adds time requirement
• Web-based set of CTSSB resources, including task criticality ratings and links to full task descriptions and other relevant resources– Caution: Retain face-to-face group discussion
21
Products
22
Summary
• SST can increase the effectiveness of site recommendations– Board members assisted in applying more rigor and thought to their
knowledge of the task and subsequent recommendations– Enables board members to have equal votes in the site selection portion
of the CTSSB• SST can increase the efficiency of the site selection portion of the
CTSSB– High consensus tasks do not require discussion– Group discussion time can be focused on those tasks on which board
members disagreed– Encourages less assertive members to voice divergent views; prevents
board from inappropriately concluding consensus• SST is available upon request
– In the future, will be accessible via Army Training Network
23
24
• Marisa L. Miller, PhD (ARI) [email protected]
[email protected] (706) 545-2450
• Gary Rauchfuss, PhD (INCOPD)
[email protected](757) 501-5439
• Phillip Finerson, CSM (R) (DRC)
(270) 268-9900
• Jennifer Phillips (CPG)
[email protected](440) 937-8434
Points of Contact