training and testing for acquisition · bmc3 spo l development l best practices-quick conversion to...
TRANSCRIPT
BMC3 SPO
Theater Battle Management Core System- T B M C S -
National Defense Industrial Association5th Annual Testing and Training
Symposium & Exhibition19-22 August 02 Orlando FL
Training and Testing for Acquisition
Marcie Zaharee, Training Manager, ESC/AC
Capt Kevin Crocco, Test Manager, ESC/AC
BMC3 SPO
Overview
l Backgroundl What is TBMCS, and what does it do?
l Pastl TBMCS 1.0.1 Training Review FY02
l Presentl In Response to the Warfighter -- Sep 11th
l Futurel Acquisition Challenges
BMC3 SPO
BackgroundWhat Is TBMCS?
CIS
• DII/COE Compliant• Y2K
• System Wide Data Access/Distribution• ATO
• Common Intel DB
WCCS
• Common Tools
Not Y2KApplication DB
An Integrated Planning and Execution System Providing the JFACCCommand and Control of All Air Operations To Include Theater Missile Defense
CTAPS
TBMCS
Stovepipe Systems
One System IntegratingAll Air Resources
BMC3 SPO
BackgroundWhat Does TBMCS do?
Air Planning and Execution CycleAir Planning and Execution Cycle
Combat PlansCombat Plans
IntelligenceIntelligenceCombatCombatAssessmentAssessment
Strategic Planning
Air BattlePlanning
MissionPreparationMission
Execution
Reporting & Analysis
Strategy DivisionStrategy Division
Combat OperationsCombat OperationsFlying UnitsFlying Units
BMC3 SPO
Section I -- The PastTBMCS Training Review
Goals for TBMCS 1.0.1 Training Evaluation FY02
l Provide an in-depth analysis in assisting futureSystem Program Directors in determining whatconditions distributed learning is likely to beeffective for C2 systems
l Provide a holistic view of TBMCS training thatshows the impact of training, not only on theindividual but on the USAF as well
BMC3 SPO
TBMCS Training ReviewISD Process
l Analysisl Best Practices-Training covers 2,140 tasks from
force to unit processesl Lessons Learned-A TASA is a critical component of
the ISD process—all future strategies are based uponthe results.
l Designl Best Practices
l Spiral development costly for paper based materials.Considerable savings resulted in using HTML materials tosupport spiral testing and fielding
l Material available anytime, anywhere, anyplace
l Lessons Learnedl Limited communities of practices to compare web design
techniques due to new technology
BMC3 SPOl Development
l Best Practices-Quick conversion to web and availabilityto students
l Lessons Learned-Lack of technology planning. Needfor: infrastructure assessment, necessary bandwidth,and AFCA involvement in local computer security.
l Evaluation of Trainingl Preliminary assessment users disliked self-paced and
web-based environmentl Customer Expectations High
l Affected system of record decisionl ESC required to revert to instructor led hands-on approach and
maintain web developmentl Very costly decision
l Material development per student $2471.25l Cost of MTT's per cost of student $6046.37l Cumulative cost per student $8517.62
TBMCS Training ReviewISD Process, cont.
BMC3 SPO
TBMCS Training ReviewData Collection
l Kirkpatrick Level IVl I – Reaction
l End of Course Questionnairel Focus Groups
l II – Learningl Pre/Post Test
l III -- Transferl Self-assessment “ability to perform task”
l IV – Business Resultsl Not Collected
l Total Trained 812 (Operator, SA, and PSS)l Air Force 443l Marine 307l Navy 34l NORAD 38
BMC3 SPO
Findings Question #1Were the Majority of Students Satisfied
at the Completion of Training?
l Agreed range is low, there is room for improvement in the coursesl Focus groups and observations revealed:*
l Pre-conceived and/or negative attitudes by students against TBMCSsystem and/or against LMMS
l Students showed resistance to learn without the instructor (web)l Persistence and voluntary engagement in task was seldom noticed.
4.26%10.23%22.02%Disagreed tng metexpectations
53.57%
24.07%
Ops
69.76%
19.9%
SA
65.7%Agreed tng metexpectations
30%Did not Respond
PSS
*Exception was 152, 157 ANG. Students were motivated, eager to learn, great attitudesand disposition. Facilities were also the most prepared.
BMC3 SPO
Findings Question #2Will there be a difference in the student test
scores after completing the training?
l With all students shifting from a below average score(<75%) to above average (>75%) data suggests thatlearning objectives were met as a result of the instructionAND a knowledge transfer took place.
87.5%87.62%Post Test
32.7%
54.87%
Ops
42%
45.5%
SA
Avg Gain
Pre Test
*PSS was a new skill set. Students did not have an experience prerequisite. They were notprovided pre/post-tests.
BMC3 SPO
Findings Question #3Will users be confident in their ability to perform
key tasks upon completion of the training?
l With a cumulative total of 91.4% students stating that they couldcomplete the key tasks with over-the-shoulder help, on-line help, orwithout help vs. 1.4% stating they could not accomplish the keytasks, data suggests the students perception of their ability toperform key tasks is high.
Overall Performance22 Jan to 29 Jun 2001
0102030405060
1. Could notaccomplish
2. Couldaccomplishw/over-the-shoulder
help
3. Couldaccomplish
w/on-linehelp
4. Couldaccomplishwithout help
5. Notattempted
Air ForceNavyMarine CorpsNORAD
Total students 248Total ops/plans tasksevaluated 66
Total possibleresponses 16,949
Total responsesreceived 7,645Response Rate=45%
1.4% 15.3% 24.1% 52% 7.3%
BMC3 SPO
Findings Question #4Is there a correlation between userexperience and EOC satisfaction?
l Data suggests a correlation exists, however without access toraw data a true correlation could not be determined
l Observations and focus groups revealed:l Students who did not meet the prerequisite of 12 mo legacy/or
TBMCS experience displayedl Lack of knowledge of duty positionl Higher frustration levelsl Quick to judge instructors knowledge
l The didactic personality in most SA led to collaboration/ teamworkto resolve problems/differences during training
64.5%
12.54%
Ops
48.96%
31.43%
SA
Less Than12 Mo Exp
12 Mo Exp
Experience
22.12%
53.57%
Ops
10.23%
69.76%
SA
Did NotMeet
Met
Expectations
BMC3 SPOFindings Question #5. Will the students perceive thefacilitator as knowledgeable about the course content?
l Did Not Respondl 27.38%
ll Strongly Agreed/AgreedStrongly Agreed/Agreedll 72.55%72.55%
l Strongly Disagreed/Disagreedl 3.15%
Findings Question #6. Will students perceive the coursecovered key TBMCS skills specific to their work center?
l Did Not Respondl 23.59%
ll Strongly Agreed/AgreedStrongly Agreed/Agreedll 61.73%%61.73%%
l Strongly Disagreed/Disagreedl 13.92%
Findings Questions 5 & 6,cont.
BMC3 SPOFindings Question #7. Will students perceive that their unitsprovided a workspace that supported a successful trainingenvironment?
l Did Not Respondl 30.58%
ll Strongly Agreed/AgreedStrongly Agreed/Agreedll 57.93%57.93%
l Strongly Disagreed/Disagreedl 11.31%
Findings Questions 7, cont.
BMC3 SPO
TBMCS Training ReviewBarriers and Issues
l Inconsistent Fundingl Evaluation
l Lack of clarity in regulationsl No established criteria from test communityl Performance standards not identified at NAF/Service
l Lack of OJT and Continuation Training Plansl Technology Planning
l “hop” on the web bandwagon with out a long range planl Policy and Management
l Change agentl Enforcementl Lack of CONOPSl No certification program
l Changing Roles of Presentation Media, Instructors, andStudents
l Design of Web-based Training Materials
BMC3 SPO
Section II -- The PresentIn response to the Warfighter
l Trainingl Change in Acquisition Strategyl Testing
Provided HW, SW, installation support, training, and over the shouldersupport for real world events such as Noble Eagle and Operation
Enduring Freedom
BMC3 SPO
In Response to the WarfighterTraining
l Just in Timel Over the shoulderl Subject Matter Experts
l CD/Web
Multi-Service Support
BMC3 SPO
l Loss of fundingl Initial reduction in
manpower for testingl Implemented spiral
development earlier thananticipated
l Focused on smallersystem builds notrequiring reinstallation
In Response to the WarfigherAcquisition Strategy
BMC3 SPO
AFI 63-123 Evolutionary Acquisition“Spiral Development”
In Response to the WarfighterAcquisition Strategy, cont.
One Increment
EstablishPerformanceObjectives
Experiment
SystemEngineering
Risk Analysis&
Trade Offs
CostEstimate
FieldingDecision
Test Test OperationalTest & Eval
Day 0Valid Rqmt
$s
Goal:18 Mosor less
RefineRqmts
RefineRqmts
RefineRqmts
RefineRqmts
Experiment
SystemEngineering
SystemEngineering
SystemEngineering
SystemEngineering
Risk Analysis&
Trade Offs
Risk Analysis&
Trade Offs
Risk Analysis&
Trade Offs
Risk Analysis&
Trade Offs
CostEstimate
CostEstimate
CostEstimate
CostEstimate
Spiral
Acq StrategyApproval
(Feedback)
ContractAward
(Feedback)
SDIPTDecision
(Feedback)
SDIPTDecision
(Feedback)
Spiral Spiral Spiral Spiral
BMC3 SPO
l Three Spirals, One Increment every Two Years
l Spirals; September-March-Septemberl 2nd March; an Increment
l We will schedule back from these dates therequired actions to include requirementsdefinition
l When requirements will be cut off to meet releasel Meeting dates will be known months in advancel Personnel needed for testing will have plenty of notice
l If one of the anticipated enhancements is notready, there will be another “bus” six monthslater
TBMCS
In Response to the WarfighterAcquisition Strategy, cont.
The Bus Stop Schedule
BMC3 SPO
Evolutionary AcquisitionTesting
l Pro’sl User Centricl Less manpower intensive testingl Less time needed for testing
l Con’sl Service concerns AFI is not a regulationl Product driven vs. schedule drivenl Fielding too fast to become proficientl Service Pack Distribution
BMC3 SPO
Section III -- The Future
Training and Testing forEvolutionary Acquisition
BMC3 SPO
Section III -- The Future Testing & Acquisition
• Testing
• Insure spiral test team members areknowledgeable about the system under test
• Some test manager responsibilities mayneed to be delegated to “increment programmanagers”
• Acquisition
• Ensure funding, resources, anddocumentation are in place prior to fieldingC2 systems.
BMC3 SPO
Section III -- The Future Training
• Training• Develop and procure training systems (e.g.,
simulators and trainers) to emulate thecharacteristics of the system vs. MTT reliance.
• Must incorporate a “train the way we fight”mentality
• Budget for initial qualification training prior tosystem fielding
• Implement certification program
BMC3 SPO
The FutureChallenges
l Paradigm changel Mentoring others on processl Doing more with less
l Moneyl People
BMC3 SPO
SummaryTraining and Testing for Acquisition
l Past -- TBMCS Training Reviewl TBMCS Training Analysis, Design Development and Fieldingl Data Gathering and Findingsl Barriers and Issues
l Present – In Response to the Warfighterl JIT support to deployed locations
l Program Office response to real world eventsl Training and Testing for Spiral Development
l Pros & Cons
l Future – The Challengesl Training and Testing for Evolutionary Acquisition