evaluation plan: nemesis precision shooting...

2016

DARLENE FERRI-KURJACK

EDCI 57300

2/8/2016

Evaluation Plan: Nemesis Precision Shooting System

EVALUATION PLAN: NEMESIS PRECISION SHOOTING SYSTEM 1

Contents Executive Summary ..................................................................................................... 2

Training Product ............................................................................................................................... 2

Highlights – Evaluation Need ........................................................................................................... 2

Objectives ........................................................................................................................................ 2

Evaluation Goals .............................................................................................................................. 2

Nemesis Precision Support System (NPSS) Product Overview ............................................................................................................. 3

PARTICIPANTS ................................................................................................................................... 4

Evaluation Process Overview ................................................................................ 7

Measurement Instruments, Administration

Procedures, Reporting .................................................................................... 8

PARTICIPANT CHARACTERISTICS ..................................................................................................... 9

PRE-TEST .......................................................................................................................................... 9

ADDITIONAL MEASURES ............................................................................................................... 10

KIRKPATRICK MODEL .................................................................................................................... 11

REFERENCES: .................................................................................................................................. 20

Appendix A: Measurement Instruments – Baselines/Participants ....................................................................................... 21

Appendix B: Measurement Instruments – Level 1 – Gauging Satisfaction .......................................................................................... 57

Appendix C: Measurement Instruments – Level 2 Skill Acquisition ........................................................................................................ 71

Appendix D: Level 3 - Behavioral Transfer/Skill Retention .......................................................................................................... 77

Appendix E: Reporting Recommendations ................................................................. 84

Appendix F: Planning Charts ...................................................................................... 86

Appendix G: Product Figures ...................................................................................... 89

Appendix H: Liability Waivers .................................................................................... 97


Executive Summary

Training Product

The Nemesis Precision Shooting System (NPSS) is a training support product developed by. Norrad Inc. The

technology is cutting edge to collect biometric data and audio/video in a remote manner to provide multimodal

information to the trainee or the instructor. This capability, assists in the effectiveness of Marksmanship Training

for Military, Law Enforcement, and Private domains. The tool is designed to limit the gap of cost and venue for

training to time critical stress based environments where marksmanship, team dynamics, and efficient decision

making must take place within seconds.

Highlights – Evaluation Need

Norrad has created a cutting edge technology that offers a solution to training needs in marksmanship domains

restricted by funded budgets, such as Law Enforcement and the Military. The domestic threats of late also require

available training in Private Security sectors. Preliminary subjective responses from subject matter experts yield

promise to the marketing and usefulness of the product. Evaluating a product with credible research holds value to

those domains reliant on funding. This plan serves the preparedness of future evaluative needs relative to the

marketing of the NPSS training tool. The proposed Evaluation Plan focuses on the product’s metrics relative to

performance gauged with use of this training support product.

Objectives Gauge the satisfaction of trainees and instructors with their respective use in roles with the NPSS tool.

Identify training efficiency in terms of time to competency with the NPSS product over traditional training

Identify the cost effective margin of training with the NPSS product over traditional means (time to train,

performance effectiveness, spent ammunition, satisfaction)

Define any deficiencies that exist in training or instruction with the NPSS tool.

Evaluation Goals

This proposal conveys an evaluation strategy to assess the benefit of a training performance support product for

fielded use among varied domains that require marksmanship expertise. This plan serves as a framework of

consideration for evaluating the NPSS product. A framework approach is justified given the broad range of

domains that may require specialized evaluation considerations in terms of variable training objectives, target

accuracy requirements, scoring mechanisms, and environmental challenges and conditions. The measurable

factors among this variability are limited to satisfaction in training with the product, efficiency in time to train,

performance improvement (skills acquired and retained) and a measure of spent ammunition.

This evaluation prescription will serve as a means to collect data and feedback to evolve the NPSS training product.

The evaluation plan proposed will gauge the satisfaction and efficiency of training with the tool. This part of the

evaluation provides a formal means of gaining instructor/ trainee feedback regarding the training aspects of the

NPSS tool and the evolution of this product to meet the goals of training. Metrics such as these have primary focus

to gauge the subjective measures of effectiveness in training with the tool and to identify any decrements to

training with the tool. These results will serve to glean insights to the needs for product use aligned to the most

current technological standards, and future marketability. The plan will also serve to obtain objective metrics and


subjective data relative to defined training objectives for the purpose of gauging the training potential of the

product among Levels 1-4 of the Kirkpatrick process, respectively being satisfaction, skill acquisition, skill transfer

to real world live demonstration, and the results of training measured as a quantifiable benefit to the training goals

of the specified client. In addition, the plan can serve as a foundation of evaluation prescription to serve future and

more evolved business needs.

The evaluation strategy will be guided by Kirkpatrick’s Four Level Evaluation Model (Kirkpatrick & Kirkpatrick,

2006). For the purpose of evaluating the NPSS tool, evaluation Levels 1 and Level 2 respectively represent

Instructor and Trainee Reactions to the Training and Training Efficiency (Kirkpatrick & Kirkpatrick, 2006). Evaluation

methods will be employed to respectively gauge performance improvement to long term behavioral transfer (Level

3) and quantifiable benefits of the training product to stakeholders (Level 4) (Kirkpatrick & Kirkpatrick, 2006).

Outcomes of proposed analysis should subscribe to the following goals for future use of this training platform:

Level 1 Gauges Satisfaction and Reaction to use with the NPSS training support product

Trainee Satisfaction of Training with the NPSS Content in the 90th percentile or higher (Survey)

Instructor response to Frequency of use with Biometric and Audio/Video capabilities for Training with the

NPSS tool in 80th percentile or higher (Survey and comments of instructors/Experimental Group)

Level 2 Gauges Skill Acquisition

Training Efficiency (efficiency of training as measured by time and accuracy correlates over NPSS product

and traditional training methods)

Improved efficiency to training time with the NPSS product = cost improvement

o Decreased training time by 5% (NPSS over control group)

Ammunition Savings of 5% with NPSS product Training over traditional training vs = Cost Improvement

Level 3 Transfer of Training to Transfer Effectiveness Evaluation for Skill Retention)

A minimum of 5% increase of noted skill acquisition of NPSS training participants over traditional method

(post-test evaluation equates to performance improvement (live rehearsal)

Long term Retention skills supported by using NPSS product over traditional method of retention testing

Level 4 Benefits to the Organization

A conveyance of reported improvements to skill retention as a result of training with the NPSS product

(live rehearsal) (Level 3) = negates training challenges to venue, diminishes cost of refresher training

A conveyance of reported training efficiency as measured by decreases in training time and spent

ammunitions, skill acquisition (Level 2)

A conveyance of satisfaction with the training product. (Level 1) = Usefulness

Nemesis Precision Support System (NPSS) Product Overview NPSS uses sensors to obtain biometric data as well as video and audio for increased performance and data

management. The information is sent to a server and used for training feedback and stored for future use. The

data can be used for the instructor to create personalized training scenarios that scaffold marksmanship training

objectives to the needs of the individual or group, in a face to face or remote venue. The product considers use

with augmented and real physical training environments with live or dry fire training experiences. Collected real-

time data includes ECG, audio, video, trigger pull, EMG, and GSR. Training parameters are definable for

ammunition, weapon type, shooting session, and shot calculations.

This technology is advanced to collect data for interpretation to the instructor and instant feedback to the trainee,

which assists both the instructor and the trainee to have mutual understanding of what is taking place during a

training exercise. Efficient feedback has value to the learner and the instructor to decrease training time and

effect physiological conditioning to training, as well as increase the aptitude and diminish the time for students to

meet competency milestones relative to marksmanship training among a variety of domains and specified training

objectives.


The biometric information, combined with video and audio convey information that may be not perceptually

captured by instructors, but is required to convey proper feedback and guidance to trainees. Basic marksmanship

training fundamentals support certain behavioral skills such as: trigger control, breath control, aiming, steadiness

and physical position that play into accurate targeting (James, D., Dyer, J., & U.S. Army Research Institute for the

Behavioral Social Sciences, 2011). Research and synthesis support marksmanship as a complex skill and that

biometric data supports aspects of marksmanship that are difficult to observe, such as, trigger control, breath

control, muzzle wobble, and the brain states of the shooter (Chung, G., et al, p. 31, 2011). The NPSS product also

addresses these challenges to instructor observation for trainee feedback. The biometric overview of the product

consists of data collection for breathing, and trigger pressure which can influence target zeroing performance. In

addition, the biometric data identifies if stress or anxiety are contributing to performance issues through the

collection of heart rate data, Galvanic Skin Response (GSR) and Electrocardiogram (ECG).

Training Audience Given that the demographic identified by Norrad, Inc. supports the intended use of the NPSS product to an audience of broad scope; to range from private use, law enforcement, the military and private security sectors. This evaluation provides a holistic framework for evaluating the NPSS product. The support for the product to provide merit for a gap to training needs among demographics is defined for the purposes of this evaluation. Several points of rationale exist in support of the NPSS product to meet the identified training needs among venue, safety, and instructor and trainee needs to improve training efficiency and cost. Given the shared commonality among the needs of the target markets, the scope and focus of this evaluation plan does not solely focus on participants but on second layer of consideration, being the shared mutual needs of training that are limited by funding and resources. The purpose of the evaluation plan is to scope the ability of the NPSS product to fill identified training gaps in venue, safety, and training efficiency which relates to cost in training time and misspent ammunition. Budget cuts to both military training and law enforcement require more cost effective resources to aid in training gaps that allow for training in remote venues, as costs of training in formal ranges presents a high cost (Smith & Hagman, 2003). The cost challenge for live marksman training to be available to forces exists in the maintenance and logistics of ranges as well as for the sustainability of ranges that mimic real world military environments (Lepore, B., & United States. Government Accountability Office, 2011). Law enforcement agencies are challenged by training that is funded to meet regulations as opposed to marksmanship training that meets the need of refresher training that can transfer to the rare time where an officer is forced to draw a gun (Veit, J, 2008) Instructors are limited to what they can perceive is taking place when multiple trainees co-exist. A research study funded by the Army to re-establish training guidelines for marksmanship, revealed that many instructors were unable to diagnose Soldiers’ marksmanship problems, which led to training that did not necessarily address the specific problem (James & Dyer, 2011). Another theme was that many instructors misunderstood parts of rifle marksmanship doctrine and inconsistently applied training techniques and procedures (James & Dyer, 2011). Even with one trainee and an experienced instructor, the necessary information needed to facilitate optimal training is not realized without the use of sensors and biometric data. The goal of the NPSS product is to provide this data as a resource to instructors so that they can monitor the information they require for a line of shooters in real time aligned to audio/video and provide marksman trainees feedback that is goaled to progress competency milestones more efficiently. This is specifically relative to training scenarios that mimic real world dynamics, in order to leverage transfer to the real world.

PARTICIPANTS

Participant Characteristics

Domain Variability


Due to the variable market that exists for assessment of the NPSS product among trainees and instructor types,

the evaluation plan supports various scenarios and levels of expertise that exist among these domains. The

evaluation is versatile and the plan or components of the plan can be facilitated for specific populations at the

discretion of cost and business needs among an identified target market.

A. Military

B. Law Enforcement

C. Private sector

D. Personal Use (carrying Novice)

Experience Levels

Experience levels among domains are identified by established criteria. There is variability that exists in describing

marksmanship expertise among domains. For example, the Army has different qualifying parameters among the

Navy or the Airforce. In addition, law enforcement domains have their own set of descriptions among levels of

classifying expertise as does the private sector, including the National Rifle Association (NRA). To address this issue

of variability, a template will serve for the creation of such classifications as these pertain to the specific

demographic of evaluation as noted in Appendix A, Artifact 1 – Template for Parameter Classification: Levels of

Expertise.

Training and Compliance

Certain domains may follow certain regulations or guidelines dictated to the governance of Human Capital. It is

therefore necessary to convey consistency to these considerations for the purposes of training. Therefore, any

preliminary requirements regarding safety training, waivers, or compliance forms should be addressed for the

purposes of evaluation.

Training Objectives

In addition to the variability that exists among the classification of marksmanship experience among domains;

there is also variability in training objectives that serve individual domains. For this reason, all forms will have

training objectives and domains defined. Training objectives also serve to convey the needs and goals of training as

these apply to trainees among their existing and evolved post-training levels of expertise.

Trainees

It is understood that in some domains and among levels of expertise participants may have previous or required

preliminary training gained through classroom requirements for marksmanship fundamentals or individual

experience with weapon types and shooting. Qualifying surveys and marksmen trials will demonstrate pre-

requisite knowledge of skill level for matched assessment pools. In addition, domain specific certifications and

experience will serve to classify skill level. Weapons used for the purpose of gauging expert ranking are also

identified in forms, as these can be variable, but for the purposes of evaluation should be consistent for use in pre-

test and post training assessment.

Skill Level

Subjective assessment does not always serve to convey accurate measures of skill level but do serve to provide the

instructor information regarding the existing knowledge of individual students. See Skill Level Interview Forms -

Trainees in Appendix A, Artifact 2 – Training Participants Skill Level Interview.

Pre-Test – Baseline for Marksmanship Skill

Given the nature of subjectivity, objective measures employ a more reliable baseline of expertise in pretest for

individual participants among the qualifying standards of the domain and training objectives.

Shooting skill level is to be gauged to the training objectives to assess a baseline of participants’ skill and for the

purposes of classifying subject participants into populations reflective of expertise prior to using the NPSS product.


As noted earlier, classifications may be described differently among a variety of domains. A representative

example of classification may exist among levels such as (Novice, Intermediate, Sharpshooter, Marksman, Sniper).

The assignment of classification serves to assist in pre-test and post-test comparison where increases in skill level

demonstrate a mark of improvement in skill acquisition. Weapons used for the purpose of gauging expert ranking

are also identified in forms, these can be variable, but for the purposes of evaluation should be consistent for use

in pre-test and post training assessment.

Defining Levels of Expertise establishes a performance ranking for individual classifications that exist for the

purposes of grouping. Criteria for levels of expertise may leverage the example used in this evaluation or be

gauged according to the specific needs of a domain. A form is provided to assist in defining parameters for the

purpose of establishing criteria for evaluation. See Appendix A, Artifact 3 – Marksmanship Ranking Scorecard.

Leveraging from the range of classifications established in Artifact 1, the Marksmanship Ranking Scorecard serves

as a template for pre-test ranking of participants based on skill level assessed by results of baseline testing set to

training objectives. Baseline testing occurs in a live fire environment consistent to training and post-test in terms of

weapons, ammunition, and conditions that facilitate the objectives of training. The number of sessions in training

for both groups should be established and consistent for the requirement of the stated objective. In this aspect

facilitators must seek agreement on the parameters of training in terms of time allowed and number of sessions

and this agreement must be consistent for both the control group and the experimental group and throughout all

phases of evaluation noted by post-tests. During pre-test, the facilitator will record a standard rate of accuracy as

defined by average score for a shooting session for the stated objective(s). Qualifying surveys and marksmen trials

will demonstrate pre-requisite knowledge of skill level for matched assessment pools. In addition, domain specific

certifications and experience will serve to classify skill level.

Instructors

Qualification is gauged through training certifications, badge qualifications, training experience as it applies to a

specific domain and training objective. In result, the instructor must have qualifications that are acceptable for

training in a specific domain and to specified training objectives. The instructor must possess higher expertise over

training participants. In most instances, the domain targeted for evaluation of the NPSS product will possess their

own set of criteria for qualifying instructors. In some instances, resource constraints may require the use of

instructors with qualifying deficits for the training at hand. The lack of ability for instructors to meet or not meet

criteria to serve as an instructor for the training at hand should be noted on the form to address any potential

barriers to matching instructors to trainees, so as to not bias the study. It is therefore good practice to gather the

information relative to the instructor for the purposes of pre and post-analysis to assess matched resources to

participant groups. See Appendix A, Article 4 – Instructor Skill Level Interview. In some instances, multiple ongoing

large evaluations may require statistical measures to ensure equal pairing to both evaluation groups. In these

instances, percentages based on demographics should be measured for comparable means as noted by the

example in Appendix A2: Instructor Characteristics: Matched Demographics.

Instructor/Trainee Pairing

Instructors must support a level of expertise as gauged through certification or experience and will be classified

according to their level of expertise and aligned to the corresponding needs of trainees and to relevant domains.

Trainees will be paired with certified instructors appropriate to their level of expertise and domain. A call to

volunteers may be required for adequate resource matches to trainees for the instructor pool. Instructors may

serve for both training in the experimental group and control group to ensure groups are consistent in matching

the skillsets of instructors equally. In other instances, where different instructors are used, consistency among

instructor demographics and skills should be validated.


Evaluation Process Overview

The following evaluation strategy uses the widely accepted evaluation methods guided by Kirkpatrick’s Four Level

Evaluation Model. For the purpose of evaluation Levels 1 evaluations are conducted post-training and respectively

represent Trainee Reactions to the Training and Learning Evaluations (Kirkpatrick & Kirkpatrick, 2006). Level 2

evaluation methods and processes are employed within the formal training period. Level 3 evaluations of skill

retention are employed in a specified training period that follows after formal training. Skills will be tested in live

fire demonstration immediately upon conclusion of training and retention skills are tested in a short-term

longitudinal fashion. A control group will also be utilized for all Level 3 assessments to determine the effectiveness

of the NPSS product. The goal of having a control group is to isolate the effect of training effectiveness with the

NPSS product. Level 4 evaluations will be compiled for review and assessment upon completion of data collection

methods for Levels 1-3. Level 4 data will convey to the overall performance metrics that relate to the NPSS product

in result of the evaluation. An integrative approach to assessment also considers the Attitudes of participants. As

noted by research conducted by (Baldwin & Ford, 1988) attitudes of participants can influence skill transfer. A

detailed process plan for Attitude Evaluations is noted in Appendix A.

The framework for this evaluation plan conveys to the assessment of marksmanship with the NPSS product. The

framework is made to be leveraged to a variety of domains and for use for use with future and existing products

relative to marksmanship training.

The purpose of creating an evaluation paradigm that supports best practices in the consideration of criterion,

participant characteristics, and training objectives stems from the need to meet variables that exist among domain

specific considerations. This consideration serves that consistency among such variability be addressed to

eliminate experimental bias and to ensure consistency of evaluation measures among diverse clientele in support

of the NPSS product. The contention is that inconsistent measures of evaluation serve no correlation to assessing

strategy or supporting the product in similar domains. The goal of this plan is to support two primary measures of

consideration. The NPSS tool is cost effective in terms of time to train and serves to reduction of ammunition costs

required for training. The tool itself allows for training in diverse venues.

Training Objectives

This evaluation plan does not account for training plans, however the strategy considers the incorporation of

training objectives that may serve to be considered in culmination to meet the specific needs of the evaluation for

the NPSS product. Qualification parameters and rankings to skill levels are defined by the business needs of the

organization or clientele. The evaluation proposes an adaptable paradigm for performing training evaluations with

the NPSS product to meet the variable investment cost of analysis as it relates to clients.

Single or several training objectives may be indicated that convey to simple or complex goals, respectively

exampled by establishing breathing control for a shot or as complex by marksman skills required for team

performance.

For instance, the training objective may be simple to convey breathing control or complex to a team scenario such

as clearing a room, that requires several training objectives. For this purpose, data collection processes consider

training objectives be identified for the purposes of evaluation with consistency of cost saving variables that

convey to training efficiency as noted by this plan for the consideration of skill acquisition and behavioral transfer.

Complexity ranking for scenarios

This plan considers the creation of scenarios for training purposes. It is recommended that scenarios are evaluated

and aligned to a level of complexity that correlates to the experience classification of the trainees. Several levels of

expertise may exist in a team based scenario. In such cases, the Interpretation of results will account for

performance measures on both an individual basis and as a team. Given the dynamic nature of training needs

aligned to the needs of the client, scenario creation is assumed to be constructed according to the relevance of the


training domain being assessed and the identified training objectives. Given the complex nature of designing

scenarios, and the technological variability that may co-exist with the NPSS product or other technology sources,

scenarios are accounted for the purposes of evaluation parameters described in this plan. Therefore, complexity

itself will not be evaluated as a factor among levels of experience. The level of experience for the trainee however

is considered, and it is recommended that all trainee groups consist of similar levels of experience so as to

eliminate the potential of experimental bias.

It is advised that any training objectives associated with scenario creation follow a classification process to meet

the appropriate skill level of challenge for trainees. To account for this consideration, it is proposed that scenario

complexity be ranked upon subjective assessment of instructors correlated to the experience level of a pool of

trainees based on pre-trial qualification status. These rankings may vary among domains, for the purpose of

providing examples relative to the overview this evaluation plan, the ranking examples will be conveyed as

(Novice, intermediate, Marksman, Sharpshooter, Expert, Sniper). In certain instances, evaluation needs may exist

outside of those addressed in this plan. For this purpose, the plan can be expanded upon to serve organizational

needs or the specific needs of clientele.

Measurement Instruments, Administration Procedures, Reporting

PRELIMINARY MEASURES

USING THE NPSS PRODUCT

It is advocated that a preliminary overview of the NPSS product will be conducted for both trainees and instructors

in the experimental group who need to train with the NPSS tool.

Upon completion of the overview instructors and trainees will be gauged on their comprehension of use with the

tool. An experienced user of the NPSS tool will utilize a criterion form to measure the success of instructors and

trainees using the NPSS tool for their role of required tasks associated with following models for use. The training

content required for use with this evaluation is evolving separately and will be realized at a future date. A sample

high-level criteria is indicated for the purposes of example relative to this evaluation plan.

A sample comprehensive model of qualification for trainees will include the following:

Demonstrates the ability to attach the product to the weapon (Appendix C of the instruction manual)

Demonstrates the ability to configure the product for the prescribed needs of the demonstration.

(Criterion TBD)

o To include parameter settings, target settings and connectivity to devices and the primary server.

Demonstrates the ability to interpret the feedback of the product prior to the demonstration. (Criterion

TBD)

A sample comprehensive model of qualification will include the following for instructors:

Demonstrates the ability to configure the instructor monitor for the trainee(s).

Demonstrates the ability to maneuver the software paradigm for practical use and scenario creation.

(Criterion TBD)

Demonstrates the ability to interpret trainee data for feedback and direction of instruction to trainees

(Criterion TBD)

Upon conclusion of the overview both instructors and trainees will be surveyed relative to their subjective

experience of learning to the use the product (Survey questions TBD).

Liability Waiver


See Appendix H.

PARTICIPANT CHARACTERISTICS Trainee Grouping

Participants in both the experimental and control groups are equally paired on the basis of skill level and

demographics for the purposes of homogeneity among the statistical measures employed.

Instructor/Trainee Pairing

Instructors are paired with training groups on the basis of their domain knowledge, instructor certifications and

experience aligned to the identified training objective(s). See Appendix A, Artifact 4, A2.

Personal Characteristics

Subjective assessment does not always serve to convey accurate measures of skill level but do serve to provide the

instructor information regarding the existing knowledge of individual students. See Skill Level Interview Forms -

Trainees in Appendix A, Artifact 2 – Training Participants Skill Level Interview. Additional demographic data

facilitated by interview with this form should convey to consistent demographics for both groups in terms of age,

gender, disability or corrective lens wearer, and years of experience shooting a weapon. The groups should be

assessed statistically for mean distributions and standard deviations among specified variables consistent with

both groups as noted in Appendix A, AI: Trainee Demographics- Statistically Matched Example

PRE-TEST Pre-Test Evaluation Procedures

Participants will have a 15-minute trial period to familiarize themselves with the weapon in live fire demonstration.

The instructor is refrained from giving instruction or feedback to the trainees outside of safety considerations and

the manual means to fire the weapon. During pre-test, the facilitator will record a standard rate of accuracy as

defined by average score for a shooting session for the stated objective(s). Qualifying surveys and marksmen trials

will demonstrate pre-requisite knowledge of skill level for matched assessment pools. In addition, domain specific

certifications and experience will serve to classify skill level.

Pre-Test Data Collection

Prior to the beginning of training a live-fire pre-test is conducted to baseline skills for comparison to live-fire post-

test to assess skill retention in live fire demonstration. During pre-test, the facilitator records a standard rate of

accuracy as defined by the participant characteristics and average ranking score from the shooting session for the

stated objective(s). The number of sessions in training for both groups should be established and consistent for the

requirement of the stated objective. In this aspect facilitators must seek agreement on the parameters of training

in terms of time allowed and number of sessions and this agreement must be consistent for both the control group

and the experimental group. These agreements must be consistent for pre-test and all Level 3 evaluations.

Weapons used for the purpose of gauging baseline experience level rankings are also identified in forms and are

consistent to use in pre-test and post-test trials.

Pre-Test Evaluation Instruments & Reporting Process

It is recommended that an independent sample t-test assuming equal variance between the experimental group

and control group be conducted for the baseline pre-test qualification scores (min–max) specified by the

Marksmanship Ranking Scorecard. The groups should be evenly matched according to their qualifying range of

expertise as noted by accuracy means obtained by the final qualifying score. This score is indicated on the


Marksmanship Ranking Scorecard (Appendix A, Artifact 3). It is best if this test is conducted in a preliminary fashion

to assure the experimental and control groups are evenly matched.

ADDITIONAL MEASURES

ATTITUDES

Trainee surveys are created to assess attitudes towards confidence and the marksmanship training itself. Surveys

are administered after pre-test, at post-test upon conclusion of training. The objective is two-fold: (1) Did

confidence and attitude levels influence training? (2) Was there a scalable difference in both confidence and

attitudes upon conclusion of training between the Experimental and Control groups?

Kirkpatrick points out that transfer involves several factors (1) training must be effective to promote skills, (2) skills

learned are retained by performing these on the job, (3) organizations must have processes that support the

retention of skills or the desired performance (Kirkpatrick & Kirkpatrick, 2006). Baldwin and Ford (1988) reveal that

another factor of skill transfer is supported by the characteristics of trainees in terms of ability, personality and

motivation (p. 5). These same characteristics play integrally into the success of training. Attitudes play into

motivation, and can exist based on personal reflection or outward to peers and organizations. Research notes that

learners with positive attitudes towards peers and/or the organization are more likely to succeed in learning and

maintaining new skills (Dick, Carey & Carey, 2014, p. 98). Schunk, Meece and Pintrich, (2014) note that

motivational beliefs are constructed by the learners based on their own self perceived capability of the skill to be

learned, how it should be learned, goals, self-efficacy, attributions, and self-interest (p. 41). Attribution also plays a

role on individual perceptions of self-efficacy based on the individual’s expectation of success or failure and their

perceived causes of an outcome (Schunk, Meece and Pintrich, 2014, p. 82). People appraise their own self-efficacy

from the success or failure of their performance, through observational comparison of others, and persuasion, and

these have an effect on the amount of invested effort and persistence to perform a task (Schunk, Meece, Pintrich,

2014, p. 145).

Gauging participant attitudes is therefore an important determinant to the results of evaluation in both foresight,

through baseline pairing of skills and hindsight to assess attitudes towards the training experience and assess

anomalies or insights to the results of evaluations. Based on the ability of trainees, it is important to baseline

learners to achievable objectives. Learners are also likely to disengage in achievable tasks that afford no personal

value in terms of reward or interest, therefore both expectancies and values are considerations to the training

objective and the participants matched to it based on skill level (Schunk Meece & Pintrich 2014, p.49).

ATTITUDE ASSESSMENT: EXPERIMENTAL AND CONTROL GROUPS

Evaluation Procedures

It is recommended that attitude and confidence surveys be administered in an anonymous fashion to allow for

honest feedback (Kirkpatrick & Kirkpatrick, 2006, pp. 28, 58). The facilitator of this survey alerts participants that

the survey is a requirement, and instructions convey that survey responses hold no personal or professional

liability. The instructor conveys the instructions that “You are participating in a study that evaluates the use of a

tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the

training you received. Your opinions are very important, honest responses are encouraged for the proper

assessment of this study”. The survey is presented upon conclusion of Pre-Test and upon conclusion of training

after Post-Test and upon conclusion of retention training. The survey is administered to both the control group and

the experimental group.

Data Collection

It is recommended that all Attitude and confidence data will be collected in an anonymous fashion to allow for

honest feedback and collected immediately upon conclusion of training to ensure 100 percent of responses are


obtained and utilize a Likert scale for quantifiable measurement (Kirkpatrick & Kirkpatrick, 2006, pp. 28, 58). Four

of these surveys will be conducted, one upon the conclusion of Pre-Test and another at the conclusion of training

upon conclusion of Post-Test. The others will take place at the 4 and 8-week retention testing. A facilitator will

guide trainees to the Attitude assessments. The assessments will be obtained via online manual input and stored in

an internal database. Optionally these forms can be made available for printouts that may serve for the purpose of

manual data collection. The data will be collected in an anonymous fashion to allow for honest feedback and

collected immediately upon conclusion of a module to ensure 100 percent of responses are obtained and utilize a

Likert scale for quantifiable measurement (Kirkpatrick & Kirkpatrick, 2006, pp. 28, 58). Attitude surveys for both

Pre-Test and Post-Test are found in Appendix A.

Evaluation Instruments

Attitude data will be obtained using a 5-point Likert rating system. The 5-point Likert rating system will provide an

appropriate range of responses relative to the marksmanship training content, instructor and relevance of the

training to the training needs, and the tools and resources provided for training. Questions are consistent in

representation for both groups. This method of questioning is posed to assess Attitudes in Confidence.

Each question also contains a section for optional comments or suggestions. See sample Attitude assessments in

Appendix A for the experimental group and the control group for both Pre-Test and Post-Test.

Trainee surveys are created to assess attitudes towards confidence and the marksmanship training itself. Surveys

are administered after pre-test, at post-test upon conclusion of training. All Attitude Surveys are found in Appendix

A, A3-A10.

Data Reporting

Data reporting serves the following questions: (1) Did confidence and attitude levels influence training? (2) Was

there a scalable difference in both confidence and attitudes upon conclusion of training between the Experimental

and Control groups?

The attitude data from both groups will be analyzed in an ordinal fashion for each question between groups

among the conditions that follow Pre-Test and Post-Test. For each question the median response will be calculated

as well as the Inter-Quartile Range (IQR) of each question item. The IQR will assess if responses are consistent by

being clustered together; or inconsistent as being scattered across the range of possible responses. This data will

be statistically evolved utilizing SPSS or any choice of statistical software. The results of IQR for each question will

be compared for both the experimental and control group and among the conditions of Pre-Test and Post-Test.

This first measure considers attitudinal differences between the control and experimental groups during pre-test,

the second considers the same at Post-Test, and the third attempts to derive the influence of training for both

group types on attitude and confidence by comparing attitudes for each group based on the results of Pre-Test and

Post-Test. The results should be reported relative to the consideration of confidence and attitude for both groups

in a comparable display of data. The Mann-Whitney U Test may serve best for non-parametric measures between

groups and conditions

KIRKPATRICK MODEL

LEVEL 1 – GAUGE SATISFACTION

Level 1 Research Questions

Trainee Survey

1. Overall, was training more relevant for those participants in the experimental group as compared to the

control group?

2. Overall, were the tools or resources used to assist training in the experimental group more assistive to

self-perceived performance than those used in the control group?


3. Overall, were the instructors in the experimental group more efficient at their coaching task than those in

the control group?

Instructor Survey

4. How frequently did the instructor’s leverage the data provided by the NPSS tool during instruction?

(Optional)

Level 1 Evaluation Procedures

Experimental Group & Control Group

The facilitator of this survey alerts participants that the survey is a requirement, and instructions convey that

survey responses hold no personal or professional liability. Honest responses are encouraged for the proper

assessment of this study. It is recommended that all level 1 Reaction data will be collected in an anonymous

fashion to allow for honest feedback and collected immediately upon conclusion of training to ensure 100 percent

of responses are obtained and utilize a Likert scale for quantifiable measurement (Kirkpatrick & Kirkpatrick, 2006,

pp. 28, 58).

Experimental Instructor Group (Optional)

This type of assessment may be limited for use as you need a pool of instructors to gain accurate statistical

measures and requirement may be limited to availability depending on the size of the study and the amount of

available instructors.

Upon immediate conclusion of Level 2 training the facilitator will overview the survey process with instructors The

facilitator will convey that purpose of this survey is for instructors to evaluate the efficiency of instructing with the

NPSS product. The survey is assistive in defining the benefits or needs of product improvement, so please be frank

in your answers. Read each item carefully. Reflect on the data feed described. Determine what level of frequency

you accessed the described data for instructional purposes. Place an (x) under your corresponding choice. If you

note a lack of use for a survey query, place an (x) under the column that indicates “was not useful to the training”

If you have any comments or suggestions relative to an item listed, please indicate these below in the comments

section.

Level 1 Data Collection


Once the Level 2 training is completed, reaction data will be collected from the trainees. Initially, a facilitator will

guide trainees to a quick Level 1 Satisfaction assessment. The assessments will be obtained via online manual input

and stored in an internal database. Optionally these forms can be made available for printouts that may serve for

the purpose of manual data collection. All level 1 Reaction data will be collected in an anonymous fashion to allow

for honest feedback and collected immediately upon conclusion of a module to ensure 100 percent of responses

are obtained and utilize a Likert scale for quantifiable measurement (Kirkpatrick & Kirkpatrick, 2006, pp. 28, 58).

Training surveys for Level one data are found in Appendix B.

Experimental Instructor Group (optional)

Upon immediate conclusion of Level 2 training the facilitator will administer and collect survey data to instructors.

Initially, a facilitator will guide trainees to a quick Level 1 Satisfaction assessment. The assessments will be

obtained via online manual input and stored in an internal database. Optionally these forms can be made available

for printouts that may serve for the purpose of manual data collection. All level 1 Reaction data will be collected in

an anonymous fashion to allow for honest feedback and collected immediately upon conclusion of a module to

ensure 100 percent of responses are obtained and utilize a Likert scale for quantifiable measurement (Kirkpatrick

& Kirkpatrick, 2006, pp. 28, 58). Training surveys for Level one data are found in Appendix B.


Level 1 Evaluation Instruments

Experimental & Control Groups

Trainees

Level 1 data will be obtained using a 5-point Likert rating system. The 5-point Likert rating system will provide an

appropriate range of responses relative to the marksmanship training content, instructor and relevance of the

training to the training needs, and the tools and resources provided for training. Questions are consistent in

representation for both groups. This method of questioning is posed to assess differences between the

experimental and control groups as these relate to the reactions of training with and without the NPSS product.

Each question also contains a section for optional comments or suggestions. See sample Level 1 assessments in

Appendix B for the experimental group and the control group.

Experimental Instructor Group (optional)

Level 1 data will be obtained using a 5-point Likert rating system. The 5-point Likert rating system will provide an

appropriate range of responses relative to the marksmanship training materials and resources, trainees and

relevance of the training to the training needs for the specified objective. The Scale Range of Frequency consists

of: [1 – Never, 2 – Seldom,3 – Occasionally, 4 – Often, 5 – Always]. Each question also contains a section for

optional comments or suggestions. See sample Level 1 assessments Appendix B-B3: Instructor Feedback on

Frequency of Use of NPSS Biometric Information-Experimental Group.

Level 1 Reporting Process


The Level 1 Reaction data for each question will be analyzed in an ordinal fashion for each question and between

groups. For each question the median response will be calculated as well as the Inter-Quartile Range (IQR) of each

question item. The IQR will assess if responses are consistent by being clustered together; or inconsistent as being

scattered across the range of possible responses. This data will be statistically evolved utilizing SPSS or any choice

of statistical software. The results of IQR for each question will be compared for both the experimental and control

group. The results should be reported relative to the assigned topics of Relevance, Instructors, and Tools for both

groups in a comparable display of data.

Experimental Instructor Groups (optional)

The use of this form depends on the size of the study, whereas a large sample of instructors supports unbiased

analysis. The form may be used for correlation to other studies to gauge instructor feedback on the use of the

data associated with the NPSS product. An example of this form is Appendix B-B3: Instructor Feedback on

Frequency of Use of NPSS Biometric Information-Experimental Group.

The Level 1 Reaction data for each question will be analyzed in an ordinal fashion for each question. For each

subjective question the median response will be calculated as well as the Inter-Quartile Range (IQR) of each

question item. The IQR will assess if responses are consistent by being clustered together; or inconsistent as being

scattered across the range of possible responses within the group being evaluated. This data will be statistically

evolved utilizing SPSS or any choice of comparable statistical software. The results should be reported relative to

the assigned topics of Audio/Video and the Biometric Data categories of:

Breathing during the shot,

Trigger pressure applied during the shot.

Heart rate during the shot


Pupil dilation/eye movement during the shot

Weapon movement during the shot (acceleration, gyroscope effect)

Galvanic Skin Response - GSR

Tacit Response – EMG

Electrocardiogram (ECG) Detects increased heart rate

Shooter Movement during the Shot

The results of IQR for each question may be compared to gauge consistency to objective responses gained in the

feedback data for the Level 2 Skill Acquisition, experimental group where the data collection form requires the

instructor to note forms of data used for feedback.

LEVEL 2-SKILL ACQUISITION


5. How much improvement in performance did training have from the onset to the end for the experimental

group over the control group as noted by improvements (accuracy, time to complete training)?

6. What is the estimate of saved ammunition in the experimental group as compared to the control group?



Evaluation procedures start upon completion of assessing baseline participant and instructor characteristics, pre-

tests and any preliminary training required for use with the NPSS product. Participants will have a 15-minute trial

period to familiarize themselves with the weapon in live fire demonstration. The Instructors will have been briefed

on the proper use of the form. During training, the Instructor (potentially assisted by a facilitator) will record

accuracy, as defined by the training session for the stated objective(s). Venue may be a consideration for the

purpose of this training in that the instructor in the experimental group may be conducting the training session in a

venue remote from training participants. The number of sessions in training for both groups should be established

and consistent for the requirement of the stated objective. In this aspect facilitators must seek agreement on the

parameters of training in terms of time allowed and number of sessions and this agreement must be consistent for

both the control group and the experimental group. Several outcomes relative to training efficiency with the NPSS

product are determined by this training to include: Time Efficiency, Performance Improvement, and Ammunition

Usage.

Time Efficiency

Time to train is established by the time it takes for the instructor and trainees to complete the parameters

prescribed in training, to include the objectives and the number of sessions prescribed. It is therefore the time it

takes to train to the prescribed number of Rounds or Sessions. The results will determine the efficiency of the

NPSS product for the purpose of diminishing training time. Time is recorded in seconds.

Performance Improvement

Performance improvement is realized by determining if training with the NPSS product (experimental group)

realized higher marksmanship scores over the control group. Marksmanship scores convey to the number of target

hits during the Level 2 training session.

Ammunition Usage

Ammunition usage is the comparable difference in misspent ammunition between the experimental group and the

control group as determined by target misses during the Level 2 training session.



Experimental Group and Control Group

The NPSS product has the capability for the remote collection of data, however for consistency between groups,

data will be collected manually or digitally with forms in Appendix C. Thereby data can be collected via software

for online or database storage. The data collected from both groups will be statistically evolved utilizing SPSS or

any choice of statistical software. Facilitators will ensure all forms are properly completed and query any

discrepancies to instructors.



The Marksman Score Card used for Level 2 evaluations is leveraged from common marksmanship score cards used

for training. The fields for manual or digital entry allow for the instructor to data capture individual records for the

training session to include: The time start and End of Training, Target Hits, Target Misses, No Fire, and final

marksman totals and scores for required data fields. In addition, data fields include the recorded range of the shot

in meters, and the time it takes for the shooter to make the shot in seconds. Shots can be scored in options of

single or group to meet the variable needs of training objectives listed on the form. As previously mentioned

identified training objectives should be consistent throughout out all study trials as well as Unique Identifiers

associated with the study. In addition, the baseline rank for each trainee is made available for the purpose of

instructor knowledge, data collection and later analysis relative to the study.

The form requires the instructor to note feedback provided to trainees relative to performance correction during

training for both the experimental and control groups. The form for the experimental group varies to include

additional data collection relative to noting any assistive data received from the NPSS product to diagnose

performance problems. For both groups, the number of targets hit and missed are totaled for each trainee upon

completion of training. Instructors in the experimental group will additionally total the use of any assistive data

conveyed by the NPSS product.

The total number of hits convey to a total score and an assigned ranking level established at the onset of the study.

The total number of misses conveys to spent ammunition and is totaled on the form. The total time to train is

established by the difference between the start and end of a training session. All Level 2 Evaluation Instruments

are found in Appendix C.



An overview of each variable is depicted for the purpose of statistical analysis and reporting. The analytical

processes recommended support the following questions:

How much improvement in performance did training have from the onset to the end for the experimental

group over the control group as noted by improvements (accuracy, time to complete training)?

What is the estimate of saved ammunition in the experimental group as compared to the control group?

The recommendation serves a basic understanding of training effectiveness based on the reflective research

questions. Additional analysis may evolve more in depth understanding based on the needs of business objectives

and certain clientele. In addition, certain instances of analysis may reveal data anomalies that require additional or

alternative statistical measures, it is expected that these will be handled on an individual basis.

Time Efficiency


Relative to the variables of time to train, the (the start – end time) is converted to seconds for a fine tuned gauge

of analysis. The time for both the experimental and control group participants is statistically analyzed with an

independent samples T-Test. This measure serves to determine if training with the NPSS product decreased

training time.

Performance Improvement

The number of hits for the experimental groups are compared also with an independent samples T-Test to gauge

performance differences relative to increased scores overall for use with the NPSS product.

Ammunition Usage

Relative to the correlation of misspent ammunition between both the experimental group and the control group

an independent samples T-Test to statistically analyze differences in ammunition use between both groups.

Biometric Data

This data serves as a measure of frequency of use within the experimental group by the instructors. Independently

the data use frequency should be totaled for each category. A Frequency Distribution will assist in determining

varying levels of use consistent among instructors for each category of information the NPSS product provides.

This data can be used as an objective measure to the subjective assessment of Frequency of NPSS data resulting

from the analysis of Level 1 Instructor Survey found in Appendix B-B3: Instructor Feedback on Frequency of Use of

NPSS Biometric Information-Experimental Group. These comparisons are optional but may serve to provide useful

information relative to the utility of the product from the perspective of instructors.

LEVEL 3-SKILL RETENTION


7. Did training in the experimental group transfer to improvement in marksmanship performance over the

control group as noted by range qualification scores gained from pre-test to post-test?

8. Can NPSS (experimental group) be used to sustain skills over time and yielding higher or equal retention in

the experimental group compared to the control group?



Post-test

Evaluation procedures start the day after training is complete. A domain expert will oversee the marksman post-

test and convey an overview demonstrating the purpose of this post-test is to determine the efficiency of training

for a product, the number of sessions and time parameters are conveyed. It will be explained that an average of

range scores will be taken to assess experience level based on the established number of sessions defined for post-

test. Participants will have a 15-minute trial period to familiarize themselves with the weapon in live fire

demonstration

Four week & Eight Week Retention Tests

The procedures for 4 & 8-week retention testing convey in the same manner as post-test. In this instance, the

difference is that the experimental group will use the NPSS mounted mobile device product in both the 4-week


and 8-week post training assessment. The tool will be used in conjunction with a live fire demonstration consistent

with the control group. The control group will perform the same evaluation using traditional resources for the

specified domain. An image of the NPSS product can found in Appendix G.

.



Post-Test

Leveraging the data obtained from baseline participant characteristics the initial ranking in pre-test is identified.

Pre-test scores resulting from live-fire demonstration provide a baseline of marksmanship performance for

comparison to live-fire post-test and to assess long term skill retention in live fire demonstration. Relative to the

stated objective(s), the number of sessions in post-test and retention tests for both groups should be established

and consistent. In this aspect facilitators must seek agreement on the parameters of training in terms of time

allowed and number of sessions and this agreement must be consistent for both the control group and the

experimental group. These agreements must be consistent for pre-test and all Level 3 evaluations.

Shooting skill level upon conclusion of training is to be gauged to the training objectives to assess the effect of

training on participants’ skill for both the experimental and control groups. The assignment of classification serves

to assist in pre-test, post-test, and retention skill comparison where increases in skill level demonstrate a mark of

improvement in skill acquisition received from training. Weapons used for the purpose of gauging expert ranking

are also identified in forms and are consistent to use in all trials. Both post-test and pre-test consist of live-fire

demonstration, however the form allows for variable environments. It is recommended that if an alternative is

used that there be consistency in pre-test and post-test criteria.

Post-test retention assessments are provided 4 and 8 weeks after the conclusion of training and the initial post-test. Data Collection Procedures are consistent for all post-test assessments. Forms for post-test data collection are found in Appendix D. The data (manual or digital) is collected during training, compiled and stored in a database for future analysis.



Marksmanship improvement is gauged from pre-test, the initial post-test and the 4 and 8-week post-training retention tests. Standard practice of evaluating skill acquisition recommends the use of control groups, the time allowance or behavioral change to occur, and repeat evaluations (Kirkpatrick & Kirkpatrick, 200, p. 53). The first measure considers differences between the baseline pre-test and post-test determines if training conducted in the experimental group assisted in the improvement of qualification scores as compared to the control group. Higher mean targeting scores gained from pre-test to post-test indicate improvement metrics for each group. The difference between improvement scores yielding from the experimental group and control group determine performance effectiveness of the product (experimental group). Performance improvement is realized when participants receive higher marksmanship scores on the post-test than they did on the pre-test. These measures serve to support Research Question 7 “Did training in the experimental group transfer to improvement in marksmanship performance over the control group as noted by range qualification scores gained from pre-test to post-test?”. In order to assess the usefulness of the NPSS product to promote skill retention, the 4 and 8-week retention assessments will consist of a comparison of the experimental and control groups. The participants in the experimental group will be fitted to use the NPSS attachment, which consists of Mobile touch device attached to a


handgun or rifle or at the user’s wrist (see Appendix G). In this instance the device will serve for personal support of marksmanship in the absence of an instructor. The level of biometric feedback is determined by the parameters established in the evaluation as these align to training objectives. The assignment of groups is consistent with prior testing, whereas those initially assigned to the experimental group have received training with product use. Those previously assigned to the control group serve the same assignment for the purpose of post-training retention tests. The NPSS product has the capability for the remote collection of data, however for consistency between groups data will be collected manually with forms in Appendix D. Experimental group data can be validated against remote data as needed. Relative to retention skill data the post-test measure completed at the end of training will serve as the baseline to the 4-week post training retention test and the 8-week post training retention test. Higher scores received in the 4-week post training retention test indicate improvement metrics for each group (experimental & control). Performance retention is realized when participants receive equal or higher marksmanship scores on the 4-week post training test than they did on post-test. Higher scores received in the 8-week post training retention test indicate improvement metrics for each group (experimental & control). Performance retention is realized when participants receive equal or higher marksmanship scores on the 8-week post retention training test than they did on the 4-week post retention training test. These measures serve to support Research Question 8:” Can NPSS (experimental group) be used to sustain skills over time and yielding higher or equal retention in the experimental group compared to the control group?” Level 3 instrumentation is found in Appendix D.


Data measures in support of questions 7 an 8 will calculate: (1) The difference between pre-test and post-test upon immediate conclusion of training and (2) the 4-week post training retention scores, (3) the difference between the 4-week post training retention scores and the 8-week post training retention scores. are calculated to determine any difference in skill retention strength among the two groups (experimental and control). The recommended statistical measure to compare the experimental and control groups will consist of Independent Sample T-Tests supported initially by F-tests to assure equal variance for all comparisons. Comparisons made for each group, where participants metrics are consistent to either the control group or the experimental group, will be measured for performance gains under the variables of Pre-Test and Post-Test. In this aspect Control group participants scores will be compared for differences between Pre-Test and Post-Test and the Experimental group participant scores will be compared for the same differences separately. The measure of statistical analysis will be a Dependent Measures T-Test. Certain instances, may have data anomalies that require additional or alternative statistical measures, it is expected that these will be handled on an individual basis. Certain domains may request specialized intervals to assess retention.

LEVEL 4-COST BENEFIT

Level 4 Questions

What cost benefit does training with the NPSS product yield to stakeholder? (Time to train, satisfaction,

ammunition cost, improved accuracy and retention and satisfaction)

Level 4 Benefits

The accumulation of data from Levels 1-3 data will be interpreted and extend to Level 4 evaluations to determine

the benefit of training for individual clients as noted by the outcome of analyses that promote training efficiency

with the NPSS training product (Kirkpatrick & Kirkpatrick, 2006, p.65). Level 4 analysis will also provide information

that serves Norrad, Inc., in terms of product satisfaction. The feedback gained from study participants and the

results of analyses of Levels 1-3 will offer insights to the potential needs of evolving the product.


The condition of domain variability, that exists in the construction of this plan, does not allow for the benefits to a

stakeholder to be formulated in detail but hypothetically some considerations are recommended.

Performance improvement is realized when use of the NPSS product enhances marksmanship skill and

preparedness for the task and stakeholder it supports. In order to assert a realized benefit of the product to the

client, training effectiveness realized by use of the NPSS product should be exampled by study results in favor of

the following:

Improved confidence of training participants noted by attitudes (Level 1)

Satisfaction of Training in terms of Relevance, Instructor Satisfaction, and satisfaction with the tool

(Level 1)

Satisfaction of Instructors using the NPSS product to improve training effectiveness and decrease

workload (Level1)

Effective Use of Ammunition during training (Level 2)

Diminished time to train (Level 2)

Improved scores during training (Level 2)

Improved Transfer from training to real world as noted by score differences between Pre-Test and Post-

Test administered upon completion of training. (Level 3).

The demonstrated capability of the NPSS product to promotes skill retention over traditional domain

culture paradigms as noted by retention assessment (Level 3)

The comparative analyses of the NPSS product to traditional training realized by this evaluation plan will yield

significance to the applicable benefits of the product based on percentiles. These percentiles can convey to cost-

benefits for the domain they serve. For instances relative to results that support no statistical difference or a

positive outcome; one can assess the cost of training with the product as opposed to traditional training methods.

Tangible Benefits

Tangible benefits are often realized by conducting a Cost-Benefit Analysis. For instance, relative to the results of

training with the NPSS product, a client may assess the savings they could realize in terms of ammunition cost,

training efficiency and the time to train, and weigh this relative to the investment in the NPSS product. By example

a client will assess the costs of projected misspent ammunition that result from not investing in the product

against the investment cost. A calculation that applies to the efficiency of training time relative to personnel costs

can be conducted in the same manner. The client will consider the investment in the NPSS product correlated to

not investing in the product and paying more in man hours for personnel to deliver traditional training. Results of

this evaluation can also assume that if the NPSS product can be used to retain or improve training skills, there is a

cost benefit associated with the NSS product in terms of personnel and venue training costs. The culminating

savings can be estimated over years.

Intangible Benefits

Intangible benefits are often difficult to measure as they pose indirect connections to positive results. Attitudes

and satisfaction improved by use with the NPSS product can convey to improved skills. Research notes that

attitudes play into motivation and learners with positive attitudes towards peers and/or the organization are more

likely to succeed in learning and maintaining new skills (Dick, Carey & Carey, 2014, p. 98). This conveys to

satisfaction of training and the attitudes that are influenced by training. Therefore, we could say there is a benefit

to employee dedication to the task and employee retention that convey indirectly to organizational benefits.

Planning Documents

To assist in the planning of evaluation, supporting documents that represent Data Reporting Recommendations

are found in Appendix E, and a Planning Chart can be found in Appendix F.


REFERENCES:

Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63-105. Retrieved February 22 from: http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6570.1988.tb00632.x/pdf

Chung, Gregory K. W. K., Nagashima, Sam O., Delacruz, Girlie C., Lee, John J., Wainess, Richard, & Baker, Eva L.

(2011). Review of Rifle Marksmanship Training Research. CRESST Report 783, National Center for Research on

Evaluation, Standards, and Student Testing (CRESST). Retrieved from:

http://files.eric.ed.gov/fulltext/ED520427.pdf

Dick, W., Carey, L. & Carey, J.O. (2009). The systematic design of instruction. Upper Saddle River, NJ: Pearson Education, Inc. James, D., Dyer, J., & U.S. Army Research Institute for the Behavioral Social Sciences. (2011). Rifle marksmanship

diagnostic and training guide (Research product; 2011-07). Arlington, Va.]: U.S. Army Research Institute for the

Behavioral and Social Sciences. Retrieved from: http://www.dtic.mil/dtic/tr/fulltext/u2/a544533.pdf

Kirkpatrick, Donald; Kirkpatrick, James (2006). Evaluating Training Programs: The Four Levels. Retrieved from

http://www.eblib.com

Lepore, B., & United States. Government Accountability Office. (2011). Military training DOD's report on the

sustainability of training ranges meets annual reporting requirements but could be improved. Washington, DC: U.S.

Govt. Accountability Office.

Schunk, D.H., Pintrich, P.R., & Meece, J.L. (2008). Motivation in education: Theory, research, and applications.

Upper Saddle River, NJ: Pearson Education, Inc.

Smith, M., Hagman, J., L-3 Communications Corp Herndon VA Link Simulation Training Div., & U.S. Army Research

Institute for the Behavioral Social Sciences. Reserve Component Training Research Unit. (2003). Using the laser

marksmanship training system to predict rifle marksmanship qualification (Research report (U.S. Army Research

Institute for the Behavioral and Social Sciences); 1804). Alexandria, Va.: Reserve Component Training Research

Unit, U.S. Army Research Institute for the Behavioral and Social Sciences.

Veit, J. (2008, November). Unsafe at any Distance‐ The Failure of Police Handgun Training. Retrieved FEBRUARY 27, 2010, from: http://ohioccw.org/200811244136/unsafe-at-any-distance-the-failure-of-police-handgun-training.html

http://onlinelibrary.wiley.com/doi/10.1111/j.1744-6570.1988.tb00632.x/pdf

http://files.eric.ed.gov/fulltext/ED520427.pdf

http://www.dtic.mil/dtic/tr/fulltext/u2/a544533.pdf

http://www.eblib.com/


Appendix A: Measurement Instruments –

Baselines/Participants

PRELINMINARY ARTIFACTS

Artifact 1: Parameter Classification of Marksmanship

PARAMETER CLASSIFICATION FOR MARKSMANSHIP

Unique ID: (Please present a numeric or alpha

unique identifier for this training evaluation) Note:

Use this ID consistently throughout evaluative

materials

Date: Click or tap to enter a date.

Domain: (briefly describe branch and or unit of the

domain being evaluated)

Supporting Standard:

(Provide a reference supporting identified rankings for

measures of standard accuracy for the domain of

evaluation. )

Training Objective (s): (Outline objectives for training)

Weapon Description: (Identify the weapon type used for this training)

RANKING CLASSIFICATION (list appropriate

ranking classifications) see examples

RANGE OF CLASSIFICATION (Indicate numeric ranges

of accuracy for the specified ranking) see examples

Expert 160-200

Sharpshooter 120-159

Marksman 80-119

Unqualified 79 or below or less than 24 hits

Artifact 2 – Training Participants Skill Level Interview


Training Participants Skill Level Interview Assigned Skill Level: (completed by interviewer)

Unique ID assigned to this study:

Name of Interviewer:

Date of Interview:

Specify domain for training:

Describe the Training Objective (s):

Identify weapons used for training:

Personal Information Name or Unique ID assigned to Participant: Age: 18-24 25-31 32-38 39-45 46-52 53+

☐ ☐ ☐ ☐ ☐ ☐

Duty Position: Gender: Male ☐ Female ☐

Phone: Wear Corrective Lenses: Yes ☐ No ☐

Email Address Disability: No ☐ If Yes ☐ Describe:

Skills Review Provide Knowledge and Skills that exist for

training to this task (as noted by the domain

relent to the evaluation):

What training certifications does the

participant possess?

What badge qualifications if any does the

participant hold?

How much training experience does the

participant possess relative to the identified

training objective (s)? Explain:

How many years of experience do you

have shooting a weapon?

☒ ☐ ☐ ☐ ☐ ☐ ☐

0 1-3 4 -7 8-10 11-13 14-16 17+


Appendix A: Artifact 3 – Marksmanship Ranking Scorecard.

Marksmanship Ranking Scorecard: Pre-Test

Unique ID: (Please present a numeric or

alpha unique identifier for this training

evaluation) Note: Use this ID consistently

throughout evaluative materials

Date: Click or tap to

enter a date.

Trainee Name or

Unique ID:

Final

Ranking:

Training Objective (s): (Outline objectives

for training)

Domain: (briefly describe branch and or unit of the domain being

evaluated)

Testing Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☒

Post-training ☐

phase

Post-training ☐

phase (post 4 weeks)

Post-training ☐

phase (post 8 weeks)

Supporting Standard: (refer to reference document or resource in

support of the raking standards)

Describe the weapon used for training:

Indicate range (meters) required for training objective:

(Provide a visual reference conveying identified rankings for

measures of standard accuracy for the domain of evaluation.) see

example below

Expert 160-200


Marksman 80-119


Session Number RANKING (Assign a measure of

ranking as noted by the supported

standard)

RECORDED STANDARD

ACCURACY

1 Unqualified 60 with 20 hits


3 Marksman 92

4

Appendix A, Article 4 – Instructor Skill Level Interview


. Instructor Skill Level Interview Meets Criteria ☐ check here Does not meet criteria ☐ check here


Date of Interview:

Name of Interviewer:

Interviewer Title/Rank:

Personal Information

Name or Unique ID assigned to Instructor: Age: 18-24 25-31 32-38 39-45 46-52 53+

☐ ☐ ☐ ☐ ☐ ☐

Duty Position: Gender: Male ☐ Female ☐

Phone Wear Corrective Lenses: Yes ☐ No ☐

Email Address Disability: No ☐ If Yes ☐ Describe:

Training Overview

Specify domain for training:

Specify weapon used for training:

Describe the Training Objective (s):

Skills Review

Provide Knowledge and Skills that exist for instructing to this task (as noted by the domain

relevant to the evaluation):

What training certifications does the instructor

possess?

What badge qualifications does the instructor

hold?

How many years of training experience does the

participant possess relative to the identified

training objective (s)?

Identify any gaps the potential instructor has

relative to training to objectives (as noted by a

lack of knowledge, skills, and/ or certifications).

How many years of experience do you have

instructing marksmanship skills?

☐ ☐ ☐ ☐ ☐ ☐ ☐

0 1-3 4 -7 8-10 11-13 14-16 17+

How many years of experience do you have with

the specified weapon for this training?

☐ ☐ ☐ ☐ ☐ ☐ ☐

0 1-3 4 -7 8-10 11-13 14-16 17+

Interview Section: (Interview instructor to discuss any gaps to experience that may not be

identified in the form required to meet training objectives)

List gaps:


Record instructor response:

A1: TRAINEE DEMOGRAPHICS-STATISTICALLY MATCHED EXAMPLE

TRAINEE DEMOGRAPHICS: STATISTICALLY MATCHED SAMPLE

Assigned Skill Level for Matched Group: Note skill level assigned for this pairing. (Novice, Sharp shooter etc.)

Training Objective:


Experimental Group Control Group

Age% Years

18-24 16 17

25-31 25 23

32-38 18 17

39-45 26 25

45-52 10 14

53+ 5 4

Gender %

Male 92 92

Female 8 8

Corrective Vision %

Yes 25 25

No 25 25

Disability %

0 0

Years of experience shooting a weapon%

0 4 4

1-3 13 13

4-7 21 23

8-10 19 17

11-13 22 21

14-16 21 22

17+ 0 0

Optional: Mean and Standard Deviation for each category


A2: INSTRUCTOR CHARACTERISTICS: MATCHED DEMOGRAPHICS


INSTRUCTOR DEMOGRAPHICS: STATISTICALLY MATCHED SAMPLE

Instructor Pool Size: 20 (note number of instructors available)

USE THIS FORM TO VALIDATE INSTUCTORS ARE EQUALLY PAIRED TO BOTH GROUPS/OR TO NOTE DIFFERENCES in multiple ongoing large size evaluations

Training Objective:


Experimental Group Control Group

Age% Years

18-24 1 2

25-31 25 25

32-38 28 27

39-45 16 17

45-52 16 16

53+ 14 13

Gender %

Male 8 7

Female 2 3

Corrective Vision %

Yes 50 50

No 50 50

Disability %

0 0

Years of experience instructing

marksmanship skills.

0 4 4

1-3 3 3

4-7 6 8

8-10 29 28

11-13 12 11

14-16 23 22

17+ 23 24

Years of experience with the specified weapon for training marksmanship

skills%

0 4 4

1-3 3 3

4-7 6 8

8-10 29 28

11-13 12 11

14-16 23 22

17+ 23 24


Optional: Mean and Standard Deviation for each category

A3: PRE-TEST ATTITUDES-CONTROL GROUP


PRE-TEST – ATTITUDES: CONTROL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.

Please complete this form and provide your input to assist us to evaluate our efforts. No liability exists for your honest response. Any comments and suggestions you provide will better assist in the evaluation of the training and tools. Unique ID assigned to this study: Training Domain:

Instructor Name or Unique ID: Date: 00/00/0000

Objective(s) Associated with Training: (should be pre-filled)

1.How confident are you that you will like this

training?

Very Much Somewhat Neutral Not So

Much

Not Very Much

4 3 2

COMMENTS:

2. How important do you think this training is to

your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:

3.How much better do you feel your

performance will improve as a result of this

training?


Much

Not Very Much

4 3 2

COMMENTS:

4. How much do you anticipate instructor

feedback will assist in your final scores?


Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5


5.How much difficulty do you anticipate you will

have with this training?


Much

Not Very Much

4 3 2

COMMENTS:

6. How much confidence do you have in your

ability to meet target requirements for the

training objective(s)?


Much

Not Very Much

4 3 2

7. Do you feel you did much better on the pre-test

than expected?


Much

Not Very Much

4 3 2

COMMENTS:

8. Do you feel you did much worse on the pre-

test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

9.How much comfort do you have about firing

the established weapon for this training?


Much

Not Very Much

4 3 2

COMMENTS:

10.How much do you think this training will

improve your current level of skill?


Much

Not Very Much

5

5

5

5

5


4 3 2

COMMENTS:

11.How much comfort do you have with your

current level of basic knowledge to assist you in

this training?


Much

Not Very Much

4 3 2

COMMENTS:

12.How nervous were you during the pre-test

qualification rounds?


Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:

Please provide your overall concerns (if any)

5

5

5


A4: PRE-TEST ATTITUDES-EXPERIMENTAL GROUP


PRE-TEST – ATTITUDES: EXPERIMENTAL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.




1.How confident are you that you will like

this training?


Much

Not Very Much

4 3 2

COMMENTS:

2.How important do you think this training is

to your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:



training?


Much

Not Very Much

4 3 2

COMMENTS:

4.How much do you anticipate instructor



Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5


5.How much difficulty do you anticipate you

will have with this training?


Much

Not Very Much

4 3 2

COMMENTS:

6.How much confidence do you have in your




Much

Not Very Much

4 3 2

7. Do you feel you did much better on the

pre-test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

8.Do you feel you did much worse on the

pre-test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

9.How much comfort do you have about

firing the established weapon for this

training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

5

5

5

5

5


4 3 2

COMMENTS:

11.How much comfort do you have with

your current level of basic knowledge to

assist you in this training?


Much

Not Very Much

4 3 2

COMMENTS:

12.How nervous were you during the pre-test



Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


A5: POST-TEST ATTITUDES-CONTROL GROUP


POST-TEST – ATTITUDES: CONTROL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.

Please complete this form and provide your input to assist us to evaluate our efforts. No liability exists for your honest response. Any comments and suggestions you provide will better assist in the evaluation of the training and tools.

Unique ID assigned to this study: Training Domain:



1.Overall, how much did you like the

marksmanship training?


Much

Not Very Much

4 3 2

COMMENTS:

2.How important do you think this training was

to your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:


performance improved as a result of this

training?


Much

Not Very Much

4 3 2

COMMENTS:

4.To what extent do you feel the instructor’s

feedback assisted in improving your final

scores?


Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5


5.How much difficulty do you feel you had

with this training?


Much

Not Very Much

4 3 2

COMMENTS:



training you had?


Much

Not Very Much

4 3 2

7. Do you feel you did much better on the post-

test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

8. Do you feel you did much worse on the post-

test than expected?


Much

Not Very Much

4 3 2

COMMENTS:


the established weapon after this training?


Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5

5


10.How much do you think this training

improved your current level of skill?


Much

Not Very Much

4 3 2

COMMENTS:

11.How much do you feel the training assisted

in improving your level of basic knowledge?


Much

Not Very Much

4 3 2

COMMENTS:

12.How nervous were you during the post-test



Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


A6: POST-TEST ATTITUDES-EXPERIMENTAL GROUP


POST-TEST – ATTITUDES: EXPERIMENTAL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.





1.Overall, how much did you like the



Much

Not Very Much

4 3 2

COMMENTS:

2.How important do you think this training

was to your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:


performance improved as a result of this

training?


Much

Not Very Much

4 3 2

COMMENTS:

4.To what extent do you feel the instructor’s

feedback assisted in improving your final

scores?


Much

Not Very Much

4 3 2

5

5

5

5


COMMENTS:

5.How much difficulty do you feel you had

with this training?


Much

Not Very Much

4 3 2

COMMENTS:



training you had?


Much

Not Very Much

4 3 2

7.Do you feel you did much better on the

post-test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

8.Do you feel you did much worse on the

post-test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

9.How much comfort do you have about

firing the established weapon after this

training?


Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5

5


10.How much do you think this training

improved your current level of skill?


Much

Not Very Much

4 3 2

COMMENTS:

11.How much do you feel the training

assisted in improving your level of basic

knowledge?


Much

Not Very Much

4 3 2

COMMENTS:

12.How nervous were you during the post-

test qualification rounds?


Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


A8: POST-TRAINING ATTITUDES (4 WEEKS)-CONTROL GROUP


POST TRAINING – ATTITUDES 4 WEEK: CONTROL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.





training?


Much

Not Very Much

4 3 2

COMMENTS:

2.How important do you think this training is to

your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:



training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5





Much

Not Very Much

4 3 2

COMMENTS:





Much

Not Very Much

4 3 2

7.Do you feel you did much better on the post-

training test than expected?


Much

Not Very Much

4 3 2

COMMENTS:

8.Do you feel you did much worse on the post-



Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

5

5

5

5

5


A8: POST-TRAINING ATTITUDES (4 WEEKS)-EXPERIMENTAL GROUP

4 3 2

COMMENTS:



this training?


Much

Not Very Much

4 3 2

COMMENTS:


training qualification rounds?


Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


POST TRAINING – ATTITUDES 4 WEEKS: EXPERIMENTAL GROUP

You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.






training?


Much

Not Very Much

4 3 2

COMMENTS:


your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:



training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5





Much

Not Very Much

4 3 2

COMMENTS:





Much

Not Very Much

4 3 2




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5

5


A9: POST-TRAINING ATTITUDES (8 WEEKS)-CONTROL GROUP




Much

Not Very Much

4 3 2

COMMENTS:



this training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


POST TRAINING – ATTITUDES 8 WEEKS: CONTROL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.





training?


Much

Not Very Much

4 3 2

COMMENTS:


your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:



training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5





Much

Not Very Much

4 3 2

COMMENTS:





Much

Not Very Much

4 3 2




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

5

5

5

5

5


A10: POST-TRAINING ATTITUDES (8 WEEKS)-EXPERIMENTAL GROUP

4 3 2

COMMENTS:



this training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


POST TRAINING – ATTITUDES 8 WEEKS: EXPERIMENTAL GROUP You are participating in a study that evaluates the use of a tool to improve marksmanship training. This questionnaire is designed to obtain your attitudes and opinions of the training you received. Your opinions are very important.





training?


Much

Not Very Much

4 3 2

COMMENTS:


your job or goals?


Much

Not Very Much

4 3 2

COMMENTS:



training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

5

5

5

5





Much

Not Very Much

4 3 2

COMMENTS:





Much

Not Very Much

4 3 2




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

5

5

5

5

5


4 3 2

COMMENTS:



this training?


Much

Not Very Much

4 3 2

COMMENTS:




Much

Not Very Much

4 3 2

COMMENTS:

COMMENTS:


5

5

5


Appendix B: Measurement Instruments – Level 1 – Gauging

Satisfaction

B1: LEVEL 1 - REACTION /SATISFACTION OF MARKSMANSHIP TRAINING-

EXPERIMENTAL GROUP


LEVEL 1 – SATISFACTION: EXPERIMENTAL GROUP Please complete this form and provide your input to assist us to evaluate our efforts. No liability exists for your honest response. Any comments and suggestions you provide will better assist in the evaluation of the training and tools.




RELEVANCE

1 How satisfied are you with the content

of this training module to meet your

training needs.

Very

Satisfied

Satisfied Neutral Dissatisfied Very Dissatisfied

4 3 2

COMMENTS:

2. How satisfied are you that the training

content provided in this training module

will assist you in performing your job.

Very

Satisfied


4 3 2

COMMENTS:


you received relates to the stated learning

objectives?

Very

Satisfied


4 3 2

COMMENTS:

4. How satisfied are you with the

relevance of the training to convey to

your performance improvement in

marksmanship skills?

Very

Satisfied


4 3 2

COMMENTS:

5


5. How satisfied are you that the

objectives stated in this training are

relevant to the marksmanship tasking

needs of your job?

Very

Satisfied


4 3 2

COMMENTS:

6. How satisfied are you that the skills

you learned in this training as these apply

to marksmanship tasking?

Very

Satisfied


4 3 2

7. Overall. how satisfied are you with the

marksmanship training you received?

Very

Satisfied


4 3 2

COMMENTS:

INSTRUCTOR

8. Overall. how satisfied are you with the

marksmanship coaching you received

from your instructor?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to gauge

marksmanship training issues?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s attentiveness to assist you to

succeed in this marksmanship training?

Very

Satisfied


4 3 2

COMMENTS:


11. How satisfied are you with your

instructor’s ability to progress your

performance capability?

Very

Satisfied


4 3 2

COMMENTS:


knowledge and skill your instructor

possessed relative to marksmanship?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to communicate

guidance?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to recognize any

individual performance problems.

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to communicate

guidance?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to provide corrective

feedback regarding marksmanship

performance issues?

Very

Satisfied


4 3 2

COMMENTS:


17. How satisfied are you that your

instructor’s feedback assisted in your

ability to make corrective action?

Very

Satisfied


4 3 2

COMMENTS:

TOOLS/MATERIALS (Identify tools and provide specific comments if any)

18. How satisfied are you with the tools

you were provided for use in this

training?

Very

Satisfied


4 3 2

COMMENTS:


tools reflected the needs of the training

objectives?

Very

Satisfied


4 3 2

COMMENTS:


tools use for this training assisted in

improving your performance?

Very

Satisfied


4 3 2

COMMENTS:


tools/resources for training assisted in

your ability to make corrective action??

Very

Satisfied


4 3 2

COMMENTS:


tools were easy to use?

Very

Satisfied



4 3 2

COMMENTS:

23. How satisfied are your ability to

interpret training resources/tools (please

list in comments)

Very

Satisfied


4 3 2

COMMENTS:

COMMENTS:

Please provide your overall feedback or suggestions (if any)


B2: LEVEL 1 - REACTION /SATISFACTION OF MARKSMANSHIP TRAINING-

CONTROL GROUP


LEVEL 1 – SATISFACTION: CONTROL GROUP Please complete this form and provide your input to assist us to evaluate our efforts. No liability exists for your honest response. Any comments and suggestions you provide will better assist in the evaluation of the training and tools.




RELEVANCE


content of this training module

to meet your training needs.

Very

Satisfied


4 3 2

COMMENTS:


training content provided in

this training module will assist

you in performing your job.

Very

Satisfied


4 3 2

COMMENTS:


training you received relates to

the stated learning objectives?

Very

Satisfied


4 3 2

COMMENTS:


relevance of the training to

convey to your performance

improvement in marksmanship

skills?

Very

Satisfied


4 3 2

COMMENTS:


objectives stated in this training

are relevant to the

marksmanship tasking needs of

your job?

Very

Satisfied


4 3 2

COMMENTS:


skills you learned in this

training as these apply to

marksmanship tasking?

Very

Satisfied


4 3 2

5


7. Overall. how satisfied are you

with the marksmanship training

you received?

Very

Satisfied


4 3 2

COMMENTS:

INSTRUCTOR

8. Overall. how satisfied are you

with the marksmanship

coaching you received from

your instructor?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to gauge

marksmanship training issues?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s attentiveness to

assist you to succeed in this


Very

Satisfied


4 3 2

COMMENTS:

11. How satisfied are you with

your instructor’s ability to

progress your performance

capability?

Very

Satisfied


4 3 2

COMMENTS:


knowledge and skill your

instructor possessed relative to

marksmanship?

Very

Satisfied


4 3 2

COMMENTS:



communicate guidance?

Very

Satisfied


4 3 2

COMMENTS:




recognize any individual

performance problems.

Very

Satisfied


4 3 2

COMMENTS:



communicate guidance?

Very

Satisfied


4 3 2

COMMENTS:


instructor’s ability to provide

corrective feedback regarding

marksmanship performance

issues?

Very

Satisfied


4 3 2

COMMENTS:

17. How satisfied are you that your

instructor’s feedback assisted

in your ability to make

corrective action?

Very

Satisfied


4 3 2

COMMENTS:

TOOLS/MATERIALS (Identify tools and provide specific comments if any)


tools you were provided for use

in this training?

Very

Satisfied


4 3 2

COMMENTS:


training tools reflected the

needs of the training

objectives?

Very

Satisfied


4 3 2

COMMENTS:


training tools use for this

training assisted in improving

your performance?

Very

Satisfied


4 3 2

COMMENTS:


tools/resources for training

assisted in your ability to make

corrective action??

Very

Satisfied



4 3 2

COMMENTS:


training tools were easy to use?

Very

Satisfied


4 3 2

COMMENTS:

23. How satisfied are your ability

to interpret training

resources/tools (please list in

comments)

Very

Satisfied


4 3 2

COMMENTS:

COMMENTS:

Please provide your overall feedback or suggestions (if any)


B3: LEVEL 1 – INSTRUCTOR FEEDBACK ON FREQUENCY OF USE OF NPSS

BIOMETRIC INFORMATION- EXPERIMENTAL GROUP (OPTIONAL)

INSTRUCTOR FEEDBACK ON FREQUENCY OF USE OF NPSS BIOMETRIC INFORMATION Unique ID assigned to this study: Instructor Name or Unique ID:

Training Domain: Date: 00/00/0000

Objective(s) Associated with Training: The purpose of this survey is to evaluate the efficiency of instructing with the NPSS product. The survey is assistive in defining the benefits or needs of product improvement, so please be frank in your answers. Read each item carefully. Reflect on the data feed described. Determine what level of frequency you accessed the described data for instructional purposes. Place an (x) under your corresponding choice. If you note a lack of use for a survey query, place an (x) under the column that indicates “was not useful to the training” If you have any comments or suggestions relative to an item listed, please indicate these below in the comments section.

Level 1

Instructor Survey - Frequency of Use Scale Range [1 – Never, 2 – Seldom,3 – Occasionally, 4 – Often, 5 – Always]

OBSERVED FREQUENCY

Was not useful to the training

Indicate by checking the

box

☒

Alw

ays

Oft

en

Occ

asio

na

ll

y Se

ldo

m

Ne

ver

Audio/Video

1 Audio conveys information to assist in training ☐ 2 Video conveys information to assist in training ☐ 3 Audio assisted in providing guidance to the trainees ☐ 4 Video assisted in providing guidance to the trainees ☐ 5 Overviewing audio playback with the trainees assisted in a mutual shared

awareness of competency issues ☐

6 Overviewing video playback with the trainees assisted in a mutual shared awareness of competency issues

☐

7 Audio analysis assisted in trainees overcoming competency issues. ☐

8 Video analysis assisted in trainees overcoming competency issues ☐

Performance Skills- Biometric Data 9 Breathing during the shot. ☐

10 Trigger pressure applied during the shot. ☐

11 Heart rate during the shot ☐

12 Pupil dilation/eye movement during the shot ☐

13 Weapon movement during the shot (acceleration, gyroscope effect) ☐

14 Galvanic Skin Response - GSR ☐ 15 Tacit Response – EMG ☐ 16 Electrocardiogram (ECG) Detects increased heart rate ☐

17 Shooter Movement during the Shot ☒

COMMENTS

Relative to any of the numbered items (1-8) above, please provide any comments or suggestions you have regarding your use of audio/video for the NPSS tool.

1

2

3

4

5

6

7

8

Relative to any of the numbered items (9 – 16) above, please provide any comments or suggestions you have regarding your use of biometric sensing data for the NPSS tool.

9

10

11

12


13

14

15

16

17

ADDITIONAL COMMENTS


Appendix C: Measurement Instruments – Level 2 Skill Acquisition C1: Marksman Scorecard: Control Group

Marksman Scorecard – NPSS Product (Control Group)

Trainee Name or Unique ID: (Enter the Unique ID or name of trainee)

Qualifying Experience Level: (Indicate the experience level assigned by preliminary interview and pre-test qualification ranking)

Date: 00/00/0000 Total Hits:


Instructor Name or Unique ID:

Training Start Time: (Indicate the time training started HH: MM: SS

Total Misses:

Training Domain: Qualification Score:

Expert ☐ 160-200

Sharpshooter ☐ 120-159

Marksman ☐ 80-119

Unqualified ☐ 79 or below or less than 24 hits

Training End Time: HH: MM: SS

Score:

List Objective(s) Associated with Training: Duration of Training: HH: MM: SS (the difference between the start and end of training.)

Describe Weapon:

Instructions for this form: Fill out the top completely. If a section does not apply, put N/A. For each session please indicate the range, the time to fire and allocate a check mark to the appropriate section for (Single or group shot strategy). If feedback was provided to the trainee, please provide a brief explanation.

Session or

Round

Range (Meters)

Time (Seconds)

Hit Single Shot

Miss Single Shot

No Fire Group Shots # of Hits

Group Shots # of Misses

Feedback Provided to Trainee

1 ☐ ☐ ☐

2 ☐ ☐ ☐

3 ☐ ☐ ☐

4 ☐ ☐ ☐

5 ☐ ☐ ☐









Total Misses:


Expert ☐ 160-200


Marksman ☐ 80-119



Score:


Describe Weapon:


Session or

Round

Range (Meters)

Time (Seconds)

Hit Single Shot

Miss Single Shot




6 ☐ ☐ ☐

7 ☐ ☐ ☐

8 ☐ ☐ ☐

9 ☐ ☐ ☐

10 ☐ ☐ ☐

11 ☐ ☐ ☐

12 ☐ ☐ ☐

13 ☐ ☐ ☐

14 ☐ ☐ ☐

15 ☐ ☐ ☐









Total Misses:


Expert ☐ 160-200


Marksman ☐ 80-119



Score:


Describe Weapon:


Session or

Round

Range (Meters)

Time (Seconds)

Hit Single Shot

Miss Single Shot




16 ☐ ☐ ☐

17 ☐ ☐ ☐

18 ☐ ☐ ☐

TOTAL>>>> <<<<<<<<<TOTAL


C2: Marksman Scorecard: Experimental Group

Marksman Scorecard – NPSS Product (Experimental Group) Trainee Name or Unique ID: (Enter the Unique ID or name of trainee)





Training Start Time: HH: MM: SS

Total Misses:


Expert ☐ 160-200


Marksman ☐ 80-119



Score:

List Objective(s) Associated with Training: Duration of Training: HH: MM: SS (the difference

between the start and end of training.)

Describe Weapon:

Instructions for this form: Fill out the top completely. If a section does not apply, put N/A. For each session please indicate the range, the time to fire and allocate a check mark to the appropriate section for (Single or group shot strategy). If feedback was provided to the trainee, please provide a brief explanation. Check all biometric data that assisted to the feedback provided to the trainee. Session

or Round

Range (Meters)

Time (Seconds)

Hit Single Shot

Miss Single Shot

No Fire

Group Shots

# of Hits

Group Shots

# of Misses

Feedback Provided to Trainee What data assisted in your constructive guidance to the trainee (check all that apply) Weapon Shooter Trigger Movement Movement Pressure Breathing Video Audio ECG EMG GSR Pupil Data

1 30 3 ☒ ☐ ☐ Hold breath before shot ☒ ☐ ☐ ☒ ☐ ☐ ☐ ☐ ☐ ☐

2 200 6 ☐ ☒ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

3 100 4 ☒ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

4 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

5 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

6 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

7 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

8 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐








Total Misses:


Expert ☐ 160-200


Marksman ☐ 80-119



Score:



Describe Weapon:


or Round

Range (Meters)

Time (Seconds)

Hit Single Shot

Miss Single Shot

No Fire

Group Shots

# of Hits

Group Shots

# of Misses


9 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

10 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

11 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

12 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

13 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

14 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

15 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

16 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

17 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

18 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐








Total Misses:


Expert ☐ 160-200


Marksman ☐ 80-119



Score:



Describe Weapon:


or Round

Range (Meters)

Time (Seconds)

Hit Single Shot

Miss Single Shot

No Fire

Group Shots

# of Hits

Group Shots

# of Misses


TOTAL <<<<<<<< TOTAL>>>>>>>>

Continue on another card in succession if needed


Appendix D: Level 3 - Behavioral Transfer/Skill Retention

D1: Skill Retention-Post-Test- Control Group

Marksmanship Ranking Scorecard: Post-Test/Control Group



evaluation) Note: Use this ID

consistently throughout evaluative

materials


enter a date.

Trainee Name or

Unique ID:

Final

Ranking:

Training Objective (s): (Outline

objectives for training)


evaluated)

Testing

Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☐

Post-training ☒ phase

Post-training ☐ phase (post 4 weeks)


Supporting Standard: (refer to reference document or resource

in support of the raking standards)





example below

Expert 160-200


Marksman 80-119


Session Number Range

(meters)

RANKING (Assign a measure of


standard)

RECORDED STANDARD

ACCURACY



3 Marksman 92

4


D2: Skill Retention-Post-Test- Experimental Group

Marksmanship Ranking Scorecard: Post-Test/Experimental Group





materials


enter a date.

Trainee Name or

Unique ID:

Final

Ranking:




evaluated)

Testing

Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☐

Post-training ☒ phase









example below

Expert 160-200


Marksman 80-119




standard)

RECORDED STANDARD

ACCURACY



3 Marksman 92

4


D3: Skill Retention-Post-Training (4-weeks)- Control Group

Marksmanship Ranking Scorecard: Skill Retention (4 weeks)/Control Group





materials


enter a date.

Trainee Name or

Unique ID:

Final

Ranking:




evaluated)

Testing

Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☐

Post-training ☐ phase

Post-training ☒ phase (post 4 weeks)








example below

Expert 160-200


Marksman 80-119




standard)

RECORDED STANDARD

ACCURACY



3 Marksman 92

4


D4: Skill Retention-Post-Training (4-weeks)- Experimental Group

Marksmanship Ranking Scorecard: Skill Retention (4 weeks)/Experimental Group





materials


enter a date.

Trainee Name or

Unique ID:

Final

Ranking:




evaluated)

Testing

Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☐










example below

Expert 160-200


Marksman 80-119




standard)

RECORDED STANDARD

ACCURACY



3 Marksman 92

4


D5: Skill Retention-Post-Training (8-weeks)- Control Group

Marksmanship Ranking Scorecard: Skill Retention (8 weeks)/Control Group





materials


enter a date.

Trainee Name or

Unique ID:

Final

Ranking:




evaluated)

Testing

Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☐










example below

Expert 160-200


Marksman 80-119




standard)

RECORDED STANDARD

ACCURACY



3 Marksman 92

4


D6: Skill Retention-Post-Training (8-weeks)- Experimental Group

Marksmanship Ranking Scorecard: Skill Retention (4 weeks)/Experimental Group





materials


enter a date.

Trainee Name or

Unique ID:

Final

Ranking:




evaluated)

Testing

Environment:

Check One:

Augmented ☐

Simulated ☐

Live Fire ☒

Dry Fire ☐

Check one:

Pre-test phase ☐










example below

Expert 160-200


Marksman 80-119




standard)

RECORDED STANDARD

ACCURACY



3 Marksman 92

4


Appendix E: Data Reporting Recommendations Levels 1-4

DATA REPORTING RECCOMMENDATIONS Training Effectiveness

Purpose Analysis Level

Objectives to Measure

Trainee satisfaction Level 1 Overall satisfaction of training content, delivery and facilitation (instructor and online).

Skill Acquisition Level 2 Measures of time to time, ammunition usage, performance efficiency relative to accuracy.

Transfer of Behavior/Skill Retention Level 3 Influence of training on skill retention behaviors associated to training objective(s).

Results of Training Level 4 The benefit of training with the product vs traditional training on skill retention, time to train and ammunition costs.

ATTITUDES (1) Did confidence and attitude levels influence training? (2) Was there a scalable difference in both confidence and attitudes upon conclusion of training between the Experimental and Control groups? 9. Have the attitudes of participants changed since the conclusion of training.

REPORTING DATA GRAPHICAL DEMONSTRATION

PRE – TEST & POST-TEST COMPARISON (CONTROL GROUP & EXPERIMENTAL GROUP)

DEMONSTRATE THE 12 QUESTIONS ON A SCATTERPLOT WITH THE CONDITIONS OF POST-TEST AND PRE-TEST MATCHED TO THE EXPERIMENTAL &CONTROL GROUP

4 WEEK POST-TRAINING COMPARISON (CONTROL GROUP & EXPERIMENTAL GROUP)

DEMONSTRATE THE 12 QUESTIONS ON A SCATTERPLOT WITH THE CONDITIONS MATCHED TO THE EXPERIMENTAL &CONTROL GROUP


DEMONSTRATE THE 12 QUESTIONS ON A SCATTERPLOT WITH THE CONDITIONS OF MATCHED TO THE EXPERIMENTAL &CONTROL GROUP

4 WEEK and 8 WEEK POST-TRAINING COMPARISON (CONTROL GROUP)

DEMONSTRATE THE 12 QUESTIONS ON A SCATTERPLOT WITH THE CONDITIONS OF 4 WEEK AND 8 WEEKS MATCHED TO THE CONTROL GROUP

4 WEEK and 8 WEEK POST-TRAINING COMPARISON (EXPERIMENTAL GROUP)

DEMONSTRATE THE 12 QUESTIONS ON A SCATTERPLOT WITH THE CONDITIONS OF 4 WEEK AND 8 WEEKS MATCHED TO THE EXPERIMENTAL GROUP

LEVEL 1 1. Overall, was training more relevant for those participants in the experimental group as compared to the control group? 2. Overall, were the tools or resources used to assist training in the experimental group more assistive to self-perceived performance than those used in the control group? 3. Overall, were the instructors in the experimental group more efficient at their coaching task than those in the control group? 4. How frequently did the instructor’s leverage the data provided by the NPSS tool during instruction?


PRE-TEST AND POST TEST SURVEY EXPERIMENTAL & CONTROL GROUP

CREATE A TABLE THAT DEMONSTATES EACH SURVEY QUESTION AND THE IQR FOR EACH ONE UNDER THE CONDITIONS OF THE EXPERIMENTAL AND CONTROL GROUP ALIGNED FOR COMPARISONS INSTRUCTOR, RELEVANCY AND TOOLS/RESOURCES

INSTRUCTOR FREQUENCY SURVEY OF USE WITH BIOMETRIC DATA

CREATE A TABLE THAT DEMONSTATES EACH CATEGORY OF NPSS DATA AND THE IQR FOR EACH ONE OR DEMONSTRATE THE SAME WITH A BARGRAPH OR SCATTERPLOT

LEVEL 2 5 How much improvement in performance did training have from the onset to the end for the experimental group over the control group as noted by improvements (accuracy, time to complete training)? 6. What is the estimate of saved ammunition in the experimental

REPORTING DATA GRAPHICAL DEMONSTRATION COMPARISON OF TIME EFFICIENCY DATA BETWEEN THE CONTROL GROUP AND EXPERIMENTAL GROUP

CHART DEMONSTRATING TIME IN 95% OF THE TWO TAILED T-TEST

COMPARISON OF TARGET EFFICIENCY DATA BETWEEN THE CONTROL GROUP AND EXPERIMENTAL GROUP

CHART DEMONSTRATING TARGET EFFICIENCY IN 95% OF THE TWO TAILED T-TEST

COMPARISON OF MISSPENT AMMUNITION DATA BETWEEN THE CONTROL GROUP AND EXPERIMENTAL GROUP

CHART DEMONSTRATING AMMUNITION IN 95% OF THE TWO TAILED T-TEST


group as compared to the control group? LEVEL 3 7.Did training in the experimental group transfer to improvement in marksmanship performance over the control group as noted by range qualification scores gained from pre-test to post-test? 8.Can NPSS be used to sustain skills over time and yielding higher or equal retention in the experimental group compared to the control group?

REPORTING DATA GRAPHICAL DEMONSTRATION PRE – TEST & POST-TEST COMPARISON (CONTROL GROUP & EXPERIMENTAL GROUP)

COMPARISON OF EFFECT OF TRAINING BETWEEN THE EXPERIMENTAL GROUP AND CONTROL GROUP PRE-TEST AND POST-TEST CONDITIONS IN A (TABLE AND GRAPHIC CURVE DEMONSTRATING IMPROVEMENT ON ONE CONDITION OVER THE OTHER REPRESENTATIVE OF T-TEST


COMPARISON OF EFFECT OF TRAINING BETWEEN THE EXPERIMENTAL GROUP AND CONTROL GROUP PRE-TEST AND POST-TEST CONDITIONS IN A (TABLE AND GRAPHIC CURVE DEMONSTRATING IMPROVEMENT ON ONE CONDITION OVER THE OTHER REPRESENTATIVE OF T-TEST


COMPARISON OF EFFECT OF TRAINING BETWEEN THE EXPERIMENTAL GROUP AND CONTROL IN A (TABLE AND GRAPHIC CURVE DEMONSTRATING IMPROVEMENT ON ONE CONDITION OVER THE OTHER REPRESENTATIVE OF T-TEST

4 WEEK and 8 WEEK POST-TRAINING COMPARISON (CONTROL GROUP)

COMPARISON OF EFFECT OF TRAINING BETWEEN THE EXPERIMENTAL GROUP AND CONTROL GROUP 4 & 8 WEEK-TEST CONDITIONS IN A (TABLE AND GRAPHIC CURVE DEMONSTRATING IMPROVEMENT ON ONE CONDITION OVER THE OTHER REPRESENTATIVE OF T-TEST

4 WEEK and 8 WEEK POST-TRAINING COMPARISON (EXPERIMENTAL GROUP)

COMPARISON OF EFFECT OF TRAINING BETWEEN THE EXPERIMENTAL GROUP AND CONTROL 4 & 8 WEEK-TEST CONDITIONS IN A (TABLE AND GRAPHIC CURVE DEMONSTRATING IMPROVEMENT ON ONE CONDITION OVER THE OTHER REPRESENTATIVE OF T-TEST

LEVEL 4 10. What cost benefit does training with the NPSS product yield to stakeholder? Time to train, satisfaction, ammunition cost, improved accuracy and retention)


LEVELS 1-3 CONVEYED TO THE BENEFIT OF SELECT CLIENTELLE OR A DOMAIN THAT CORRELATES TO THEIR BUSINESS COST

’


Appendix F: Planning Charts

Planning Chart for Product Evaluation – Marksmanship Project Performance Effectiveness

Goal Purpose Analysis Level Objectives to Measure Trainee satisfaction Level 1 Overall satisfaction of training content, delivery and

facilitation (instructor and online).

Skill Acquisition Level 2 Measures of time to time, ammunition usage, performance efficiency relative to accuracy.

Transfer of Behavior/Skill Retention

Level 3 Influence of training on skill retention behaviors associated to training objective(s).

Results of Training Level 4 The benefit of training with the product vs traditional training on skill retention, time to train and ammunition costs.

Preliminary Measures

Training for use with the tool. Prior to the start of the evaluation (Time for Training is TBD)

Training content is constructed independent of this evaluation plan

Gauging competency with the NPSS tool

Following training but prior to the start of evaluation. Training and testing should be completed for use with the NPSS too and within of the start of evaluation testing.

Training competency for use with the product is constructed independent of this evaluation.

Training Objectives are identified Training objectives and domain information are identified for the purposes of evaluation, participant characteristics, and pretest. These issues are agreed upon on the basis or the needs of the organization (Norrad) or the client.

Agreement is made prior to further preliminary processes.

Participant Characteristics are baselined for instructor and trainees

Prior to evaluation participant characteristics relative to the classification of subject pools is conducted.

Conducted prior to training the size of the research population will determine a time scale for this activity

Pre-test is conducted A single session pre-test is conducted in live fire to assess a baseline of use for entry level skills with the weapon and aligned to training objectives. Attitude measures are conveyed after pretest.

Conducted prior to training the size of the research population will determine a time scale for this activity

Process & Data Collection

Facilitator Steps Time Range (from Start)

Corporate Sponsor Finalize Evaluation Objectives with Stakeholders Week 1

Domain Expert Conduct assessment of participant characteristics TBD

Domain Expert Conduct Live fire pre-test TBD

Domain Experts Assign groups and instructors TBD

Training Team Train and assess knowledge of use with the NPSS Product TBD

Training Team Deliver Pre-training attitude assessment (Experimental & Control Group)

TBD

Training Team Conduct Level 2 training and data collection with the NPSS product (Experimental Group)

TBD

Training Team Facilitate level 1 surveys to trainees and collect data Immediately upon conclusion of training

Training Team Conduct Level 3 Post-Training and Attitude Assessment/Collect Data

Immediately upon conclusion of Level 3 Post-test

Conduct traditional training and data collection with traditional training methods (Control Group)

Upon conclusions of experimental group evaluations

Training Team Facilitate level 1 surveys to trainees and collect data Immediately upon conclusion of training


Training Team Conduct Level 3 Post-Training and Collect Data Immediately upon conclusion of training

Analysis Team Compile and Analyze Data Levels 1-3 Upon conclusion of all training

Training Team Level 3 Skill Retention Post -Test and Attitude Assessment/Collect Data (Experimental & Control Group)

4 weeks upon conclusion of initial Level 3 post-test

Analysis Team Compile and Analyze Data Levels 3 (4-week post-training retention and attitude assessment)

Upon conclusion of 4- week retention assessment

Training Team Level 3 Skill Retention Post -Test and Attitude Assessment/Collect Data (Experimental & Control Group)

8 weeks upon conclusion of initial Level 3 post-test

Analysis Team Compile and Analyze Data Levels 3 (8-week post-training retention and attitude assessment)

Upon conclusion of 8-week retention assessment

Analysis Team Compare initial level 3 post test results to 4 &8-week post training retentions scores

Upon conclusion of testing Level 3 retention skills

Analysis Team Compile, Analyze and Interpret Finding TBD

Analysis Team Report Findings in Formal Document TBD

Analysis Target Findings Calculation Method

Eliminating External Influences

Level 1 1. Overall, was training more relevant for those participants in the experimental group as compared to the control group?

Likert Rating Taken in anonymity, immediately upon completion of training. Required to ensure 100% participation.

2. Overall, were the tools or resources used to assist training in the experimental group more assistive to self-perceived performance than those used in the control group?


3. Overall, were the instructors in the experimental group more efficient at their coaching task than those in the control group?


4. How frequently did the instructor’s leverage the data provided by the NPSS tool during instruction?


Level 2 5 How much improvement in performance did training have from the onset to the end for the experimental group over the control group as noted by improvements (accuracy, time to complete training)?

Statistical comparative analysis (experimental & control group)

Experimental results are compared to a control group Preliminary measures ensured matched groups and paired instructors to trainees.

6. What is the estimate of saved ammunition in the experimental group as compared to the control group?


Experimental results are compared to a control group Preliminary measures ensured matched groups and paired instructors to trainees.

Level 3 7.Did training in the experimental group transfer to improvement in marksmanship performance over the control group as noted by range qualification scores gained from pre-test to post-test?


Experimental results are compared to a control group Level 3 data is obtained in anonymity Evaluation questions are aligned to training content

8.Can NPSS be used to sustain skills over time and yielding higher or equal retention in the experimental


Experimental results are compared to a control group Level 3 data is obtained in anonymity Evaluation questions are aligned to training content


group compared to the control group?

9. Have the attitudes of participants changed since the conclusion of training.

Attitude Assessment Likert Rating

Taken in anonymity/Compared to pre-training data/post-training assessments.

Level 4 10. What cost benefit does training with the NPSS product yield to stakeholder? Time to train, satisfaction, ammunition cost, improved accuracy and retention)

Summation of levels 1-3 Statistical Analysis

Experimental results are compared to a control group Taken in anonymity/Compared to pre-training data/post-training assessments.

Report Data

High Level Findings

Stakeholder Interest

Communication of Results

Satisfaction of use with the NPSS product (Instructors & Trainees)

Satisfaction Training with the product streamlines processes

Finding report for stakeholders Stakeholder meeting to convey preliminary findings

Performance improvement – Marksman efficiency

Better scores = higher performance yield

Efficiency in time train

Quicker training means a savings cost and allows for more training participation

More efficient shooting

Ammunition cost savings


Appendix G: Product Figures

Note: All figures displayed here relative to the product are retrieved from the NPSS User’s Manual

Demonstrates instructor access to data that serves multiple students at a time.


ATTACHMENTS

Shows hand gun with the NEMESIS wrist attachment to hold your Mobile Touch Device


Shows hand gun with your Mobile Touch Device inserted into the NEMESIS wrist attachment with NEMESIS running

Shows an Android touch phone running NEMESIS while attached to a rifle with the NEMESIS attachment device.


Shows an Android 7.7 tablet attached to a rifle running NEMESIS


Show the rifle tilt while in a shooting position. Note: Rifle icon should be in the middle and straight up as shown in the picture. Any tilt of the rifle will be picked up and

displayed under the tilt functionality.

FEEDBACK


Conveys feedback to the trainee relative to motion.


If the device is held straight up, you will see a handgun straight up indicating that the gun is in the correct position, which is what you want. Moving the device moves

the handgun and therefore affects your shooting accuracy. The following shows two examples of the gun being held straight up and another showing the gun is tilted to

the right and of center.


Overview of data tracking options for a Mobile Touch Device


Appendix H: Liability Waivers

Official Waiver of Liability and Release of All Claims Use of NEMESIS-SE Adapter and Software

Instructions: Please read this form carefully and completely.

I acknowledge that shooting guns’ entails known and unanticipated risks which could result in physical or emotional injury, paralysis, death, or damage to

myself and others, to property, or to third parties. I understand that such risks simply cannot be eliminated without jeopardizing the essential qualities of

the activity.

The risks include, among other things: Being shot, dying or becoming permanently injured and or impaired physically from the discharge of bullets. NPSS

Furthermore, range owners, range employees and instructors expressly agree and promise to accept and assume all of the risks existing in this activity.

My participating in this activity is purely voluntary and I elect to participate in spite of the risks.

I hereby voluntarily release, forever discharge, and agree to indemnify and hold harmless NEMESIS-SE, from any and all claims, demands, or causes of

action, which are in any way connected with my participation in this activity or my use of the NEMESIS-SE adapter or software, including any such claims

which allege negligent acts or omissions.

I certify that I have adequate insurance to cover any injury or damage I may cause or suffer while participating, or else I agree to bear the costs of such

injury of damage myself. I further certify that I have no medical or physical conditions which could interfere with my safety in this activity, or else I am

willing to assume and bear the costs of all risks that may be created, directly or indirectly by any such condition.

By acknowledging have read this document, I acknowledge that if anyone is hurt or property is damaged during my participation using NEMESIS-SE

provided adapter or software, I may be found by a court of law to have waived my right to maintain a lawsuit against NEMESIS-SE on the basis of any

claim from which I have released.


I have had sufficient opportunity to read this entire document. I have read and understand and agree to be bound by its terms.

I have read and agree to the EULA and Release of Liability

Sign: _____________________________________ Date: _________________________________