air force institute of technology - dtic · managers, estimating needs is typically an exercise in...
TRANSCRIPT
0CX)
II
A DECISION SUFPORT SYSTEM (DSS) FORS pERsoNNEL ACQJISITION M
THESIS
Kevin M. Holt, B.S.Captain, LSF
PF IT/G]R/13'S/90M-10
-- '
DISTIli!:IN STATE~iENT A.. .
Approved fcr public release; k, * - .:2 oDistribcution UnLim-ited "
DEPARTMENT OF THE AIR FORCE 4e
AIR UNIVERSITY
AIR FORCE INSTITUTE OF TECHNOLOGY
Wright-Patterson Air Force Base, Ohio
,V .l .
A JECISION SaFPPcRT SYSTEM (1)66) FORP14ACS PERSONNEL ACGWISITIEIJ M1GEMENT
THESIS
Kevin Mi. Holt, 8.6.
AIT/GR/E8/90M--1O
AFI T/GR/ENS/90M- 10
A DECISION SU1PPORT SYSTEM (DSS) FOR
AWAICS PERSONNEL ACQUISITION MANAEME
TI-ESIS
Presented to the Faculty of the School of Engineering
of the Air Force Institute of Technology
Air University
In Partial Fulfillment of the Degree
Requirements for the Degree of
Accession ForMaster of Science in Operations Research - S
DTig T.3
J i o
Kevin M. Holt, B.S. rA -i I .. . 'c- de
Captain, USAF -. ~ .- " 1.or
March 19% b ni i
pproved for public release; distribution unlimited
Preface
Personnel acquisition planning for AWACS crew positions is
currently based on "best guesses" driven by near term loss expectations.
For this reason, managers largely restrict themselves to dealing with
short term corrections in manning level deficiencies. Long term
management is largely a matter of manager intuition. This research was
conducted to extend management capabilities to handle long range
acquisition planning in a structured environment.
The process of this research was fraught with difficulties and
dead ends. That these were overcome is largely due to the guidance
provided by my advisor, LtCol Skip Valusek (who I think could teach
Machiavelli a few tricks).
Finally, I owe a debt of thanks to my wife Robin, for her support
and understanding throughout my AFIT experience, and a debt pure and
simple to my children, upon whom I will lavish attention, now that I am
done here.
ii
TABLE OF CONTENTS
Page
Preface ....................................................... ii
List of Figures ............................................... iv
List of Tables ................................................. v
Abstract ........................................................ vi
Introduction ................................................ 1Background .............................................. 1Programmed Flying Training .............................. 2
AWACS and the PFT ................................... 3AWACS Personnel Structure ............................... 5
Factor Diversity among Crew Positions .............. 3Problem Statement ...................................... 11Research Objectives ..................................... 12
TI Methodology ................................................ 13Suitability of a DSS Approach ......................... 13Adaptive Design ........................................ 15
The Vehicles of Adaptive Design ................... 18Concept Mapping ................................ 18Storyboarding ................................... 18
ROMC . ..................................... 19The "hook book" . ............................... 20
Establishing a Paradigm ................................ 21Factor Identification ................................. 22
Manning Level Factors .............................. 23Experience Level Factors ............................ 24
MADA Methods .................................... 25Regression Methods ............................. 27
Logistic Regression ....................... 28Multiple Linear Regression
with a Binary Response ................. 28
III Overall System Design ...................................... 30General ................................................. 30Conceptual Design Considerations ....................... 31
Concept Mapping .................................... 32Storyboarding ...................................... 32
Representative Storyboards ..................... 34Storyboarding's Contributions to
Understanding System Nepds ............... 3PThe Decision Process ....................... 31The Decision Paradigm ...................... 40
Kernel Selection ................................... 43
iii
Continued Adaptat ion .............................. 43Technical Design Considerations ....................... 44
Environment ........................................ 45Data ............................................... 47Models............................................... 47
IV Kernel Design .............................................. 49General ................................................. 49Kernel Selection ....................................... 49
Rationale for the Kernel Selection ................ 51Experience Score Modeling Examination ................. 53
Logistic Regression ............................... 53MCDM ............................................... 55Multiple Regression with Binary Responses ......... 57
Kernel Implementation ................................. 58Pulling the Pieces Together ....................... 59
V Conclusions and Recommendations ............................ 63Background .............................................. 63General DSS Development ................................ 64The Specific DSS ...................................... 73Summary ................................................ 75
Appendix A Concept Map ......................................... 76
Append ix B Storyboards ......................................... 79
Appendix C Hook Book ......................................... 122
Appendix D Data Requirements ................................... 131
Appendix E Model Requirements ................................. 134
Appendix F MADA Preliminary Survey ........................... 144
Bibliography ................................................. 193
Vita ......................................................... 196
iv
List of Figures
Figure Page
i Crew Position Manning Fluctuations ........................ 52 First Term Personnel in Crew Position ..................... 73 An Adaptive Framework for a DSS .......................... 154 Hook Book Entry Format .................................... 205 Experienci, g F-rofil ng Process ........................... 256 High Level Concept Map of Acquisition Problem ............ 337 Home Representation of DSS ............................... 358 Capabilities Representation of DSS ....................... 369 Force Structure Representation of DSS .................... 38
10 Scheduling Representation of DSS ......................... 4211 Concept Map ............................................... 7712 System Structure Representation .......................... 8213 Example Help Screen -- Decision Command Descriptions ... 8614 Example Help Screen -- Manning Profile Options ......... 8815 Aggregate Experience Profile for AWACS ................... 9116 Setting a Position Filter for Experience ................. 9417 Current MCC Experience Profile ............................ 9618 MCC Experience Profile Projections ....................... 9819 Scheduling Acquisitions .................................. 10120 High Level Capabilities Assessment ...................... 106
21 Changing Capabilities Assessment Resolution ............. 10922 Detailed Capabilities Assessment ........................ Il123 AWACS Force Structure Representation .................... 11424 Manning Profile Sub-options .............................. 11725 Selecting a Rank Filter .................................. 11726 A Low Level Data Representation ......................... 12027 Unweighted Regression Predictions ........................ 14028 Weighted Regression Predictions ......................... 142
V
List of rables
Table Page
1 AWACS Crew Component Information ...................... 22 Unweighted Regression ................................. 139
3 Weighted Regression .................................... 1414 Survey Results, Part I(a) .............................. 1555 Survey Results, Part I(a), Modified ................... 160
vi
- (
FIT/GOR/ENS/gOM-1O/
1- Abstract
- The purple of this rearch was to define an appropriate tool to
assist AWACS'personnel managers in determining the personnel
acquisitions that zre required for AWACS operations. The approach used
to structure the decision process, was that of a Decision Support System
(DSS). The goal of the specified DSS was to provide an initial point to
better state personnel acquisition requirements, particularly with
regard to the experience of the crew force, and to provide a structure
for improvement in the DSS itself.
The dec isions supported by the designed DSS go beyond what has
been attempted oy ALYNCS air crew managers in the past, explicitly
addressing experience level inventories and the need to stabilize both
manning levels and experience levels in the various AWACS crew
positions. The overall design of the DSS to support these goals is
specified through the mechanisms of concept mapping and storyboarding,
and a kernel system is developed. Adaptive design is adopted as the
means for continued system development and evolution. -,
vii
A DUCTSION SU1PPR SNSTEM (SS) FORAIMYCS PERSONNEL ACDJISITION MAN41EENT
I. Introduction
Backqround
The Airborne Warning and Control System (AWACS) is an air battle
management system. Its functions are to provide airborne surveillance
and to control friendly aircraft in air battles in its surveilldnce
area. These functions are tactical in nature; therefore, AWACS belongs
to the Tactical Air Command (TAC), with the 28h Air Division (28 AD) at
Tinker AFB providing primary management of the system.
While the functions of AWACS are tactical, the personnel
requirements of the system are largely alien to TAC. TAC is primarily
oriented to fighter aircraft with one or two rated crew positions.
AWACS, in contrast, has 12 crew positions (Table 1), only two of which
are rated. Managing this diverse crew force requires the ability to
plan new personnel acquisitions for eac crew position with enough
accuracy to keep the crew force "stable" (i.e., maintain manning and
experience levels near nominal values). One of the consequences of the
TAC/AWACS personnel orientation mismatch is an absence of management
tools to support the personnel acquisition planning process across the
entire spectrum of AWACS crew positions. It is this absence of
management tools in the AWACS personnel management system which is the
concern of this study.
Table I -- AWACS Crew Component Information*
Author izat ionsFliQht Cvew Positiuns #/Crew Line Staff
1. Pilot' (P/CP) 2 103 182. Navigator* (NAV) 1 50 143. Flight Engineer (FE) 1 55 3
Mission Crew Positions4. Mission Crew Commander (MCC) 1 78 285. s enior Director (3D) 1 50 96. Weapons Director (WD) 4 2-8 87. Air Surveillance Officer (ASO) 1 54 88. Air Surveillance Technician (AST) 3 219 239. Communications System Operator (COSO) 1 101 610. Ccmmunications Technician (CT) 1 50 611. Computer Maintenance Techr ician (CDMT) 1 50 612. Airborne Radar Technician (ART) 2 100 5
* Rated PositionsMod 30-35
Programmed Flying Training
The Programmed Flying Training (PFT) program is a personnel
management effort originally designe< &o allow major commands to
identify new requirements for rated officers to the Air Training Command
(ATC). It was later expanded to cover non-rated crew positions. The
stated goal of the PFT program is to "sustain an inventory ... to meet
renuirements while retaining a credible combat posture" [Dept of the Air
Force, AF Planning Document, Change 6 : 31. For any weapons system
manned through the PFT, this amounts to maintairing each crew position
at an adequate manning level with an ewperience profile that is
sufficient to ensure the system is effective. Assuming ATC can provide
the training capacity to fill the requirements identified by the major
commands, the PFT may be capable of achieving its stated goals.
However, regardless of ATC capabilities, the PFT can only function
2
properly if the requirements identified by the major commtnancs are
accurate. Accurate requirements identification requires that personnel
managers have adequate management tools that allow them to properly
identify personnel needs. These tools have never been developed fully,
particu~larly for the non-rated positions. Major factors in the lack of
development of management tools for he non-rated positio; - have been
the lack of good criteria for measuring the experience level of the
force and the specialized (small sized) nature of the positions.
AACS and the PFT. Lkder the current AWACS PFT process, the 28 AD
hosts annual PFT conferences and quarterly review confereices at Tinker
AFB :o plan new crew acquisitions for a five year period. Participants
are from TAC, the Air Force Military Personnel Center (AFMPC) and the 28
AD. These three participants produce a consensus figure for the
appropriate numbers of new acquisitions required to keep the AWACS crew
force healthy with the constraint that the desired acquisition is
achievable. (By achievable, it is meant that constraints that bind each
organization in the acquisition process are not violated, e.g., more
bodies are not "bought" than TAC or AFMPC can afford, or more are not
bought than the 28 AD can give final training to in a given acquisition
cycle).
The primary 28 AD organization that deals with PFT issues consists
of representatives from 1) the 552 Tactical Training Squadron (TTS) and
the 966 Airborne Warning and Control Training Squadron (AWACTS), t-,
cover 28 AD controlled training issues (i.e., annual capacities,
changes, etc.), and 2) the personnl managers for each crew position,
who are tasked with providing thE= basic information on which the whole
3
PFT conference outcome is based, i.e., identifying the necessary new
acquisitions needed to keep their respective crew positions "healthy".
For AWACS, the before-mentioned lack of appropriate personnel
management tools has resulted in managers determining personnel
requirements through the use of individually developed, unstructured
heuristics. Each of the crew position managers has his own "rules" for
arriving at a best guess; these rules are not stated anywhere, and
change every time the manager for the position changes. For these
managers, estimating needs is typically an exercise in maintaining a
personal knowledge of all members of the crew force with respect to the
factors that impact the manning status of the crew position (factors
such as PCSs, separations, upgrades, ratings on evaluations, etc). This
information is "filed" by each manager in his own unique format for his
sole use. Fundamentally, the personnel requirements determination
process is based on the manager's intuition.
Given the diversity of factors that impact the status of a crew
position (discussed in the next section) and manager's lack of tools, it
is not surprising that the personnel requirements identified by the
managers are driven by short range considerations. Consequently, the
crew positions suffer from severe long term fluctuations in total
manning and in experience base. The manning fluctuations shown in
Figure 1 are representative of fluctuations that have been detected in
the AWACS crew force in past analyses of the crew force [Stone]. The
shortfall from the authorized manning level is plotted at six month
intervals. Of special note are the severe peaks at five year intervals.
Under current planning processes, these peaks are built into the system,
4
PERCENT SHORTFALL FROM AUTHORIZED LEVEL
18
14
CI
2
a-2
-4
82-1 B3-1 B4-1 85-1 86-1 87-1B2-2 83-2 84-2 85-2 88-2 87-2
SEM I ANNUAL I NCREMENTS
Figure 1 -- Crew Position Manning Fluctuations (representative)
and are repeated every five years.
A comparable graph for experience level is not directly available,
due to the lack of experience level criteria for most of the crew
positions. However, it is instructive to look at the proportion of a
crew position that is made up of first term personnel (for positions
that acquire personnel through new Air Force accessions), as an
indicator of the level of experience. For crew positions such as the WD
position, there is a direct correlation between the periods of high
manning in Figure 1 and a high representation of first term personnel in
the crew position. This is to be expected, since acquisitions for this
position have historically been made up of personnel new to the Air
Force. A representative graph for this condition is shown in Figure 2.
As in the previous figure, significant fluctuations in the data of
interest (assuming the proportion of first term personnel is accepted as
5
PERCENT OF FIRST TERM PERSONNEL IN
CREW POSITION
30
2S-
20
15
10
5
0 I TI - I I TI I82-1 83-1 84-1 85-I 86-1 87-1
52-2 83-2 84-2 85-2 86-2 57-2
SEMI ANNMUAL INCREMENTS
Figure 2 -- First Term Personnel in AWACS Crew Positions (representative)
an indicator of overall experience in the crew position) are observed.
AIWACS Pe-sonnel Structure
AWACS' mission obligations and capabilities are determined by
higher headquarters, and result from specific defense objectives
(ranging from NATO support to showing the flag in hot spots around the
world). Specific personnel authorizations for the system depend on the
extent of the mission for which the system is responsible. The number
of authorizations an air crew manager deals with varies by crew
position. This number has four primary factors:
1) the number of members required in each position for a full
crew;
2) the number of overhead positions authorized for stafffunctions;
3) the nuner of crews authorized per primary assigned aircraft(PAA), and;
6
4) the number of PAA.
Keeping these authorizations filled is a basic part of the air crew
manager's function. The personnel acquisition requirements he
determines for the PFT are driven at a very basic level by the need to
predict when some of these authorizations will become vacant. However,
he must also deal with changes in authorizations. All four of the
factors that drive authorizations are subject to change. The creation
of new units requires new staff authorizations (e.g., the creation of
the 962 AWACS at Elmendorf AFB created new staff authorizations to
provide the squadron staff) and changing missions can alter crew
compositions, PAA and crew/PAA ratio (e.g., the transition of airframes
from Block 20-25 to Block 30-35 configuration, reflecting mission
requirements for more surveillance and control capacity. The change in
mission will require authorization increases of more than 251. for both
the WD and AST positions).
The AWACS crew positions were presented in Table 1. As stated,
there are 12 crew positions; all are required for AWACS to perform its
mission. The current authorizations for each crew position are also
presented in Table 1. Factors to note in the table are 1) the absolute
differences in number of personnel being managed in each crew position,
2) the number of personnel required per crew in the position, and 3) the
size of the staff authorization for each position. These factors are of
interest because they indicate the differences in problem scope faced by
the individual crew position managers who collectively are required to
keep the AWACS crew force healthy and the system mission effective. For
(1), the number of authorizations in different crew positions vary by a
7
factor of close to five; some managers have many more people with whom
they must maintain contact under current management methods. For (2),
the number of authorizations per crew position per crew indicates that
some positions have more leeway in their manning tolerances than others.
This is also the case for (3), since positions with large staff
authorizations have reserve personnel in the system to perform flying
missions if shortages occur). These issues are discussed more
extensively in the following section.
Diversity of Factors among Crew Positions. All of the AWACS crew
positions must remain healthy for the system to perform its mission
effectively. However, the considerations facing each position manager
to achieve a healthy status for his position are not the same. A
contrast of Pilots and Weapons Directors provides valuable insights into
the differences in the factors that affect each crew position.
Pilots have well-defined performance factors that indicate the
competence of individual crew members and that can be aggregated to
indicate the experience level of the AWACS pilot crew force. Measures
such as total flying hours, number of landings, number of touch-and-goes
and number of air-refuel ings accomplished provide a direct measure of
the pilot's experience because the pilot is exercising the skills that
affect his experience level for every measure mentioned. The pilot
manager has reasonable measures for determining the experience profile
of the crew position he manages.
In the numbers game, however, the pilot manager has severe
problems predicting losses over time. Pilots have fairly extensive non-
AWACS assignment opportunities (e.g., staff and command jobs and flying
8
jobs oi other airframes are filled by AFMPC with AWACS pilots). With
pilots in short supply Air Force wide, AFMPC juggles its supply of
pilots, often giving the losing organizations short notice. More
critical to the pilot manager however, is the demand for the pilot's
skills in the civilian sector. The myriad of factors that change the
demand for pilots in the civilian sector changes the "leak rate" the
manager must contend with, and generally, he has no way of predicting
these changes effectively.
The story for the WD manager is much different. In the experience
profile area, as an example, even though total flying hours are logged
for WDs as they are for pilots, flying hours may be a poor measure of
the individual's competence for the following reasons:
1) the time spent by the WD exercising his primary airborne
skills, i.e., controlling other aircraft, may only be a small fraction
of the total time he spends in the air; it is not uncommon for the WD to
control no aircraft due to factors such as weather and AWACS systems
mal functions;
2) the work load an individual WD experiences is not uniform
because of different mission profiles; and
3) since there are multiple WDs on a given crew, the
accumulation of experience will, in general, not be uniform; for a
difficult control problem, the MCC may have his most experienced
operator be in charge of directing the problem.
This lack of good experience measures has resulted in the current
practice in the AWACS community of designating WDs as fully experienced
after they have been flying for ne year following graduation from
9
flight training at the 552 TTS. (In contrast, non-airborne WDs,
operating at ground TACCs, are rated for experience by the number of
controls they direct. With current data availability constraints, this
is not an option for rating the AWACS WD force).
In the manning level arena, however, the WD manager is faced with
a crew force that has limited non-AWACS assignment opportunities and
almost no demand in the civilian sector for its military skills. Losses
are much more predictable over a longer period of time in the WD crew
force than they are in the pilot crew force.
In general, the management of each crew position entails
consideration of a unique mix of factors. Some of the critical factors
that have differing impacts on the different crew positions are:
1) Non-AWACS assignment opportunities -- the personnel system is
closed for some crew positions (ARTs, AST) and crew losses are mostly
due to separations (which are fairly predictable), while the system is
open for others (P, FE, CDMT) and short notice PCS losses are a major
factor in disrupting the position's manning stability. (Closed and open
refer to whether the AFSC for a particular crew position is "unique" to
AWACS. If the ASC is closed, then the personnel in it are "trapped" in
the AWACS personnel system).
2) Experience measures - some crew positions have well-defined
measures for determining experience of its members (e.g., Flight Deck),
others have less well-defined measures (CDMT, ART), but operate in a
technical area that lends itself to rather unambiguous measures (repair
record), and still others have arbitrarily defined measures that provide
no guidance (WD).
10
3) Redundancy in the crew position -- AWACS mission capabilities
are sensitive to crew positions. Crew positions that are manned one
deep per crew can degrade system performance more rapidly than crew
positions having multiple members per crew. If there are insufficient
FEs, the E-3 will never get off the ground; if there are insufficient
WDs, the system can still operate at some degraded level with a crew
that is undermanned in the WD position.
4) Crew position independence -- manning of many of the crew
positions can be managed totally independently of all of the other
positions; the manager's only consideration is the health of the
position he manages. Other positions are not independent. For example,
a significant portion of the MCC force is drawn from the SD population,
while almost all SDs come from the WD position. Therefore, the WD
manager may be faced with a deficiency in WD manning because of a
decision outside of his control to rob the WD personnel pool to fix the
SD pool. Further, this deficiency is generally not only a numbers issue
but involves the WD experience base because the most experienced Wf are
selected for upgrade to the SD position.
Fundamentally, the structure of the crew is non-homogeneous; crew
positions can be grouped into different categories (flight - mission;
rated - non-rated; officer -- enlisted), and each category imposes
special considerations on how personnel are managed. Regardless of
differences in the positions, all should be managed with equal skill.
Problem Statement
If AWACS is to be as effective as possible in its mission, it must
be properly manned. The goal of the PFT is to achieve this manning,
11
quantitatively and qualitatively, in such a manner that the crew force
is stable. With the current lack of management tools, AWACS managers
have not been able to realistically determine their manning needs.
Consequently, the AWACS crew force has been chronically unstable, both
in absolute numbers and in the experience base of the personnel
constituting the crew force. If AWACS is to be properly manned, the air
crew managers require tools to integrate, present, manipulate and
analyze information that is pertinent to the personnel acquisition
process.
Object ives
To approach the design of a tool to assist in accurately prc-dict
manning acquisition requirements for the PFT effort. To meet this
objective, the following sub-objectives must be attained:
1) Establish a "management paradigm" for determining needs that
is simple, effective and uniform for all air crew managers, regardless
of the idiosyncrasies of the position structure;
2) Identify factors impacting manning level;
3) Identify factors impacting experience level;
4) Identify where appropriate data is located;
5) Determine appropriate models for analyzing the data, and;
6) Provide an adaptive design "road map" so the tool remains
viable in changing circumstances.
The following chapters will discuss the methods for accomplishing
the above steps, the specific design, conclusions reached during the
research effort, and recommendations for further areas of research on
this subject.
12
II. METVCDCLDGY
Sitability of a Decision Support System (DSS) Approach
The problem the AWACS managers face in determining personnel
requirements is not "well structured". Each manager deals with factors
impacting manning that are common to all positions, and with factors
unique to his own. Some factors are dynamic, and possibly worse, the
importance of factors are also dynamic. Because of the diversity of
factors and their dynamic nature, the users of the system to be designed
(crew position personnel managers) cannot provide a functional
specification of the decision process to be supported. Keen argues that
the user's inability to provide a functional specification is one of the
hallmarks of a process for which a DSS is appropriate [Keen : 15). The
lack of a functional specification prohibits the use of a purely
prescriptive modeling approach; the problem's nature is such that an
interactive, semi-structured process is called for.
Current practices in the AWACS personnel requirements process have
never emphasized long term effects of the factors managers consider in
making their manning decisions. The impact of different factors on
short term manning profiles are reasonably well understood by the
managers -- in most instances, the short term condition of the force is
all that is managed. However, to be effective in stabilizing the crew
force, managers must understand the long term impact of their
acquisition decisions. They must understand the long term impact of
managing crew positions for short term health.
Any system that implements a capability to examine the effects of
current management decisions on the future condition of the crew force
13
can help the manager understand long term effects of his decisions. In
particular, a DSS is an ideal tool in this respect, since it can
implement the desired capability in a system that is designed to
"facilitate the ... [decision] process rather than to force-fit it into
a designer's notion of the best process" [Young : 1]. This allows the
user to focus on the problem, further clarifying his understanding of it
(possibly gaining new insights), and make decisions. This is a distinct
departure from traditional "support systems" where the user is often
consumed by the mechanics of the system, to the detriment of process
understanding. Traditional systems design is a "builder" driven
process; as a result, systems are designed to support the builder, not
the user.
A further reason for choosing a DSS approach is the necessarily
evolutionary development of the management system. Changes in any
system developed will have to be made, since:
1) the importance of factors will change as corrections are made
in the force structure, with new factors requiring consideration and
some current factors becoming unimportant;
2) the current understanding of factor importance is such that
mistakes will be made. The system will need to be modified to adjust
for the user's better understanding of his needs; and
3) the use of the system will change the user's decision process
from whatever process is first "captured" in the DSS. The DSS will need
modification to support new processes its use engenders.
A DSS, oriented around adaptive design, provides a structured framework
to evolve the system being developed.
14
Adaptive Desicin
Adaptive design is a systems design approach where the philosophy
is "start small and grow" [Valusek : 105 - 111]. Adaptive design shares
the elements of the traditional systems design approach, i.e.,
requirements analysis, design, development and implementation. However,
in contrast to the traditional approach, it does not fix a rigid systems
specification before development starts. Recognizing the user usually
cannot fully specify the system desired before development starts, the
adaptive approach stresses putting a kernel system into the user's hands
and evolving it as a result of the user's reactions.
Adaptive design is critical if the decision process being
considered is to be effectively supported. Keen provides a succinct
graphic that describes why this is so (Figure 3) [Keen : 16). The three
elements in a system design process are the user, builder and the system
User
user middle-outlearning design
personalized facilitatesuses implementat ion
pressure for evolutionSystem e Bu iilder
evolution of system function
Figure 3 -- An Adaptive Framework for DSS
Adopted from CKoon 1 163
being developed. In design processes to which the DSS approach is
suited, there are necessary two way linkages between all three elements.
15
"A system is a "DSS" only if each ... is relevant to the situation"
[Keen : 17].
For the decision proce 3 discussed here, all linkages are
necessary for the production of a functional system. The System - User
linkages are required because the AWACS manning determination process is
not well structured. Users will gain insights as the system starts to
structure the process; the user will therefore change his use of the
system. The User - Builder linkages are required to ensure the proper
system is implemented. The middle-out design approach enhances
communication by placing a kernel system in the user's hands, providing
a design element that cai be used by both 'he user and builder to define
the problem. Proper definition of the problem facilitates system
implementation. Finally, the System-Builder linkages must exist. The
altered use of the system seen in the User-System loop can only progress
so far before the system requires adaptation to support the new process.
The builder then provides new functions to the system, and the cycle
starts over again.
Expanding on the above theme, Ackoff makes the following
observation about designing support systems:
It must be assumed that the system that is being designed will be
deficient in many significant ways. Therefore it is necessary toidentify the ways in which it may be deficient, to designprocedures for detecting its deficiencies, and for correcting the
Jystem so as to remove or reduce them. Hence the system should be
designed to be flexible and adaptive. ... No completelycomputerized system can be as flexible and adaptive as can a man-machine system. [Ackoff : 15)
The key points here are 1) that the system must allow for evolution if
it is to be effective, and 2) that systems that are designed as man-
machine systems are more capable of adaptation than are purely
16
mechanical ones. Adaptive design in the DSS context .s highly attuned
to both of these factors. It requires an evolutionary framework for thE
sy.e. that is part of the system and can easily allow fur the framework
in the man-machine inter lace of the support system.
An adaptive desiign appyoach also has "immediata gratification"
attributes that make it particularly desirab'e when contrasted to
traditional systems development approaches. Some of these attributes
are:
1) Short response times to user inputs. Where problem
understanding is "foggy", extracting the process from the users is a
time intensive process. If the user is to make a time commitment, he
must see a return in the short term.
2) User participation in the design of his system. The user
probably is aware of how foggy the problem is. If so, he kncws mistakes
wil7 be made. Providing a system that is responsive to his changing
perceptions, many of which will result from the process of defining the
problem, allows the user to "get the system he wants" not the one he
thought he wanted (or the designer thought he wanted). The prorass of
watching the system evolve, due to his inputs, assures the user that the
system will be what he cesires.
3) User perception that he is being understood. As a corolla ry
of (2), the communications between the "ower" of a decision process and
the builder of the support tool will be poor because of the different
realms of expertise. Mistakes will occur because of poor communication
even in problems that jre well understood by the user. According to
Cerveny, et al, "the use of the traditional [systems design] approach
17
does not adequately address the underlying issue of poor communicat irn
between the user and the analyst" [Cerveny, et al : 54]. Adaptive
design provides a framework for identifying mis-communications and
correcting them. The feedback in the communications process, indicated
by evolutionary changes in the system being designed, reassures the user
that he is getting what he wants.
The Vehicles of Adaptive Design. Adaptive design, as being
researched at the Air Force Institute of Technology (AFIT), above all,
provides a means for structuring the development of support systems.
For the purposes of this study, three major structuring devices were
e ployed. These devices are concept mapping, storyboarding via ROMC,
and the "hook book".
Concept Mapping. The first stage in the adaptive design
process is developing some understanding of the decision process being
supported. Concept mapping is an educational tool that is adaptable to
developing this understanding (Valusek : 107) It provides a medium for
identifying elements of the decision process and describing the
relations between the elements. With a map of the decision elements,
the builder and user can pick an appropriate "initial kernel" (subset)
system to be developed. (However, the actual kernel that provides the
seed for the system to be developed is usually not fully specified until
after storyboarding [described in the next paragraph] is couplete).
Stor-yboarding. After concept mapping, the initial kernel
system is described pictorially on paper to ensure that the kernel is
what the user actually desires. This pictorial description of the
system is called "storyboarding" and was proposed by Andriole as a
18
technique for capturing the users views on the important aspects of the
decision process being supported [Andriole : 463-469]. Storyboards, in
the adaptive design realm, provide the specification of the system to be
designed.
The process of storyboarding, with its graphical presentation of
the DSS, helps clarify the decision process layout, and usually leads to
iterative changes in the initial kernel that is identified for
development. At the point where the user is satisfied that the
storyboards reflect his desired system, the "final kernel" has been
identified. System development then starts, and further adaptation of
the kernel can proceed until the desired system is evolved.
ROMC. In conjunction with storyboarding, the
Representation, Operations, Memory aids and Control (ROMC) structures
described by Sprague and Carlson provide a valuable method of ensuring a
storyboard design is complete [Sprague, et al : 103-118].
Representations are the presentation of the information the manager
requires. The focus is on what form the presentation takes to convey
the required information. Operations are the functions executed by the
DSS to provide representations. The focus is on how the decision
maker's information is produced. Memory aids are the mechanisms
required to ensure the user knows where he is and what he has considered
in the decision process. The focus is on providing orientations for the
decision process. Finally, Control mechanisms are the mechanisms that
allow the user to direct the functioning of the DSS. The focus is on
facilitating system use. All four of the ROMC factors must be
co-sciously arcounted for in the design if an implemented systan is to
19
be functional.
The "hook book". Finally, the system must have a mechanism to
facilitate adaptation. A method for identifying new requirements and
system deficiencies, that is readily available while the system is in
use, is necessary if the system is to adapt rapidly. A "hook book",
described by Valusek, provides the necessary mechanism for determining
new systems requirements [Valusek : 109J.
The hook book is a software facility embedded in the implemented
system that is accessible from any point in the system. Fundamentally,
it is an on-line memory aiding facility for capturing user thoughts
about the system (or the decision being supported). The notes to be
captured are structured (see Figure 4), and have four distinct parts,
these being:
1) the date of the note entry;
2) a label for the entry;
3) a brief description of the idea that prompted the note, and;
4) the circumstances that led to the idea.
DATE: LABEL:IDEA:
C IRCU"MSTANCES:
Figure 4 - Hook Book Entry FormatAdapted fro CValueok 1 102
Each part serves a specific purpose. The date allows
chronological sorting of ideas, the label allows sorting of ideas by
20
system function, the description provides the basic memory aid as to
what the user wants, and the circumstances provide the key to "detailed
recall of the (user's] idea during requirements elicitation" [Valusek
109). Together, the four parts of the hook book are sufficient for the
orderly identification of new system requirements, and ensures the
system can evolve and remain useful.
These three "vehicles" together are the mechanism through which an
adaptive design approach can be implemented. They ensure a systems
design environment that fosters ready communication and rapid
implementation of Lutabiguously communicated desires (user's) and
intentiors (builder's).
Establ ishing a "Paradicim"
To design a proper paradigm, it is necessary that there be a clear
understanding of who the users are, and what their areas of expertise
(and limitations) are. If users are properly identified, a paradigm
that provides "procedures and data representations that fit well with
specific managers' established activities" can be designed [Meador, et
al : 162). This is important if the DSS is to be accepted. A support
system that presents a non-technical manager with statistical summaries
to guide his decision process is an effective way of ensuring the system
is not used. Likewise, providing a system that 1) allows the manager to
view data pertaining to his problem (even at some level of aggregation
and in various formats), without 2) providing a method of synthesizing
its impact on the decision, is an exercise in Management Information
Systems (MIS), not DSS. The above MIS type systems are designed under
what Ackoff calls the "give them more" syndrome, which causes managers
21
to "suffer ... from an over abundance of irrelevant information [Ackoff
: 147). A proper management paradigm will make a management system
easily used; a poor paradigm will condemn the system to oblivion because
it requires too much of the user.
Establishing a proper paradigm rests on the identification of
users. For this study, the users to be supported and their "general
characteristics" are readily definable because of the 28 AD air crew
management structure; identification is basically a matter of
observation to be detailed in the following chapter.
Factor Identification
Many of the factors that bear on this problem were identified in a
previous research effort in this area [Schneider]. Some were addressed
adequately, some need more detailing before they can be useful in an
implemented system. The factors that received the most attention in the
previous research effort were those that dealt with manning level.
While the issue of experience level was raised, methods for determining
these levels were treated scantily. Identification of measures of
experience and methods for profiling the experience base of the crew
force were not adequate to guide a decision process attempting to
stabilize experience in the crew force.
The factors were identified in discussion with air crew managers
and analysts at the 28 AD during the course of multiple PFT conferences.
In most cases, the 28 AD PFT participants indicated that they were able
to identify factors that were important to them, and that they attempted
22
to estimate their levels and impact on the AWACS crew force in making
PFT recommendations. They also indicated that many of the estimates
were indefensible, as they were generally "best guesses".
Maning Level Factors. For the purposes of this research, the
factors previously identified as drivers of manning level were accepted
for the kernel design. Phone interviews with AWACS managers verified no
significant changes in the set of perceived factors have occurred
[Guzec]. These factors are related to two primary areas: 1) personnel
losses from the system and 2) changing authorizations.
The personnel loss factors are derived from historical databases
and are: 1) the time spent in the Air Force, and 2) average "tour"
length. The first factor indicates the losses AWACS will suffer due to
separations (end of enlistment, retirement, etc). The second factor
indicates the losses due to Permanent Change of Station (PCS) out of the
AWACS community.
The factors that bear on authorizations for ejJh crew position
were discussed briefly in chapter one. They are PAA, crews per PAA,
staff authorizations, and number of personnel per crew. These factors
are set outside of the PFT process, being part of overall Air Force
structuring policy. As such, they are planned well in advance of any
PFT planning on which they bear and are matters of record.
In this study, none of the above factors will be addressed from
the perspective of why thay occur. Rather, they will be used solely for
the purpose of predicting manning shortfalls. The issue of (for
example) why some mid-career flyers choose to separate before retirement
is not examined. The fact that some percent historically do separate
23
is not examined. The fact that some percent historically do separate
early is, however, of use in predicting acquisition needs.
Experience Level Factors. Estimating the experience level of a
crew position entails identifying the factors that can describe
experience, building a model to estimate the individual crew member's
experience (as a "score"), and then using the estimated score to profile
the experience of the position. This profiling can be done in any
nuner of ways, e.g., it can be described in terms of mean experience,
median experience, or via a graphical presentation of the experience
distribution. This process is depicted graphically in Figure 5.
Unfortunately, experience level quantification is difficult to
accomplish for most of the mission crew positions. This is because no
direct measures for assessing experience of the individual crew members
exist. Estimates of individual experience must be based on surrogate
measures, where the choice and significance of these measures are
unc lear.
The first step in assessing the experience of a crew position is
determining what surrogate measures are useful in developing "experience
scures" for the individual crew members and then developing the scores.
Examples of possible surrogate measures are: rank, years of service,
Standardization Evaluation (StanEval) ratings, number of flying hours,
number of missions flown, number of exercise missions flown, etc.
Since appropriate surrogate are uncertain and the number of possible
surrogates needed to produce an "experience score" may be large, any
method used to identify factors and produce scores will have to be
capable of identifying and aggregating multiple factors in the face of
24
user uncertainty. Two possible methods that can identify the necessary
factors and directly aggregate them into usable scores are considered.
Identify Factors Usefulin Describing Experienceif
Define Relationships Amongthe Factors in a Model
Calculate an "Experience Score" for eachIndividual in a Crew Position
Personal are inserted theAttributes from D b Modelfrom Databases into
assigned as a Experience rSrultingScore e
(not for use in in anthe model, i.e.,
no recursion
Analyze Distribution ofExperience Scores
Figure 5 - Experience Profiling Processs
These are 1) Multi-Attribute Decision Analysis (MADA), and 2)
regression. These methods are discussed below.
25
MIDA Methods. Estimating experience for the crew members can be
approached from the perspective of "expert judgement". There are
experts in the AWACS community who are familiar with the jobs each crew
member performs and who routinely make mission decisions based on their
estimates of who is most capable. If an expert judgement approach is
used, the issue becomes one of eliciting from the expert the uncertain
factors that enter his decisions, their relative weights, and the trade-
offs (utility) within them. Multi-Attribute Decision Analysis is an
analysis discipline geared to structuring problems such as this, where
decisions (answers) are plagued by uncertainty [SmartEdge Documentation
: B-2].
To elicit a set of possible factors from the experts, surveys are
used. The intent of the surveys is to establish ranking among the
possible factors (to reduce the set of possible factors to manageable
proportions), to establish that the expert being surveyed is consistent,
and to look for possible factors that have not been considered. Two of
the standard surveying methodologies for developing measures on 1) the
ranking of factors and 2) respondent (and factor) consistency are based
on semantic scales and pair comparison, respectively [Chan].
Solicitation of new factors to be considered can be achieved through
simple questionnaires.
Appropriate relative weights for the factors and the utility
function that describes the relationship of the states within each
factor can be determined through a series of reference lotteries. In a
"basic" reference gamble procedure, sequences of reference gambles are
conducted where, prior to the gamble, payoffs for the reference gamble
26
are established. Then, iteratively, probabilities (p) for winning the
gamble are assigned, and the certainty equivalent (CE) for the gamble
evaluated (where the CE is the point where the expert being consulted
indicates he is indifferent to the lottery and the certain value).
Plotting p verses CEyields the utility curve for the factor being
examined [Holloway : 422).
Modifications of the basic reference are available. Holloway
suggests that the 50-50 is often the best gamble, since people often
think in terms of go-no go choices [Holloway : 428). Here, Cs are
first determined for p = 0, .5, and 1. For the rest of the procedure, p
is fixed at .5 and the CEs for each gamble elicited. The probability
for the certain value can be determined from the gamble conducted.
The lotteries described above are conducted in a structured
question and answer session, where the expert indicates his preferences
to the series of proposed gambles. This lottery structure elicits both
the factor weights and the utility functions that underlie the expert's
estimations. Combination of the weights and utilities in an additive
formulation produces a "score" for individual crew members when the
formula is evaluated using the crew member's attributes for each factor.
(Reasonably easy to use commercial software packages that conduct these
types of question and answer session exist, e.g., SmartEdge, ver (3)).
Regression Methods. Two classes of regression treatments suggest
themselves as candidates for determining appropriate factors for
constructing an experience score. Both require that the crew force be
"categorized" in some manner prior to regressing to identify factors.
The "categories" are then used as the response to be fit by a regression
27
of personnel data. The set of regressors that are significant are the
factors to be used for producing experience scoyes.
Logistic Regression. Typically this type of regression falls
into the "logistic regression" arena. In cases where the response
variable is categorical, "theoretical and empirical considerations
suggest ... the shape of the response function will frequently be
curvilinear Neter, et al : 361]. Logistic transformations of the
response function produce responses of appropriate shape and have the
desirable property of being asymptotic to the extreme values of the
response, i.e., predicted values produced using the logistic response
function will not fall outside of the range of the categories of the
original response Neter, et al : 356-363].
Multiple Regression with a Binary Response. If the initial
categorization of the crew force is restricted to two groupings, (i.e.,
split the personnel into an "expert group" and a "less than expert
group"), the response is a binary response. In this case, a multiple
regression is feasible Neter, et al; 354-361). As in all regression,
the factors to be used are those that test significant in predicting the
responses.
The methods discussed above for determining appropriate experience
factors and producing experience scores are only some of the possible
methods for dealing with the problem at hand. Certainly, they do not
constitute the entire spectrum of options. Each has problems associated
with its use. In some cases, the problems are inherent in the method
itself (regression using binary categorical responses). In others, the
problems encountered in using a particular method are related to the
28
environment in which the method will be applied (both MADA and logistic
regression share this feature).
In the following chapters, which deal with design and
implementation of the DSS, the particular problems with each method is
discussed, and a choice of method for the kernel design is selected and
explained.
29
I II. Overal 1 System Design
General
Designing for adaptation is critical for the decisions being
supported by this research effort. The decisions to be made go beyond
what has been attempted by AWACS air crew managers in the past,
explicitly addressing experience level inventories and the need to
stabilize both manning levels and experience levels in the various AWACS
crew positions. Since the decisions being supported cannot follow an
already established decision process, an initial process must be
established and then adapted as the users determine exactly what is
required to fully support their decisions.
The necessity for supporting system adaptation is also inherent in
the goals of the decisions being supported. The goals of stabilizing
manning and experience in the AWACS crew positions are geared to change.
Since the system will be changing the environment it addresses, it must
also change to continue to address the environment of interest and
successfully achieve its goals.
The specific design of the DSS examined in this research effort is
driven by two major concerns: conceptual design considerations that are
related to information requirements and techniral considerations that
deal with how the system is implemented. Both of these "design
elements" must be addressed in the adaptive design environment. The
conceptual aspects of the design effort deal with determining user
requirements, identifying a decision process to support the
requirements, and establishing the infrastructure necessary to support
the system's evolution. The technical design considerations deal with
:30
providing the environment where the decision process occurs.
The above delineation of design concerns is made only to indicate
that there are two dimensions to the design problem, not to imply that
the design issues can be treated separately. One (the conceptual
portion) is primarily the realm of the user and designer, while the
other (the technical portion) is primarily the realm of the builder.
However, there is not a clean division in these two efforts. F ir
example, the act of specifying that the system be able to adapt has an
impact on the technical portion of the design effort, and requires
choosing specific facilities to ensure the necessary adaptations to the
system are supported.
The following sections of this chapter will discuss each of these
design areas. While the areas are not cleanly divided in practice, the
discussion of each area will b' coarated for clarity.
Conceptual Desicin Considerations
The concept .1 design issues that need to be considered in
developing a DSS can best be addressed from the context of two elements
of the adaptive design process. These elements are the information
requirements determinations (IRD) phase and the information requirements
analysis (IRA) phase. During the "earlier" IRD phase, user requirements
are expressed as general I -ts of possible requirements. These
"candidate" requirements are subjected to scrutiny during the IRA phase.
It is here that the actual "succinct" list of requirenents defining the
system's form is generated. Each of these phases are distinct in their
goals. However, a common set of design "tools" can be used for each,
and are those that were discussed under the methodology of adaptive
31
design.
The conceptual design issues in this research were addressed in
four distinct phases. These are: 1) concept mapping, 2) storyboarding,
3) kernel selection, and 4) establishment of a paper hook book to
capture the road map for future development of the system. These four
phases are discussed below.
Concept Mapping. Initially, a concept map was constructed to
describe the elements of the problem and their interactions. The
concept map was generated by this researcher and another former 28 Air
Division analyst [Sumner]. (In this instance, these analysts acted as
proxies for the actual decision makers to be supported). Both
participants in the concept map's generation have had experience with
the AWACS PFT cycle, and have participated in AWACS PFT conferences in
advisory capac it ies.
An aggregated concept map for the acquisition problem is shown in
Figure 6. Relationships among the major elements are easily seen in
this high level presentation.
The detailed concept map for the overall acquisition problem is
included in Appendix A. The map is by no means complete; its function
is to delineate the problem space sufficiently to a'low the design
process to continue with some assurance that the problem's total scope
has beer, considered.
Storybtarding. Storyboard production for this system occurred in
three cycles. The initial storyboards for the system discussed here
were conceptualized at the 28th Air Division in mid-1987 in response to
dissatisfaction with aspects of the storyboards produced by Schneider
32
Internal IMPACTED BY External(2BAD) (AF/TACIMPC/...)Policy Policy
AWACS CHARACTERIZED) BY I Eprec --
ct oAcquisitionNeeds
DE T v I of People T EAE PO02 E NE
I DN S
Y EE
l EStructuref ll REFLECTS Gains
Figure 6 - High Level Concept Map of Acquisition Problem
[Schneider]. This early definition of the desired system occurred
during brainstorming sessions among the 28th Air Division analysts (Capt
Geo. Mark Waltensperger, and Lts Kevin M. Holt and David L. Sumner).
The evolution of the storyboards continued at AFIT when Holt and
Sumner were formally exposed to DSSs and storyboarding. Here, the AWACS
PFT problem was used as a vehicle to better understand the structures
and goals of storyboarding. In the process of developing an
understanding of storyboarding, a system that would support 28 AD
personnel acquisition was captured in a "complete" set of storyboards.
33
Finally, the process of this research led to modifications of the
storyboards as the problem was addressed in more detail. The final
storyboards were evaluated by Sumner, acting as a proxy decision maker,
to ensure that they continued to define the form of the desired system
and to ensure that the system addressed "all" of the user's decision
support needs. Three of the storyboards are shown below, and the ROMC
(as discussed in chapter 2) of each is detailed. The complete set of
"final" storyboards are included as Appendix B.
RIpresentatLve Storyboards. Figure 7 is the home display for
the system when it is first entered. (As such, it is also the kernel of
the DSS, which is discussed in detail in the next chapter). The user
enters the system to a display that graphically represents the current
experience profile of the crew force he manages.
Here, the state of the crew position is shown by the distribution
portrayed in histograms. The left histogram indicates the impact of
known losses (i.e., known retirements, known PCSs, etc.) on the
position's experience while the right indicates a best case for
experience based on recapturing all known "eligible" ex-AWACS personnel
in the Air Force manning pool. (The top portion of the stacked bars
indicate experience contributions by mebers being lost or potentially
recaptured).
Operations are minimal for this screen. At the decision command
level (top menu), the user has the option of changing the crew position
being profiled and of selecting different manning profiles. At the
systems support command level (bottom menu), he can select any option.
34
POS: MCC RANK: ALL FUNCTION: MANNING PROFILE - CURRENT EXPERIENCE
POSITION MANNING PROFILE
EXPERIENCE LEVEL HISTOGRAMSAuthorized: 86 Authorized: 86
Total graphed: 80 Total graphed: 76I
1 16N
El 12 (PE 10R1 0EN 6CE 4
G 2R0 0---U I t I , . .P
EXPERIENCE FACTOR EXPERIENCE FACTOR - PROJECTEDCURRENT WITH KNOWN LOSSES RECAPTURING ALL TRACKED PRIOR AWACS
THE COMING YEAR MEMBERS ABSENT AT LEAST 3 YEARS(THIS IS A BEST CASE PROJECTION)
LOSSES BALANCE GAINS
Help Notepad dookbook I Assuertions I Prior Screen Prin Q it
Figure 7 - Home Representation of the DSS
Mevory aids are embodied in three separate areas of the screen.
First, the top line of the screen indicates to the user specifically
what he is looking at. Second, the command bar has the applicable
command that generates the representation highlighted (and also
indicates decision commands that can be executed during the current
operation, i.e., change the crew position to be examined). Last, the
coding of the bar segments is documented below the histograms.
35
Finally, the control mechanism of the system is embodied in the
command selection procedures and options available to the user. In this
instance, the user can select aiy of the system support options from the
bottom command bar (and these commands are purposely separated from the
commands that bear on the decision process), or he can examine other
manning profiles that are of interest to him.
Figure 8 is the display where, ultimately, the final decision i5
made about which acquisition strategy is best. Manning AWACS is not an
Capabilities: Zoom engaged -- Details of tagged time + 6 months
fACS CAPAIILITIES
94------------ -- ---------------e i
r aCudi
86 <UN
86------------ ----- --- - ----------------- ar nt ii ma ue m
I,-+J--+I i i t i i t iLIMITING POSITIONS CENTER DATE: APR 91 DATE STEP SIZE: I MONTH
1. P -6-5-4-3-2-1 01234562. FE3. CDNT 89 89 87 86 685 893 84 6 98 99 90 924. AST .2 .4 .9 .3 .3 .1 .8 .2 .1 .3 .7 .8 .1
Help Notepad HOOkbook Assumptions Prior Screen Main Menu I Print QuitFlI F? F3 F4 F5S MiF6 Vi IO
Figure 8 - Capabilities Representation of the DSS
36
end to itself; a mission effective AWAC capability is the goal, and this
goal is why manning acquisition is an issue.
For each of the acquisition strategies the manager considers, he
can examine the manning consequences on AWACS mission capabilities.
Capabilities information is represented three ways: graphically by a
line that is plotted relative to reference values, numerically at a
detailed level in a table below the graph, and, finally, in a table that
shows crew positions in the order in which they limit capabilities the
most.
Operations include the ability to "zoom" to each crew position to
examine its specific impact on capabilities and accessing system support
functions such as getting printouts of the representation or capturing
thoughts in the hookbook or notepad. ("Zooming" is accomplished by
selecting the appropriate number from the capabilities limitation list).
Memory aids and control mechanisms are largely the same as those
discussed in Figure 7.
Figure 9 is the storyboard for the DSS function of keeping the
user abreast of scheduled changes in the manning structure of the AWACS.
It is a tabularly oriented representation of the changes in the four
factors that cause manning requirements at the systems level. The user
is shown the current values for each of the factors in the BASE column,
and is shown when and by how much (relative to the current values)
changes in factors will impact each crew position.
Operations are restricted to causing the crew positions and their
relevant information to scroll in their windows and to accessing system
37
FORCE STRICTURE
1989 1991 1996 2000 BASE
PAA __ f _ +3 29
CREMS +.2 +.2 1.7
POs AIRCRAFT MODIFICATIONS
30 -- 35SD 1 1 -WD +2 3ASO IAST +2 2ART 2COHT I (--CSO +1 1CT I
POS I ORGANIZATIONAL CHANGES -
S ADD 962 '
NAY +1 14 (--FE +1 3MCC +1 28SD 9WD 8ASO i 23 T
I I F2 F3 4F5 n P F7 F10
Figure 9 - Force Structure Representation of the DSS
and decision commands. Mewry aids and ccntrol mechanisms remain the
same as in figure 7.
These three storyboards are representation oriented, having
minimal operations input from the user. A heavily operations oriented
figure is discussed later in this chapter.
Storyoarding Contributions to Lhwrstanding System Needs. As
indicated in the previous chapter, both the concept map and storyboards
38
impact the selection of the kernel system to be developed. The
storyboard process immediately reinforced this fact. This process
reduced the scope of what was addressed to a small portion of the
concept map, choosing not to directly address why perturbations (from
desired stable conditions) of AWACS manning occurred, but rather
focusing on how to correct perturbations when they occur. The
storyboards focus on a system that allows managers to identify manning
deficiencies, examine means of correcting the deficiencies, and portrays
the consequences on manning structure of whatever acquisition decisions
are made.
It was also during the process of storyboarding that the issue of
exactly "who" the users being supported were was addressed in detail.
Until this point, the problem was looked at largely from the perspective
of stabilizing the crew force in a prescriptive nwrner, not from the
perspective of a decision process. User needs and impact on the system
had not been considered. The specific characteristics of the users were
addressed at this point, and the "shape" of the DSS that could support
them determined. Consequently, storyboarding was critical to guiding
the formulation of the decision process to be implemented in the initial
system and the paradigm through which the decision cycle would be
expressed. The issues of decision process and paradigm are discussed
next.
The Decision Process. As stated above, this system
addresses areas in AWACS crew management not addressed previously.
Lacking a process to model, an initial process to support the user's
39
decisions had to be developed. This decision process could then be
modified (or totally new processes implemented so that the problem can
be addressed multiple ways) as the users determined new needs (or
desires) for support in addressing personnel acquisition decisions.
The issue of supporting particular decision processes is widely
discussed in the DSS literature. It is generally held that, to provide
effective support, a DSS must not constrain decision makers to any one
view of how to approach a decision. The user should be allowed to
establish his own "best" way to address the problem being supported
[Robey, et al; Cohen]. In the initial stages of the design of this DSS,
however, some process, not necessarily the "best", was required as a
starting point from which growth and adaptation could proceed. A
decision process was therefore established.
The basic decision cycle established by the storyboards is to
identify the current state of the crew position with respect to
experience and manning level, and then proceed to a "what-if" cycle to
examine various decision alternatives. During the "what-if" cycle, a
manager can assess multiple strategies for correcting crew deficiencies
based on a clear understanding of the crew's current state. Support is
also provided for direct examination of the personnel databases and for
examining scheduled changes in the AWACS force structure that will alter
the manning state.
The Decision Paradigm. Typically, the managers to be
supported by the proposed DSS are senior crew mebers assigned to a
Tinker AFB AWACS unit. Uniformly, the managers assume the job of
40
managing their respective crew positions as a duty in addition to their
normal flying responsibilities. To support these managers effectively,
system design must occur with the understanding that crew management is
only one facet of the managers job, and not necessarily the most
important (in the managers eyes). First-and-foremost, it must be
recognized that the managers are crew memnbers, not personnel
specialists.
The management paradigm for this system must reflect the user's
capabilities and allow him to make decisions without being burdened by
the system. A proper management paradigm makes the management system
easily used; a poor paradigm will condemn the system to oblivion because
it requires too much of the manager.
To ensure the system is useful to the identified user, a simple
graphical paradigm was adopted. The estimated state of the crew
position is depicted as a line that can be compared to a line
representing the desired crew state. (It is possible that as managers
become familiar with the system and gain expertise in making decisions
with its aid, they may find statistical summaries useful, particularly
,.hen trying to defend their need estimates to higher headquarters. This
can be added to the design at a later date. Initially, however, the
system should be limited to graphical portrayals of decision results.
These are easily understood and tend to limit the manager's sense of
informat ion overload).
In Figure 10, the dashed line represents the desired manning level
for the crew position, and is specified by authorization levels mandated
41
by force structure considerations (show in Figure 9). The solid line is
POS: NCC
POSITION SOEPLE MANNING PROFILE ACSAPABILITIEST
AU 106
100""
E
80
AN 82NIN 76S - j . -t I I I j I I
YEAR (INDEXED WITH RESPECT TO CURRENT YEAR)
INTERNAL EXTERNAL CURRENT SAVE SAVE SAVE
TRAINING CODE 55 20 4 22 5FLY HRS AVG TOS .131 4.5 .181 5
BODIES 20
LABELS RED (NON) BLUE 'IF' YELLOW 'IF' GREEN 'IF'
Help otepadUiiookbo& 1 Assumftions 1Prior Screen M ain Mlenu IPrinft OUitF1 F 34F5 F6 F7 FIO
Figure 10 - Scheduling Representation of the DSS
the projected manning level based on current acquisition decision
parameters. The manager's objective is to make the solid line match the
dashed line by suitable choice of values for the acquisition decision
parameters. The manning level projections that arise due to the manager
changing parameters are graphed as the dotted line, and can be compared
to the desired levels and to the levels that result from current
42
parameter settings. (Relative to the previously discussed storyboards,
this screen is where the user's major operations occur. He can change
multiple parameter values, and save intermediate combinations of
settings for comparison of the results of his various "what-if"
examinations of acquisition possibilities).
Kernel Selection. While the concept map addresses the overall
problem space and the storyboards restrict the space to a subset the
users want supported, the overall system specified is still a major
undertaking. To attempt to give the users a tool that allows them to
start addressing their problem, it was decided that an appropriate
kernel would be the capability to assess current experience in the
various crew positions. The kernel's selection and development is
discussed fully in the next chapter.
Contiued Adaptation. The mechanisms for fostering continued
system adaptation are embodied in two capabilities included in the
system. These are the hook book discussed previously, and an explicit
assumption review capability triggered by the "ASSUMPTIONS" button
described in the storyboards.
The hook book was started in a paper form at the beginning of the
design process to capture needs for system evolution. (These hook book
entries are included in Appendix C). This was important to the system's
development in that it captured evolutionary needs without squandering
design resources on redefinition of the problem. In other words, it
allowed kernel development to continue, rather than causing the kernel
to be continually redefined. In its system's integrated form, the hook
43
book serves the same function, allowing the user to capture needs
without unduly interrupting the decision cycle in which he is engaged.
The explicit inclusion of an "ASSUMPTION" button was modeled after
that described by El Sherif, et. al, in their discussions of support
systems developed for the Egyptian cabinet [El Sherif, et. al. : 560).
It was included as an adaptive design mechanism because it forces the
user into an awareness that decision making depends on more than the
data that is manipulated. How and hy the data are manipulated, and the
meaning of the results can be very assumption sensitive. If the system
is to adapt successfully, any assumptions that are inherent in the
system's implementation must be open to scrutiny and change.
Having discussed the adaptive design elements of this research,
the remainder of this chapter is devoted to a discussion of the
technical design considerations of the DSS considered in this research.
Technical Desion Considerations
The technical design considerations for this effort are as
extensive as those of adaptive design, and are equally as important.
Technical elements of a system's design can relegate a conceptually
elegant system to a less-than-sterling system when implemented, with
incorrect selection of technical design components hindering system
development and use. Technical design considerations include:
1) choosing the particular hardware and software environment to beused;
2) identifying and planning access to data necessary to supportthe decision process;
3) identifying places in the decision process where models are
44
necessary (or desirable) to support decisions, and
4) choosing appropriate model formulaticns and managing the "modelbase'.
Environment. Choosing the hardware environment to be used was a
non-issue for this DSS. The 28 Air Division has invested heavily in
Zenith Z-248s, with networking of the systems in a beginning stage. All
air crew managers have ready access to these machines.
The software environment is not as ciear cut, other than the
requirement that the software run on Z-248 computers. Ideally, a DSS
generator would be employed to design and implement the desired system.
However, this option was not examined for two reasons. First,
procurement of the software in time for use in this effort presented
problems. Second, it was beyond the scope of this effort to even
identify an effective DSS generator (if it existed) that could support
the requirements embodied in the storyboards. (In another Air Force DSS
development effort, two years were spent evaluating numerous software
packages "claiming" to be DSS generators [Walker, et all. A large
number of candidates were examined, and more than hal f were el iminated
from the evaluat ion process, because they were clearly not DSS
generators, regardless of the vendor's claims).
Another environment rejected for the implementation of this DSS
was examined in the previous research effort in this area. Schneider
examined the feasibility of using an off-the-shelf software package,
"ENABLE", as the environment for generating a DSS. The windowing
abilities and the integrated database, spreadsheet, graphics and word
processing features in this package were exercised extensively in
45
developing a prototype DSS, and "ENABLE" was found adequate to produce a
workable DSS.
From the user's perspective, however, this environment had severe
shortcomings that would preclude its use as the environment of final
implementation. The test system that was demonstrated at the 28 AD was
driven with small test databases and simplified models. It was, none-
the-less, incredibly slow, which aggravated the operators, and it was
uncomfortable to operate because of the way "ENABLE" transitions between
its various "integrated" modules (i.e., screen bounce and flash, and
"ENABLE" environment menus flickering past as the package made
transit ions).
In general, the man machine interface can be a major factor in a
system's success, and speed is a major component of this interface.
Slow systems are perceived as being "unresponsive" to the user, and are
therefore avoided. For "ENAFLE" (and probably any integrated "office"
software), speed was a compound issue. In the integrated environment,
trade-offs are made in the capabiLties of the various modules, with no
module being the equal of the best stand-alone packages that are
available to perform the same function. Further, the very fact of
having an integrated environment in a general package entails overhead
to exercise all of the package's abilities, including those that are not
needed for a particular application (say, a DSS). This is probably the
case for any general purpose system; they will always have overhead to
support requirements not necessary for a DSS, and the DSS suffers.
For the reasons alluded to above, the final environment for
46
implementing the DSS examined here should be provided by a stand-alone
program that provides the user interface prescribed by the storyboards,
which executes all of the desired functions, and which supports other
functions only as their needs are identified during the adaptive design
process. Unfortunately, this has undesirable consequences for systems
adaptations. These consequences are discussed in Chapter 5.
Data. The data to support this DSS is available at the 28 AD.
However, it is not in a form that can be directly used by any automated
system. (A large part of the data is in word processing documents that
are updated daily).
An AWACS wide, standardized database for crew management functions
that contains most of the necessary data has been designed using DBASE
III. When it is in place, implementation of this DSS can begin. The
other necessary data elements that are not part of the above database
can be incorporated into a small database, or are on base level
computers from which they can periodically be down-loaded in Z-248
readable formats and converted to DBASE III file formats for use in the
DSS. (The necessary data elements and their sources are included in
Appendix D).
Models. This DSS requires three major model types to synthesize
personnel data and drive the displays providing the decision testing
mechanism. First, models to project losses in each crew position are
necessary. These models will be used to provide the loss estimates to a
simple additive model that drives the manning level displays, and will
also feed the models driving the experience profile displays. (These
47
are the displays shown in Figures 10 and 7 respectively). Since these
models tie into experience level projections, they will necessarily be
complex. They will have to project losses in distinct categories in
each crew position, where the categories are defined by a personnel
factor that is important in estimating experience.
Second, models for each crew position that assign "experience
scores" to the members of the crew position are needed. These scores
will provide the data used to profile the experience in the position.
(These scores are aggregated to produce the histograms shown in Figure
7).
Finally, a model that evaluates AWACS mission capability as a
function of manning and experience levels is required. (This is the
model that drives the display shown in Figure 9).
Each of these models will require extensive mwnagement in that
they require continual updating until such time as the crew positions
they address become stable (and periodic examination there after).
Model management will entail AWACS analysts periodically rebuilding the
models. Model accuracy will change as factors pertinent to the model's
function change, due to outside forces (e.g., force reduction that
remove classes of people in an unexpected manner), or due to actions
resulting from the use of the DSS that alter the force structure.
The models that drive the system are the major "repository" of
assumptions that the system functions under. Therefore, in addition to
updating the models, the analysts have a critical responsibility to
document the changes in assumptions that are inherent in changing the
48
models. (As stated previously, review of assumptions is critical if the
system is to remain useful). More information on models can be found in
Appendix E.
Having discussed the overall system design, the following chapter
deals with the design of the kernel DSS.
49
IV. Kernel Desion
General
The overall system discussed in the previous chapter will entail a
large and time consuming effort to implement in its entirety.
Therefore, some portion of the system needed to be addressed in. detail
to: 1) demonstrate to the user that progress was being made, 2) to help
the user solve some portion of his problem, and 3) to put an initial
tool into the user's hands so that the tool's use would help the user
refine his perceptions of his needs. To get the system up and running,
a kernel was selected that provides managers an initial capability in
assessing acquisition needs where they currently had none.
Kernel Selection
Managers currently deal with acquisition planning solely from a
short term "numbers" perspective. They plan personnel acquisitions to
correct any shortfall from authorized manning levels that they see
arising over the near term life of the crew force. Even though
acquisitions are managed to correct short falls in manning level, there
is a widespread recognition in the AWACS community that experience in
the crew force should also be managed.
Because of the difficulties in determining experience, experience
profiles of the crew positions have been measured against arbitrarily
specified references, (i.e., all WDs having one year's flying experience
are "experienced" and all CTs with 600 E-3 hours are "experienced". In
general, from the author's experience at the 2BAD, these measures of
50
experience are viewed with varied levels of disbelief).
These references are so broad as to be meaningless. The only
people that they exclude from the "fully experienced" category are those
recently graduated from primary crew training at the 552 TTS.
Therefore, the kernel DSS that was designed was developed to address the
crew experience issue. Specifically, the kernel system allows managers
to graphically profile current experience of crew positions with a fair
degree of detail, rather than examining them in the current arbitrary,
binary fashion. (At some future time, a referent curve of minimum
expertise levels in each experience category should be developed to
allow the manager to assess how close he is to a tolerable manning
profile. Until that time, managers will have to make determinations on
the desirability of the experience profile from the gross pattern
evidenced in the graphical display).
Choosing a kernel DSS to address current experience in a crew
position makes good sense for many reasons. These reasons are
introduced here, and discussed in the following section of this chapter.
The first reason selecting the above kernel is that it gives managers a
capability that they want. Second, even though it does not directly
address the need to correct the current short term management approach
to the acquisition process, actions taken as a result of decisions based
on experience level estimates will not aggravate AWACS manning level
conditions. Third, it allows managers to use an experience profiling
tool and refine it before work is expended in building the more complex
profiling tool needed to project experience levels in future years.
51
Lastly, it is one area of the acquisition planning process where a
concerted effort is needed to achieve any results. The value of giving
users something they desire is self evident. The other three issues are
discussed next.
Rationale for the Kernel Selection. The capability to examine
current experience levels allows managers to approach the acquisition
process with an ability to address needs that he cannot currently
assess. With this capability, he can take action to correct experience
deficiencies by altering the types of people acquired. This augments
his management capabilities without altering his current management
scheme, which only addresses identifying the number of people to be
acquired. Since the manager is not changing the manner in which he
decides how many accessions are needed, experience profiling cannot
aggravate AWACS manning relative to the current management scheme. The
current acquisition procedure is simply not affected by it.
The net result of the kernel DSS is an ability for the manager to
identify times when AFMPC must make some of the annual crew acquisition
come from sources with either 1) AWACS experience (i.e., allow AWACS to
recapture previous AWACS resources), or with 2) transferable experience
(e.g., pull computer technicians into the CDMT position, rather than
bringing in brand new Air Force acquisitions). These conditions can
occur when the current aggregate experience of the crew position is low
and the addition of totally inexperienced personnel to the position will
only aggravate the condition. Therefore, the effect of the kernel DSS
will be to elevate the experience level of the crew force for some
52
extended period of time. (Even if elevating the experience level of the
crew force were not necessary, it would certainly not be damaging to
have a crew force "overly rich" in experience).
Referencing the point about letting managers use a "simple"
experience profiling tool, there are two reasons for implementing the
chosen kernel. In the first place, the basic work required for the long
term experience profiling is embedded in the kernel system. The basic
issue of how to build an experience "score" is addressed here.
Second, projecting profiles requires the ability to accurately project
manning losses and the development of an apportionment scheme for any
factors that impact experience, e.g., if flying hours are a significant
factor in determining experience, all projected AWACS flying hour
allocations must be distributed over the crew force in some manner to
project how the individuals in it gain experience. Resolving these
issues will entail a significant effort. Before the effort is expended,
the basic experience profiling capability should be used extensively to:
1) familiarize managers with experience profiling, and 2) refine the
experience scoring methods if managers feel it is necessary.
Finally, experience is an acquisition issue requiring a concerted
effort if the manager's capabilities to address the issue are to be
improved. While the crew managers currently deal with acquisitions on a
near-term manning level basis, improvements in AWACS manning stability
can be achieved with relatively modest effort. As discussed in chapter
two, fluctuations in AWACS manning are cyclic in nature. Recognizing
this, if the past annual final PFT acquisitions are examined to
53
determine where peak acquisition demands occur, the cycles can be damped
by over-acquiring personnel in the period just before these peaks. In
contrast, there is no usable information on experience, since experience
has been ignored in past acquisition management. Any attempt to
directly deal with experience required a "start from scratch". The
kernel DSS provides that start.
With the above understanding as to why the particular kernel DSS
was chosen, the remainder of this chapter discusses the modeling methods
examined to develop experience scores and a test implementation using a
spreadsheet environment to examine the aggregated experience scores.
Experience Scare Modeling Examination
Three different modeling approaches were examined for developing
experience scores for the individuals in each crew position. These
were: 1) logistic regression with more than two categorical responses
(10 in this case), 2) MCDM techniques that elicit and quantify experts
opinions to produce "scores", and 3) multiple regression with a binary
categorical response. Each is discussed below.
Logistic Regression. Logistic regression approaches to developing
experience scores suggest themselves because, as stated in chapter 2,
empirical evidence suggests that response functions developed using
categorical dependent variables will frequently be curvilinear.
Logistic transformations of the response function produce responses of
appropriate shape and have the desirable property of being asymptotic to
the extreme values of the response, i.e., predicted values produced
using the logistic response function will not fall outside of the range
54
of the categories of the original response [Neter, et al : 356-363].
Taking a logistic regression approach to this problem requires
that the current crew force be separated into distinct categories of
experience. Therefore, this approach would not be purely a regression
approach, but would require some preliminary work to categorize the
members of each crew position. Multi-Attribute Decision Analysis (MADA)
techniques could be used to get "expert evaluations" of the various crew
mefmbers.
Unfortunately for the current problem, logistic regression
requires large samples at each of the possible combinations of input
factors to weight the regression appropriately MNeter, et al : 364).
For the current problem, the largest crew force is in the 250 person
range. If only four predictors at two levels are used, the largest
sample for a given combination of inputs is only 15, which is
insufficient. (Given the current perceptions among AWACS experts as to
what factors may be useful for determining experience, it is unlikely
that a reasonable regression can be achieved without more than four
predictors constrained to two levels).
Another problem with this approach is related to the expert
support required to do the initial categorization of the crew members in
the various crew positions. Discussion of this problem is deferred to
the next section, where MCDM methods are discussed.
These difficulties with the logistic regression approach removed
its use from consideration.
55
MC)M. Multi-Criteria Decision Making techniques suggest
themselves as possible candidates for addressing this problem because of
the lack of definitive experience measures for AWACS crew members.
Lacking suitable measures, the problem can be approached from the
perspective of expert opinion. MCDM techniques, particularly rulti-
Attribute Decision Analysis (MDA), are ideally suited for this type of
approach. Therefore, MDA approaches to this problem were examined.
The details of this examination can be found in Appendix F. The major
conclusions of the examination are repeated here.
The MADA methodology examined gave results that made sense.
However, while the factor set that was examined was quite small, the
techniques still consumed a significant block of an expert's time.
Further, when multiple experts differ on utilities and weights, MADA has
some problems. Various concordance measures are available for
determining whether differences observed between the elicited values are
significant. Unfortunately, when the concordance measures indicate
significant disagreement, MADA currently has no generally accepted means
for resolving the dispute. These differences in expert opinion are
bound to arise when these techniques are applied to the DSS considered
here. How they are resolved is critical if these techniques are used.
The most significant problem for this particular application was
the time consumption mentioned above. The level of expert involvement
required to implement this type of strategy is enormous. For the
examination of MADA, weights and utilities were only developed for two
factors. The process still took over six hours to accomplish, using a
56
reasonably friendly software package. This time issue is critical for
the DSS. The people who will be queried in this process at Tinker AB
are, first-and-foremost, flyers who have an erratic and very full flying
schedule. Attempting to corral experts to repeat the MADA approach for
each of the 12 crew positions, using multiple factors and repeating the
process multiple times as the system evolves, would probably kill the
DSS development outright.
The reason the process consumed so much time was that the experts
being consulted, in general, have no background in probability. Both
the utility and weighting elicitation process are heavily dependent on
the expert's ability to assign probabilities to the choices presented
during the elicitation process. The major problem encountered was that,
even using a 50-50 gamble (which Holloway indicated is generally the
most easily understood), the decision maker being queried could not
build a utility curve that matched his estimations, even when he had a
clear perception of what the curve should look like. People in general
do not deal with probabilities well, and this experience indicates they
have particular problems estimating the relative probabilities for two
different bounded ranges (not including either extreme) in the interior
of the space they are considering.
For these reasons, M was removed from consideration for this
application in the DSS. However, MCDM in general, is a tool that will
necessarily be appliec to other areas of the final DSS. This is
discussed in chapter 5.
57
Multiple Regression with Binary Responses. During the examination
of MCDM techniques the idea of using a multiple regression approach with
a binary response surfaced as an appropriate technique for experience
modeling . One fact that was uncovered while surveying experts for the
MADA evaluation was that a fairly rigorous categorization based on
expert assessment of crew member expertise already occurred in AWACS.
This being so, there existed a binary response that categorized crew
members as: (l's) - definite experts or as (0's) -- the rest of the
crew force. (All experts do not necessarily fall in the expert
category). With a binary response, a multiple regression is feasible
Neter, et al; 354-361).
While a multiple regression is feasible for the binary categorical
response, there are two problems. First, two of the assumptiuns of
least squares regression do not hold. Least squares regression assumes
that the error in the regression is normally distributed with a mean of
zero and with constant variance. Since the response is binary, the
error in the regression can only take on two values. This being the
case, the error is not normal. Similarly, the variance of the error is
dependent on the response level, and is therefore not constant.
Second, there are constraints on the possible values of the response.
The interpretation of the output of the regression model is somewhat
different than in other regression formulations. Since the response is
binary, the expected value produced by the regression model is just the
probability that a particular combination of regressors will result in
an "expert" classification. Since the output of the response function
58
is a probability, it is constrained to the region from 0 to 1.
These problems can be successfully dealt with. The lack of
constant variance can be corrected for by using a weighted least squares
fit. The constraint issue can be dealt with by ensuring that the
response cannot fall below zero or above one over the range of the
predictors or by transforming the predictors. Finally, while the error
terms are not normal, for sufficiently large samples inferences about
the regression results can be made as if the error were normally
distributed. (In general, sample sizes greater than 30, are sufficient
for treating the results as having come from normal populations [Devore
260). Samples for all crew positions are larger than this).
The approach just discussed is feasible and easily implemented.
It is transparent to the user in that it does not consume valuable
expert time, but its assumptions can easily be documented for expert
review. It was therefore chosen as the nodeling technique to be
employed in the kernel DSS. Implementation is discussed in the
remainder of this chapter.
Kernel Inylementat ion
The kernel was implemented for test purposes using Borland
International's Quattro Pro spreadsheet environment. This environment
was chosen for multiple reasons. First, Quattro Pro acts directly on
multiple file formats, one of which is DBASE III. An abbreviated form
of the standardized personnel management database, which is based on
DBSE0 III (and which was discussed in chapter 3) was available to
develop a prelimiiary kernel DSS.
59
Second, as compared to ENABLE, which was used in the previous DSS
in this area, Quattro Pro has much better response times. Borland's
memory management system used in Quattro Pro is much more sophisticated
than E'BLE's, significantly improving Quattro Pro's speed. Possibly
more important Quattro Pro does not support all of the capabilities that
ENABLE does, reducing its overhead, and its graphics capabilities are
much better integrated to the spreadsheet than ENABLE's were.
Finally, Quattro Pro is almost totally configurable to specific
applications. Full menuing capabilities (that replace the normal
spreadsheet environment menus) are provided, multiple windows are
available for changing displays rapidly, and mice are fully supportEd
(this feature was not used). As such, the user environment in Quattro
Pro could be tailored to match the environment envisioned in the
storyboard almost totally (the main exception was to place the system
support options as a menu item on the top menu bar).
Pulling the Pieces Together. In the kernel DSS, multiple
regression with binary responses was conducted in a PC based statistics
package, Statistix II, using the abbreviated database mentioned above.
(Results of this regression are discussed in Appendix E). This modeling
technique is implemented with relaxed tests for confidence intervals, as
suggested by Deming [Deming]. (This issue is discussed in Appendix E).
After ixamining the regression results and determining an
appropriate model, the appropriate database fields were read into
Quattro Pro, where the model was applied to each record of the database
to build an experience score.
60
Scores in the database were separated into three distinct groups:
people who were known to be departing AWACS, people who were currently
not in AWACS (whose records were in the inactive portion of the
database) but who nad been gone long enough to be due to PCS, and the
rest of the people in the crew position. The scores for each group were
then aggregated using Quattro Pro's capability to examine frequency
distributions, and the results written to a histogram to display the
score distribution. (This graph is part of the spreadsheet. A
transition to another program module is not necessary before it can be
viewed, in contrast to ENABLE). The resulting graph was similar to that
shown in Figure 7.
The results of this effort were a system that operated and which
provided an understandable display for the current experience of the
crew position profiled. Building the system in Quattro Pro involved two
deviations from the design specified by the storyboards. First, the
user interface defined in the storybodrds was modified. It was built as
specified with the exception of placing the system support commands in a
sub-menu off of the decision command menu and placing the memory aid
line below the main menu, rather than above it.
It was not absolutely necessary that the system support commands
be moved to the top menu; a menu at the bottom of the screen could have
been implemented. However, flexibility in selecting from this menu
would have been limited, and would have "felt" different from the
selection process used in the rest of the system (i.e., mouse and cursor
selection of the system support opt ions on a bottom of the screen menu
61
would not have been possible).
In contrast, the memory aid line had to be moved. This is because
Quattro Pro reserves the line above the main menu line for its own use.
The other deviation from the storyboards involved placement of the
two graphs being displayed. For resolution reasons, the graphs were
placed one below the other instead of side-by-side. While Quattro Pro
is exceptionally flexible, it does place some restrictions on the user.
One of them is the lack of control offered to the user over some of the
"cosmetics" the system's graphics. Large boarders are placed around
graphs; this required that the two graphs be stacked if they were to be
viewed at the same time. Further comments on Quattro Pro's usefulness
as a DSS environment can be found in the following chapter, which
details the conclusions and recommendations that resulted from this DSS
research effort.
62
V. Conclusions and ecommidations
When a request for an A-IT thesis proposal was received by the 28
AD in early 1987, the author proposed that the issue of supporting AWACS
personnel managers be examined. The proposal was prompted by the lack
of understanding of long term personnel issues exhibited by PFT
attendees during the course of numerous PFT conferences. When the
proposal was acted upon by Schneider, the author was the point of
contact at the 26 AD for the research effort, acting as the user's
"representat ive".
Many of the objectives outlined in the original proposal were
never addressed (prior to embarking on a thesis, the author had no
appreciation of the constraints that limit a thesis' scope). Further,
the majority of the issues that were addressed were approached from a
perspective alien to that envisioned when the proposal was made (i.e.,
Schneider's research addressed the problem largely from the perspectives
of TAC, not the 28 AD).
The ways in which the previous research was limited, the
directions it took, and the apparent inability of the "user" to
influence the course of the research were never understood by the author
and prompted him to re-address the problem in this thesis. The process
of viewing the same problem from two perspectives has been an eye
opening experience, and has certainly impacted the types of conclusions
and recommendations that resulted from this research.
This chapter contains the conclusions and recommendations that
63
resulted from this research effort. The observations documented here
are separated into two major groups: 1) those that deal with
development of DSSs in general, and 2) those that address the future of
the specific DSS examined in this research.
General DES Develvniwt
1) DSS development and separation from the user's environment.
Conclusions. DSSs cannot be successfully developed at a location
removed from the user. This conclusion results from three areas of
difficulty which are discussed next.
First, communications and accountability problems arise. While
storyboarding is an outstanding communications tool (discuss-d later in
this chapter), it cannot correct the communications problems related to
timely feedback. Without constant communication, a necessary sense of
accountability between the developers of a system (designer, user and
builder) is lost, resulting in a design process that wanders. (This
issue is also discussed later in this chapter).
Second, if the DSS is developed away from the user's environment,
the system will assume the designer's character, regardless of the
designer's good inlentions. Without constant communication and
immediate feedback, the designer has to act on his perceptions, with the
result that the designer's views are built into the system. (At the
very least, the adaptive development cycle is prolonged while the user
weeds out those elements that don'" support his needs).
Third, constant interaction with the user is the only way to
determine what he knows. A general complaint of system builders is that
64
users do not know what they want. After the experiences involved in
this research, another observation should be "users do not know what
information they have that is applicable to the solution of their
problem".
All three of these difficulties are at the root of why the author
didn't understand the course Schneider's research took, which resulted
in his pursuit of the same research area.
Further Ibservations. Valusek makes a distinction between rapid
prototyping and adaptive design [Valusek]. Both design methods are
geared to developing portions of a system and "growing" the final system
by merging the previously delivered portions. The distinction that
Valusek makes is that rapid prototyping occurs at the developer's
facilities and is "delivered" to the user in pieces, whereas adaptive
design occurs at the user's facilities. This difference is critical,
and makes one wonder if rapid prototyping isn't traditional systems
development in disguise. If the designers and builders are not at the
user's facility, they cannot "wallow" in the problem or determine what
data the user actually has that may be applicable to the problem. They
have to totally rely on the user's description (read specification) of
his needs. Separating a DSS's development from the user's environment
results in a rapid prototyping approach, and entails building the DSS to
a "spe." in that the system is built to some static standard between
deliveries, just as is done in traditional systems development. (Not
only is the system being built to "specs", it is being built to a spec
multiple times).
65
Adaptive design is geared to getting away from specs, or at the
very least, letting the spec "float". The constant interaction between
the system developers results in a more "even evolution" of the system,
rather than a process that involves taking two steps forward and falling
one step back when it is discovered "too much progress was made" (in the
wrong direction).
Both Schneider's and this research effort to support the AWACS
personnel acquisition process are guilty of falling back to a rapid
prototyping approach, particularly when they are viewed together. This
research was pursued because Schneider's DSS evolved in a direction that
did not suit the users, which required it to be corrected. The DSS
developed as a result of this research may be equally guilty. The only
correction for this situation is for any further development of the DSS
to occur at the 2SAD, utiere cozmmuication with the user and system
assessment by the user can occur during development.
Conclusion. Research into DSS development, particularly with
regard to adaptive design, needs to be done at a real user's site, and
removed from the purely academic environment. This is a direct
corollary of the above discussion.
Recommendations. This DSS should not be examined for further
development in thesis efforts. All future evolution of the system
should be done by 28 AD analysts (28 AD/XOS). While the analysts at the
28 AD may not have familiarity with the Operations Research tools that
bear on the problem, learning specific tools is easily corrected. The
difficulties in acquiring data and understanding of user's needs from a
66
distance is not.
2) Storyboardina as a communications tool.
Conclusion. Storyboarding is a tremendous tool for clarifying
user desires and designer intentions. The author's first exposure to
storyboards was during Schneider's research in the AWACS acquisitions
area. At all times, the intent of the designer (Schneider) was clear to
the user (Holt), and desires for the system could be clearly stated by
referencing desirable and undesirable features of the storyboards. (As
discussed above, however, the power of storyboards to communicate
intentions and desires was insufficient to overcome other communications
obstacles).
Conclusion. Storyboards fill a very fundamental role in the
adaptive design process. Determining user requirements is a difficult
process that entails two distinct phases (IRD and IRA). Both the IRD
and IRA processes can be well structured by storyboarding. The
storyboards lend a "tactile" element to the study information
requirements. Users and designers can "grasp" their needs because they
are clearly and simply stated, rather than floundering to an
understanding of needs while wading through a morass of verbiage that
obscures the communications of needs. Pictures are worth a thousand
words.
Recommendation. An effective, general purpose storyboarding tool
should be developed. The ability to build a running "dummy" system in
storyboard form that responds to user inputs would be a valuable system
development tool. Further, it could significantly speed up the
67
storyboarding process, by placing the storyboards in an environment
geared to building and editing them. (Word processors are not effective
environments for building storyboards).
3) DSS design as an individual effort.
Conclusion. The importance of synergy can never be overrated.
Dealing with complex issues as individuals is not as productive as
dealing with it as a team, if the tean is built froan the right elements.
In the adaptive design context, the design team can be considered to
consist of the designer, builder and user. Together they capture user
requirements, determine user capabilities, identify useful data, choose
applicable modeling techniques, identify technical constraints that may
impact a system's development, ... and address the many other functions
that must be considered in a design process.
Each of these people brings his own domain expertise to the design
process. However, the user comes up short in the domain area when the
design process leaves the "this is what I want" stage and enters the
"how do we give him what he wants" stage. The user's domain expertise
is crucial to a system's development. It is, after all, his problem that
is being addressed. But for systems design to transition successfully
to implementation, many questions need to be answered in areas where the
user has no expertise, and more, does not even understand the language.
For these questions to be resolved effectively, designers and builders
need to communicate with others who speak the language of systems
development so they can sharpen their insights and ferret out their
errors and false assumptions. The lack of a "sounding board" to help
68
sharpen insights was a significant handicap in approaching this research
as a lone designer.
Recommendation. DSS development should proceed from a team
perspective, whkere the tean consists of xiltiple desig7ers, builders,
and users, not one of each.
Conclusion. A clear distinction in the functions of each member
of a design team is a necessity, and builders should never be confused
with designers.
This research was conducted by a "builder", i.e., someone used to
dealing with the solution of problems, not their formulation. Builders
assume that if somebody has a problem to be solved, they know what the
problem is and how they want it solved. Their job is to implement the
desired solution. Therefore, builders view the system as the product,
totally overlooking or ignoring the fact that, for a DSS, the decision
is the product. Building becomes the important task, resulting in
design functions being subsumed by building functions.
Builders also tend to look at problems from a depth first rather
than breadth first perspective. They focus on small portions of a
problem and deal with it, approaching problem solution "modularly". A
consequence of this approach is that, until it is dealt with
successfully, the smaller problem becomes "the problem". This cripples
a builder's ability to act as designers, since the designer must be
concerned with the entire problem area.
To overcome these problems, there must be a clear distinction as
to who the designer is, and that person cannot be the builder. Design
69
functions are too important to be entrusted to somebody with a builder's
perspect ive.
Recommendation. Separate responsibilities - and never let a
builder be responsible for design functions (otherwise "design" will
occur in an erratic manner).
4) DSS development and technical constraints.
Conclusion. Because of the speed with which basic blocks of a DSS
can be examined, spreadsheets allow rapid assessments of the concepts
embodied in the storyboards.
The value of the spreadsheet environment lies in the speed with
which a system can be prototyped. For the DSS developed in this
research, a spreadsheet (specifically Quattro Pro) was flexible enough
to implement the entire system's control structure (embedded in the
menuing system and the memory aid system) as specified in the
storyboards, with only minor deviations. The environment also allowed
the disparate data necessary for the kernel's functions to be viewed and
manipulated, with the necessary models being built into the spreadsheet.
Conclusion. Spreadsheet environments are currently inadequate for
final implementation of DSSs. Current user programming capabilities
(macros) in spreadsheets are not powerful enough to build systems that
need minimal support. This is particularly apparent when external
databases are being queried by the spreadsheet.
For this DSS, building a general database interface with a great
deal of conditional flexibility was needed if a system were to be built
that required minimal modification of the core spreadsheet environment.
70
Model specifications (that were subject to change) needed to be passed
to the spreadsheet. The same was true for data specifications (which
were dependent on the model specified). Since both of these elements
were subject to change in dimensionality, the specification of
calculation areas in the spreadsheet for specific modeling operations
became a messy issue that could only be solved by execution of
conditional programs that exceeded the capacities of the macroing
language of the spreadsheet.
Racomuiiation. Until such time as effective DSS generators are
available, spreadsheet environments should always be considered for
quick assessments of DSS design elements.
Further Observations. Development environments not specifically
designed to support design in a particular application area are
dangerous to the overall design process. They impose technical
constraints on a system's development that may not be appropriate.
Worse, they breed commitment to the system in its current stage of
development, when this commitment is not justified. It is very hard for
a system's developer to abandon past development because of the
constraints of the environment he is operating in; it is easier to
change the "requirements" of the system to be developed. This is
particularly apparent in the DSS Schneider designed for AWACS.
Schneider's environment, ENABLE, imposed significant constraints on the
DSS design (this is conjecture on the authors part, but is consistent
with his experience, and goes a long way to explaining the course
Schneider's DSS took).
71
Conclusion. DSS development in the adaptive design paradigm
without effective DSS generators is severely crippled. Turn around time
for useable system enhancements are simply not short enough for the
adaptive design process.
Until such time as a development environment is available that is
flexible and extensive, development will have to occur in one of two
environments. First, design can occur in spreadsheets, as discussed
above, where assessments of DSS design elements can rapidly be made.
When the design element has been "shaken down", it can then be
implemented in stand alone code for integration into a final system.
Unfortunately, this is very wasteful of the effort that goes into
developing the spreadsheet capability.
The other possibility is to build the system in stand alone code
from the start. This is a possibility, but only if large programming
toolboxes applicable to the task are available. Without the necessary
toolbox (which constitutes the beginnings of a DSS generator),
development time for the basic system components will make delivery of
DSS components impossible in adaptive design time frames.
Recommendation. A major research effort should be started in the
area of developing specifications for DSS generators.
5) The Lritical nature of assumptions.
Cownclusion. The critical nature of assumptions cannot be
overstated for DSSs. When a user knows what decision needs to be made
and resorts to a DSS to arrive at his decision, he has lost control of
the decision process and rendered the decisions a, rived at suspect if he
72
does not know what assumptions have been implemented in the DSS's
design. All modeling techniques entail assumptions; some are related to
how data is treated, and some are related to the applicability of the
various modeling techniques used. All of the assumptions made using the
1SS modeling technique need to be "examinable" if the decision maker is
to make valid decisions.
Recomit dation. Some structure similar to the "ASSUMPTIONS"
button presented by El Sherif, et al, needs to be incorporated in all
DSSs, and all assumptions need to be fully documented.
The Specific ES
6) This DSS and where to cao in the future.
Conclusion. Expv-,rt opinion is going to play a major part in
whatever quantification scheme is proposed for the AWACS capabilities
assessments. This is also true for determining an appropriate baseline
for desired experience distributions in the different crew positions.
This is because there are currently no hard and fast measures for these
areas. MCDM techniques are very relevant to this area.
Conclusion. Synergy, as discussed for design team issues, also
exists among modeling techniques. The examination of MADA for
quantifying experience led to identification of criteria by which a
subset of a crew position could be categorized in experience. The
regression technique specified for use in the kernel (which overcomes
the time issues that make application of MCDM techniques to the problem
untenable) was only able to be used because of information uncovered
using MADA techniques.
73
Recommendation. rultiple modeling techniques should be examined
for applicability to each phase of this DSS's development.
Conclusion. The regression method employed in the kernel has some
potential stumbling blocks.
As discussed in the previous chapter, regression results only make
sense if the response is bounded by zero and one. When a set of factors
is determined to be significant in predicting membership in the "expert
class" (defined by StanEval personnel and instructors), the range of the
factors needs to be examined to ersure that they don't force the
response into an un-allowed range. If infeasible responses are
encountered, transformations of the predictors may be necessary. Another
possible strategy is to examine the predictors, to see if it makes sense
to "cap" the values they can take on. For example, if flying hours are
a factor in the regression model, it may be that it makes no sense to
count flying hours above 2000 hours. Marginal flying hours above this
number may not impact experience significantly. This type of "call" is
an expert judgement call -- MCDM techniques may be useful in evaluating
these types of questions.
Recommendation. Always keep in mind that regression (like all
other model ing techniques) only provides guidance. The analyst must
bring his/her judgement to bear on what the results mean, and must also
keep data implications in mind, if effective models are to be built.
Conclusion. This DSS is going to require extensive support from
the 28 AD analysis office. Implementing the entire system discussed
here will be a major undertaking. This effort is oie that the 28 AD
74
analysis shop, as it is currently configured, is well equipped to
address, given the combination of analysts, programmers and a supervisor
with extensive systems development background. However, the commitment
required from this office is going to extend well beyond "system
delivery", where system delivery refers to a time when it appears that
the system has evolved to a fairly stable state. At this point, when it
appears that user needs are being met, support requirements for this
system are still going to be extensive.
As changes in the crew force come about due to changed acquisition
practices, it must be realized that models developed to address earlier
crew compositions will have to change - and regression cannot be
automated. As discussed above, the analyst is an integral part of the
regression modeling approach, and will therefore have to be an integral
part of the DSS's operation. The analyst must be involved to interpret
the regression results and to determine the most effective model to
describe what is observed.
As a learning experince, the value of this research cannot be
overstated; its impact on the author has been immense. The different
natures of design and building were "discovered" (and constantly re-
discovered) during the research, resulting in the realization that good
design does not often occur by happenstance, it is hard work.
For the Operations Researh world, the insights pertaining to
division of labor in DSS development and the necessity for good DSS
development tools are important, and are areas requiring further study.
75
Appendix A
Concept Map
The concept map describing the AWACS personnel acquisition process
is included on the following two pages. This concept map is presented
in a hand drawn form for two reasons. First, a concept map is only a
guide to understanding the intricacies of the problem being addressed --
it should never be viewed as a product that defines the problem.
A "perfect" concept map could be construed to be a product which
fully describes a problem and implies that the problem is completely
understood. This is nonsense, and could stifle further exploration of
the problem. Making the concept map "perfect" is, therefore, counter
productive. (Also, effort expended making the concept map "pretty" can
better be spent addressing how to solve the problem which the concept
map describes.
Above all, the concept map must be viewed for what it is -- a
useful tool, not a product in its own right.
76
I tt'
Figue 1 (prt ) - oncpt ap or (PJ~S Prsonel cqusition
4 j 7A
I a.j
td
A Q
iq.
Figure 11 (part b) - Concept Map for AWACS Personnel Acquisitions
78
Appendix B
Staryboards
The storyboards describing the AWACS PERSONNEL MOAGEMENT
ACQUISITION SYS1-M follow. They are divided into five functional areas
that indicate the issues they address in the system's operation. These
areas are:
1) System functions -- addressing how the system is structured
and how it is used;
2) Assessments of current crew position profiles - the user
needs to understand where he currently stands before he attempts to plan
acquisitions;
3) Acquisition planning and assessment of long term impact - the
user schedules acquisitions under different assumptions and examines the
impact the planned acquisition has on the crew position;
4) Assessment of personnel issues on mission capabilities -- the
manager needs to know how to balance conflicting acquisition needs.
System's performance provides the measure, and;
5) Provisions for the managers to examine low level information
that impacts their decision -- e.g., examining the personnel databases
and accessing information on scheduled changes to AWACS manning that
will impact his needs assessments.
Each storyboard has a description of its major features. These
descriptions are not stated explicitly in terms of ROMC. Rather, the
79
descriptions are in terms of %.hat the user needs to know to determine
that the system meets his needs. (The correlation of the description
elements to the elements of ROMC should be self-evident to readers
familiar with the ROMC structure).
80
&ystem Control and He~lp
AWACS PERSONNEL ACQUISITION MANAGEMENT SYSTEM (-- (1)
POSITION SCHEDULE IMANNING PROFILE IAWIACS CAPABILITIES IFORCE STRUCTURE (-(2)
Welcome to the 28 Air Division AVACS PersonnelAcquisition Management System
System files are now being accessed
Please wait <-- (3)
(Approximate time : I minute)
H~lp Notepad IHookbook Assumrt ions Prior Screen MiMeu in Qut1-(41 IF F5 F6 F7 FI1O
Figure 12 - System Structure Representation
82
This is the entry screen for the support system. The system is
spec if ied on the memory aid bar (1) and the major system options are
designated on the system connand menu bar (2). The system work and
display area (3) contains a message welcoming the user informing him
that there will be a brief wait while data files are accessed. The
major system support functions are contained on the bottom menu, the
system support menu bar (4).
System Comwnid Bar: The major decision areas are separated from the
rest of the system and placed on the top command bar. They are also in
all caps as a further reminder that they are decision area commands.
Descriptions of the commands can be accessed through the help command
(Fl). Added information for selected options follows.
POSITION: The POSITION option allows the user to filter the databaseso only the crew position of interest is evaluated during the currentsession. A specific position must be designated for the schedulingroutine and some of the manning profile routines. If a position hasn'tbeen named prior to selecting the SCHEDLLE option or the affectedsubsets of MANING PROFILE option, the system will self-select thePOSITION option before continuing.
SCHElULE: This is where the bulk of the work with this system willbe conducted. Current position in manning and future projections areused in a "What-if" cycle to arrive at manning acquisition decisions. Asignificant feature of this capability is the ability to show TAC/MPCthe immediate and long term consequences of their acquisition policies,in a quantitative format.
MANING PROFILES: This options has a two-fold function: to providebackground at a low level (a subset of the actual database driving themodels), and to allow the user to make high resolution assessments ofthe experience (current and near term change) of the crew position he isresponsible for.
83
System Spport Bar-: The system support bar provides the support
required of the system that is not directly related to the decision
elements of the process being supported. These commands are separated
from decision commands by position and by the typographic form of the
commands.
Help: Provides context sensitive insights and direction that clarifythe decision process and the software's operation.
Notepad: Provides user with a pad to document his insights,questions, and ideas about personnel. acquisition.
Hookbook: A notebook for system related ideas or comments.(Any sequence of menu selections in a particular session are stored in abuffer and recorded as memory aid entries on the notepad and hookbookwhenever they are opened).
Assumptions: At any point, the user can query the system to seewhat assumptions have been made in the way data is treated. This isparticularly useful for ensuring the user understands what models aredoing, and what the weaknesses of their operations may be.
Prior Screen: Allows user to backup 1 screen per keystroke. Ascreen is defined as any display that is uniquely different from theprevious one, i.e., if a display has a sequence of commands performeH onit that adds new material, each stroke of F5 will back the user out tromthe results of the last command.
Main menu: Takes user to the system command bar for his next action.
Print: Make hard copy of current screen and the materials beingreferenced, e.g., entire roster being referenced, of which only aportion is visible on the screen.
Quit: Returns user to operating system or shell, with an option tosave any work.
Major System Cantrol Features: The major control features of the system
are embodied in the menuing structure displayed on the screen. Further
structure is provided by limiting access to certain system command
options when others are active. In these instances, commands that
84
cannot be selected from within current operations disappear from the
command menu. For example, when MANNING PROFILE is selected, the only
other system command option that remains on the menu is the POSITION
option, allowing profiles to be specified by crew position (see the
following story board).
M y Aids: Memory aids are also largely embodied in the systems
presentation format. Main system commands that are currently active
remain highlighted; sub-level options and the choices made at the sub-
level will be displayed on the Memory Aids Bar (2). (See the third
story board for clarification). As a further aid to memory, levels of
sub-menus are kept to a minimum; the user's understanding of where he is
in the decision cycle in enhanced by keeping command trees short.
85
AWACS PERSONNEL ACQUISITION MANAGEMENT SYSTEM
IPOSITION ISCHEDULE IMANNING PROFILE AWJACS CAPABILITIES IFORCE STRUCTUREI
Main menu options may be chosen from the system menu bar by selectingthe highlighted character, by pointing and clicking with a mouse or byplacing the cursor in the desired box and striking return.
POSITION - Select the crew position of interest. Defaultis all positions being evaluated.
SCHEDULE -Plan personnel acquisitions strategies.
MANNING PROFILES - View current manning, experience profiles, gainand loss estimates and their impacts.
AWACS CAPABILITIES -Assessments of ANACS systems effectiveness asimpacted by personnel issues.
FORCE STRUCTURE -View weapons system factors driving personnelrequirements.
System support bar options may be selected by designated F keys and bythe methods for the main menu (except there are no highlighted letters)
Help INotepad IHookbook Assaf io Prior Screen IMain Menu IPr int]IGiF2? F3 4 F5 Fi F7 F10
Figure 13 - Example Help Screen -- Decision Command Descriptions
86
This is the main help screen that gives a basic descriptici of the
main commands on the decision command bar. This is the help screen that
is activated if no sub-menu optiais have been user selected. More
detailed information is available by selecting (via mouse or cursors)
the command of interest. In general, the help system can be accessed at
any time and is context sensitive, with help being provided for the
currently active command.
87
POS: COT
POSITION NKING PRUFILE-The MANNING PROFILE option allows you to examine different aspects ofhow experience is dis ributed in a crew position, and by whom. Differentfilters are provided to help you assess experience distribution issues.
Rank filter : Specify a rank or range of ranks you wishto examine. Default is no filter set; allranks are examined.
Losses : Examine roster of crew members who are projectedas losses in the database.
Gains Examine roster of crew members that are currentlyprojected as inbound.
Display : Roster of all AWACS crew members.
Experience : Graphic display of experience profiles forcrew positions.
Current : Shows the experience profile for the crew positionas it is currently manned.
Projections: Shows projected experience resulting from the manager'sproposed acquisitions.
Help NoteFa Hok3o Assumptions Prio cren Maneu rnt Qi
Fokbo I I34F F6 F7I FIO
Figure 14 - Example Help Screen -- Manning Profile Options
66
This is the help screen that is displayed if help is activated while MANNING PROFILE is
selected on the decision command bar. (he memory aid bar indicates that the MANNING PROFILE option is
indeed active). Each of the sub-menu options under MANNIN6 PROFILE is briefly described. More
detailed descriptions are available by selecting the sub-menu option of interest (by highlighting the
appropriate title).
89
Assessment of Current Crev Position Status
(Entry Level Assessments)
90
CURRENT EXPERIENCE STATUS (POS: ALL FUNCTION: MANNING PROFILE -EXPERIENCE)
POSITION MANNING PROFILE
EXPERIENCE LEVEL HISTOGRAMSAuthorized: 1282 Authorized: 1282Total shown: 1200 Total graphed: 1200I
1 225N
E 180 -XPE 135 -RIE 90 <NCE 45 -
6R 0
a .6 5 t5 .4 .5 A.6 5 4UP EXPERIENCE FACTOR EXPERIENCE FACTOR - PROJECTED
CURRENT WITH KNOWN LOSSES RECAPTURING ALL TRACKED PRIOR ANACSTHE CONING YEAR MEMBERS ABSENT AT LEAST 3 YEARS
(THIS IS A BEST CASE PROJECTION)
LOSSES BALANCE GAINS
Help Notepad Hookbook Assumptions Prior Screen Main Menu Print quitlI F2 I F3 1 F4 1 F5 F6 F7 I No
Figure 15 - Aggregate Experience Profile for ANACS
91
This is the main entry screen, and is entered by system default after
all necessary data files have been loaded. The first time into the
system, the current aggregated experience profile for all AWACS crew
positions is presented. After a particular user selects the crew
position he will use, the system will subsequently come up with his
position profiled. (For high level managers, the profile for all
positions is available by setting the position filter to "ALL").
This entry point to the decision process was selected allow managers to
understand the current experience profile of the positions they manage
before becoming involved in acquiring bodies. Managers will therefore
know when acquisition planning starts whether special attention must be
given to the types of acquisitions offered by MPC.
The manager gets two profiles. The left profile shows the current
status of the crew position and the impact known losses will have on it.
The right profile indicates the best profile available if all known
"eligible" former AWACers from this position are recaptured from the Air
Force manning pool. (If the corrections are not enough, the manager may
have to examine the possibility of capturing people with applicable
experience).
Position is the only system command menu directly available to the user
from the home display. Position filters can be changed directly by
hitting "P" or selecting "POSITION" with a mouse or cursors.
92
Leaving this display requires F6 to be selected to enable access to the
wh-ol e system c ommand menu bar.
93
CURRENT EXPERIENCE STATUS (POS: ALL FUNCTION: MANNING PROFILE - EXPERIENCE)
POSIT N MANNING PROFILE
P/CP EXPERIENCE LEVEL HISTOGRAMSNAY Authorized: 1282 Authorized: 12B2FE Total shovn: 1200 Total graphed: 1200"CCSD
ASOASTARTCDIIICSOCT (
ALL
NCE 45(
R 0-0 .65 1 5 ..45 1 .655 1. 65 .5UP EXPERIENCE FACTOR EXPERIENCE FACTOR -PROJECTED
CURRENT WITH KNOWN LOSSES RECAPTURING ALL TRACKED PRIOR AWICSTHE CONING YEAR MEMBERS ABSENT AT LEAST 3 YEARS
(THIS IS A BEST CASE PROJECTION)
LOSSES BALANCE GAINS
Help INotepad Hookbook IAssumptions IPrior Screen 1Main Menu IPrint IquitFI F2 F3 1 F4 1 FS F6 F FT1 FO
Figure 16 - Setting a Position Filter for Experience
94
Selection of POSITION drops a pop down menu containing the AWACS
abbreviations for all crew positions on the aircraft. Selection is by
command letter (highlighted) or by point and click. The default
selection is "All".
Selecting Help at this point would spell out the positions associated
with each abbreviation, describe the filtering that occurs if a position
is selected, and describe why filtering might be desired.
After a position filter is selected, the choice made will appear on the
memory aide bare on all subsequent screens as long as the filter is
active.
95
CURRENT EXPERIENCE STATUS (POS: MCC FUNCTION: MANNING PROFILE - EXPERIENCE)
POSITION MING PRUVILE
EXPERIENCE LEVEL HISTOGRAMSAuthorized: 86 Authorized: 86
Total graphed: 80 Total graphed: 76
1 16 (N
14 (EX 12 (PE 10 (R1 8(EN 6 (CE 4 -
6 2 (R0 0u 5 ..555 v .45 .5 1 AP
FIPFRIENCE FACTOR EXPERIENCE FACTOR - PROJECTEDCURRENT WITH KNOWN LOSSES RECAPTURIN6 ALL TRACKED PRIOR AIACS
THE COMING YEAR MEMBERS ABSENT AT LEAST 3 YEARS(THIS IS A BEST CASE PROJECTION)
I ---1MLOSSES BALANCE GAINS
Help | Notepad | Hookbook I Assumptions | Prior Screen Main Menu 1 Print QuitFlI F2 F3 F4 F5 F6 Fl I FO
Figure 17 - Current MCC Experience Profile
e9
The user operation executed in the previous storyboard selected MCCs for
profiling - this is indicated on the memory aid bar. The displayed
profiles now reflect MCC experience.
This screen also provide baseline information that is useful to the
manager when evaluating a crew position. Information on current
authorizations and manning are included above the profiles. In this
instance, the MOC manager can see that he is losing key experienced
people, and that he is already short on people. In the normal course of
the acquisition process, the manager could expect to get novices that
would enter the experience profile at the low end of the scale. If he
corrects his initial manning deficiency and his expected losses with
this Lype of acquisition, he easily see that the inexperienced end of
the experience profile will become very heavy. He must do something to
acquire experienced people if he is to avoid this situation.
97
POS: MCC RANKS: ALL FUNCTION: EXPERIENCE PROJECTIONS
POSITION MANNING PROFILE
EXPERIENCE LEVEL HISTOGRAMS
1 20
I 1N
16EI 14PE 12RI 10EN 8CE 6
G 4R0 2UP 0 - -
EXPERIENCE LEVEL GROUPINGS
1909 19" 1991 1992 1993
Help Notepad Hookbook Assumptions JPrior Screen Main Nw Print QiI F2 F3 F4 I FS I ,F6 I P iF
Figure 18 - CC Experience Profile Projections
| p z98
This is the experience profiling screen for evaluating the long term
impact of the acquisitions that the manager plans for the five year PFT
cycle.
The experience level projections are driven from the scheduling entries
that the manager makes during his "what-if" examination of acquisition
scheduling options (see the next sto'yboard). His control of numbers of
people acquired, their experience, when they are acquired, and his
suggested control of factors that impact how a position loses people all
impact the projection.
In this instance, the manager was looking at MCCs. The left most bar in
each experience group is for the current year (1969 in this instance).
Looking across the different groups at the left most bar indicates that
the MCC position is heavy on inexperience people. The right most bar
gives the profile after five years if the managers acquisition plan is
followed. In this instance, experience for the position is distributed
more evenly across the groups, the number of people in the three most
experienced groups has doubled, and the average experience of the crew
position has increased.
99
Scheduling kquisitions
(and Hid-level Assessents of Crew Position Status)
100
POs: MCC
POSITION SCHEDULE MANNING PROFILE AWACS CAPABILITIES
AU 106
1 94
E
AN 82N
N 76
YEAR (INDEXED WITH RESPECT TO CURRENT YEAR)
INTERNAL EXTERNAL CURRENT SAVE SAVE SAVE
TRAINING CODE i5 20 4 22 5
FLY HRS AVG TOS .13% 4.5 .181 5BODIES 20
1989
LABELS RED (NOW) BLU 'IF'd YELW i' REN'
Heil Ntead Hookbook I Assumpt ions IPrior Screen Main Menu Print QuitF I FF34F5 F6i F7 FIG
Figure I9 - Scheduling Acquisitions
101
Selecting SCEDLE runs a model that plots projected manning based on
current manning and acquisition policy against the planned requirements
derived from FORCE STRUCTURE considerations. The requirements are in
black and the current projections are in red. Levels for all current
policy parameters are shown in the parameter box labeled "FED (NOW)".
A table of labels for the parameters that quantify policy positions is
provided, along with a corresponding table of current values. The
parameters are broken into two categories, internal and external. The
internal policies are things that the 28 Air Division has control over,
and the external policies are those whose change would require
concurrence from outside agencies, such as TAC and MPC. As examples,
the 28 Air Division has some control over how it allocates flying hour
budgets and has some flexibility in how many people it can train in each
crew position each year. However, MPC controls the length of the CODE
55 incurred for training, and also determines the average tour length
through its PCS actions. The breakout is provided so the user knows
which parameters he has the most control over.
The parameters that appear in the boxes are chosen to fit at least one
of three criteria. First,they can be things that impact acquisition
capabilities in some way (such as training capacity that limits how many
people can be bought in a year). Second, they can be things that will
alter demand for personnel (such as Code 55s, which tend to impact the
average time in AWACS). Third, they can be things that impact
102
experience of the crew force (such as number of acquired bodies in a
given year with some specified experience score). (Actually, it is
necessary that some of the parameters be directly related to the factors
that the experience scoring model uses to produce scores. There must be
some mechanism between the scheduling process and the experience
projections that allows individual experience to be incremented as time
goes by).
MIltiple ".hat-if" tables are provided to allow the user to make
comparisons among various settings of the parameters. For the current
screen, assume that the current training structure is set up to train 20
CDMTs a year and that they incur a 4 year CODE 55 after training. The
user can look at the impact of changing these values to 22 and 5
respectively, and observe the impact on the manning chart. (The
parameters can be scrolled and are synchronized across the boxes to
facilitate comparison of the parameter values for a given plot). The
objective is for the user to come up with an achievable set of
parameters that cause the projection line to match the requirement line
as nearly as possible.
Each of the "what-if" boxes is plotted in its ow color; a new box will
be entered when the SAE command of the current box is selected.
Previously saved boxes can be re-entered by backing up through the boxes
using the Prior Screen (F5) command.
103
While this screen's operations is to schedule acquisitions, total
manning is not the whole picture. Therefore, the MANING PROFILE and
AWACS CAPABILITIES menu options are available to examine the impact of
changing parameters. Selecting the AWACS CAABILITIES option will drop
a window over the graph produced by SCHEDULE that contains the same
capabilities graph previously described, with plots for the CLNT box
and any "what-if" boxes with changed parameters in them. Sekc:tion of
the MANNING PROFILE option from within SCHEDULE will drop a window over
the current plots and contain a line graph of the SCHEDULLE and AWACS
CAPABILITIES format. The graph will indicate a reference "average
desired experience level" line for the crew position, and display the
plots for the CURRENT box parameters and any "what-if" box parameters
that have been saved.
By assessing his choices graphically with direct comparison of the
results of his choices, the user can arrive at a "good" answer, and
support it. He knows what he has considered, and has built in
documentation showing the results of his various choices. He also has
quantitative estimations of the consequences of any parameter value
forced on him from outside the Division, e.g., TAC decides to buy only 4
new CDMTs for a given year.
104
Systems Operat ions Impacts
(End-game Assessments of Acquisition Strategies)
105
AIJACS CAPABILITIES
934 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- (0 Me i
eudi
96 --------------------------------------------------- (w Ma ir nt i
YEAR (INDEXED WITH RESPECT TO CURRENT YEAR)
Help INotepad I Iookbook IAssumptions IPrior Screen I ai enu IPrint IQiIl F2 F3 F4 F5 fliF6 F7 Fuit
Figure 20 - High Level Capabilities Assessment
106
Selection of AWACS CAOABILITIES produces a graphic plot of AWACS mission
capability as affected by personnel issues. The plot is from the
current date minus 2 year to current date plus 5 years (1 long range
acquisition cycle). The capability factor is plotted relative to two
reference levels (scaled to 100): desired day to day minimum capability,
and minimum war fighting capability for effective operation. The plot
is driven by a Multi-Criteria Optimization model that accesses the
personnel database. Major inputs are crew experience profiles and
absolute manning levels.
The impact of personnel issues on AWACS capabilities are of paramount
importance. This combined assessment of manning levels and experience
provides an important tool to structure new accessions. Quantity may
have to be balanced by quality; re-acquiring of previous AWACS personnel
may be the only solution to a decrease in capability caused by a lack of
experience or a shortage of bodies. The timing of acquiring new bodies
or attempting to re-acquire old ones will be critical in balancing
capability shortfalls that occur in the future due to current personnel
managements. The AWACS CP°ABILITIES graphics provides a means for
identifying the shortfalls and the specific causes.
The memory aid bar indicates there is no crew position filter set. If a
filter has been previously specified, selection of AWACS CAOABTLITIES
disables the filter.
107
Systems capabilities is a composite estimation of ability to perform a
mission. No other menu options are available from this command.
A zoom feature is supported to allow for increased resolution in a
selected area. This facilitates examination of critical areas in the
future. Selecting the Zoom command will position cross-hair on the
screen that can be positioned with mouse or cursors. A special key to
execute Zoom is provided; this is as a courtesy to facilitate execution
of a command from the live window. It is also a reminder that there is
more to this screen than the presented graph. All commands offered as
sinole commands on the live screen can be executed by clicking or
striking the return key.
108
AVMCS CAPABILITIES
94 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -- (D Iei
r ae u
86 - - - - - - - - - - - - - - - - - - - - -- - - - - - - - - - -- (W Ma ir nt i
mu
YEAR (INDEXED WITH RESPECT TO CURRENT YEAR)
Help INotepad IHookbook IAssumptions IPrior Screen R ain Nenu I Print IQuitF I F2 I 3 F 1 4G 1i F5Frio
FVre 21 - Changing Capabilities Assessment Resolution
109
Selection of the Zoom feature placed cross-hairs on the screen. After
positioning, clicking, hitting return, or hitting F8 executes the zoom
at the specified point.
F8 option is again the "hot key" from the live screen. Again, all
commands offered as single commands on the live screen can be executed
by clicking or striking the return key.
110
Zoom engaged -- Details of tagged tie + 6 months
AIMCS CAPABILITIES
94 ---------------------------- I ------------------------ ( MIe i
s ni i
e ud a
96------------------------------------ a i
r nt ii 0I • Ue •
- - 4 A -t -1 6 1 t S ILIMITING POSITIONS CENTER DATE: APR 91 DATE STEP SIZE: 1 MONTH
1. P -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 62. FE3. CDNT 89 89 697 96 96 95 93 94 8 89 9 90 924. AST .2 .4 .9 .3 .3 .1 .8 .2 .1 .3 .7 .8 .1
Help Notpa Ho3 0k Assumptions Prior Screen Main Menu Prin QuitF1 F6 F37 Ff Fnt IO
Figure 22 - Detailed Capabilities Assessment
111
The screen displayed as a result of the Zoom sequence show a roughly 7
to I increase in date resolution Cn a re--scaled graph. The display
centers the selected point from the low resolution graph on the new
plot, indicated by vertical tics crossing the reference lines at the 0
coordinate. The plot is shown for six months on each side of the
selected point.
The fact that the user is viewing capabilities at increased resolution
is provided as a reminder on the memory aid bar.
Discrete values for each month are provided in a table to remove
interpolation problems. Integer parts are place on one line and
fractional parts on another. This improves readability of the screen,
and also speeds up scanning for trends, ignoring fine detail.
Finally a table is provided, showing the four crew positions that the
model driving the assessment being plotted is most sensitive to at the
current point. (Sensitive to with respect to improvement). The crew
positions are rank ordered.
Detailed assessment of the impact of specific crew positions is
available at this point by selecting the position of interest from the
"LIMITING POSITIONS" list.
112
Low level Supporting Information
113
FORCE SMTRUTR
1989 1991 199 2000 BASE
PAA j +3 29
ICREWS j +.2 +.2 1.7
POS AIRCRAFT MODIFICATIONS30 -- 35
SD + IUD +2 3ASO IAST f2 2ART 2CONT I (--CSO +1 ICT IPOS ORGANIZATIONAL CHANGES [
AN %2P +1 NAY +I 14 (-"FE +1 3"cC +1 28So 9ID0 8 ,-
ASO 23 _
Help l oea ookbook IAssumptions IPrior Screen m a in menu IPrintFI F2 3F4 F5 F6i F7 I Flo
Figure 23 - ANACS Force Structure Representation
114
The FORCE STRJCTLRE option is selected to view the prime factors that
drive changes in AWACS manning. Theses are:
PAA: Primary Assigned Aircraft - The number of aircraft used in
conducting AWACS mission. Purchases of new aircraft would require more
manning.
CREWS: Number of crews assigned for each PAA. Changes in AWACS
mission could warrant changes in the number of crews assigned to each
aircraft.
AIRCRAFT MODIFICATIONS: Changes in AWACS mission can require changes
in the crew structure. As an example, the block 30-35 mod on the E-3
will call for more W)s and ASTs to enhance weapons control and
surveillance capabilities.
ORGANIZATIONAL CHANGES: Structural changes in the 28 Air Division
may impact manning. The creation of the 962 AWAC Squadron increased the
overhead for certain crew positions to provide the squadron staff.
The memory aid bar indicates there is no crew position filter set. The
POSITION command is present, indicating FORCE STRUCTLRE can be filtered
if desired. (If selected, the position pop down menu appears). If a
position is selected, the PAA and CREW elements remain unchanged. Only
the crew position of interest would appear under AIRCRAFT MODIFICATIONS
and ORGANIZATIONL CHANGES.
The AIRCRAFT MODIFICATIONS and ORGANIZATIONAL CHANGES sub-charts have
scroll bars to scroll the crew position charts, if necessary. Point and
115
click (mouse or cursor/return) on the up or down arrows to scroll in the
indicated direction, with the current position in the list being
indicated by the left pointing arrow. (As in Macintosh operations).
The sub-chart heading will remain to indicate which sub-chart is which.
The BASE column holds the appropriate values for the current force
structure. In the AIRCRAFT MODIFICATIONS sub-chart, the number for each
position per crew will be displayed. In the ORGANIZATIONAL CK-iNGES
sub-chart, current overhead in each position will be displayed. Crew
positions will be coordinated across all columns and scrolling
synchronized.
Columns are only provided for years in which changes occur. Only the
change from the previous level is indicated, and changes are cumulative
across the columns. (For example, a change in CRW is indicated for
1996 and 2000. With BASE indicating a BASE RW of 1.7 per aircraft,
the figure indicates a force structure of 1.9 crew/aircraft for 1996 and
2.1 for 2000). Change, and not level, is shown because the intent of
this chart is to indicate what changes are going to occur and when they
will occur.
No operations occur under FORCE STRUCTURE. It is provided as a means of
representing critical background information.
116
POS: CDMT
POSITION MING PRUFILE
Rank filtersLossesGainsAllExperienceCurrentProject ions
Help Notepad Hookbook Prior cre Man enu Pit Qi
Figure 24 - Manning Profile Sub-options
POS: CDNT
POSITION RMIING WRIILE
Rank filters
AE 2 3 4 5 6 7 8 9E
Highlight ranks you want to see.Select either ta single rank or acontinuous set of ranks.
Help I Notepad I Hookbook | Assumptions I Prior Screen I Main Menu I Print I Quit IF1 I F2 F3 F4 I FS FG I F7 FIOj
Figure 25 - Selecting a Rank Filter
117
The MANNING PROFILE features allow the user to explore different aspects
of his problem at the database level. He can look at the data and make
direct correlations with the people he knows. He can look directly at
short term losses and determine how well his known gains balance losses.
Support is provided to look at all people in the database. Finally,
experience profiles are provided so the user has a feel for some of the
factors involved in the AWACS CAPABILITIES assessment. Selection of the
MANNING PROFILE option drops a pop down menu of sub-options.
Basic descriptions of the sub-options are given on as part of the help
system, (accessed via F1, and sho.n as the last storyboard). The
position filter can be set from within the MNNNING PROFILE command, as
indicated on the menu bar. In this case, the filter is currently set to
CDMT, as indicated on the memory aid bar.
One of the primary purposes of this section of the support system is to
provide the user with basic data, in a structured environment, to allow
him to exercise his intuition. His intuition may become fine-tuned in
the process of using the system, or it may prove to be better than the
models driving the system, which should prompt him to make hookbook
entries concerning his perception of system deficiencies.
Selecting 'Rank filters' causes a pop down rank selection area to
appear. Rank is selected by clicking and dragging with a mouse or by
placing the cursor on the desired end points of the range and hitting
118
return. (Hitting return twice selects the single rank designated by the
cursor). The ranks displayed in the select ion area are keyed by the
selected crew position. (The rank filter is the deepest branch of the
decision command tree, and is only three levels deep).
Rank filtering is provided to allow the user to break the crew position
being examined into smaller chunks that have special significance to
him. Rank appears to be the best discriminator, since it is linked to
respc-sibility level, time in service, experience, and special areas of
qualification within the crew position.
The capability of selecting a range of ranks is provide to allow
resolution flexibility; looking at the top three may be sufficient for
some judgments, while the crew positions status with respect to E-9 may
be critical in others.
119
POS: CDIT RANKS: E7 - E9 FUNCTION: LOSSES
POSITION ANNIN G PROFILE EX RI N E L SEXPERIENCE LOSSRANK NAME FACTOR DATEI I *. I --.
Help Notepad I Hookbook i P o n ain Menu Print quitF1 P ee F3F F6 H FIO
Fipre 26 - A Lov Level Data Representation
120
This is a representative screen for the GAINS and LOSSES options of the
MANNING PROFILE choice. Both of these options require a crew position
to be selected, and are two of the three mentioned from the opening
screen description. Position can be changed, but the "All" option will
not appear on the menu.
This screen is displayed as a loss screen. The memory aid bar reminds
the user that he is looking at CDMTs in the grades of E7 to E9 who are
projected as losses.
A scroll bar is provided to scroll through the list. Point and click
(mouse or cursor/return) on the up or down arrows to scroll in the
indicated direction, with toe current position in the list being
indicated by the left pointing arrow. (As in Macintosh operations).
The sub-chart heading will remain to stationary during scrolling. The
roster is indexed on RAW then LOSS DATE and then NAME. An indexing
command can be supported and implemented through a pop-up priority
selection table.
121
Apendix C
Hodoxx Entries
This appendix contains the aggregated contents of the hookbook
entries made during the research effort. The IDEA portions of the
individual hookbook entries have been grouped by the system's adaptation
areas that were evident to the author.
122
RESOLUTION - DON'T BECOME CONShEED WITH BUILDING HIGH RESOLUTIONMODELS. CERTAINLY, AT THE BEGINNING, LOOKING AT NEEDSBASE ON AN ANNUL TIME FRAME IS ADEQUATE.
RESOLUT ION - SUPPORT CHANES IN THE CURRENT YEAR RESULTING FROMQATERY UPDATED MEETINGS.
RESOLUT ION - STAT LOOKING AT TRAINING FLOW AND THE PROBLEMS ITINTRODUCES IN THE SYSTEM. THESE TYPES OF FACTORS WILLFOR AN INCREASE IN RESOLUTION, SINCE TRAINING ISN'TGENERALLY UNIFORM ACROSS A YEAR.
RESOLUTION - fANL SPECIFICATIONS INITIALLY, WITH ASSUMPTIONS TIHATTRAINING IS UNIFORM ACROSS THE YEAR (NOT TRUE, BUT IN THESCOPE OF THIS PROBLEM AT THE INITIAL STAGES, THEVAIATIONS IN MANNING AND EXPERIENCE CAUSED BY NON-UNIFORM TRAINING THROUGH THE YEAPR E INSIGNIFICANT WHENCOMPA ED TO THE PROBLEMS CAUSED BY MANAGRS NOT LOOKINGAT THE ACQUISITION PROCESS IN ANY DETAIL BEYOND THEIMMEDIATE YEA'S NEEDS.
MODELS -- THREE BIGGIES - EXPERIENCE - MANNING LEVEL -
CAPABILITIES
MODELS - EACH OF THE THREE MAJOR MODEL HAVE TO BE ABLE TO LOOK AT"NOW" AND PROJECT TO THE FUTURE
MODELS - IN ORDER TO MAKE PROJECTIONS, THE MODELS ARE GOING TOHVE TO INTERAT
MODELS - PERSONNEL ISSUES ARE "SOFT" ISSUES. IF MCDMMETHODOLOGIES A USED IN THE MODEL DE.ELOPMaT, THEMODELS MY BENEFIT FROM A STATISTICAL VALIDATION OF THEVALUES THEY PRODUCE.
MODELS -- THE CAABILITIES MODEL CAN CERTAINLY BE EXPRESSED AS A
MLTI-CRITERIA OPTIMIZATION PROBLEM (OR SOME OTHER FORM
OF MI" MODEL). IT IS CONCERNED WITH BLICINGCONFLICTING OBJECTIVES - MANNING LEVEL AND EXPERIENCE,FOR EACH CREW FORCE - WHERE -R RESOURCES TO BEALLOCATED.
MODELS C- AABILITIES NEEDS TO BE ASSESSED AGAINST MULTIPLEEXPERIENCE CRITERIA. MEAN EXPERIENCE IS NOT SUFFICIENT,SOME DISTRIBUTION MEASJRES WILL ALSO BE NEEDED.
MODELS - OPPS!! SCHNEIDER'S DEPATURE MODELS ARE NOT GOING TO BEADLEQLATE. DEPARTS ARE GOING TO HPE TO BE BROKEN OUTINTO CATEGORIES THAT CAN BE CORRELATED TO EXPERIENCE IFEXPERIENCE PROJECTIONS A GOING TO BE MADE.
123
MODELS - ASSUMPTIONS: PROGRESSION IN EXPERIENCE WILL BE BASED ONAL PERSE*E.. IN THE POSITION DOING THINGS THROUGH TI ME,SUCH AS MEETING THEIR MINIMUM QULIFICATION FLYINGHOLR/MISSIONS GOALS. ANY EXCESS FLYING HOURS WILL BEAPPORTIONED UNIFORMLY AMONG THE CREW FORCE IN THIS CASE.THIS WOULD BE A MODELING ASSUMPTION, AND NEEDS TO BESTATED AS SUCH.
MODELS - ASSUMPTIONS: MODELS M<E LOTS OF ASSUMPTIONS - DOCIUENTTHEM.
MODELS MODELS ARE GOING TO HAVE TO HANDLE LOTS OF TRICKQUESTIONS TO MAKE PROJECTIONS. TRANITIONS ACROSS CLASSBOUNDIES WILL HAVE TO BE HANDLED AND DOCUMENTED.
MODELS - DISCRIMINIT ANALYSIS BEAS EXAMINATION IN THE FUTURE,BUT NOT NOW. UNTIL THE SYSTEM IS CIGED ENOUGH THATTHERE IS SOME PROBABILITY OF RECOVERING PEFA-3NEL FROMTHE AIR FORCE MANIING POOL, PRIORS FOR THE DISCRIMINANTFUNCTION CANNOT BE ESTIMATED. (ACTUALY 0 CIJRN1LY,WHICH DOESN'T HELP DISCRIMINATE).
MODELS -- RTIFICIAL INTELLIGENCE MODELS P NOT A PLAYER IN THISSYSTEM UNTIL IT, AND TIE USERS MATURE QUITE A BIT. AI ISA POSSIBILITY AT SOME FUTURE DATE FOR SPEEDING UP THE"WHAT-IF" CYCLE IN THE SCHEDULING OPTION.
MODELS A GOD EXPERIENCE MODEL - PROBABILITY OF BEING ANINSTRUCTOR / STAN EVALER. ASSUMPTION - THESE PEOPLEAT THE HIGH END OF THE EXPERTISE SCALE FOR THE CREWPOSITION.
MODELS -- QUESTIONS - ANSWERS REQUIRE STATEMENT OF ASSUMPTIONS1. WHAT ARE THE MECHANISMS FOR PREDICTING LOSSES IN EACH
CREW POSITION- RATES BY RAK SEPARATIONS AND LOSSES - BASEDON AWACS DATA - OR IS AIR FORCE WIDE DATASUFFICIENT UNTIL AWACS DATA IS DEVELOPED?
- RATES BY YEARS OF SERVICE ??- IMPACT OF THINGS LIKE FORCE REDUCTION
2. SAVE FOR GAINS -- PROJECTING PROMOTIONS IS IMPORTANTIF LOSS RATES AR BY GRADE.
MODELS -- LOOK AT THE RESULTS'' JUST BECAUSE THE MODELS SAYSOMETHING, DON'T TAI<E IT AT FACE VALE. USINGINSTRUCTORS AS AN EXPERT BASELINE COULD REALLY GIVESCREY RESULTS IN A REGRESSION IF A) BEING AN INSTRUCTORIS VIEWED AS A PROMOTION POSITION, AND B) YOU DON'T CPFACTORS SUCH AS YEARS OF SERVICE FOR PASSED OVEROFFICERS.
124
MODELS M<E MODELS ABLE TO RECOMMEND OPTIMUM STRATEGIES, THENRESPOND TO "WHAT-IFS" FROM THA4T INITIAL STARTING POINT ?DO I REALLY WANT TO IMPINGE ON THE USERS WHAT IFEXPLORATION ? HOW DO vOU KEY THE USER TO STAT HISPROCESS? THIS AT LEAST GIVES HIM A STARTING POINTBESIDES A BLANK SCREEN.
MODELS - POSSIBLE PRETERS FOR SCHEDULING SCREENCODE 55AVG ON STATIONACCESSION SOURCESEDUCATIONSTRAININGFLYING HOURSNEW/USED SPLITS
THESE ARE JUST ODD THOUGHTS ON POSSIBLE PAETERS THATLEND TIHEMSELES TO MANIPULATION BY THE USER TO AIVE ATA DECISION. THESE ARE THE PPRAETERS THAT VAY THEMODELS OUTPUTS. FINAL DETERMINATION OF APROPRIATEPAR!AETERS WILL HAW TO BE BASED ON THE FACTORS THAT GOINTO QUANTIFYING EXPERIENCE. THE DATA IS GOING TO DECIDETHIS. PARAMETERS WILL HAVE TO HAVE SOME DIRECTCORREJATION TO THE INPUTS TO THE MODELS.
MODELS - OTHER PARAMETERS - CONCEIVABLE MORE IMPORTANfT THAN MAJORPARAMETERS ABOVE BUT THEY WOULD REQUIRE POLICY CHANGSOUTSIDE OF THE W400RS SCOPE OF OPERATION
CODE 55AVG TOUR LENGTHACCESSION SOLRCESMISSIONS FLOWN
MODELS - DON' T FORGET DOS ETC AS INPUTS TO LOSS CALCULATIONS MPi<ESURE ALL DRIVERS FOR GAINS AND LOSSES ARECONSIDERED.
MODELS - MAJR PARAMETERS TO BE MAIPULATED BY CIRRENT IAGER -BODIES AND TIMING.
MODELS -- PRINCIPLE COMPONENTS ANALYSIS COULD HELP SCREEN THEPOSSIBLE FACTORS TO BE CONSIDERED IN THE EXPERIENCEANALYSIS. THIS COULD SIGNIFICANTLY SIMPLIFY DATABASEACCESS ISSUES.
EXTENSIONS - GJPHS PROVIDE QUANTITATIVE AMMO IN BA INING WITHMPC/TAC CONSEENCES RE IMMEDIATELY DISCEBLE. OCCURSTO ME THAT THESE GRAHIC L PRESENTATIONS CAN PROBABLY BELNERSTOOD BY TAC AND FWC - GOOD AIMMO. THIS CAN ALSOPROBABLY BE CONSIDERED A CONTROL MECHANISM.
125
EXTENSIONS - AN EXTENSION OF THE USE OF THIS SYSTEM WOULD BE TOIDENTIFY INDIVIDLWqS WHO ARE "BEHIND THE CURVE" AND FLYTHEM MORE OFTEN.
EXTENSIONS -- THERE IS A DANGER THAT THIS SYSTEM COULD BECOME A SOARETO BE FILLED. HOW TO AVOID THIS, IF IT IS SOMETHING TOBE AVOIDED. IF FILLING THE SQUARES BECOMES THE GOAL, AIDIT IMPROVES "EXPERIENCE", IS THIS A PROBLEM. THEVULNERABI IL I TY OF THE SYSTEM BE ING CAMED FOR OTHERPURPOSES IS VERY DEPENDENT ON THE FACTORS CHOSEN TOINDICATE EXPERIENCE. FOR FACTORS SUCH AS TIME INSERVICE, THERE IS NOTHING THE MEMBER CAN DO ABOUT ITSIMPACT ON HIS EXPERIENCE SCORE.
EXTENSIONS - EXPERIENCE SCORES COULD BE USED TO HELP IDENTIFYCANDIDATES FOR "SPECIL JOBS" SUCH AS INSTRUCTOR, STANEVAL OR SCHOOL ATTEMNrCE. THIS COULD ALSO BE USED AS AMODEL VALIDATION TOOL, SINCE CONSISTENT DISAGREEENT MAYINDICATE EXPERIENCE IS BEING SCORED WRONG.
PRESENTATION - SPLIT SCREEN FOR CHARTS -- ENCOURAGE DIRECT COMPARISONS,DON'T HAVE TO GO TO PRINTER IF POSSIBLE - PARTICLLPRLYFOR EXPERIENCE AND MANNING
PRESENTATION - TOGGLE ON YEAR TO BLOWUP PROFILE IN 1 YEAR PERIODS POP UPTABLE OF MIN/MAX VALUES, AVG, STD, OTHER PERTINENT DATATO CLARIFY/SPPORT GRHICS DISPLAYS. CLOSE LOOKS ATDATA WILL HAVE TO BE S IPPORTED, AN1D INTERPOLATIONPROBLEMS SHOULD BE AVOIDED.ADD TABLES OF CRITICAL INFORMATION KEYED TO GR HS.
PRESENTATION - HOW TO DISPLAY EXPERIENCE DISTRIBUTION ON PROJECTIONS- HAVE A BAND INSTEAD OF A LINE TO ACCOUNT FOR VAR ABOUTAVERAGE VALUE ??
PRESENTATION - PEJalPS A COLOR GRADIENT IN THE LINE IS MORE EFFECTIVETHAN A BAD. MAYBE A SIMPLE GREEN TO YELLOW TO RED TOSIGNAL STRUCTURAL HEALTH OF THE CREW PROFILE
PRESENTATION - DISPLAY EXPERIENCE AS A HISTOGRAM -- PROJECTIONS BUILD
HISTOGRAMS THAT CAN BE COMPARED ACROSS YEARS
PRESENTATION - IF NECESSAY, INCREASE THE VERTICAL RESOLUTION ON THEHISTOGRAMS BY STARTING THE SCALE AT SOME VLUE OTHER THAN0. FOR THE FEW BARS THAT ARE SHORTER THAN THE BASEVALUE, WRITE THE M RGINAL POlNT ABOVE THE BAR.
126
PRESENTATION --A POSSIBLE REFINEMENT OF THE EXPERIENCE LEVELPRESENTATION IS A GO-NO G0 INDICATOR FOR THE DISTRIBUTIONPARAMETERS (KIRTOSIS AND SKEWNESS IN PARTICULA) THATINDICATES WHETHER THE CREW PROFILE IN A GIVEN YEAR ISGOOD. FOR THE CURRENT WORK, A GRZPHICAL LOOK AT T-EDISTRIBUTION IS AL THAT WILL BE PRESENTED.
PRESENTATION -- SLPPLYING E ES FOR THE MODELING RESULTS TO BEMEASURE AGAINST CAN BE TRICKY. FOR BODIES, IT IS EASYSINCE THE REFERENCE IS JUST THE AUTHORIZED MANNING LEVEL.FOR EXPERIENCE, TIE REFRNCE WILL HAVE TO BE BUILT SOMEHOW -- EXPERT OPINION (MADA) - USE AUTHORIZED RANK
STRUCTURE, ASSUMING THAT RAN STRUCTURE IS HOW MPIC TRIESTO M AGE EXPERIENCE ??
PRESENTATION - ALLOW THE USER TO EXAMINE HIS "WHAT-IFS" ON TIE SAMESCREEN. USE MULTIPLE WINDOWS FOR THE USER TO LOADDIFFERENT PARETER VALUES INTO, AND DRAW THE LINE FOREACH WINDOW ON A COMMON GRPH. COLOR CODE LINES ON THEGRAH WITH THE WINDOW IT CORRESPONDS TO. THIS ALLOWS THEUSER TO KEEP CLER WHAT HE HAS CONSIDERED AND HOW THERESULTS COMPARE.
DESIGN -- THE SCHEDULING PROCESS REQUIRES MORE THAN A SIMPLECONSIDERATION OF MANNING LEVEL IN TERMS OF BODIES.EXPERIENCE HAS TO BE OBSERVABLE AT THE SAME TIME.
DESIGN - PRADIGM - MATCH LINES FOR DECISION - PRE-AFITTHOUGHT ON HOW TO BEST MAKE THE DECISION BEING SUPPORTED.KEEP IT SIMPLE!"! THIS ENTRY ALSO PROPERLY FALLS UNDEROPERATION.
DESIGN -- PARADIGM IS CRITICAL
DESIGN -- DESIGN TOOLS APPEAR TO BE IN SHORT (NON-EXISTENT ?)SUPPLY. WHY? THEY ARE NEEDED, SO WHY AREN'T THEYAVA ILABLE.
DESIGN -- STORYBOdING IN A WORD PROCESSOR IS A HORRIBLEEXPERIENCE AGIN, WHY AEN'T THERE GOOD TOOLS.STORYBOARDING TOOLS THAT BUILD INTERACTIVE STORYB(ADSSHOLLD BE AVAILABLE !
DESIGN -- DSS IS DEAD IF SOMEBODY DOESN'T BUILD AN EFFECTIVE DSSGENERATOR. WRITING CODE FROM SCRATCH TAd<ES TO LONG.WITHOUT A GENERATOR, AN OUTSTANDING SET OF TOOLS WILLHAVE TO BE BUILT IN ADVANCE OF ATTEMPTING TO DESIGN ADSS.
DESIGN - RERDLESS OF HOW HARD IT IS TO CODE, SPREADSI-EETENVIRONENTS ARE A STOP GP AT BEST. MACOS AREINSUFFICIENT, AM BLOCK OPERATIONS ON DATABASE ELEMENTS
127
ARE DIFFICLT SINCE A) FIELD NAMES MUST BE KNOW, B)OPERATIONS E GENERALLY DIFFICULT TO DEFINE UNLESS THEYARE HADWIRED INTO THE PREADSHEET.
DESIGN -- QUATTRO PRO MAY BE CLOSE TO A FULLY AUTOMATEDSTORYBOARDING TOOL
DESIGN -- DON'T FORGET THAT MANNIING IS CYCLIC -- PAY SPECIALATTENTION TO PERIODS OF HIGH ACQUISITION DEMAD; DAMPFLUCTUATIONS -- IF 20% OF THE CREW FORCE NEEDS TO BEREPLACED THIS YE, THE SAE WILL BE TRUE IN ROUGHLY FIVEYEARS WHEN AL OF THE NEW BODIES PCS. THIS "BOOM ORBUST" CYCLE MUST BE OUGHT LNDER CONTROL. CURRNTLY,THE CYCLIC NATURE OF MANNING IS IGNORED BECAUSE OF THESHORT TERM ORIENATION OF THE PLANNING PROCESS.
DESIGN -- DISPLAY - 1 YEAR + 5 SCHEDULING GRPH, NOT 10 YEARS ASORIGINALLY CONCEIVED -- THE CHAIN OF ASSUMPTIONS IS GOINGTO BE EXTENDED TO LONG FOR RESULTS TO HA-VE ANY MEANING INTHE 10 YEAR TIME FRAME.
CONTROL -- MAKE SELECTION OF POSITION IN SYSTEM RL3PONSIVE TO FIRSTLETTER STRING FULL BLOWN COM)ND LANGUAGE TO COMPLEX FORUS TO ACHIEVE
CONTROL - AUTO ENTRY IN HOOK OF SCREEN FROM WHICH ENTRY IS MADEAUTO ENTRY IN HOOK OF PROGRESSION TO SCREENDATE/TIME/OPERATOR STAMP IN HOOK ENTRIES
CONTROL -- ASSUMPTIONS MUST BE VISIBLE IF USER IS TO SUCCESSFULLYNAVIGATE THROUGH A DECISION PROCESS -- ADD A ASSUMPTIONSBUTTON AS IN EL SHERIF
IMPLEaENTATION-WHAT IS NEEDED IS AN ENVIRONMENT T-AT FOSTERS DEVELOPMENTOF AN OPERATIONAL SYSTEM, AND THEN GENERATES THE SPECIFICSTAND ALONE CODE TO EXECUTE AL OF THE FUNCTIONS THATHAVE BEEN DEVELOPED.UNTIL THEN, BUILD PROTOTYPES IN SOME ENVIRONMENT THATPROVIDES ALL OF THE NECESSARY FUNCTIONS, AND THENREPRORA, EVEN THOUGH THIS EFFECTIVELY LOOSES THEDEVELOPER A MAJOR PART OF THE EFFORT DEVOTED TOPROTOTYPING.
IMPLEMENTATION-REGADLESS OF W-ERE A DSS IS DESIGNED, THE IMPLEMENTEDVERSION NEEDS TO BE IN STAN) LONE CODE FOR PERFORMANCEAD FLEXIBILITY -- WHERE IS MY D GENERATOR?
VALIDATION - THE PROCESS OF REPEATEDLY OPERATING THE MODELS WITHVARIED PARAETERS WILL PROVIDE INSTRUCTION IN THE SYSTEMSUSE AND VALIDATION FOR THE MODELS. OBSERVABLECONSEQUENCES OF PARAMETER CHANGES CAN BE CHECKED AGINST
128
THE MANR'S INTUITIONS, WHICH WILL MA TURE AS THE SYSTEMIS EXERCISED.
EVOLUTION IT MUST BE MADE CLEAR TO THE USERS THAT THE SYSTEM WILLONLY BE AS GOOD AS T-EY IAK<E IT. USERS MUST KNOW THATAL USER I NPUTS E I MPORTAnr, AN'JD MORE, THA T AT AMINIMUM, USER INPUT IS NECESSAY TO FINE TUNE THE MODELSUSED TO SUPPORT THEIR DECISIONS.
EVOLUTION - AUTO ENTRY IN HOOK OF SCREEN FROM WHICH ENTRY IS MADEAUTO ENTRY IN HOOK OF PROGRESSION TO SCREENDATE/TIME/OPERATOR STPP IN HOOK ENTRIES
EVOLUTION - ASSUMPTIONS MUST BE VISIBLE FOR EVOLUTION TO OCCUR ADDAN ASSUMPTIONS BUTTON AS IN EL SHERIF
DATA - POSSIBLE FACTORS - JUST A DUMP
FORCE STRUCTURE- OVERHEAD/STAFF-SOF-DETCO-ETC
( PPA/SCHEDLLED CHANGES)( MOD/SCHEDILED UPGRADES)( CREW COMPOSITION BY MOD)
MAtNING LEVEL - BODIES VS AUTHORIZATIONS- RANK<
- A-SC- SCHEDULING ACQUISITIONS
-- WHO'S LEAVING(CODE 55, TOS, AVG TOL.R LENGTH, LOSS OR
SlPf~tI)
-- NEW REQUIREMENTS(NEW PFT, COLLATERAL ACCESSION, RECPTURE)(NEW REQUIREMENTS FROM FORCE STRUCTURE /BUILD-iP ISSUES)
- PROFILES (PROJECTIONS/CONSE(IENCES)- TABLES AND LINE
EXPERIENCE LEVELS - FULL UP CREWS- RA l<
- A-SC- LOSSES- MODELS - RECOMMENDATIONS OR RESPONSE- MODEL (ALL ELEMENTS DATABASE INPUTS TO MODEL/
ALSO POSSIBLE PAETERS)- TRAINING- EDUCATION- YEARS AWACS- # MISSIONS- # HOURS FLOWN- # INTERCEPTS- YEARS COLLATERAL EXPERIENCE
129
- EROR HISTORIES
-- TAPE-LOS
MAINT INPUT (E.G., CND)
- PROFILES(PROJECTIONS)/CONSEGELCES- TABLES AND LINE
SYSTEM CAPABILITIES/PERSNNEL IMPACT-MODEL
(EFFECTIVE INTERCEPTS) (REGRESSIONS)(CONTINLOUS HARDWARE OPS)(C3 BRE KDOWNS)
- PROFILES(PROJECTIONS)/CONSEJ.ENCES- TABLES Pt-M LINE REPRESENTAT IONS
DATA - DON'T FORGET DOS ETC AS INPUTS TO LOSS CALCULATIONS IAKESURE ALL DRIVERS FOR GAINS AM) LOSSES ARECONSIDERED.
130
Appendix D
Data Requirements
This appendix lists the types of personnel data that can be useful
for each of the major models needed to make the DSS function. The list
is not comprehensive; only extensive "data snooping" at the user site
can ensure all useful data is identified.
Data elements suggested for examination are listed with the
location of the data. The 28 AD database mentioned is not currently in
place, although it has been fully developed. Until such time as this
database is fielded, the needed data can be found in the squadron ORCs.
As a general point, where data elements pertaining to time are
concerned, the databases will often contain originating dates rather
than cumulative time. Arriving at the necessary data will require
subtracting the originating date from the current date.
131
DATA BEARING ON MANNING LEVEl_
SOURCE
2SADA-FORMS CBPO DB
FORCE STRUCTURE CHANE S1. E-3 MODIFICATIONS CRRNTLY, NOTE OF THIS2. ORGANIZATIONAL CHANGES INFORMATION IS AUTOMATED. IT3. PAA CHANGES DOES EXIST AT THE DIVISION4. CREWS/PAA CA-ICNGES AT 28 AD/XO.
LOSS FACTORS5. AVERAGE TIME IN AWACS X X6. AVERAGE TIME AT TINKER X X7. CODE 55 LENGTH X8. RAN X X9. PLANNED RETIREMENT DATES X X
10. PLPNNIED SEPARATIONS DATES X X11. SCHEDULED PCS DATES X X12. LOSS RATES
BY TIME IN SERVICE MPC CURENTLY XBY RAW .. .. XBY CREW AFSC .. .. XBY # OF FLYING HOLRS X X
13. NUMBER OF REMOTES X X
TRAINING LIMITATIONS12. CAPACITY 552 TTS
PERSONNEL DATA POSSIBLY BEARING ON EXPERIENCE
SOURCE28AD
A-FORMS CEPO DBFLYING HISTORY
1. TOTAL YEARS X2. E-3 YEARS X3. TOTAL HOURS X4. E-3 HOURS X
SERVICE PIND ASSIGNMtENT SPECIFICS5. YEPRS AWACS
CONUS X XNATO, PACPF, ICELAND X X
6. NLI13ER SHORT TOURS X X7. NMBER LONG TOLRS X X8. YEARS CURRENT CREW POSITION X X9. YEARS OTHER AWACS POSITIONS X X
10. MAX STAF LEVEL X X
132
SQ (1); WING (2);DIVISION (3); HQS (4)
11. YEARS IN STAFF POSITIONS X X12. APPLICABLE NON-AWACS EXPERIENCE
YEAS GROLND TACCs (WD, SD, AST) X XYEARS C3 (FOR MCC, SD) X XABCCC X X XRIVIT JOINT X X XCOMPASS CALL X X XETC.
13. YEARS IN SECONDAY RELATED AFSC X X X(RADAR, CUIUTR)
14. GRADE X X15. TIME IN SERVICE X X16. TIME IN GRADE X X
MISSION HISTORY17. # SORTIES X X18. # MISSIONS X X19. # EXERCISE MISSIONS X X20. # TEST MISSIONS/EVAL MISSIONS X X21. # SIMULATOR SESSIONS X X22. # OF CONTROLS/INTERCEPTS
(SD, WD, ASO, AST) X X
TRAINING AND PERFORMANCE SPECIFICS23. FINAL TRAINING EVALUATION X X24. ATTENDANCE OF SPECIAL SCHOOLS
FWS, TACATC (WD, SD, AST) X XTYPE 2 TRAINING (CDMT, CT, CSO, ART) X X
25. STANEVAI_ RATINGS X X26. YEAS SPECIAL POSITION (AAST) X27. APR/OER X
133
AENDIX E
Model Requiremts
A description of the requirements for the three major models are
included in this appendix. The descriptions for the manning level
models and the capabilities models are very general. The modeling
methodology for the kernel is discussed more fully.
134
Mannina Level Models
The manning level models drive the scheduling displays, and show
the projected manning status of the crew positions. They must 1)
project losses from the crew position, and 2) roll the projected
acquisitions described by the manager during his "what-if" sessions back
into the crew position to arrive at a manning level projection across a
five year PFT planning period.
There are multiple available strategies available for projecting
losses (and this is obviously the hard part of the model). For example,
loss rates developed from historical data can be used. This can come
from AWACS data, or, if that data is not currently available, MPC data
may be available by general AFSC (to be used until AWACS specific
historical data is collected). Another possibility is to use average
time in AWACS estimate identify individuals who are likely to leave.
The second method is probably more complex to implement, but should give
truer results since it is know that loss rates are not constant across
the years.
However losses are projected, care must be taken to make individual
projections for distinctly different groups, and then aggregate these
losses for an annual loss picture. As an example, first time AWACers
accrue a service commitment (in the form of a code 55) that locks them
into the AWACS system for some period of time. People returning to
AWACS from other assignments do not have this commitment. Therefore,
there may be a significant difference in the average time in AWACS for
these two groups before they become lost to the AWACS manning pool.
Care must also be taken to project losses by personnel groupings
135
that are distinctly different in experience, so that the experience
level models can make experience level projections. For example, if a
major factor in experience level is rank, projected losses could be
broken out by rank. Identifying losses by experience categories will
allow the experience models to assess losses in experience do to
departures from AWACS.
Exerience Level Models
The experience level models need to accomplish three distinct
functions. First, they need to assess experience for each of the
individuals in a crew position. This portion forms the kernel of the
DSS presented here, and is discussed more extensively below.
The second function the model must perform is to manage chai-qes in
factor levels while projections are being made. (How these changes are
managed is a key area of system assumptions that must be documented).
As an example, if flying hours are significant in assessing experience,
a distribution scheme for allocating flying hours beyond bare
qualifications necessities is required. Obviously, the easy approach is
to average the total flying hours available over the entire crew force.
The question is, is this what actually happens, and if not, how
different are the experience profiles developed using averages from
those using a more difficult allocation scheme (over a five year period,
this may be a moot point).
Finally, the model must make projections for experience levels
across a five year period, using the factors discussed in the above two
paragraphs, and the information of losses by category that the manning
level model must provide.
136
With this general overview of the experience level model
requirements, the rest of the discussion relative to experience modeling
presented here pertains to the experience scoring method developed for
the kernel DSS.
Experience ScorinV. As discussed in chapters two and four, the
approach adopted in the kernel DSS for developing experience scores was
to use multiple regression with a binary response. Fundamentally, the
approach used is to look at personnel data for factors that describe the
difference between an identified set of suspected experts and the rest
of the crew force, and to develop a probability measure that any given
individual is a member of the expert group. It must be kept in mind
here that there is no intent to ascribe cause and effect in this model.
Using the model to allocate more flying hours (if flying hours are a
factor in describing experience) to individuals with low experience
scores may not be proper. The model may indicate that flying hours are
useful for describing who experts are; it does not say that flying hours
cause expertise.
As in all regression, the factors to be used in this particular
formulation of a multiple regression are those that test significant in
predicting the responses. However, Deming suggests that traditional
statistics are in many ways poorly suited to describing non-static
systems (i.e., systems that are not in a steady state). For non-static
systems, he suggests that confidence intervals should not be an over
riding consideration. Rather, emphasis should be placed on detecting
gross patterns. For this DSS, with a goal of changing the system it
examines, the system is known to be non-static. Therefore, significance
137
levels for determining factors impacting experience should be relaxed.
The goal here is not an exact answer, but rather a better answer.
552 AWACW/DO supplied a partial database on MCCs (with only a few
personnel factors that could possibly bear on experience). Most of the
factors available in the database were related to time in some manner;
many of the factors discussed in the data section that should help to
build a good scoring model were not available.
Using this data, a preliminary examination was conducted to test
the usefulness of the scoring approach. As discussed in chapter 4,
there are some problems with the technique used here that require
remedial measures. One of the problems is that variance is non-uniform.
This was addressed by using a weighted regression.
An unweighted regression was first conducted; the resulting
predicted response was used to develop the appropriate weignts
w, = 1 / r Y,(1-Y') ]
for the weighted regression, where w, is the weight for the it h
observation and Y, is the predicted response for the same observation.
Corrections for responses out of the allowed range were not
necessary for the data tested. If out of range responses had been
observed, appropriate transformations are available to convert the
formulation to a probit or logit regression. If this were not desired,
other remedies could be available, depending upon the factors, and their
real world implications for experience. As discussed previously,
capping the maximum and minimum values certain variables can assume may
make sense.
Results for the regression of the MCC data are shown below. For
138
the examination conducted, a p value of .4 was accepted as significant.
While th_ weiohted regression only accounts for 307. of the variance
observed, this approach was still viewed as feasible (it is actually
surprising to the author that 30% of the variance could be explained
with the data at hand. This may bode well for the confidences that can
be applied with better data).
FACTOR LABELS
D = YEARS PRIOR SERVICE DD = D2 DI = Dxl interactionE = TIME IN SERVICE EE = E EI = ExI interactionF = TIME IN GRADE FF = F2 IH = HxI interactionH = YEARS ON SHORT TOURSI = YEARS ON LONG TOURS
This is the regression that was conducted to establish weights.
Factor I was retained (with a p value of .61) because of the interaction
that were significant. R2 is .21, and the model is significant at .15.
Table 2 - Unweighted Regression
PREDICTORVARIABLES COEFFICIENT STD ERROR STUDENT'S T P
CONSTANT -4.6440 1.5634 -2.97 0. 004?E 6.1335E-01 1.9060E-01 3.22 0.0020F -2.4483E-01 1.2412E-01 -1.97 0.0528I 6.8528E-02 1.3411E-01 0.51 0.6111DD -2.7636E-02 1.0818E-02 -2.55 0.0130EE -1.3576E-02 4.6283E-03 -2.93 0.0046FF 1.6030E-02 9.5699E-03 1.68 0.0987DE 1.8232E-02 1.3590E-02 1.34 0.1844DI 2.2211E-02 1.2307E-02 1.80 0.0757D -2.9569E-01 2.5881E-01 -1.14 0.2574El -1.7528E-02 1.054GE-02 -1.66 0. 1013H -3. 7225E-01 2. 1815E-01 -1.71 0.0927HI 1.3263E-01 6.3068E-02 2.10 0.0393
OVERALL F 1.488 P VALUE 0.1514R SQU EED 0.2155
139
As can be seen here, there is not a large separation in the
lexperience scores of the two populations. The lower grouping is for
the unknown portion of the crew, while the upper grouping is for the
instructors and StanEval personnel. While the upper group has more
occurrences (as a percent of the upper group) at the high end of the X
axis, the range extend all the way to 0 (approximately).
SI3.0 +
++* ++
2++ 4+2 +1.0+ 2 + +
4 2+2+
+2445S ++43 ++
-1.0 + 254++* +3+ +
+ +
-3.0 +I, I I I'
0.0 0.3 0.6 0.9 1.2P1 78 CASES PLOTTED
Figure 27 - Unweighted Regression Predictions
The second regression, using the weights developed in the first,
has an elevated R2 (= .30) and a p value of .013.
140
Table 3 - Weighted Regression
PREDICTORVARIABLES COEFFICIENT STD ERROR STUDENT'S T P
CONSTANT -5.2769 1.3196 -4.00 0.0002E 6.7984E-01 1.6138E-01 4.21 0.0001F -2.2977E-01 1.0795E-01 -2.13 0.0371I 1. 17=-01 1.1150E-01 1.06 0.2946DD -2.4499E-02 1.0397E-02 -2.36 0.0215EE -1.5422E-02 3.9321E-03 -3.92 0.0002FF 1.5526E-02 8.8081E-03 1.76 0.0827DE 2. 4878E-02 1. 334EE-02 1.86 0.0669DI 1.2531E-02 1.1232E-02 1.12 0.2687D -4.1 32-01 2.5779E-01 -1.60 0.1138EI -I.8017E-02 8.6067E-03 -2.09 0.0402H -3.7255E-01 2. 1781E-01 -1.71 0.0920HI 1.1347E-01 5.4351E-02 2.09 0.0407
OVERALL F 2.360 P VALUE 0.0130R SQUARED 0.3053
As can be seen in Figure 28, the result of weighting the regression
was to better separated the two groups. The center of mass of the upper
group has shifted higher on the X axis, while that of the lower group
has moved down the axis.
Capabilities Models
The capabilities model falls into the realm of MCDM, since it deals
with balancing multiple objectives. This model should only address
capabilities as impacted by personnel issues.
AWACS capabilities as a function of personnel is driven by multiple
personnel issues, such as: 1) are there enough people to man a crew
position; 2) are there enough experienced people in a crew position
(average issues); 3) are there enough experts to handle exceptional
cases and to pass on knowledge (distribution issues); 4) are there
particular crew positions that are critical in impacting capabilities,
141
S24.0+ +
+
2.0+ ++
++2++2
23 + +24- + 3+ +
0.0 +9++3 +2+
* + +++34+244+2
* + 22 +2I+ +
-2.0 +
0.0 0.3 0.6 0.9 1.2P2 78 CASES PLOTTED
Figure 28 - Weighted Regression Predictions
and 5) if so, do they need special attention to fix their deficiencies
relative to the vest of the crew positions (allocation of scarce
resources).
The objective of this model is not to provide an absolute answer on
capability. Rather, the goal is to specify some measure that allows
trends to be assessed. As such, the scale for "CAPABILITY" is not
critical, but is sure to be a management stumbling block if some
reference is not provided.
Approaches for providing reference include:
1) defining the 100 level to be 100% manning;
2) with a specified experience distribution (use the same
distribution against which experience profiles are measured by
acquisition managers), and;
3) with a "table" of trade offs between changes in these values
142
clearly stated. Eliciting these trade offs (utilities and weights) is
critical, and may well be the most difficult portion in developing this
model. Every one of the assumptions made in estimating trade offs
between these factors must be available to the decision makers that use
the model.
The discussion above presents general guidelines for model
development. Developing them will be a difficult and time consuming
task.
143
APPENDVIX F
fC1M Examinat ion
This appendix discusses the examination made of MCDM techniques for
the purpose of assessing experience levels in the different crew
positions. While these methods were not adopted, they are applicable to
other portions of the overall DSS. For example, MADA methods will
probably be necessary if a reference (desired) experience profile is to
be developed. Also, MCO techniques (not discussed in this appendix,
since the area of concern was of a descriptive rather than prescriptive
nature) will surely be necessary for developing the capabilities
assessment models.
144
Table of Contents
Page
Larger Systems Issues .............................................. 146
The Kernel Problem ................................................. 148
An Assessment of a Multi-Criteria DecisionAnalysis (MDA) Approach to the Kernel ........................ 149Formulation of an "Experience Score .. ......................... 150Eliciting Factors Bearing on Experience ....................... 151
Part one of the Survey ................................... 151Part Two of the Survey ................................... 152Selection of Factors ..................................... 152
Eliciting Factor Weights and Utilities ........................ 153
Survey ............................................................. 154
Survey Results for Part I(a) .................................. 155Survey Results for Part I(b) .................................. 156Reduced Response Set .......................................... 156Survey Clarification .......................................... 157Survey Results for Part II .................................... 156Independence Issues ........................................... 158Altered Factor Selection ...................................... 159
SmartEdge Session .................................................. 161
Conclusions ........................................................ 162
Tab A - The Survey ................................................ 165
Tab B -- Example Transitivity Examination .......................... 172
Tab C - SmartEdge Output .......................................... 174
145
Larger System Issues
The overall problem addressed in this research is the development
of an effective decision aide to support AWACS air crew managers in
determining personnel acquisition needs. (The manager's goal is to
maintain AWACS with an crew force that can effectively execute the AWACS
mission).
The acquisition requirements of AWACS are driven by two issues: 1)
the number of people required to man a crew position, and; 2) the
experience these people must possess for an AWACS mission to be
effective. The first issue is concerned with personnel flow through the
crew system. Managers attempt to keep crew positions fully manned.
Quantitative shortages due to mis-timing of acquisitions (to fill
manning vacancies) need to be minimized. The second issue is concerned
with the distribution of experience in the crew force. All members of a
crew position need not be expert, but some minimum set must be. There
is also some limit to the number of novices the position can tolerate
before the AWACS mission becomes ineffective due to inexperience in the
crew position.
The overall measure of a successful manning acquisition program is
how capable AWACS is of performing its mission (limited to effects based
on personnel considerations). The AWACS capabilities issue involves
some balance among the above mentioned factors (absolute numbers and
experience). If the air crew managers are to be successful in managing
acquisition, they must be able to balance experience and number issues
to arrive at the best acquisition strategy.
In the overall problem, "optimizing" AWACS capabilities is a Mi1ulti-
146
Criteria Optimization (MCO) problem, possibly addressed best as a
compromise program. There are two criteria for each crew position in
this problem. These criteria are*
1) The absolute manning level. The "ideal" and "goal" values for
this criterion are the same. It is the manning authorization level for
the crew position, as specified by higher headquarters.
2) The distribution of the experience in the crew position. The
mean experience level of the personnel in the crew position is a
possible measure of composite experience in the crew force. It can
provide a basic measure of the ability of the current crew force to
accomplish the AWACS mission effectively. The "goal" for the mean
experience level will need to be extracted from the AWACS managers, and
can be referenced against a constructed experience scale ranging from 0
to 1, where a .9 or greater is the "experience score" of the "ideal
expert". (The distribution of experience is also critical, particularly
with respect to a heavy representation in the crew force of novices, or
a light representation of "expert" experts. However, all distribution
issues will not be considered initially. Refinements of the models in
the DSS will be necessary to account for more than central tendency.
Some other moment will undoubtedly need to be considered to ensure
unacceptable experience distributions in the crew force do not occur.
The introduction of a "distribution" criteria in no way changes the
basic approach needed; it only causes the dimensionality of the MOD
problem to increase).
For managers to structure personnel acquisitions in a manner that
"optimally" balances (under some MCO methodology) the two criteria just
147
discussed, models are required that show the impact of a manning
acquisition decision on each of the criteria. A first step is
identifying the factors that bear on each of the criteria, i.e.,
identifying the attributes in the alternative space of each.
The Kernel Problem
The kernel of the overall problem (discussed above) concerns the
"experience" issue. For most of the crew positions, there are no good
measures of the crew member's competence in his job performance. The
goal in the kernel development is to find and combine some set of
factors in an "experience score" that will provide a surrogate measure
of the individual crew member's effectiveness at his job. (The
individuals to be "scored" are the AWACS air crew members within each
crew position. Ratings are only relevant within the crew position, not
between positions. The score of interest is a rough estimate of each
individual's ability to competently handle all of the myriad of
contingencies that face crew members in his position in an AWACS
mission). To establish "experience scores", three tasks must be
accomplished. These tasks are to:
1) establish a set of factors that are reasonable indicators of
performance. Examples of possible candidate factors are: 1) rank, 2)
years of service, 3) Standardization Evaluation (StanEval) ratings, 4)
number of flying hours, 5) number of missions flown, 6) number of
exercise missions flown, etc.;
2) determine appropriate weights among the factors (if the experts
feel all are not of the same weight), and;
3) determine utility functions for each factor that describe the
148
relationship between the states of the factor.
The "experience score" that is generated for each crew member will
become part of his database attributes, so that the experience level of
the c-ew position as a whole can be judged by simple graphic
representations of the database contents. After a valid scoring method
is arrived at, the experience profile of the crew force can be presented
to the manager to enhance his acquisition recommendations. He will be
able to identify what impact policies controlling acquisitions and
losses have on the experience profile of the AWACS crew position. He
will be able to assess issues such as:
1) whether he needs to take special actions to recapture experience
(by identifying past AWACS members with the requisite experience who are
at large in the A- manning pool, and whom MPC can reassign to AWACS to
correct an imbalance), or whether he can "grow" that experience;
2) whether he can live with a manning acquisition that is composed
entirely of novices, or;
3) whether he must get MPC and TAC to give him some mix of
personnel that balances novice and experienced "acquisitions".
nn Assessment of a Multi-Criteria Decision Analysis (MAA) _Pa~oach tothe Kernel
MADA, in this context, is oriented to the determination of
experience based on expert opinion. Here, the objectives are to elicit
appropriate experience factors from "the experts" and to determine the
weights and utilities the experts hold for the factors. In general, the
possible factors are only indicators of experience; rigorous descriptors
of experience are not available. With no rigorous measures of
149
experience, it is critical that a good set of indicators be identified,
and their relative importance assessed.
The experts to be consulted are managers of the personnel to be
rated. [For the purposes of assessing MADA in this research, the
"experts" consulted were two Weapons Directors (WD) and two former AWACS
analysts who were available locally. The crew position considered was
the WD position]. Initially, the set of possible factors must be
limited to those for which data is collected as an attribute of each
individual. (The specification of a factor from the set of all
conceivable factors is not useful at this point in time. If an expert
feels that some attribute is valuable in determining experience, and
data is not currently collected for the desired attribute, the
identiFication can be u= d to instigate data collection. Better factors
can be incorporated into the model at some later time).
The formulation of "experience score" needed in the kernel and the
MADA methodology for arriving at the required weights and factors are
presented in the following sections.
For=Aat ion for an "'Ecience Score"
The formulation of the "experience score" to be developed for each
crew member is a simple additive model:
E = £ C (w°), I I u,(a,) I
where E is the "score", w,° is the "standardized" weight for the i*'
factor, and u,(a) is the i t "utility value" that is a function of the
i,, attribute level in an individual crew member's attribute vector. In
150
the above formulation, w, and u, are the weights and utilities that must
be extracted from the experts. (Here, we° is w, "standardized" to give
a maximum value of 1 to the score E when u,(a1 ) = 1 for i = 1,...,q;
where q is the nutmber of factors in the model; in other words, the
standardized weight is w,*= E w, I / [ Ew, I ). This model, while only
being applied within a crew position, can be written in a general form
that will score all crew positions. It does not have to be specialized
by crew position. It can be written incorporating all factors
identified for each crew position. All that is required to get
consistent scores in a specific crew position is to maintain unique
weighting and utility function vectors, and set the weight of any factor
that is not applicable to 0, which removes the factor from the scoring
model.
Eliciting Factors Bearing on Experience
To elicit a set of possible factors from the experts, surveys were
conducted. The surveys had two portions, one based on Semantic Scales
and the other a Pair Comparison of factors [Chan].
Part One of the Survey. Part one of the survey contains a set of
possible factors that may be of value and for which data is known to
exist. The survey respondents completed two different evaluations of
the factors.
The first evaluation of the factors was a semantic scaling survey
to elicited perceived importance rankings for the factors. Rankings are
on a five point scale (seven point scales are often used) where the
extremes of the scale are described verbally. To complete the survey,
the respondents were asked to establish rankings independently of all
151
other factors.
The second evaluation of the factors was based on pair-wise
comparisons of the factors. All factors are paired, and the respondents
required to choose the preferred factor from the pair.
The two portions of this section of the survey have two purposes.
These are: 1) to assess the importance of the various factors (both
portions), and; 2) to check for consistency in the rankings. This is
accomplished by comparing the two portions of the survey, and by
assessing the transitivity of the factor rankings in the pair-wise
comparison.
In the scheme of the overall survey, the first portion serves two
further purposes. First, it may produce a good set of factors on its
own, and second, it will get the expert thinking on the experience
factor selection issue.
Part Two of the Survey. The second portion of the survey is
basically unstructured. It consists of an open ended request for
factors the experts deem important, such that the factors suggested are
at least as valuable to the respondent as the most important factor
identified in part one. This will mainly be used as a means of
identifying possible new factors that have not been considered in the
first section of the survey. However, under certain conditions, these
factors may be included in the model to be developed.
Selection of Factors. The selection of factors will be based on
two criteria. For factors evaluated in the first portion of the survey,
any factor that is in the top two categories of the majority of the
responses will be included. This selection criterion is used for the
152
following reasons: 1) the response sample will be to small for
statistical methods to be useful, and; 2) if a majority of the
respondents agree that the factor is of above average importance, it
should be considered until a larger response can be achieved.
For the factors identified in the second portion of the survey, any
factor that appears on two or more surveys will be included. Here, the
selection criterion is more liberal than in the first portion of the
survey. This is done because if some factor appears on more than one
survey, solely from the respondents assessment of what is important and
independent of survey pronpts, the factor is probably significant.
(While an explicit pair-wise comparison is not conducted, guidance will
be given that steers the respondent toward a mental pair-wise assessment
of the relative strengths of the factors. This being so, a duplicate
identification of the same factor will be considered sufficient
justification for selecting the factor. Given more time and resources,
this initial effort would be used as a pilot survey, and further
surveying, incorporating the results of the pilot and with a larger
sample, would be conducted. For the purposes of assessing MADA as an
approach, this was not done).
Elicitin Factor Weiahts and Utilities
Appropriate relative weights for the factors, and the utility
function that describes the relationship of the states within each
factor, can be determined through a series of reference lotteries.
(This was discussed in the methodology portion of the main text, but is
discussed here again).
In a "basic" reference gamble procedure, sequences of reference
153
gambles are conducted where, prior to the gamble, payoffs for the
reference gamble are established. Then, iteratively, probabilities (p)
for winning the gamble are assigned, and the certainty equivalent (CL)
for the gamble evaluated. Plotting p verses CE yields the utility curve
for the factor being examined [Holloway : 422). (Holloway also
describes a variation on the basic reference gamble that fixes a
sequence of certain values (CV and then determines the probability p at
which the decision maker is indifferent to the lottery. Plotting p
against CV again yields the utility curve).
The reference gamble to be used is a 50-50 gamble. This is another
modification of the basic reference gamble. (Holloway suggests that
this is often a better gamble, since people often thing in terms of go-
no go choices [428]). Here, CEs are first determined for p = 0, .5, and
1. For the rest of the procedure, p is fixed at .5 and the CEs for each
gamble elicited. The probability for the certain value can be
determined from the gamble conducted. The 50-50 gamble was used to
determine both the factor weights and the utility functions that
describe the factors. This was done in an automated question and answer
session. A sequence of structured questions were asked of the experts,
and appropriate lotteries conducted. The results of the sessions were a
set of weights and utility functions to be used in the additive
"Experience Score" described first in the "Formulation" section of this
dppend ix.
The software package "SmartEdge" (ver 3.0) was used to accomplish
the structured question and answer sessions. Prior to the experts being
asked to participate in the SmartEdge session, initial setup work was
154
completed on a "Decision File" so that nothing was required of the
experts but that they answer a the programs sequence of questions (i.e.,
all entries required up to the "DECISION MAKER DATA" were entered).
In a SmartEdge session, the expert being interviewed answered
questions in the "DECISION MA<ER DATA" segments of the software package.
The results of the interview session defined the expert's utility
function for each attribute listed. (A similar interview develops the
appropriate weights for the factors).
The survey included in at the end of Lhis appendix (Tab A) is aimed
at eliciting a set of factors that indicate experience in WDs. It was
completed by four "experts". The results of the survey are presented
next.
Survey Results for Part I(a). The results of the survey for Part
(A) shown in Table D-1.
Table 4 - Survey Results, Part I(a)
IMPORTWNCE
FACTOR (RPEN >) 1 2 3 4 (TOTAL)
A) StanEval ratings 4 5 5 5 (19)B) E-3 flying hours 2 3 4 4 (13)C) Years in position 4 3 4 1 (12)D) Instructor qualified 4 5 5 5 (19)E) # of E-3 missions 3 3 4 3 (13)F) # of exercise missions 4 4 4 4 (16)G) # of Simulator sessions 2 2 3 3 (10)H) Training evaluations ... 3 4 3 3 (13)
I) ... ground controller ... 4 4 2 3 (13)
Based on the selection criteria stated previously in this appendix,
155
factors A, D and F appear to be a good set of indicators for WD
experience. As stated previously, with a larger sample, a statistical
evaluation of the results would be valuable. From the above table, it
looks like factors E , H and I may also be possible experience
indicators. A larger sample and a statistical measure would help
determine if they are indeed useful indicators.
Survey Results for Part I(b). Evaluation of the results of Part
I(b) resulted in a change in the factor selection criteria, and in the
factors selected. Examination of the paired comparisons showed that
three of the respondents surveyed considered the set of factors to be
fully transitive. Further, the preference structure established by the
comparisons were consistent with each respondents answers in Part I(a)
of the survey Ewithin the resolution of the scale of Part I(a)]. (The
transitivity of the factors was established on a spreadsheet. One
compound sort of the factors ranked the respondents preference for the
factors. The sorted responses for one survey are included in Tab B).
The fourth respondent's survey results demonstrated
intransitivities and also indicated inconsistencies with his responses
in Part I(a) of the survey. This respondent was not available for
further questioning (member was TDY) that might have revealed why the
intransitivities existed and why there was not good correlation pattern
between his Part I(a) and Part I(b) responses. His responses were
therefore removed from consideration in the study.
Reduced Respoise Set. While three of the respondents were
internally consistent n their rankings, their rankings in the pair-wise
comparison did not agree fully with each other. Two agreed exactly in
156
the ranking of the top four factors. The third included only one of
these factors (A) in his first four choices. The factor preferences
established by the pair-wise comparison are:
1) A > I >D >C >F >E >H >B >G2) D > A > F > E > C > I > B > H > G3) D >A >F >E >H > I >C > B >G
Survey Clarification. Further questioning of the respondents
showed that there was a fundamentally different perception about the
scenario under which they were operating in Part I(b) of the survey.
The two respondents who were most in agreement operated from the
assumption that there was no guarantee that any of the crew members they
were to choose from would have ground control experience, and would
therefore not attempt to base a decision on this factor. Further, they
both indicated that this assumption was based on their knowledge that
there are not many people in the AWACS community with ground control
experience. (When asked if this wasn't also true for basing a decision
on instructor qualification, they indicated that they were indeed
operating from the same assumption. However, the indicated that 1)
there are many more people in AWACS who have been instructors at some
time, so they felt there was a much better chance that pool of crew
members to be drawn from would include members with instructor
backgrounds, and 2) that instructor qualification was such a good
discriminator of expertise, that they were willing to "gamble"
on finding someone with this background).
The respondent who was not in agreement with the other two, on
considering these issues, indicated that factor I would move down in his
rankings if it were unknown if any crew members in the pool to be drawn
157
from had ever been ground controllers. He had based his rankings on the
assumpt ion that the pool to be drawn from contained members in every
factor category.
Survey Results of Part II. Part II of the survey produced two more
possible factors. One factor appeared on three surveys and one appeared
on two. These new factors were:
1) Has the WD ever been a StanEval flight performance evaluator.
If so, this is a highly reliable indicator of experience, based on
comments included on the surveys. This factor appeared on three
surveys.
2) Has the member attended the Fighter Weapons School (FWS) or the
Tactical Air Traffic Control (TACATC) training courses. If so, this is
also a reliable indicator of experience. This are very limited courses
in advanced control tactics that only a chosen few get into. Special
schools appeared on two surveys.
Independence Issues. During the survey clarification discussed
above, another issue that came out had to do with the independence of
the factors. Consideration of the factor set led to the following
conclusions:
1) number of missions flown, number of exercise missions flown,
and number of simulator sessions could be considered to be largely
independent, but all three were related to number of years in the crew
posit ion;
2) factors like instructor qualification, special school
attendance and the respondent suggested standardization evaluator
qualification factor were not independent of most of the other factors,
158
and;
3) any crew members who were instructor or standardization
qualified or who had attended any of the special schools could be
considered experts because of the screening they had been through before
they were selected for these positions. Members selected for these
functions were highly qualified before their selection; the experience
gained by filling the positions or going to schools only reinforced
their expertise.
Fundamentally, member's who were ever instructors or
standardization evaluators, or who had been to special schools should be
considered "experts", and removed to a separate "pot". These factors
can be used in a Lexicographic screening of the A population7. They are
sure indicators that a mEmber is "experienced". The Instructor
quaiification and Standardization Evaluator qualification factors can be
rolled into one factor that guarantees preference for As with either
qualification, and the Special Schools factor can be used as a second
factor in the lexicographic ordering of "experts" with attendance of
more special schools preferred. (Further distinction among experts may
be possible. The survey participants felt the only factors among those
listed or elicited that might distinguish experience among the experts
would be B, E, and possibly F).
Altered Factor Selection. Given the above discussion, the criteria
for factor selection -was altered to pick the factors that met the
original criteria and that had minimum score of 3. When there were four
respondents, and all three of the four agreed that a factor was of above
average importance, the presence of an "outlier" could be tolerated.
159
With only three respondents left in the sample, the criteria was altered
so that one rating of below average for a factor raises sufficient
question about its worth to remove it from consideration. Further,
having identified that factor C was not independent of some of the other
measures, it was removed from the set of factors to be considered.
Finally, the rating question was divided into two areas: 1) separate
out the guaranteed "experts" using factor D (and experience as a
Standardization Evaluator and attendance of one or more of the premier
controller schools, as discussed in the next section).
The results of Part I(a) of the survey with the inconsistent
respondees data removed are shown in Table D-2, below. The factors that
meet the selection criteria are indicated.
Table 3 - Survey Results, Part I(a), Modified
IMPORrT[E INCLUDED
FAC1R (REE'0(NT >) 1 2 3 (TOTAL) NOW PREVIOUSLY
A) StanEval 4 5 5 (14) * *B) Fly Hours 2 3 4 (9)
V.iM4 ::~:I......L.........Qu fl:5..I
E) Missions 3 3 4 (10)F) Exercise 4 4 4 (12) * *G) Sim 2 2 3 (7)H) TrainEval 3 4 3 (10)
The pair-wise comparison results with factors C, D and I removed (C
for non-independence, D because it will be used as part of a
Lexicographic screening of the WD population, and I because of the
difference in respondent treatment of the factor) is presented below.
Transitivity of the results for each respondent is maintained since
removal of factors causes no changes, and no new factors were added.
160
1) A > F > E > H > B > G2) A >F >E >B >H >G3) A >F >E >H >B >G
Using the above information, SmartEdge was configured to elicit the
weights and utilities of the remaining factors. The SmartEdge sessions
are discussed next.
SmartFgM Session
Assessment of utilities and weights for use in a value function
formulation was accomplished in a SmartEdge session with the two
remaining factors, i.e., A) StanEval ratings, and F) # of exercise
missions. (This assessment is done for the WD population left after the
population has been screened for "experts" as discussed above).
Independence issues are treated at the Utility Independence level
by SmartEdge. Preferential Independence is not explicitly checked,
since given Utility Independence "it is difficult to construct a
realistic exanple where pair-wise utility independence would be
violated" [SmartEdge Documentation : B-11]. Given that the set of
factors has been reduced to two, this is a moot point. (With more than
two factors, Preferential Independence may have to be verified prior to
using a package like SmartEdge). Utility Independence and Preferential
Independence are the same in this case.
The SmartEdge session was conducted by defining a set of
alternatives in the CASE DEFINITION area of the program. The
alternatives in this case are "WDs" to be ranked by preference.
("Representative WDs" were built at 10 different levels of experience.
Each was assigned representative attribute values that might be expected
161
at these levels of experience). This constituted the necessary set-up
work to allow the session to enter the weight and utility determination
phase.
Establishing utilities for each factor and the weights between
factors entailed the surrogate decision maker (only one of the WDs had
time for the SmartEdge session) passing through two sequences of
reference lotteries. Utilities were arrived at by establishing the
decision maker's (DM) trade-offs within a specific factor, and the
weights were developed by assessing preferences expressed by gambles
between specified trade-offs between the factors.
Having produced the utilities and weights, SmartEdge was asked to
rank the representative WDs that were defined as the alternative space
of the problem. All were ranked in the order expected, and the assigned
utility scores were proportional to the expected difference in
experience levels among the alternatives. Data for real WDs, obtained
from Tinker AFB, WK, was then entered into the alternative space, and
the program was executed again. The real WDs were also ranked
appropriately (Tab C).
Corclusions
The methodology examined here gave results that made sense.
However, there are probably many better discriminators of experience for
the "non-experts" that should be examined. As these factors are
included, the dimensional ity of the problem increases rapidly, making
solutions more difficult. Further, when multiple experts differ on
utilities and weights, MDA has some problems. Various concordance
measures are available for determining whether differences observed
162
between the elicited values are significant. Unfortunately, when the
concordance measures indicate significant disagreement, MADA currently
has no generally accepted means for resolving the dispute.
Possibly a more significant problem in this instance is the level
of expert involvement required to implement this type of strategy, and
the difficulty the experts have in understanding the utility and
weighting elicitation process. For this simple test, weights and
utilities were only developed for two factors. This process took over
six hours to accomplish, using a reasonably friendly software package.
The major problem encountered was that, even using a 50-50 gamble (which
Holloway indicated is generally the most easily understood), the
decision maker being queried could not build a utility curve that
matched his estimations, even then he had a clear perception of whtat the
curve should look like. People in general do not deal with
probabilities well, and this experience indicates they have particular
problems estimating the relative probabilities for two different bounded
ranges (not including either extreme) in the interior of the space they
are considering.
The time issue is critical for the DSS. The people who will be
queried in this process at Tinker AFB are 1) non-technical, and 2)
flyers who have a erratic, and very full, schedules. Attempting to
corral the personnel support from the experts to repeat the MADA
approach for 12 crew position with multiple factors, multiple times as
the system evolves, will probably kill the DSS development outright. As
such, this approach should only be used as a last resort, when other
less time intensive (flyer's time) methods have been found insufficient.
163
Having said this, it must also be said that, while these techniaues
are difficult to apply to the current problem, they do offer an
alterative means of looking at the personnel data that applies to the
problem. The process of dealing with experts is illuminating, and
results in a much better understanding of a problem than results from a
bare examination of personnel data using statistical techniques. As
such, there is some value in an analyst "squandering" (from the expert's
perspective) a little expert time to gain the insights can be gleaned
using these techniques.
164
TbA - The Suirvey
165
To get a measure of the "experience level" of the WD crew force, a"quantifying" scheme is being be developed that calculates anSexperience score" for every WD in AWACS. The quantifying scheme willdepend on factors that you indicate provide valuable indications ofindividual experience. Please give your undivided consideration to eachanswer you provide. Your contributions will be invaluable in managingnew personnel acquisitions.
Assume you must pick the WD's for the crew going on a criticalmission. Further, assume you have no personal knowledge of the WDs youhave available to form the crew. All you have is information fromvarious databases that contain career information on each candidate.
PAT I(a).
The following questions show a scale relating to the importance ofvarious data elements. These data elements are available, and may behelpful to you in forming the best crew possible from the availablepersonnel. With the situation describe above, select the scale valuethat best describes the value you would place on the having access tothe indicated data to help you assess the candidates experience. Thescale values correspond to the following descriptions:
Data in this area is in selecting the best men for this job.
1 -- not useful
2 -- somewhat useful
3 -- useful4 - very useful5 -- exceptionally valuable
Note: Make your assessment based only on the data element beingconsidered. Do not make your assessment to the other data elementsdiscussed in the survey.
DATA EL..ENT
1) StanEval ratings on last three check rides
Not useful 1- 2 3 4 5 very valuable
2) otal E-3 flying hours
Not useful 1- 2 3 4 5 very valuable
3) Years in position
Not useful 1 2 3 4 5 very valuable
166
The scale values descriptions are repeated here for your convenience.Data in this area is in selecting the best men for this job.
1 -- not useful2 -- somewhat useful
3 -- useful4 - very useful5 -- exceptionally valuable
Note: Make your assessment based only on the data element beingconsidered. Do not make your assessment to the other data elementsdiscussed in the survey.
4) Instructor qualified at some time
Not useful I 2 - 3 4 5 very valuable
5) Number of E-3 missions flown
Not useful 1 2 - 3 4 5 very valuable
6) Number of exercise missions flown
Not useful 1 2 - 3 4 5 very valuable
7) Nimber of Simulator sessions
Not useful 1- 2 - 3 4 5 very valuable
8) Training evaluations for courses in the WD training sequence
Not useful 1- 2 - 3 - 4 - 5 very valuable
9) Number of controls if from a ground controller background
Not useful 1 2 - 3 - 4 - 5 very valuable
167
PART I(b).
In this portion of the survey, you are asked to conpare the dataelements in the previous section two at a time. Based on the pair-wiseccarison stated in each of the following questions, please select thedata element you most value in selecting the WDs for the crew you areforming. Indicate your choice by circling the appropriate choice in thecenter column. Try to answer all questions. Ties are not allowed.
If I could only have one piece of information, I would prefer to knoweach candidate WD's
CIRCLE I or II
1) Years in position I or II Instructor qualificationstatus (all of career)
2) Total E-3 flying hours I or II Training evaluations forall courses in the WDtraining sequence
3) StanEval ratings on I or II Instructor qualificationlast three check rides status Call of career)
4) Years in position I or II Numiber of controls ifpreviously a ground
controller
5) Wirber of E-3 missions I or II Training evaluations forflown all courses in the WD
training sequence
6) StanEval ratings on I or II Training evaluationslast three check rides for all courses in the
WD training sequence
7) Instructor qualification I or II WNmber of simulatorstatus (all of career) sessions
8) Total E-3 flying hours I or II Years in position
9) StanEval ratings on I or II Total E-3 flying hourslast three check rides
10) Total E-3 flying hours I or II Nubmber of exercisemissions flown
11) Training evaluations for I or II Number of controlsall courses in the WD if previcisly a groundtraining sequence controller
168
If I could only have one piece of information, I would prefer to know
each candidate WD's
CIRCLE I or II
12) Instructor qualification I or II Number of E-3 missionsstatus (all of career) flown
13) Number of exercise I or II Training evaluations for
missions flown all courses in the WDtraining sequence
14) Total E-3 flying hours I or II Number of E-3 missionsflown
15) StanEval ratings on I or II Number of exercise
last three check rides missions flown
16) Years in position I or II Number of simulatorsessions
17) Number of simulator I or II Number of controlssessions if previously a ground
controller
18) StanEval ratings on I or II Years in position
last three check rides
19) Instructor qualification I or II Number of exercisestatus (all of career) sessions flown
20) Number of simulator I or II Training evaluations for
sessions all courses in the WDtraining sequence
21) Years in position I or II Number of E-3 missionsflown
22) StanEval ratings on I or II Number of controls iflast three check rides previously a ground
control ler
23) Total E-3 flying hours I or II Instructor qualificationstatus (all of career)
24) Number of exercise I or II Number of controlsmissions flown if previously a ground
control 1 er
169
If I could only have one piece of information, I would prefer to knoweach candidate WD's
CIRCLE I or II
25) Number of E-3 missions I or 11 Numiber of simulatorflown sessions
26) StanEval ratings on I or II Number ol E-3 missionslast three check rides flown
27) Years in position I or II Number of exercisemissions
28) Total E-3 flying hours I or II Number of simulatorsessions
29) StanEval ratings on I or II Number of simulatorlast three check rides sessions
30) Total E-3 flying hours I or II Number of controls ifpreviously a ground
controller
31) Instructor qualification I or II Training evaluations forstatus (all of career) all courses in the WD
training sequence
32) Nbumbey of E-3 missions I or II Number of exerciseflown missions flown
33) Years in position I or- II Training evaluations forall courses in the WD
sequence
34) Number of exercise I or II Number of simulatormissions flown sessions
35) Nk-ber of E-3 missions I or II Numiber of controls ifflown previously a ground
controller
36) Instructor qualification or II Number of controls ifstatus (all of career) previously a ground
controller
170
PART II.
Please take a f- mompnt' to indicate factors the+ ,.-*l,-_ bvaluable indicators of a WDs experience. If you choose to list some
factors, please limit your suggestions to those that you think are at
least as good an indicator as the one you feel is the best in the setyou have been answering questions about.
I .
2.
3.
4.
5.
171
1. ______________________________________________
Tab B13 Exanple Transit ivity Examinat ia
172
LESTION PAIR RESPONDENT PREFERENCE BYNUMBER COMPARED GROUP
9 A B A18 A C A3 A D A
26 A E A A preferred over15 A F A all29 A G A
6 A H A22 A I A
8 B C C23 B D D14 B E E10 B F F C,D,E,F,H,I > B > G28 B G B
2 B H H30 B I I
1 C D D21 C E C27 C F C D,I > C > E,F,G,H16 C G C33 C H C
4 C I I
12 D E D19 D F D7 D G D I > D > E,F,G,H
31 D H D36 D I I
32 E F F25 E G E F,I > E > G,H
5 E H E35 E I I
34 F G F13 F H F I > F > G,H24 F I I
20 G H H H,I >G17 G I I
11 H I I I >H
The overall preference order expressed by respondent one was fullytransitive, and is: A > I > D > C > F > E > H > B > G
173
Tab C - LinartEdge Catput
174
SmartEdge was exercised in four different modes to see how
different the results were for the oroject problem. These run modes
were:
1) Additive Model -- Probabilistic2) Additive Model - Deterministic3) Multiplicative Model - Probabilistic4) Multiplicative Model -- Deterministic
All four applications of returned the same ranking. Further, the
differences in the utility values differed by at most 2/.
In this case, there is no significant difference between the
additive and multiplicative model results. Further, the results were
insensitive to the uncertainty in attribute level set for the initial
"representative WDs".
The edited summary reports for these runs follow. The editing
consists of renoving Section III which reports on differences among
multiple decision makers (only one used) and the inclusion of only one
set of appendices (Section IV) for the four runs, since the information
for all runs remained the same.
175
SJMMARY REPORT CONTENTS
FOR ALL CASES
This report summarizes the results of PROBABILISTIC/ADDITIVE as follcyw.s:
I Introduction and Case Specifications
A. The Alternatives
B. The AttributesC. The Decision Makers
D. The Analysis ModeL. The Run OptionF. The Decision Model
II Individual Decision Maker Rankings
II Decision Maker Rankings As a Group (NOT INCLUDED -- ONLY 1 DECISION
MAKER)
IV Appendices (INCLUDED PETER THL AL LYSIS OF ALL FOLR RUNS)
A. The Attribute State/Distribution InputsB. The Preference Inputs for Each Decision Maker
176
SUMMARY REPORTFOR CA~SE: PROBABILISTIC/ADDITIVE
177
SECTION IINTRODUCTION AND CASE SPECIFICATIONS
4This Section presents the specifications for the decision making casedefined.
A. The Alternatives are:Alterative 1 = 1Alterative 2 = 2Alterative 3 = 3Alterative 4 = 4Alterative 5 = 5Alterative 6 = 6Alterative 7 = 7
Alterative 8 = 8Alterative 9 = 9Alterative 10 = 10Alterative 11 = BEENAROUNDAlterative 12 = EENEME IAlterative 13 = SEENSOME2Alterative 14 = OLDDOG
B. The Attributes are:Attribute 1 = StanEval RatingsAttribute 2 - #ExerciseMissions
C. The Decision Makers are:Individual 1 = DM2
D. Mode of Analysis Selected is: Detailed Analysis Mode
E. Run Option for Uncertainty is: Probabilistic
F. Type of Decision Model is: Simple Linear (Additive) Model
178
SECTION II
INDIVIDUAL DECISION MAKER RANKINGS
This Section presents the results of a multi-attribute decision analysisfor the Case PROBABILISTIC/ADDITIVE with rankings for each decisionmaker.
Rankings for Decision Maker: DM2Alternative Value Rank
1 0.0776 14+/- 0.0327
2 0.1261 13+/- 0.04573 0. 1633 12+/- 0.0596
4 0.2521 9+/- 0.07365 0.3263 8+/- 0.0767
6 0.3388 7+/- O.0988
7 0.4429 6+/- 0.12278 0.4607 5
+/- 0.12549 0.6032 3+/- 0.139910 0.6768 2+/- 0.1193
BEE4)RJJYZ 0.5342 4+/- O.0005
SEENSOME1 0.2465 10/- 0.0002
SEENStPE2 0.1758 11+/- 0.0001
OLDDOG 0. 9088 1+/- 0.0004
The rankings are in Agreement at a 95 percent significance level.
179
FRCA~SE: DETEMINISTIC/iADDITIVE
180
SECTION IINTRODUCTION AND CASE SPECIFICATIONS
This Section presents the specifications for the decision making case
defined.
A. The Alternatives are:Alterative 1 = 1Alterative 2 = 2
Alterative 3 = 3
Alterative 4 = 4Alterative 5 = 5Alterative 6 = 6Alterative 7 = 7Alterative 8 = 8Alterative 9 = 9
Alterative 10 = 10
Alterative 11 = BENROUND
Alterative 12 = SEENSOME1
Alterative 13 = SEENSOME2Alterative 14 = OLDDOG
B. The Attributes are:Attribute 1 = StanLval RatingsAttribute 2 = #ExerciseMissions
C. The Decision Makers are:
Individual 1 = DM2
D. Mode of Analysis Selected is: Detailed Analysis Mode
E. Run Option for Uncertainty is: Deterministic
F. Type of Decision Model is: Simple Linear (Additive) Model
181
SECTI ON I IINDIVIDUAL DECISION MAKER RANKINGS
This Section presents the results of a multi-attribute decision analysisfor the Case DETERMINISTIC/ADDITIVE with rankings for each decisionmaker.
Rankings for Decision Maker: DM2Alternative Value Rank
1 0. 0793 142 0.1221 133 0.1650 12
4 0.2521 95 0.3259 86 0.3532 77 0. 4262 68 0.4550 59 0.6055 3
10 0.6616 2BEENJOUND 0.5342 4SEENSOE1 0.2465 10SEENSOME2 0.1758 11
OLDDOG 0. 9068 1
The rankings are in Agreement at a 95 percei, significance level.
182
SUMMARY REPOPT- RF CA~SE: PRBBLSI/U-ILCTV
1B3
SECTION IINTRODUCTION AN CASE SPECIFICATIONS
This Section presents the specifications for the decision making casedefined.
A. The Alternatives are:Alterative 1 = IAlterative 2 = 2
Alterative 3 = 3Alterative 4 = 4
Alterative 5 = 5Alterative 6 = 6Alterative 7 = 7Alterative 8 = 8
Alterative 9 = 9Alterative 10 = 10
Alterative 11 = BEE qROUNDAlterative 12 = SEENSOMElAlterative 13 = SEENSOME2Alterative 14 = OLDDOG
B. The Attributes are:Attribute 1 = StanEval Ratings
Attribute 2 = #ExerciseMissions
C. The Decision Makers are:
Individual 1 = DM2
D. Mode of Analysis Selected is: Detailed Analysis Mode
E. Run Option for Uncertainty is: Probabilistic
F. Type of Decision Model is: Detailed Non-Linear (Multiplicative)
Model
184
SECTION IIINDIVIDLL DECISION MAKER RPANKINGS
This bection presents the results of a multi-attribute decision analysis
for the Case PROBABILISTIC/MULTIPLICATIVE with rankings for eachdecision maker.
Rankings for Decision Maker: DM2Alternative Value Rank
1 0.0739 14+/- 0.0312
2 0.1205 13+/- 0.04373 0.1565 12
+/- 0.0573
4 0.2420 9+I- (nTr
5 0.3146 8+/- 0.07416 0.3285 7
+/- 0.0964
7 0.430e 6+/- 0. 1210
8 0.4496 5+/- 0. 1235
9 G. 5941 3+/- 0.1399
10 0.6657 2+/- 0.1191
BEENI]PRJND 0. 5263 4+/- 0.0004
SEENSOMEI 0.2404 10+/- 0.0001
SEENSOME2 0.1689 11
+/- 0.0002
OLDDOG 0. 8800 1+/- 0. 0008
The rankings are in Agreement at a 95 percent significance level.
SLP!4PY REPORTFOP CASE: DETERMINISTIC/MULTIPLICATIVE
126
SECTION IINTPODUCTION AND CASE SPECIFICATIONS
This Section presents the specifications for the decision making casedefined.
A. The Alternatives are:Alterative 1 = 1Alterative 2 = 2Alterative 3 = 3
Alterative 4 = 4
Alterative 5 = 5Alterative 6 = 6
Alterative 7 = 7Alterative 8 = 8Alterative 9 = 9
Alterative 10 = 10Alterative 11 = BEEI\kODAlterative 12 = SEENSOME1Alterative 13 = SEENSOME2Alterative 14 = OLDDOG
B. The Attributes are:Attribute 1 = StaEval RatingsAttribute 2 = #ExerciseMissions
C. The Decision Makers are:Individual 1 = DM2
D. Mode of Analysis Selected is: Detailed Analysis Mode
E. Run Option for Uncertainty is: Deterministic
F. Type of Decision Model is: Detailed Non-Linear (Multiplicative)Model.
187
SECTION I I
INDIVIDUAL DECISION MAKER RANKINGS
This Section presents the results of a multi-attribute decision analysisfor the Case DETERMINISTIC/MULTIPICATIVE with rankings for eachdecision maker.
Rankings for Decision Maker: DM2A1ternative Value Rank
1 0.0756 142 0.1167 133 0.1580 124 0.2421 95 0.3143 86 0.3426 77 0.4139 68 0.4435 59 0.5963 310 0.6503 2
BEENARdOND 0.5263 4SEENSOME1 0. 2434 i0SEENSOE2 0.1689 II
OLDDOG 0.8800 1
The rankings are in Agreement at a 95 percent significance level.
188
SECTION IV
DATA I NPUTS
This Section presents the inputs for Case PROBABILISTIC/ADDITIVE used tocompute the rankings.
A. The Attribute State/Distribution Inputs:
F4ttribute States for Alternative : 1
Cumulative Distribution for StanEvalState CDF State CDF State CDF1.0000 0.00 2.0000 0.50 3.0000 1.002.0000 0.25 2.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CDF State CDF State CDF0.0000 0.00 2.0000 0.50 3.0000 1.000.0000 0.25 3.0000 0.75
Attribute States for Alternative : 2
Cumulative Distribution for StanEval
State CDF State CDF State CDF1.0000 0.00 3.0000 0.50 3.0000 1.00
2.0000 0.25 3.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CDF State CDF State CDF
0.0000 0.00 4.0000 0.50 5.0000 1.00
2.0000 0.25 4.0000 0.75
Attribute States for Alternative : 3
Cumulative Distribution for StanEvalState CDF State CDF State CDF
2.0000 0.00 3.0000 0.50 4.0000 1.002.0000 0.25 4.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CDF State CDF State CDF2.0000 0.00 5.0000 0.50 7.0000 1.004.0000 0.25 5.0000 0.75
189
Attribute States for Alternative z 4
Cumulative Distribution for StanEvalState CDF State CDF State CDF3.0000 0.00 4.0000 0.50 6.0000 1.003.0000 0.25 5.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CL* State CDF State CDF4.0000 0.00 6.0000 0.50 8.0000 1.005.0000 0.25 7.0000 0.75
Attribute States for Alternative : 5
Cumulative Distribution for StanEval
State CDF State CDF State CDF3.0000 0.00 5.0000 0.50 7.0000 1.00
4.0000 0.25 6.0000 0.75
Cumulative Distribution for #ExerciseMissions
State CDF State CDF State CDF7.0000 0.00 8.0000 0.50 10.0000 1.007.0000 0.25 10.0000 0.75
Attribute States for Alternative : 6
Cumulative Distribution for StanEvalState CDF State CIF State CDF
3.0000 0.00 5.0000 0.50 8.0000 1.004.0000 0.25 6.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CDF State CDF State CDF7.0000 0.00 14.0000 0.50 18.0000 1.009.0000 0.25 14.0000 0.75
Attribute States for Alternative : 7
Cumulative Distribution for StanEvalState CDF State CDF State CDF3.0000 0.00 6.0000 0.50 8.0000 1.005.0000 0.25 8.0000 0.75
Cumulative Distribution for #Exerc iseMissionsState CDF State CDF State CDF7.0000 0.00 14.0000 0.50 18.0000 1.00
7.0000 0.25 14.0000 0.75
190
Attribute States for Alternative : 8
Cumulative Distribution for StanEvalState CDF State CDF State CDF4.0000 0.00 6.0000 0.50 9.0000 1.005.0000 0.25 8.0000 0.75
Cumulative Distribution for #ExerciseMissions
State CDF State CDF State CDF10.0000 0.00 14.0000 0.50 16.0000 1.00
14.0000 0.25 18.0000 0.75
Attribute Stdtes for Alternative : 9
Cumulative Distribution for StanEvalState CDF State CUF State CDF4.0000 0.00 8.0000 0.50 11.0000 1.007.0000 0.2b 9.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CDF State CDF State CDF
14.0000 0.00 18.0000 0.50 21.0000 1.0014.0000 0.25 21.0000 0.75
Att,-ibute States for Alternative z 10
Cumulative Distribution for StanEvalState CDF State CDF State CDF7.0000 0.00 8.0000 0.50 12.0000 1.00
8.0000 0.25 9.0000 0.75
Cumulative Distribution for #ExerciseMissionsState CIF State CDF State CDF
14.0000 0.00 18.0000 0.50 21.0000 1.00
14.0000 0.25 18.0000 0.75
Attribute States for Alternative BEENRLND
Point Value for StanEval 7.0000
Point Value for #ExerciseMissions = 18.0000
Attribute States for Alternative SEENSIE1
Point Value for StanEval 3.0000
Point Value for #ExerciseMissions 18.0000
191
Attribute States for Alternative : SEENGOME2
Point Value for StanEval 3.0000Point Value for #ExerciseMissions = 7.0000
Attribute States foy Alternative : OLDDOG
Point Value for StanEval = 12.0000Point Value for #ExerciseMissions 10.0000
B. The Preference Inputs for Each Decision Maker
Attribute Preferences for Individual: DM2
Scaling Constraints (Heights) For Each Attribute Are:StanEval: 0. 8632#LxerciseMissions: 0.1368
The UtiL1ty Function for Each Attribute Are:ATTRIBUTE: St anEval
State Utility State Utility State Utility1.0000 0.00 7.0000 0.50 12.0000 1.00
ATTRIBUTE: #xerciseMissionsState Utility State Utility State Utility0.0000 0.00 15.0000 0.50 21.0000 1.00
192
Bibl iography
1. Ackoff, Russell L., "Management Misin'ormation Systems", ManagementScience, 14: 147-156, December 1967.
2. Andriole, Stephen J., "The Design of Microcorrputer-Based PersonalDecision--Aiding Systems", IEEE Transactions n Systems, Man,and Cybernetics, 12: 463-469, July/August 1962.
3. Burns, Michael D., Capt, USAF, Former 552 AWPO.W/DOT (Tactics)Weapons Director, Personal Interviews, AFIT, Wright-PattersonAFB OH, November 1969 - December 1989.
4. Cerveny, Robert P., Edward J. Garrity and G. Lawrence Sanders, "TheApplication of Prototyping to Systems Development: A Rationaleand Model", Journal of Management Information Systems, 3: 52-61, Fall 1986.
5. Chan, Yupo, "Psychometric Scaling in Transportation Modal Choice",Draft of working paper prepared for the PennsylvaniaTransportation Institute, February 1976.
6. Cohen, Marvin S., "When the Worst Case is Best: Mental Models,Uncertainty, and Decision Aids", Impact and Potential ofDecision Research on Decision Aiding, American PsychologicalAssociation, 1967.
7. Deming William E., CXut of the Crisis, Cambridge, MA: MIT Center forAdvanced Engineering Study, 1986.
8. Department of the Air Force, Air Force Planning Docuent, Change 6,Dated October 1965.
9. Devor, Jay L., Probability and Statistics for Engineering and theSciences. Monterey, CA: Brtzks/Cole Publishing Company, 1967.
10. El Sherif, Hisham and Omar A. El Sawy, "Issue Based Decision SupportSystems for the Egyptian Cabinet", MIS Quarterly. 551-568,December 1988.
11. Floyd, C. and Keil, R., "Adapting Software Development For SystemsDesign with the User", Information Analysis: SelectedReadings, Robert Galliers: Editor, 153-166, Reading, IA:Addison-W'esley Publishing Company, 1967.
12. Guzec, Ronald P., Lt Col, 552 AWACW/ADOC, Telephone Interviews,Tinker AFB OK, November 1989.
13. Holloway, Charles A., Decision Making Under Uncertainty: Models andChoices Englewood Cliffs, NJ: Prentice-Hall, Inc., 1979.
14. Keen, Peter G. W., "Adaptive Design For Decision Support Systems",
193
ACM/Database 12: 15-25, Fall 1960.
15. Land, F. F., "Adapting to Changing User Requirements", InformatiunAnalysis: Selected Readings, Robert Galliers: Editor, 153-166,Reading, MA: Addison-Wesley Publishing Company, 1987.
16. Meador, C. Lawrence, Martin J. &'yote and William L. Rosenfeld,"Decision Support Planning and Analysis: The Problem ofGetting Large-Scale DSS Started", MIS Quarterly. 159-177, June1966.
17. Mittermeir, R. T., Hsia, P. and Yeh, R. T., "Alternatives toOvercome the Communications Problem of cormal RequirementsAnalysis", Information Analysis: Selected Readings, RobertGalliers: Editor, 153-166, Reading: Addison-Wesley PublishingCompany, 1987.
18. Neter, John, Willi m Wasserman, and Michael H. Kutneo, AppliedStatistical Models. Homewood, IL: Irwin, 1965.
19. Robey, Danial and William Taggart, "Human Information Processing inInformation and Decision Support Systems", MIS Quarterly
20. Schneider, Willaim L., Adaptive Design in Building a DecisionSupport System for AWACS Programmed Flying Training, Master ofScience Thesis. AFIT School of Engineering, Wright-PattersonAFB OH, December 1987.
21. Siegel, Sidney, Nonparant-ic Statistics for the BehavioralSciences New York: McGraw-Hill Book Company, 1956.
22. SmartEdge User's Manual, A Decision Support System for ExpertKnowledge Acquisition and Evaluation Pasedena, CA: Havland-Lee, Inc, 198.
23. Smoot, Donald E., Major, USAF Human Resources Laboratory, PersonalInterviews, Wright-Patterson AFB, OH, November 1989 - December1989.
24. Sprague, Ralph H., Jr. and Eric D. Carlson, Building EffectiveDecision Support Systems. Englewooc Cliffs, NJ: Prentice-Hall,Inc., 1982.
25. Stone, Capt Matthew A., USAF, Former 26 AD/XOYS (Analysis) Analyst,Personal Interview, PFIT, Wright-Patterson AFB OH, January1989.
26. Sumner, 1 Lt David L., USA, Former 28 AD/XOYS (Analysis) Analyst,Personal Interviews, AFIT, Wright-Patterson AFB OH, August1989 - December 1989.
27. Valusek, John R., "Adaptive Design of DSSs: A User Perspective",
194
DSS-88 Iransactioni: Eiqhth International Conference on DES,105-112, 198.
28. Valusek, John R., Class lectures in OPER 652, Decision SupportSystems, Air Force Institute of Technology, Wright-Patte ;onAFB OA, August 1909.
2. Walker, Robert G., Robert S Barrnardt, and Warren E. Walker,
Selecting A Decision Support System Generator for the AirForce's Enlisted Force Management System", A Project AIR FORCE
report prepared for the United States Air F rce, R--3474-AF,Santa Monica: The RAM Corporation, 1986.
30. Young, Lawrence F., Decision Support and Idea Processing Systems,Dubuque, IA: Wi. C. Brown Publishers, 19e9
195
Vita
Major Barton H. Wohl
V . He graduated from high school in Hazlet, New Jersey, in 1974 and
attended the United States Air Force Academy, class of 1978. After
graduating with a Bachelor of Science degree, he worked for several
months at the Air Force Weapons Laboratory before beginning Undergradu-
ate Pilot Training at Columbus AMl, Mississippi. He completed pilot
training and received his wings in October 1979. He then served as a
Forward Air Controller and 0-2 pilot in the 21st Tactical Air Support
Squadron, Shaw AFB, South Carolina from 1980 to 1983. From 1984 to 1985
he was an F-4E pilot in te 57th Fighter Interceptor Squadron, Keflavik,
Iceland, and then served as an F-4Z pilot, flight lead and flight
instructor in the 336th Tactical Fighter Squadron, Seymour Johnson AB,
North Carolina until entering the School of Engineering, Air Force
Institute of Technology, in August 1988.
58
Vita
Captain Kevin M. Holt "1 _ I It l " -'
He graduated from Beavercreek High School, Beavercreek, Ohio, in 1973.
He entered the United States Air Force in December of 1979, and received
a Bachelor of Science in Mathematic from Saint Edward's University,
Austin, Texas, in December of 1964, while stationed at Bergstrom AFB.
He was commissioned through Officer Training School in October 1985, and
was subsequently assigned to the 28 Air Division at Tinker AFB,
Oklahoma, as an AWACS Operations Research Analyst. He entered the
School of Engineering, Air Force Institute of Technology, in August
1966.
196
UNLSIFIED
SECURITY CLASSIFICATION OF THIS PAGESForm Approved
REPORT DOCUMENTATION PAGE OMNo.0704-0188
!a. REPORT SECURITY CLASSIFICATION lb. RESTRICTIVE MARKINGS
LNPC.S II ED
2a. SECURITY CLASSIFICATION AUTHORITY 3. DISTRIBUTION /AVAILABILITY OF REPORT
Approved for public release;
2b. DECLASSIFICATION /DOWNGRADING SCHEDULE distribution unI imited.
4. PERFORMING ORGANIZATION REPORT NUMBER(S) 5. MONITORING ORGANIZATION REPORT NUMBER(S)
-FTT /r-/F/91-106a. NAME OF PERFORMING ORGANIZATION 6b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATION
(If applicable)
~~ ~ ~ ~~~ATT/1S ______________________
.?6c ADDRESS (City, State, and ZIP Code) 7b. ADDRESS(City, State, and ZIP Code)
8a. NAME OF FUNDING/SPONSORING 18b. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORGANIZATION (if applicable)
9c- ADDRESS (City, State, and ZIP Code) 10. SOURCE OF FUNDING NUMBERSPROGRAM IPROJECT TASK I WORK UNITELEMENT NO. NO. NO. ICCESSION NO.
11. TITLE (Include Security Classification)
nN r- i -a -inr-i qw Cfci- y m (D) f or ('Jgkj_ PersmnnP ACauisition Management
12. PERSONAL AUTHOR(S)'=, M I- 1 4. r_-n+:_in_ Il
13a. TYPE OF REPORT 13b. TIME COVERED 114. DATEOFREPORT(YearMonthDay)ItS'PAGE COUNT
Mqi Thwic: FRMqO9 0. Marc~h, 16I 9
16. SUPPLEMENTARY NOTATION
17. COSATI CODES 18. SUBJECT TERMS (Conbnue on reverse if necessary and identify by block number)
FIELD GROUP SUB-GROUP Airborne Warning and Control System;
Decision Support System; Adaptive Design;
19. ABSTRACT (Continue on reverse if necessary and identify by block number)
The purpose of this research was to define an appropriate tool to assist AWACS personnel
managers in determining the personnel acquisitions that are required for AWACS operations.
The approach used to structure the decision process was that of a Decision Support System
(DSS). The goal of the specified DSS was to provide an initial point to better state
personnel acquisition requirements, particularly with regard to the experience of the crew
force, and to provide a structure for improvement in the DSS itsel f.
The decisions supported by the designed DSS go beyond what has been attempted by AWACS
air crew managers in the past, explicitly addressing experience level inventories and the need
to stabilize both manning levels and experience levels in the various AWACS crew positions.
The overall design of the DSS to support these goals is specified through the mechanisms of
concept mapping and storyboarding, and a kernel system is developed. Adaptive design is
adopted as the means for continued system development and evolution.
20. DISTRIBUTION/AVAILABILITY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION
[0 UNCLASSIFIED/UNLIMITED r3 SAME AS RPT. 03 DTIC USERS I UNLASSIFIED
22a. NAME OF RESPONSIBLE INDIVIDUAL 22b. TELEPHONE( I4Atd) 22c. OFF IT SYMBOL
Kevin M. Holt 5lI AF /
DO Form 1473, JUN 86 Previous editions are obsolete. SECURITY CLASSIFICATION OF THIS PAGE
UNCLASSIFIED