13th International Command & Control Research and Technology SymposiumC2 for Complex Endeavors

Title: The Process of Sensemaking in Complex Human Endeavors

Topic: 1 (C2 Concepts, Theory, and Policy), 4( Cognitive and Social Issues), 3 (Modeling and Simulation)

Celestine A. NtuenArmy Center for Human-Centric Command & Control Decision Making

The Institute for Human-Machine Studies419 McNair HallNorth Carolina A&T State UniversityGreensboro, NC 27411Phone: 336-334-7780; Fax: 336-334-7729Email: Ntuen@ncat.edu

Sensemaking process involves the understanding of many different and interdependent factors that must be reconciled with the realities and rhythms of the problem context. For example, in the battle space, the commanders’ levels of knowledge, skill, and experience vary greatly among individuals and among battle staffs, and are required to deal with processing equivocal information, or sometimes, dealing with a lack of information. Also, collaborative information processing using a team of people such as coalition force structure often lead to different interpretations, which in turn affects the team understanding of the situational information. The existing training doctrines that address the deliberate military decision making process is not adequate, or perhaps not even relevant to the training of the military sensemakers and intelligent analysts. We need a new training strategy. To do this, we need to understand the process of sensemaking in complex human endeavors.

1. INTRODUCTION

The traditional definition of a complex system is that it involves a large number of dynamically interacting elements, leading to non-linear behaviors of the entire system or sub-systems. This is the fundamental principle of dynamic complexity.

Dynamic complexities are ubiquitous in many human endeavors and activities that experience unpredictable shifts in behaviors defined by many dynamics of human systems transformations. For example, from the domain of a peaceful social structure to an unstable cultural war (e.g., Iraq); from quite habitat of human species to catastrophic refugee immigrations orchestrated by natural calamities (e.g., Hurricane Katrina and the Gulf Coast residents in USA). The human system interacts with information that can create system transformation from a stable state to a chaotic state, or a state of calm to that of panic. Interestingly, these types of system state changes can be from human designs or as a result of natural phenomena. In the first case, we may design human organizations that are subject to stress and agitations. These may be political (e.g., recent outbreak of anarchies in Burma, Darfur, and Pakistan); economic such as the economic meltdown of South Korea in the early 1980’s; socio-cultural (e.g., managing over 200

DRAFT

ethnic languages in Nigeria); or military, such as the on-going Iraq and Afghanistan wars.

We can also design machines with complexity; some machines inherit complex behaviors from their constituent members. Examples of these kinds of machines include, e.g., air traffic control, nuclear plants, air planes, and manufacturing systems. In the second case, complexity can be natural. Hurricane Katrina in New Orleans and Tsunami in Asia are two obvious examples that define complexity in many facets of managing and controlling after effects—politically, economically, and socially. In general, the situations above are referred to as “wicked” since the state of the world is unpredictable and the information characteristics are random.

Dynamic human endeavors can best be described by how much we can predict the informational state of the world. For example, in the military environment, Alberts and Hayes1 note that “Being able to pick a nontraditional adversary out of the noise and determine its capabilities and intentions is among the greatest challenges that we face in Information age (pp. 101).”

Predictable states of the world mean that the behavior of the system is deterministic—leading us to have some sense of the next state of the system. Unpredictable states of the world mean that we have no control of what will happen next. Dynamic human endeavor is created by a system whose state changes are either predictable or unpredictable with respect to time, location, and changing patterns of information. The pattern of information can reconfigure their interactions such that the emerging behaviors become difficult for the human to understand. Table 1 below illustrates a simple two-dimensional matrix information interaction that creates such complexities.

Table 1. The dimensions of system complexity based on information statesInformation characteristics

The World

As used by many authors, notably, Waldrop in his text on complexity2, dynamic endeavor is a characteristic of complex adaptive systems (CAS) and has been used anecdotally and notionally to described, explain, and represent the human conceptual congruency in understanding the way the complex system works. In our examples described earlier—whether they are human organizations (man made) or natural, dynamic complexity can be described by the following characteristics:

1. Highly ambiguous and possible multiple information interpretations (equivocality).

2. Changing information patterns.

1 Davis S. Alberts & Richard E. Hayes, Power to the Edge: Command and control in the information age. CCRP, (http://www.dodccrp.org ), 2003.2 M. M. Waldrop, Complexity: The emerging science at the edge of order and chaos. A Touchstone Books, 1992.

Deterministic Random

Predictable Simple Complex

Unpredictable Complex Wicked

3. Possible multi-trait, multi-scale, multi-attribute information ontology.4. Creates an unstructured information states.5. Creates many forms of uncertainty.6. Creates multi circumstances.7. Cause and effect linkages are not inherently knowable

Command and control (C2) of complex systems such those described by the on-going conflicts in Iraq and Afghanistan are definitive points of what human endeavors look like in dynamic system. As noted by Hill and Levenhagen ( ), there are many factors responsible for this difficulty. However, the information management difficulty is the most damaging since “unexpected cues within organizational information processing can lead to missed opportunities as a result of disoriented or misaligned mental models—leading to loss of meaning and construction of a mapping model that identifies with the situation.” Here, unstructured information states must be reduced, abstracted, organized, and formalized to the level of human understanding. The organized information can be used to construct a meaningful scenario that maps what we know (mental model) to what is available, thereby creating actionable knowledge. In essence, we want to make sense of the situation before decision making. The process of sensemaking must be understood. Sensemaking increasingly involves dealing with, and gaining some workable understanding of, inherently complex, adaptive and interlinked systems, such as social and military organizations.

2. What is Sensemaking?

Sensemaking literary means making sense. The two words can be defined individually in order to illustrate the true meaning of the word. The American Heritage College Dictionary (2002) define the word “sense” as, (1) understanding, (2) signification, (3) present of meaning, (4) a mechanism of faculty as receiving (forming) mental impression, (5) deducing from observation or unnoted stimuli in respect to a particular field or relation, (6) instructive comprehension, (7) discerning awareness, (8) opinion, view, sentiment, of something felt and held by an individual or a group of people, (9) awareness derived through interpretation of stimuli or sensory information, (10) accustomed steady ability to judge and decide between possible courses with intelligence and soundness. The definitions 1-10 above represent the epistemological aspects of sensemaking. The same dictionary define “make” to imply, (1) to frame or formulate in the mind; (2) form as a result of calculation of design; (3) enact or establish. These characteristics represent the ontological views of sensemaking. By combining the two words, the same referenced dictionary defines sensemaking as a noun – “sensible, reasonable, and predictable”. Thus sensemaking implies the ability to design, build, and derive an understanding of situated information.

Sensemaking is a process, a design, or a technique of fusing information in context to derive understanding from fragmentary pieces of information (Ntuen, 2006). Sensemaking can be viewed as a paradigm, a tool, a process, or a theory of how people reduce uncertainty or ambiguity; socially negotiate meaning during decision making events. Weick (1995) states that sensemaking refers to how meaning is constructed at both the individual and the group levels. Through the accurate construction of meaning,

clarity increases and confusion decreases. For example, Leedom (2002) indicates that battle rhythms can best be understood through the sensemaking process. A poor sensemaking process often leads to poorly understood objectives, missions, and visions. This in turn can lead to poor framing of plans, and consequently, poor decisions.Sensemaking involves the collective application of individual “intuition”—experience-based, sub-consciously processed judgment and imagination—to identify changes in existing patterns or the emergence of new patterns (Weick, 1995).

3. Sample Relevant Past Models in Describing the Sensemaking Process

The mental aspect of sensemaking is attributed to tacit knowledge---a philosophical view of how individuals view the world based on reflexive mental models and latency lens accorded to some neural processes that control the faculty of human thinking—albeit, the mind. For example, Kelly (1955) defined this phenomenon in terms of personal constructs, an individual’s organization of unique mental models of the world that are both shaped by prior experience and are used to interpret new experiences.

Polanyi (1967) is repeatedly cited and credited for the definition of tacit knowledge and how it influences the sensemaking process. According to Polanyi, tacit knowledge is what is known but cannot be told.  The reasoning behind the statement is that the knowledge has become so personal in the unconscious mind and therefore it cannot be expressed because there is no access to it through the conscious mind.  Polanyi said "we know more than we can tell." Polanyi’s concept of tacit knowledge was reflected in three main theses (Leedom, 2005): (1) true discovery cannot be accounted for by a set of articulated rules or algorithms; (2) while knowledge is public, it is also to a very great extent personal or constructed by humans; and (3) the knowledge that underlies explicit knowledge is more fundamental.

Nonaka and Takeuchi's (1995) socialization-externalization-combination-internalization (SECI) model proposes that knowledge is created and expanded through social interaction between tacit and explicit knowledge. The SECI model are defined at four types of interaction: Socialization (tacit to tacit), Explication (tacit to explicit), Combination (explicit to explicit), and Internalization (explicit to tacit). The knowledge conversion occurs through the spiral of organizational knowledge creation, encompassing different organizational levels:1. Sharing of tacit knowledge by a group of individuals2. Conversion of tacit knowledge in teams into concepts and metaphors3. Combination of team-based concepts with existing data and external knowledge4. Articulation and development of concepts until they emerge into a concrete form5. Dissemination of new knowledge to others within organization

Sensemaking is a valid way to frame aspects or map of cognitive behavior; reciprocal process of finding a frame for data and using a frame to define the data. Here, as postulated by Klein and his Associates (2004), military data will go through the military frame of reasoning, economic data will go through economic models, political data will go through political frame, and so on. The frame paradigm is therefore sensitive to context, which makes it possible to capture the dynamics and continuity of information changes in the domain context.

The frame model structures problems into a particular set of beliefs and perspectives that constrain data collection and analysis. This coping mechanism structures problems into a particular set of beliefs and perspectives that constrain data collection and analysis. The framing usually narrows information search around local outcomes as opposed to issues further distant in effect. For example, an analyst may frame a solution for short run gains, disregarding long term consequences of the decision. For instance, to frame a message, we have to identify the frame of reference—the receiver and modality of transmission. You analyze an unstructured message; look at weak spots in a message. You resolve weak spots in messages, and then do a self assessment of your own information. You identify weak spots to recognize errors in order to realize an optimum gain in the sender-receiver performance metric. Thus, in the data/frame model, each cognitive element starts with a scenario that provides a context for you (Klein, et al., 2003). Sieck, et al. (2004) have extended the earlier work of Klein (1989) on recognition-primed decision making, to define what is now referred to as a data/frame model of focal knowledge creation that consists of various mental functions performed in a recursive manner. The data/frame model—as the name might imply—hypothesizes that creating focal knowledge involves fitting available data (environmental cues) into an explanatory frame (a constructed mental model of the situation built from fragmentary mental models).

4. The Study Rationales

While sensemaking is recently gaining a strong credo among cognitive and organizational psychologist communities, much of the discussions have centered on the attempts to define the structures of sensemaking (Leedom, Eggelston, & Ntuen, 2006 ). Little models exist that can capture the process of sensemaking. One reason can attributed to this lack—that sensemaking is cognitive task (ntuen, 2006) subject to individual perspectives, influenced by cultural lens model, and is dependent on contextual information. This paper attempts to present as a comprehensive process for sensemaking

Developing a process is like creating knowledge ontology. Creating knowledge ontology requires sensemaking activities which allows a thorough understanding of the system—through creating a common meaning, defining semantically and syntactically uniform interpretation across context, and creating a taxonomy or lexicon of common understanding—minimizing equivocality as much as possible (Weick, 1995; Choo, 1998).

The observations above often lead to the a cognitive model stance in which sensemaking starts with the fundamental assumption of phenomenology—that the actor is inherently involved in his observations, which must be understood from his perspectives and horizons (Dervin, 2003). Similar to the phenomenologist view such as advocated by Edmund Hursserl (1923), a German phenomenologist, sensemaking involves evaluation cognition based on our awareness of a context itself as disclosed in the most clear, distinct, and adequate way for something of its kind. Sensemaking, then, brings these assumptions together by asserting that—given an incomplete understanding of reality (ontology) and an incomplete understanding of what it is to know something (epistemology)—we arrive at an uncompromising situation of deciding between the “best” collection of information to support decision making.

5. The Sensemaking Process

As noted by Alberts and Hayes3, “Sensemaking is much more than sharing information and identifying patterns. It goes beyond what is happening and what may happen to what can be done about it. This involves generating options, predicting adversary actions and reactions, and understanding the effect of particular courses of action (pp. 102).” If we are to build a model that supports human endeavors, the process of sensemaking must be understood. Based on our sensemaking studies—both theoretical and experimental (Ntuen & Leedom, 2006, Ntuen, 2003), we have identified eight macro stages of sensemaking. These stages are based on a cognitive information hierarchy model of Ntuen (2007). Figure 1 is used to illustrate these stages with their interactions and feedbacks. The descriptions of these stages follow.

Figure 1. Stages in the sensemaking process

(1) Situation Framing: Sensemaking involves putting stimuli into some kind of framework (Starbuck and Milliken, 1988, p.51).When people put stimuli into frameworks, this enables them to “comprehend, understand, explain, attribute, extrapolate and predict.” Frames and cues can be thought of as vocabularies in which words are more abstract. There are at least four ways in which people impose frames on an ongoing information situation and link the frames with cues for the purpose of discovering meaning in context. Sensemaking can begin with beliefs and take the form of arguing and expecting. Or, it can begin with actions and take the form of committing or manipulating. In both cases, sensemaking is an effort to tie beliefs and actions more closely together as when arguments lead to consensus action during team problem solving. Various performance shaping factors that are relevant for situation framing outcome may include,

3 Ditto Alberts & Hayes, 2003

Situation framing based ondynamic goals

Search for relevantinformation cues

Map or clusterinformation based onsimilarity or variations

Search for meaningin the pattern

Determine conditionsFor information comprehension

Interprete information relevance to goals

Create a subset of situation understanding

Connect understandingmodels to action

8 steps to sensemaking process

for example, clarified expectations; confirming actions for effect-based objectives; justification for matching sensemaking outcomes to desired actions; or explanations for assumptions and hypotheses about a situation of interest. The failures in framing a set of hypotheses about a context can be attributable to atypical beliefs, bias, and stereotypes—not able to see things in the same fixed frame of reference.

(2). Searching for Cues: Cues provide the focal base for human reasoning strategies and also provide the focus in which a problem space is contextualized and grounded. For example, in a signal guided mapping, the sensemaker basically starts with a hypothesis, look for data to confirm the assumption. On the other hand, a cue-guided search is a bottom-up search which uses information cues as an initial data frame. From here, the sensemaker seeks linkages and patterns in the data, and classifies the available data according to saliency of the cues to sensemaking. During the process of looking for clues from the provided cues, we are likely to encounter a confirmation failure—information processing state whereby the existing information space does not match or correlate with the information in our memory. Also, when we make the wrong assumptions or hypotheses which are contradictory to the existing evidence, we are in essence subjecting sensemaking process to a first level fault which can later manifest into a serious decision making failure.

(3). Information Mapping: Available information cues allow the expert sensemakers to develop a map or clusters of similar information stimuli. The purpose of information mapping is to gain a spatial understanding of the strength of relationships in the available information space and to predict the likehood that new information will fit into a known cluster or a group. The mapping process can include link maps, conceptual maps, free body diagrams, decision trees, and semantic diagrams. Mapping information is essentially a classification problem. So, mapping an information object to a wrong cluster can lead to potential faults. For example in the medical domain, a wrong classification of a patient’s symptoms and blood types is an error that may lead to the patient developing unknown sickness (fault) or failure of organs (or possibly death).

(4) Search for Meaning in Information Pattern: Sackman (1991) talks about sensemaking as the mechanisms that organizational members use to attribute meaning to events, such mechanisms include the standards and rules for perceiving, interpreting, believing and acting that are typically used in a given cultural setting (p.33). Meaning is tied to a specific context and dependent on the sequential order of interaction between all the experts involved in the sensemaking (Arnseth and Solheim, 2002). As an epistemological construct, meaning is a subtle, loose, and diverse assignment of definition to a knowledge token, object, or artifact. In this respect, Berkeley (1710) notes that meaning exists in one’s mind, and is often difficult to explain it—an observation that leads to the paradigm that “we know more than we can tell (Polyani, 1966).” Polanyi describes the semantic aspect of tacit knowing, how meaning tends to be displaced away from ourselves, and toward the external. This is observed in the perception of using a tool, in which the meaning of the use of the tool becomes evidenced in the external impact of the tool, not in its immediacy in our hands while using it. Meaning is also realized through the process of how we describe things, objects, events, and so forth—hence, meanings are embedded in language through description (Macdonald, 1995)--

implying that meaning cannot be absolute or objective in the positivist sense (Ambrosini, 1998). Information mapping has no meaning unless it results in pattern recognition. Pattern recognition is the act of taking raw data and making an action based on the “category” of the pattern. A remarkable property of humans (and other living beings) is our ability to recognize patterns. Examples include face recognition and sound recognition. Recognizing patterns in a cluster of grouping also enables such tasks as diagnostics, search, monitoring, and inspection. Thus, the characteristics of the groups or clusters formed by information patterns have some useful knowledge for sensemaking and decision making. Such knowledge is a result of some derived statistical and syntactical properties such as features, error estimations, and grammatical inference. When patterns are irregular in form, or when we can not predict the conditions when and where an information pattern repeats itself, we can encounter gestalt type errors leading to sensemaking failures.

(5). Information Comprehension: Comprehension is a meta-cognition task explicated in the context of tasks. In the sensemaking task, comprehending a situation is synonymous to “being aware” of the situation. It involves developing rules to the cluster or pattern of information with relevance to goals. When we comprehend a situation, in a nominal sense, the abstract frame of reference is concretized. During a comprehension task, “changes in the environment will often be met by an updating of the current schema by a subconscious reaction to cues or a consciously expressed intention (Rasmussen, 1986; pp.151).” The mental aspect of sensemaking is attributed to tacit knowledge---a philosophical view of how individuals view the world based on reflexive mental models and latency lens accorded to some neural processes that control the faculty of human thinking—albeit, the mind. For example, Kelly (1955) defined this phenomenon in terms of personal constructs, an individual’s organization of unique mental models of the world that are both shaped by prior experience and are used to interpret new experiences.

(6). Interpreting Information Relevance to Goals: Interpretation reflects an approximation of individual awareness of the situation in a collective sensemaking setting while ignoring some elements and only partially ascribing meaning to the subset of external knowledge (Leedom, 2005). Signification is a function of how the sensemaker interprets the situation. Interpretation reflects an approximation of individual awareness of the situation while ignoring some elements and only partially ascribing meaning to the subset of external knowledge (Leedom, 2005). Interpretation leads the sensemakers to more focused knowledge required for the formalisms required for intended actions. Feldman (1989) views sensemaking as an interpretive process that is necessary for “organizational members to understand and to share understandings about such features of the organization as what it is about, what it does well and poorly, what the problems it faces are and how it should resolve them.” The act of interpretation may take the form of explicit sensemaking through communication; it may also take place through the transformation and integration of representation of selected information base within the defined context (Suthers, 2005). The key challenge is, however, minimizing the variance in a diversity of meanings accorded the object of interest with its different interpretative viewpoints (Malhotra, 2001). Interpretation leads the sensemakers to more

focused knowledge required for the formalisms required for intended actions. Leedom (2005) observed that “Given the difficulty in externalizing tacit knowledge, these articulations, by nature, reflect only an approximation of each individual’s activated knowledge-ignoring some elements and only partially describing the remainder.’’ The art and science of interpretation is subjective to all forms of subjectivity—opinions, estimates, guess, and so on. In the Hurricane Katrina, people in the Gulf Coast misinterpreted the hurricane forecast by lowering its expected consequences. This misinterpretation can lead to a serious system failure with catastrophic consequences.

(7). Creating a Subset of Situation Understanding: Understanding a situation means that we have a grasp of the relevance knowledge spectrum about the situation. In addition to being situation aware, we also possess meta-cognitive structures that allow us to solve problems that are not familiar—those problems that evolve according system changes, relatively unfamiliar and with novel characteristics. It is surmised here that knowledge is useful only if it can be understood in terms of the implications for action. As complexity, dynamics, or uncertainty increase, the use of the knowledge can become a burdensome and labor intensive process. The principal resource available to the sensemaker for perceiving the situation and understanding it is his or her experience and judgment. If a certain pattern of information has been encountered previously and always represented a clearly defined situation, the sensemaker will likely recognize that pattern and make the connection quickly. Accordingly, Polanyi’s (1958) definition of focal knowledge can be used to infer how individuals in an organization assign meanings to what the see and feel. As echoed by Malhotra (2001), by understanding a situation, we can form the conceptual link between information available and the expected result or anticipation of task outcomes. It could also help us to understand the gap between performance expectations based on information in context (Malhotra, 2001; pp. 120). As complexity, dynamics, or uncertainty increase, the use of the knowledge can become a burdensome and labor intensive process. The principal resource available to the sensemaker for perceiving a situation and understanding it is his or her experience and judgment. If a certain pattern of information has been encountered previously and is always represented in a clearly defined format, the sensemaker will likely recognize that pattern and make the connection quickly. As echoed by Malhorta (2001), by understanding a situation, we can form the conceptual link between information available and the expected result or anticipation of task outcomes. It could also help us to understand the gap between performance expectations based on information in context (Malhorta, 2001; pp. 120). Creating a subset of situation understanding translates into a set of hypotheses about a situation. This leads to the mapping of a problem set to the hypotheses space, and then comparing the outcomes to make conclusions based on cause-effect linkages. Here, various types of faults and failures discussed above are likely to occur. There may be several convolutions of types I and II errors that are often prevalent in statistical test of hypotheses. In the Hurricane Katrina case, we have several cases in which motor homes and several shelters were purchased based on the decision makers creating a substandard set of situation understanding. Subsequently, most of the shelters were not used as intended.

(8). Connecting Understanding to Action: Knowledge transfer is a result of implementing actionable knowledge by enacting framed or scripted focal knowledge on the task requirements. Crothy (1988) note that it is contended that “all knowledge and therefore all meaningful reality as such, is contingent upon human practices, being constructed in account of interaction between human being and their world (pp.42).” Transfer of knowledge to actions can suffer from many faults that include, cognitive mismatch, incongruence alignment due to motor skill, poor situation awareness, and/or lack of adaptation skill to cope with dynamic changes in the system information.

The focal knowledge posited by Polanyi (1966) forms the theoretical basis for describing the enactment of sensemaking process into an actionable knowledge. According to Polanyi focal knowledge is a form of articulated knowledge made explicit through implementation of actions—therefore, resulting in some observable behaviors. In terms of the sensemaking process, we can describe knowledge transfer in one or all of the following ways:

1) knowledge that provides an understanding of the task domain;2) framing strategies based on common recognizable information cues;3) providing a plausible cause-effect explanations to executed actions;4) recognizing the specificity of knowledge, that is, some knowledge is specialized

based on consensus agreement on the way standards are enforced during task performance. (In other words, there is no body of consensus knowledge specific to all tasks. This is echoed by Nonaka (1994 ), that “what makes sense in one context can change or even lose its meaning when communicated to people in a different context”);

5) although some knowledge resources could be transferred from one task domain to another, their efficiency or effectiveness would not be as great as it was before because the context as a whole would be different (Ambrosini, 1988);

6) emphasis on actions enables us to view knowledge as task-driven; this results in the so-called matter dualism that characterize empiricism and rationalism explanations of the sensemaking process (Leedom, 2005);

7) shared and collaborative knowledge is derived for a purpose based on task; the sensemaking should ask, “what is the relevant of the information in this activity?”

5. Summary

Sensemaking is a cognitive task and a human endeavor. Unlike decision making in which a rational process is used to choose a course of action among many alternatives, sensemaking does not enjoy that privilege of dealing with “yes” or “no” choice decisions. Instead, sensemaking deals with systems with asymmetric dimensions—information changing based on contexts, time, and space. These dimensions create mushroom clouds of uncertainty, complexity, and sometimes, chaos. The sensemaker does not have a formal process per se to deal with or cope with extreme cases of information characteristics: a state of information glut that requires organization and filtering in one hand, and in the other, a possible state of nature in which no information is available and must be reconstructed using prototype near-similar cases. In both extremes, the sensemaker travels along the path of the cognitive information hierarchy—from the purely intuitive to purely analytic. This continuum of intuitive-analytical creates a

pervasive complexity for the human endeavor. For example, trying to frame a situation from anecdotal information obtained from HUMINT—such as creating believable set of hypotheses about potential attack by an enemy; or using hindsight and mental models to develop clues or leading indicators obtained from past information footprints.

The interest within the military community is to train sensemaking tasks to the personnel and also to develop sensemaking support tools (FM 3.0) to support real-time sensemaking process. In order to achieve these tasks, we have developed a sensemaking process that accommodates the existing most referenced paradigms and constructs in sensemaking research. Among the various applications of the sensemaking processes are: (a) knowledge representation for constructive modeling and simulation of sensemaking tasks; and (b) real-time capture of individual or group critical thinking skills.

ACKNOWLEDGMENT:This project is supported by ARO Grant # W911NF-04-2-0052 under Battle Center of Excellence initiative. Dr. Celestine Ntuen is the project PI. The opinions presented in this report are not those of ARO and are solely those of the authors.

References

(to be added)