AD-752 '437

EFFECTIVENESS, OF AN ERROR KEY FOR IMAGEINTERPRETATION IN VIETNAM

Harold Martinek, et al

Army Behavi-or and Systems Research Laboratory LArlington, Virginia 4

September 1972

k

U.S EPRMNTO OMEC

5 Pi

cii

IIt-

DISTRIBUTIED BY: 1

3

National Technical Information ServiceU. S. DEPARTMENT OF COMMERCE5285 Port Royal Road, Springfield Va. 22151

Islam .111

0mlTechnical Research Note 250 1D

' EFFECTIVENESS OF AN ERROR KEY FOR IMAGE"INTERPRETATION IN VIETNAM

Harold Martine-.- and Richard E. Hilligoss4I' e• Behavior and Systems Research Laboratoryand -

Bruce HerrinqtonNorth American Rockwell Information Systems Company a

-2C

Rot~podvcee byNATIONAL TECHNICALINFORMATION SERVICE

U S Deportment of Zommerc#Spr;ngf.eld VA 22151

A

U.S. Army

Behavior and Systems Research Laboratory

"Septembor 197i\

Apvwov4d for PIjblic telit8; di.twlbton uwllwmited,

7'

BEHAVIOR AND SYSTEMS RESEARCH LABORATORYZAn activity of the Chief, Research and Development

J. E. UHLANERDirector

DISTRIBUTION: Primary distribution of this report has boen made by BIESAL. Plees" addromcorresporidene concernkig distribution of reports ix: U. S. Army Behavior and Systerns ResearchLaborator. Attn: RDMR-D5L2 13V) Wilson Soulevard. Arlingow, Virginia 22209.

FINAL DISPOSITION: This report may be destroyed when It is no logaw neaded. Plea"e do notreturn It to the Behavior and Systems Resarch Laboratory.

NOTE: leisfindings In thisrepo;rtannt lo be construedas anocfficlal Department of theArmyposition Wade" oo dasi~natd by other authorized documents.

UnclassifiedSecurity Classification ,

DOCUMENT CONTROL DATA- R & D(Security cle'elikellon of title, body of abstract and Indexing annotation must be entered when th( overall report to classified)

I-OIGINA TING ATVTY corpor1ateauthor)h L* t~ lg REPORT SECURITY CLASSIFICATION4• ~Behavior and Systems Research Laboratory, Arlington, Va. UnclassifiedSand North American Rockwell Information Systems Company, -ab. GROUP

Anaheim, California im3. REPORT TITLE

EFFECTIVENESS OF AN ERROR KEY FOR IMAGE INTERPRETATION IN VIETNAM

4. DESCRIPTIVE NOTES (,Je otrpert and Inclusive di•tc) 2k

5. AUTHORISI (P•iet nu.e, miCdde Initial, fact name)

Harold Martinek and Richard E. Hilligoss (BESRL), and Bruce Herrington (NARISCO)

6. REPORT DATE 7a. TOTAL NO. OFr PAGES Tb. NO. or RZFS

September 1972 2Be. CONTRACT OR GRANT NO. Se. ORIGINATOR-S RFEPORf NUMUStR(S)

Technical Research Note 230b. PROJE[CT NO.

2Q662704A721______________________C. 9b. OTHER REPORT NOMSl (Any elet nimbeore "Nt nv be Daaalmed

Advanced Surveillance Systems rep,)£ b-

10. DISTRISUTION STATEMENT

Approved for public release; distribution unlimited.

Si. SUPPLEMENTARY NOTES 12- SPONSORING MILITARY ACTIVITY

Asst Chief of Staff for Intelligence, DA,Wash. D,. C.; Office of Chief of-Researchand Development, DA, Wash., D. C.

3I. ABSTRACT

- Experimentation to develop a specialized reference key to enable interpreters toavoid certain common errors in reporting information derived from imagery taken overVietnam is reiported in the present Technical Research Note. The research study, ajoint effort of personnel of the Behavior and Systems Research Laboratory (BESRL) andof the North American Rockwell Information Systems Company (NARISCO), was undertaken toconstruct and validate a new type of reference -- an error key -- for use in Vietnam.The key was constructed from operational imagery of Vietnam territory with annotationspointing out types of error to be avoided. Evaluation was conducted using trainedinterpreters to determine if use of the error key would achiEve the following effectson image interpreter reports: I) increase number of right responses; 2) dectease num-ber of wrong responses; and 3) increase accuracy of identification.

Performance results achieved by 122 enlisted image interpreters (recent II Coursegraduates) were used in evaluating effectiveness of the error key. Each of two matchedgroups interpreted two sets of imagery, one set using the error key, the other withoutusing the key. Analysis of results showed that 1) the BESRL error key significantlyimproved the accuracy of inexperienced interpreters in dealing with operational imageryon Vietnam--39% increase in rights, 261 decrease in errors made, and 43% increase inaccuracy rate; 2) the error key significantly improved the completeness of reporting onthe subject imagery; and 3) differences in performance as a function of mission andgroup were evident.

ij

DD " 147 REPLACE00 FORM 147o. JAN 64.,,,,,0,,D IN 4,, 473 0SOOLEoZ FOR ARMY USU. Unclassified

Security Claessffcation

UnclassifiedSecurity Classification

S,.. LINK A LINK • LINK CROLE WT ROLE WT ROLE WT

*Image interpretationSurveillance systemsImage character

*Reference key*Error key*Interpreter performance*Operational imagery*Identification*Accuracy rate

Image Interpretation (II) Course*Field utilization - VietnamConventional kEvInterpretation 'reports

Laboratory facilitiesMilitary PsychologyTactical targets

!Z

UnclassifiedSecurity Classlflcation

DD Form 1475 -

13. ABSTRACT - Continued

For optimal operational use, the present key should be extended to I,include different areas of Vietnam, different scales and types of phocog-raphy, and different sensors such as infrared. There is apparent adv.atn-tage to be obtained in use of the key in the field (Vietnam) and for

training in the Image Intetpretation (II) Course.

II

II

A

Technical Research Note 230 -

EFFECTIVENESS OF AN ERROR KEY FOR IMAGEINTERPRETATION IN VIETNAM

Harold Martirek and Richard E. HilligossBehavior and Systems Research Laboratory

arnd

Bruce HerringtonNorth American Rockwell Information Systems Company

SUPPORT SYSTEMS RESEARCH DIVISIONJoseph Zekider, Chief

BEHAVIOR AND SYSTEMS RESEARCH LABORATORY

Office, Chief of Research and DevelopmentDepartment of the Army

1300 Wilson Boulevard, Arlington, Virginia 222091

September 1972

Army Project Number Advanced Surveillance Systems-b20662704A721

Approvod for public refeete; dintdbutiort unlirrited.

---- Mi

BESMI Technical Research Reports and Technical Research Notes are intended forsponsors of R&D tasks and other research and military agencies. Any findingsready for implementation at the time of publication are presented in the latter partS~of the Brief. Upon completion of a major phase of the task, formal recommends-

tions for official action normally are conveyed to appropriate military agenciesby briefing or Disposition Form.

Ii°

i

FOREWORD

The ADVANCED SURVEILLANCE SYSTEMS research program of the Behavior andSystems Research Laboratory has as its objective the production of scientific date bearingon the extract., n cl information from surveillance displays, and the efficient storage, retrieval,and transmissk - of this information within an advanced computerized *mage interpretationfacility. Research results are used in future systems design and in the development of en-haniced techniques for all phases of the interpretation process. Research is conducted underArmy RDT&E Project 2Q662704A721, "Advanced Surveillance -Systems," FY 1972 Wcrk

Program. BESRL research in this area is conducted as an in-house effort augumented bycontracts with organizations selected as having unique capabilities and facilities for researchin aerial surveillance. The present study was conducted jointly by personnel of the Behavior

and Systems Research Laboratory and North American Rockwell Information Systems Company(NARISCO).

The BESRL Work Sub-unit, "Influence of Displays on Image Interpreter Performance".focuses on research to determine how interpreter performance is affected by variations in thecharacter of the image. The present publication reports on experimentation to develop aspecialized reference key to enable interpreters to avoid certain common errors in reportinginformation derived from imagery taken over Vietnam.

J. UHLANER, Direc rBehavior and SystemsResearch Laboratory

EFFECTIVENESS OF AN ERROR KEY FOR IMAGE INTERPRETATION IN VIETNAM

BRIEF

Requirement: ITo construct and validate a new type of reference-an error key-to help image interpreters I

produce more accurate and complete interpretations of operational imagery of Vietnam.

Procedure:

Categories of error to be treated in an error key were determined from the interpreter per-formance of recent enlisted graduates of the Army Image Interpretation Course at Fort Holabird.A key was constructed using operational imagery of Vietnam similar to but not identical withthat used later to evaluate L.e key. Effectiveness of the key was evaluated using results achieved iby 122 enlisted graduates of the above course. Each of two matched groups interpreted twosets of imagery, one set using the error key, the other without using the key. Results were ana-

lyzed for number of correct identifications, number of wrong identifications, and accuracy ofidentification.

Findings:

Practical and significant improvement with use of the error key was found-a 39% increase intne number of right responses, a 26% decrease in the number of errors made, and a 43% increabein accuracy rate.

Utilization of Findings:

Use of the BESHL Error Key can improve significantly the accuracy and completeness ofthe product of school trained but inexperienced U. S. Army interpreters viewing operationalimagery of Vietnam. The key can be used to advantage in the field in Vietnam and for trainingin the Image Interpretation Course.

The technique of error avoidance demonstrated by the key can be extended both ir thefield and in the schooi to other imagery for different regions of the world.

A7

FFFECTIVENESS OF AN ERROR KEY FOR IMAGE INTERPRETATION IN VIETNAM

S= CONTENTS

Page

-BACKGROUND I

PUFWOSE 2

CONSTRUCTION OF ERROR KEY 3

IError Definition 3Error Key Preparation 6

EVALUATION OF ERROR KEY 7

i Sample Used in Evaluating Key 8Experimental Design 8Dependent Variables 8

RESULTS 9

Effect of Error Key 9

CONCLUSIONS 12

SUMMARY OF RESULTS 12

OPERATIONAL IMPLICATIONS 13

APPENDIXES 15

DISTRIBUTION 33

DD FORM 1473 (Document Control Data - R&D) 35

./ij

SM

TABLES Page

Table I. Target types involved in false alarms 4

j 2. Omission errors by mission and target type3. F values of analysis of variance 10

4. Key vs No-Key comparisons - mean rights,wrongs, and accuracy 10

5. Group comparisons - mean rights, wrongs,and accuracy 11

6. Mission comparisons - mean rights, wrongs,and accuracy 11

I

3

]

- - - - -: • : • • •

EFFECTIVENESS OF AN ERROR '£.Y FOR IMAGE INTERPRETATION IN VIETNAM

BACKGROUND

Image interpreters traditionally utilize a wide variety of informaland formal references (keys) in performing their interpretation tasks.Informal references may be notes, sketches, or annotated photographs.Formal references are pictorial depictions of various targets under vari-ous conditions. These photointerpretationL;(PI) keys came into existenceearly in World Waz II as a result of British-sponsored efforts in aerialreconnaissance. In the United States, production of keys began as amatter of expediency under a great deal of time pressure. The earliestkeys, which illustrated industrial facilities of various types, did muchto set the pattern for subsequent keys produced in the United States.As requirements for identification of new target categories evolved,military expediency still ruled out any broad review of photographic in-terpretation key techniques before a given key was prepared for use. Be-cause of extremely diverse subject matter and urgent need, little stan-dardization in format, terminology, or technology was achieved duringthis period.

The war in Korea directed attention to the need for updating and ex-panding keys to cover new targets and target areas. Again, time pressureand expediency dictated many of the ground rules. The period from 1950to 1958 was one of intensified key production, with interest concentratedon production, expeditious update schedules, preplanning of keys and waysof presenting keys. Attention was also given to the formal layout andapproach to be used in preparing keys. Considerable standardization wasachieved at this time through a joint servi=e.production.-program. Most

keys were quite elaborate, giving different views and precise descriptionsof the targets in question and their associated features, although somekeys were quite simple, showing only outline sketches of targets to beidentified and giving their length and width.

The main purpose of a key is to help the interpreter make a moreaccurate and complete identification of objects shown on the imagery.It thus serves as a standard for comparison of a suspect target with aknown target. Keys are particularly useful when image quality is.mir-ginal, when the target to be identified is atypical of its class, or whenthe target is one seldom encountered in normal work. Of course, like anyother reference material, keys also can be used for training purposes.

Since one function of keys is to reduce error in interpreter reports,it was felt that error might be still further reduced by use of an errorkey which, unlike the conventional key, depicts commonly occurring errors.

L'Later termed image interpretation (II) keys to include newer forms ofimagery such as infrared and radar.

The error key explains what the objects which resemble the target really 4are and how they differ from the target. The conventional key makes no

mention of the possibility that other objects may look like certain tar- :Sgets but only discusses attributes of actual targets. The difference be-tween the two keys is illustrated by the following statements aboutweapons positions:

Conventional Key. Two open positions joined by connecting trencheswith clear fields of fire... .

Error Key. The items at (1) are not weapons positions but aregraves with walls surrounding the grave mound with a path fromvillage to grave site. Poor tactical position (such as no fieldof fire) is clearly indicated.

An earlier studyP.', based on World War II imagery, demonstrated thevalue of an error key. Use of an error key resulted in an operationallyuseful and statistically significant decrease in errors in interpretationreports concerning convoys and artillery positions. Since the key usedwas designed to reduce false alarms (reporting a non-military object asone of military significance), its use did not affect the number of rightidentifications made.

An error key approach seemed particularly appropriate to the South-east Asia situation because of the likelihood of confusion because ofthe similarity of military and civilian enemy activities. Moreover, afteran analysis of image interpretation reports of inexperienced image inter-preters in Vietnam, it became apparent that, in addition to a need forreduction of false alarms, common errors of omission (not reporting amilitary target) were also a subject for error reduction.

PURPOSE

The objectives of this effort were to construct and evaluate anerror key for use in Vietnam. The key was to be constructed from opera-tional imagery of Vietnam territory with annotations pointing out typesof error to be avoided. The evaluation was conducted using trained inter-preters to determine if use of the error key would achieve the followingeffects on interpreter reports: 1) increase rights, 2) decrease wrongs,and 3) increase accuracy.

-,Martinek, Harold, and Robert Sadacca. Error keys as reference aIds inimage interpretation. Technical Research Note 153. Behavior andSystems Research Laboratory, Arlington, VA. June 1965.

2-

CONSTRUCTION OF ERROR KEY

In the first phase of construction of the error key, imagery takenover Vietnam territory was interpreted by 50 recent graduates of the en-listed Image Interpretation Course at Fort Holabird, and their interpre-tations were analyzed in order to-define errors. In the second, phase, -imagery selected as representative, based on the analysis of typicalerrors, was assembled into a key and instructions for.key use were pre-'pared. 1Error Definition

To identify erroru typically made by image interpreters, imagery wasselected from two missions. MSN 5536' covers an area of Vietnam located.approximately 40 miles northwest of Saigon. The terrain is mostly flatwith meandering streams. Nearly two-thirds of the area is either woodedor abandoned agricultural land. The remainder is actively cultivated intea and row crops. There are two urban centers. The other mission, I4SN6358, covers an area located 15 to 20 miles southwest of Saigon. Thisarea is intensively cultivated, with rice as the principal crop. Some Jtrees are found in the hamlets and bordering streams. The Yam Go Riverflows through a corner of this area. The approximate scale o-1 both mis-sions is 1:500G. The missions consist of prints in 9" x 18" format.Thirteen prints were used from )IN 5536 and 19 from MSN 6358. Interpre-tation reports were collected following the procedure described inAppendix A, Instructions to the Examinee. The examinees used the sametarget list (Appendix C) and report form (Appendix D) as were later usedin the evaluation experiment. -Each interpreter had available stereoscopes(2X and 4X), and 8X tube magnifiet, and a slide rule. Objects found bythe interpreters were annotated by number in grease pencil directly onthe paper prints. This procedure avoided the possibility of errors inreading and transcribing coordinates. The description of the* object foundwas recorded on a special target reporting form. Each response was re-corded on a master set of imagery to allow .nalysis to determine whatimage charccteristics tended to attract interpreters' attention most often.Tabulation of responses was then analyzed for errors of two types, falsealarms and omissions. Errors were classified, first by false responseand then by the non-military object or terrain fcature believed to havecaused the error.

Table 1 shows the false alarm targets classified into eight categories--weapons positions, vehicles, sampans, etc. Objects giving rise to falsealarms included graves, craters, brush, and trees. The two image setsproduced similar but not identical errors. For example, 10% of the falsealarms from MSN 6358 were reports of mine fields but-na-mine fields werereported on MSN 5536. For MSN 5536, total false alarms were 219 with 87%classifiable; for MSN 6358, there were 336 false alarms with 96% classi-fiable'.

Table 2 presents the omission errors by %ission and target type.The first colum tander each mission presents the number of targets presenitin that mission for each of twelve target types. Comparison of these

-3

Table 1

TARGET TYPES INVOLVED IN FALSE ALAWIS

S: False AlarmTarget Type Missions 5536 Mission 63M8

Weapons position 42% 29%

Vehicles 16 6

Sampans 13 29Supply points 10 19

SPersonnel 10 2

STunnel entrances 4 1

Mine fields 0 10

Bunkers 6

101% lO2

Object Giving Rise to IFalse Alarms

Graves 34% 29

Craters 22 3

Brush and trees (ground) 20 16

Brush and flotsam (rivers) 0 25

Straw 12 26

Wells 12 11

100% 100%••I

S-4

4.c,. 0". 4 .,I 4 .-4 ' a a 0 0 .

0 S- 4' -

coL -4J

'.0 *oj c41 0l CD 04 0n0 Ca Go ato-4 0 4 4

4 c: d d

00

E-44

0"4 0n I-a..-4 ) C4

0* 4 -4 r- 4)r o 1 CtU)0 toc

W i W- IL 1 -A

0 r03 41-

cnccc 4)4 r)ii00 M C

0 p ccz .C 0 (D

1-44 %0 r 0 -' CD %-4 0n C f)cc ru ccCý4 c c

C ~ .-4 ".4 -a . ~C ) .') si 0

4 0 030 0 c

41 G) I0 r. k~ ) ia ' 000 Aj) 0 -4 0 0 C- Sa 4i .

e) co Ci) 0l 0) W. u ci i-a 00. m m m r ) 4 r 0 1co x 13 C w 0-5_-) w "4 1

° M

figures Lidicates a difference between missions in number and kind oftargets present. Column 2 for each mission gives the percentage oferrors of omission for each target type and is an index of the difficultyof finding each target type in this imagery. Column 3 gives the percent- gage of total omissions and is. an index of how much impact this error of -

Somission will have on the total report of the mission. For example, ahigh percentage in column 2 could have little effect if column 1 indi- -cated that only one target was present. The percent omitted values(column 2) were obtained by summing separately for each target the num-ber of omissions for all examinees, and then dividing this syum by theproduct of the frequency of occurrence of the target and the number ofexaminees. The percent of total omissions values (column 3) were obtainedby again summing, separately for each target, the total number of omi~s-sions for all examinees, then dividing the sum by the total number ofomissions for all targets by all examinees.

Based on these results, vehicles, sampans, and personnel were se-lected as the targets to be emphasized in the error key because of theirhigh difficulty (column 2) and high impact on the interpreter's report(column 3). Weapons positions were initially included as a target typeto be considered, but were rejected because of disparity in results be-tween the two missions. Also, it was felt that added emphasis on thistype of target might increase one of the severest problems in falsealarms reporting (Weapons positions was the largest error category in 4

Table 1.)

In addition to the above analyses, high pan imagery from MSNA037,scale 1:8000, covering more rugged terrair with heavy forest cover, wasanalyzed in a similar procedure, although much less completely. Similarbut not identical error types were found and included in the final errorkey, to provide a more general key. However, high pan imagery was notincluded in the evaluation of the key. The additional error types foundwere identified as slow-down obstacles, mud puddles, trails, and ditches.

Error Key Preparation

The error key consists of three pages, two pages with two stereopairs on each and one page with four non-stereo images. The stereo pairsof the error key were also prepared from imagery taken from MSN 5536 andMSN 6358, but did not include imagery frames used in the study. The non-stereo chips were taken from MSN 8037.

From three to nine objects were annotated on each illustration. Abrief description of each annotation was included. A more thorough dis-cussion was presented in an attachment sheet which consisted of no morethan a four-sentence description of an individual annotation. Descrip-tions were presented in two basic formats, a cautionary note type and anot type. These two types of statement were intermingled in the narra-tive as appropriate to provide variety. Below are examples.

"-6-

Cautionary Note. Annotation (1) shows walled graves of a specialtype frequently misidentified as vehicles. The length-width ratio,

square corners, and flat appearance inside the low walls can beSvery deceptive, particularly when a few graves of this type arefound in isolation. Noting the lack of clearly associated trackagetogether with careful measurement wilU Ldeueify these objects asgraves.

Not Statement. The items at (1) are not weapons positions butmound graves which are sometimes found in groups, with the olderoues almost completely overgrown. General diagnostic features ofthe gravos--random arrangement, absence of trackage, size, siting,etc.--should be noted in waking identifications. Annotation (2)indicates not foxholes, but wells of a certain type dug into thedikes between fields. Wells are usually old and regularly spacedalong the entice length of the dike. Foxholes and weapons positionsare usually more numerous and freshly dug and not as regularlyspaced.

The verbal portion of the key was read slowly to the examinees withappropriate pauses to give them sufficient time to study the examplesshowr on the imagery. The examinees then were allowed to keep the errorkey at their work stations to use during the testing. Appendix B contains"he descriptive material of the Error Key.

EVALUATION OF ERROR KEY

The general procedures for the evaluation experimentation was tohave a new sample of interpreters extract information from photographswith and without use of the error key. In addition to the test imagery(which was the same as that used for error definition) and the error key,examinees were given a Southeast Asia target list (Appendix C) and aninterpretation report form.kAppendix D). The target list esteslished in-formation requirements to which the interpreter was to respond. Thefirst section presented a lest of specific items to be annotated on theimagery such as sampans and weapons positions. The second was a Remarkssection requiring reports of three types as appropriate, but no annota-tions. These reports dealt with the approximate number of foxholes,amount of trail activity, and number of road cuts to be found in an in-dividual photo.

The answer sheet provided space for the target entry and associatedremarks in senarate c-lumns. The associated remarks could provide suchinformation as: "Sampan is motorized". "Personnel plowing or gatheringcrops." "Object is one-man foot bridge." Space was also provided foran estimate of confidence attached to each target located, using thelevels Certain, Probable, Possible.

-7-

x

Sample Used in Evaluating Key

The image interpreters used in evaluating the error key were 122enlisted men, recent graduates from the Image Interpretation Course atFort Holabird, Maryland. During their training, they had received morethan eight hours of intensive instruction on Southeast Asia targets.Training included use of a conventional key for interpreting imageryobtained in insurgency and counter-insurgency situations. Additionaltraining on Southeast Asia targets was provided in various parts of thecourse concerned with other topics. For example, in training on theinterpretation of infrared imagery, at least some of the exa'-les pre-sented were taken in Vietnam and showed typical Southeast Asia targets.

.xperimental Design

A 2 x 2 x 2 Latin square design with replication was used, eachexaminee serving as his own control. This design allowed for a relativelysensitive test of the main variable (interpretation with key versus with-out key). The independent variables were two sets of imagery (Missions5536 and 6358), two groups of matched examinees, and the two key condi-tions. Destign is shown below:

V

Day I Day IIWithout Key With Key

Group A MSN 5536 Y-3N 6358

A possible source of confounding occurred because the test with use

of key was always administered after the no-key test. Thus, examineesunder the key condition were exposed to more Vietnamese imagery, havingpreviously been engaged in interpretation under the no-key condition.This effect was believed to be minimal because there was no feedback tothe examinees on their work under the no-key condition and thus littleif any learning. Moreover, different imagery was used for the two Icon-ditions,

Dependent Variables

Rights Score. Number of correct identifications. This score isperfectly correlated with completeness, the latter defined as the number

of right responses divided by the number of targets present in theimagery. Consequently, completeness and number of correct identifica-tions can be used interchangeably in statistical analysis. Completenessof a report is important in that it is a measure of how many targets areunreported (errors of omission), in other words, the "unknowins" a com-mander has to consider.

-8-

Wrongs Score. Number of incorrect identifications (errors ofcommission). This score gives an indication of the number of erroneoustargets an interpreter reports. Wrong interpretations can be of twotypes:

1. Misidentifications where a militarily significant target (truck,tank, etc.) is identified as some other military target.

2. False alarms where a non-military object (brush, logs, a well,a shrine, etc.) is interpreted as a militarily significant target (tank,dugout, AA position, bunker, etc.). Either of these types may cause thecommander using the intelligence information to have a wrong estimate ofenemy capability and possibly make an incorrect commitment.

Accuracy Score. Number of rights divided by rights plus wrongs.This score informs the commander of the number of true targets relativeto the total number reported, and consequently how much faith he canhave in the report.

RESULTS

Effect of Error Key

Rights. The F values in Table 3 show that the mean difference inthe rights scores between using the error key and not using the error keywas significant at the .05 level. Table 4 shows that the total number ofcorrect responses increased from a mean of 25.05 for the no-key group toa mean of 34.88 for the group using the key, an increase of 39%. Thus,use of the error key resulted is a statistically significant and opera-

tionally useful increase in the number of correct targets the interpretersfound--in other words, a useful increase in the completeness of the re-port. The complete analysis of variance tables are given in Appendix E.

Wrongs. Table 3 also shows that the mean difference in the wrongsscore between the key/no-key conditions was significant at the .05 level.From Table 4, the number of incorrect responses decreased from a mean of'45.6 in the no-key group to a mean of 33.8 for the key group, a decreaseof 26%. Thus, not only does use of the error key increase correct re-sponses but it also results in a statistically significant and operation-ally useful decrease in the number of wrong identifications an interpre-ter reports.

Accuracy. The mean difference in accuracy was significant at the.05 level (Table 3). Since use of the error key both increased thenumber of rights and decreased the number of wrongs, it was expected thataccuracy would be substantially greater with the key than without. Infact, mean accuracy increased from 35 to 51 (43%) with use of the key.Use of the error key thus produces a statistically significant and opera-tionally useful increase in the accuracy rate of interpretation.

"-9-

Table 3

F VALUES OF ANALYSIS OF VARIANCE

Rights Wrongs AccuracyF value F value F value

TJKeys 41.549* 17.906* 69.656*

Groups 8.I0* 1.614 0.741

Missions 23.789* 17.906* 3.521

*Significant at .05 level.

Table 4

KEY VS NO-KEY COMPARISONS - MEAN RIGHTS, IWRONGS, AND ACCURACY

Mean Mean MeanRights Wrongs Accuracy

No key 25.0 45.6 35.8

With key 34.9 33.8 50.8

1

I.5

- lO -i

Table 5

GROUP COMPARISONS - MEANZ RIGHTS, WRONGS,"AND ACCURACY

Mean Mean Means W Accuracy

Group A 33.1 42.8 43.7

Group B 26.8 36.6 42.3

Table 6

MISSION COMPARISONS - MEAN RIGHTS, WRONGS,AND ACCURACY

Rights Wrongs Accuracy

MSN 5536 26.2 33.8 43.7

MSN 6358 33.7 45.6 42.5

Group and Mission Effects. Analysis of the effectiveness of theError Key in terms of rights, wrongs, and accuracy, was the ,primaryobjective of the experiment. However, control or. groups and missionfrom which the imagery was taken was included in the analysis to increaseprecision of results. Results of the analysis of groups and missionswere desired as indication of the need for control on these variables infuture experiments. Table 3 shows that the mean difference between groupswas statistically significant only for the Rights score. From Table 5,Group A had 24% more rights than Group B. Future experimentation shouldtherefore use counterbalancing techniques in addition to matching tech-niques for control of examinee differences.

- 11 -

m7 -5

Mean differences between missions were significant at the .05 levelfor right4 and for wrongs, but not significantly different for accuracy(Table 3). From Table 6, MSN 6358 had 28% more rights and 35% morewrongs than MSN 5536. Reports on MSN 6358 had a mean completeness ofonly 6% compared to 23% for MSN 5536. However, these figures must beinterpreted with caution because, although equal time was allowed forinterpretation of the missions, the test imagery from MSN 6538 consiitedof 19 frames with 637 targets whereas only 13 frames with 115 targetswere presented from MSN 5536.

CONCLUSIONS

Although implications of the findings from the present study canhave only limited generalizability for Vietnam because of a multitude ofuncontrolled variables--experience, scale of imagery, type of report,time allowed, etc.--results of a specific study such as this, based onreliable data, are important as guidelines in establishing a baseline fortraining and utilization of interpreters. Average completeness was only7% for the no-key experimental condition, which may be representative ofimage interpreter performance for a novice going to Vietnam directly fromthe Image Interpretation Course and not yet immersed in the specificmilitary situation nor having gained field experience. Even with use ofthe error key, completeness of interpretation was only 10%. The need forimproved operational performance indicated ity these results is reflectedin the establishment of the present on-the-job training programs inVietnam. Unfortunately, this training reduces the productive time inVietnam during the one-year tour of duty.

A second major consideration is the level of accuracy of the reports.Mean accuracy of about 35% was found before use of the key. Even the 51%accuracy achieved with the key means that half the information providedis wrong, leaving considerable rooni for improvement and again indicatingneed for additional training, perhaps on the Job. Training should includeopportunity for feedback to the interpreter so that he can determinewhether or not he was right or wrong in his report and learn by his mis-takes. In this context, it is apparent that there is continued need forimplementation of new team approaches and other interpreter techniques tobring about improvement.

SUMMARY OF RESULTS

The bESRL Error Key significantly improved the accuracy of inexperi-enced interpreters in dealing with operational imagery on Vietnam.

The Error Key significantly improved the completeness of reportingon imagery of Vietnam.

Differences in performance as a function of mission and group wereevident.

-12 -

There is considerable room for improvement in interpreter performanceeven with the error key. n .,

OPERATIONAL IMPLICATIONS

The present key should.,be made operational.

For optimal operatinnal use, the present key should be enlarged toinclude different ireas 3f Vietnam, different scales and types of photog-raphy, and different sensors such as infrared. N

Other geographical areas of potential interest should be consfieredas topics for an error key.

Consideration should be given to incorporating more craining inerror avoidance into the curriculum at the Intelligence School.

Error analysis ehoul '-o,. app'..ed to performance in the image inter-pretation course and on-the-jrZL tr'1ning to help define areas which needimprovement.

13

iI

*: ,

-13-

APPENDIXES

Appendix Pzge

A. Instructions to the Examinee

B; Vie tnam Error Key

C. Southeast Asis Target List 27

D. Interpretation Report Form 29

E. Analysis of Variance for Three DependentVariables in Experiment 31

r• Precedinig page blank

-15

APENDIX A

INSTRUCTIONS TO THE EXAMINEE

A. Today we have a research project using imagery representative of

operational imagery used in Vietnam.

This research, w" ich we call an error study, is different from mostresearch in that we are most interested in what you do wrong. We arehoping that by analysis of the special mistakes and the general types oferrors made by you and the other classes who have worked with thisimagery, we can develop methods for improving II keys.

As you may know, there is a DIA key (Insurgency and Counter-insur-gency in Vietnam) dealing with Vietnamese imagery, and the Ist MI Bn (ARS)made a key for use in Vietnam. We are hoping that, from data determinedby the research we conduct here, we will be able to develop guidelinesfor producing better keys, including supplemental keys in the field, Wehave several studies on scales, the type of perspective, use of drawings,and other aspects of how to put-an.I.I key together. Today's research isconcerned not with how to format the key, but rather what to put in it.By asking you to interpret Vietnamese imagery with relatively littleexperience with the country and insurgency targets, we know you will makemistakes. The most common mistakes are, naturally, the topics which aremost important for inclusion in a key. Thus, we call this an Error Study.By studying the errors which you make, we hope to be able to develop moreefficient and effective keys oriented specifically to avoiding the commonpitfalls in identification of COIN type targets.

B. TARGET LIST

The target list for use in this study is divided into two sections;a list of specific items to annotate, such as sampans, weapons positions,etc., and three types of information to report when appropriate, withoutrequiring annotations. These three items, foxholes, road cuts, and trailactivity, are discussed on the bottom of your target list. After youhave looked at some of this imagery, it will be obvious that for manyareas the counting and reporting of individual foxholes is a very tedioustask, thus, we do not require you to annotate them but just to report theapproximate number on the frame. Road cuts and trails are also founi inquantity and need not be annotated individually.

Urban areas may be ignored completely, as obviously there will betoo many personnel, vehicles, bridge-, etc. to worry about in these areas.

Urban areas where they occur will be designated on your prints.Note that "weapons positions" includes covered and fortified posi-

tions, as well as open pits and firing bays along trenches, etc.

Preceding page blank

-17'

The targets which you do find are to be annotated with wax pencilson the paper prints and reported on the forms provided. The annotationsshould be made by neatly outlining the target or target complex and thennumbering the annotation. Where items occur close together, it may bedesirable to tie the number to the proper annotation with a "tick mark."Do not use large circles for annotations as they may include more targetsthan you report.

C. REPORTING FORK

The reporting form we will use for this test is fairly self-explan-• ~story.

Mission number is the mission number from your envelope of imageryand the copy number--for instance, MSN 5536 Copy 19.

Frame number is indicated in the titling strip on each frame.Please use a separate form for each frame on which you report targets sothat you can go back for additional items.

II name is your name.

Annotation number (col I). You may annotate as you choose, but itwill probably be easier to start over with number 1 on each frame.

Identification (col 2). It is important that you use the terms on•, :the target list.

.quantity (col 3). Indicate quantities of weapons positions, person-

nel, etc. in this column.

Confidence (col_4). Indicate certain, probable, or possible, orsome reasonable abbreviation thereof, for each target reported.

Remarks (col 5). The last column can be used for remarks about thetarget, the annotations, and any other comments you may wish to make.

The bottom three lines are already labeled with foxholes, road cutsand trails. Fill in confidence and quantities, etc., as indicated in thetarget list.

D. INTERPRETATION LOGGING FORMI' In addition to our main research, we are also interested in theamount of time required to work with this type of imagery. We have pre-pared a logging form to record the time as you go along. The most effi-cient way is to tape the form down at the edge of your table. Pleaseenter the clock .time when you start, and each time you pick up a newprint or go back to an old one. Enter the time and the frame number inthe spaces provided. Record also the time when you start breaks and whenyou start to interpret again. From these data we hope to determine anaverage time per frame for various types of imagery and various targetdensities, for JI's of your experience level.

- 18 -

We do ask that you work thoroughly and carefully. In all otherclasses, working with this Vietnamese imagery, the interpreters rushedthrough their interpretations and reported less than 10% of ihe actualtargets present. You will have three hours of working time, plus ahalf-hour break. From time to time, we will tell you how many framesyou should have completed so you can adjust your work pace as necessary.

E. MISSION 5536

This mission covers an area located approximately 40 miles northwestof Saigon. You will have 13 9 x-18 prints of the 'left split of a bplit-vertical mission flown at approximately 1:5,000 scale. You will havethree hours to work with this imagery. You may take breaks as you desire,,but do record the time oft your loggingisheet.

F. MISSION 6358

You will have 19 9 x 18 prints from this split-vertical misslon.The area is located 15 to 20 niles southwest of Saigon and is mostlyagricultural. Scale is approximately l:5;000. You will have three hourqto report tar!ets from this mission, which is ,sufficient time to allow athorough studyof all frames. Take breaks as you see fit, but be sure to

[ log the time.

19I

III I,

I ,

- 19- .1

I

4 7

APPENDX B

SHVIETNAM ERROR KEY - I

TOP Figure

* Annotation (1) shows a special type of walled graves which are frequentlymisidentified as vehicles. The length-width ratio, square corners, andflat appearance inside the low walls can be very deceptive when s fewgraves of this type are found in isolation. Noting the lack of clearlyassociated trackage and careful measurement will indicate these objectsto be graves.

At (2) are some wells of a type ofte. mistaken for weapons positions.The lack of trackage, lack of association with other military activities,and absence of any military reason for being where they are, are oftenthe best features for distinguishing these wells from military emplace-ments.

Personnel are often difficult to detect at this scale (1:5000). Theirmovement between exposure of a stereo pair, coupled with their smallsize, is often the best means of detecting them. Sometimes shadow orlogical location are also helpful clues. At (3) are personnel along aroad and in the courtyard of a building.

The rea annotation (4) indicates some slow-down obstacles on a road.These serve the same function as road cuts but are built up on the topof the road. Note that these items at (4) are not vehicles, even thoughthey have the same general shape and size. Indicators of their truenature are their staggered pattern on opposite sides of the ro3d and themanner in which the traffic marks tend to go around ýhem.

Bottom Figure

This illustration shows cultural activities in more open country. Theitems at (1) are not weapons positions but a type of mound grave whichis sometimes fotind in groups, the older oneb almost completely overgrown.General diagnostic features of graves--random arrangement, absence oftrackage, size, siting, etc.--should be noted in making identifications.

Annotation (2) indicates not foxholes, but a type of well dug ýnto thedikes between fields. Wells are usually old and regularly spaced alongthe entire length of the dike. Foxholes and weapons positions areusually more numerous and freshly dug.

At (3) are wells out in the fields, which illustrate the tendency to becentral to the fields even more clearly than those in the top figure.Non-strategic location, lack of crackage, etc. are further evidencethat these are neither weapons positions nor foxholes.

Preceding page blank- 21 -

Personnel such as those at (4), are always difficult to detect at thisscale.

Carts, as at (5), should be easily detected on roads and somewhat so in

fields. particularly when hitched to oxen and moving. Small size, lackof regular shape w1hen loaded, and lack of tonal contrast with many back-grounds, apparently cause many interpreters to overlook these items.In general, extra care must be taken to avoid missing such small itemsas personnel and carts.

At (6) in red are some mudpuddles which are not vehicles. Note the lackof sharp outline or square corners and in particular the way the pathwaysskirt these spots, indicating that they have been present for some time.

22I

- 22 -

r 7%-~

Top FigureThd rectangular grave mounds at (1) can be distinguished from weapons

pits by the fact that the shadow indicates their mounded nature. Thelack of trackage or weapons positions precludes their beirg fortifica-tions despite appearance of substantial concrete constrtcion. Thenumber and siting are other clues to consider. Note also. how some gravemounds at the very top of the frame are completely overgrown with brushwhich may look like camouflage.

Often an experienced interpreter may mistake a round earth scar for a"weapons position. The irregular size, pattern, and lack of trackage orany distinct shadow help identify the features at (2) as craters, notweapons. positions.

At (3), the mine field pattern results from stacking grain in shocks.- Note the lack of strategic value to the location and the slight irregu-

larity in the size and arrangement of the spots. Considering that thisis 1:5,000 scale imagery, the spots are very large for mines. A

Many things are found in the water other than sampans. Logs, brush, and9garbage float down the streams, rocks protrude in spots, and tree shadowsare frequently present along the margins. At (4) are indicated treeshadows and a log. Actual sampans are usually clearly identifiable whenpresent.

At (5) are found a sampan in the water and one along side the river bank.Why sampans are frequently omitted on II reports is not clearly under-stood.

Personnel such as those at (6) are best identified by their small size,movement between exposures, and association with cultural activitiessuch as houses or crops.

Carts may also be detected by movement or association with transportationroutes or, as in (7), agricultural activities.

At (8), the clustered dark objects are not personnel but bunches of riceseedlings that have been collected together in bunches in the seedbpdready for transfer to the paddy below. Note that these riqe bunches aretoo blocky in appearance to be personnel, cast no noticeable shadow be-cause they are so short, lack the white spot of a coolie hat, show nomotion in stereo, and are too neatly organized to be a bunch of personnel.

At (9), the objects are not vehicles because the "road" shows only trailtraffic. There is a severe road cut at the upper 1/3 of the annotationand the outline of the objects is fuzzy compared to the sharp outline ofthe graves at (1). The objects at (9) are bushes commonly found on thebroader, higher dikes.

- 23 -

% • ,•.•,-- . • ' I '• •' r' -•'• ''•fi•'ff • •'' " - "-'.. = - - ' --- "- - -.. .

6tBottom Figure j

Th- items at (I) are not weapons positions but rather grave moundssimilar to those in the upper stereo pair, and the same indicatorsapply. This is another example of the serious error of calling a grave aa weapons position.

The shrines at (2) are larger and more elaborately constructed than mostgraves but lack the trackage, field of fire, or strategic location of aweapons position.

Tree shadows and logs such as at (3) are often mistaken for sampans butlack the size and clarity of outline of a true sampan. Usually, thetree casting the shadow can be found.

At (4) are personnel distinguished by their size and location on trailsor !.n the fields.

The sampans at (5) are clearly sampans, in contrast to those pseudo-sampans at (3), but have nevertheless been omitted in the reports ofsome interpreters. Note that sampans do occur on land where they arepulled up for maintenance or storage.

The objects at (6) are neither vehicles nor personnel. By comparisonwith the houses nearby, they are too small to be other than little carts,but too big and blocky to be personnel. Noting the similar regularlyspaced pattern at the top side of the same paddy confirms their identifi-cation as strawstacks.

The items at'(7) are too tall to be vehicles and too blocky and lackingin movement to be personnel. They are rather neatly aligned to bebushes and rather tall and spindly to be strawstacks. Perhaps they hadbest be called Unidentified objects (UIO's).

- 24 -

VIETNAM ERROR KEY-3

Top Left Figure

This photo, at 1:8000 scale, illustrates graves and craters, two typesof circular objects which inexperienced interpreters often call foxholesand weapons positions. At (1), the quantity and highly random arrange-ment of the 'revetted' walled entrances protruding from the circle is aprominent feature with this type of grave.

At (2), the random size and arrangement, spoil, and lack of symmetryidentify these objects as craters. When weapons positions or foxholesdo occur in and around cratered areas, great care must be taken todistinguish between the two types of holes in the ground.

Vietnamese traila (3) often look very much like roads and do fulfillmany of the functions of roads, even though much narrower. However,their carrying capacity is limited and if considered a road and used asan indicator of scale, gross errors may result. Compare these trailswith the tank tracks in upper center.

Shrines such as at (4) are found infrequently. They are distinguishedfrom fortifications by the lack of defenses other than the walls them-selves, and the absence of military clutter.

Annotation (5) indicates personnel on a trail. Note that if the trailwas erroneously considered a road, the personnel might be misidentifiedas vehicles.

Top Right Figure

Graves such as those at (1) may occur in large or small groups. Inlarge groups they are readily distinguislted from military positions bytheir random pattern, even if individual graves do look like AA, AW, orartillery positions. In smaller groups, proper identification may takemore care, and even masses of graves should be checked for militarypositions which may be hidden amongst them.

Size is the most important factor in identifying annotation (2) as atrail. Compare its idth with the width of a dike in a rice paddy;such dikes are rarely more .han 24'" wide.

Size is also the most significant feature in identifying the brush and/or grass hummocks at (3). After the trail had been identified as a roadby several inexperienced interpreters, these objects were called aconvoy of vehicles along the road, when actually they are less than twofeet wide.

-25-

Bottom Left Figure

In this illustration are numerous craters which have been misinterpretedas other military activity. At (I), air-burst bombs have left large

circular scars with scalloped edges and no noticeable pit. Particularlyiiv more heavily vegetated areas, such craters have been mistaken for

assembly points or supply points when the radial marks of the thrown-outspoil were thought to be trails.

At (2) we have shell craters which are distinctive because of theirappearance like an edge view of a flying saucer. This is because themain bursting force goes to either side with a secondary force forwardand very little force extending back toward the weapon. These shell

holes are misidentified as foxholes or weapons positions when the lateralextensions are thought to be access trails.

Comparison with the tank tracks at (4) or the shell holes and bomb cratersindicates that the trace at (3) is far too narrow to be a road and thusmust be a trail.

Bottom Right Figure

Many civilian features are misidentified as military.

The round-walled graves at (1) look very much like AA positions but lackthe proper arrangement, spacing, conmunications trenches, ammo revetments,and fire control center which would be associated with a true AA position.

Rectangular or square walled shrine/graves such as those at (2) may looklike fortifications but have their own characteristic appearance, differ-ent from the characteristic military installation with its firing bays,trenches, supplies, strategic location, etc.

At (3) is a temple complex which has the appearance of parade grounds,

barracks, and other components of a military installation. However, themanicured gardens, sculptured ponds, complete absence of defenses or

military clutter, and the large number of graves surrounding the complexindicate its non-military nature.

The clumps of bushes at (4) are not vehicles. Look closely to note theindividual bushes and do not be misled by the length-width ratio andsquare corners into thinking that they are vehicles. Note that theseobjects leave no trackage.

The drainage ditch or ravine at (6) is not a trench. It is too broad,irregular in outline, and mottled in texture, Compare with (5) whichis a trench.

- 26 -

APPENDIX C!

SEA TARGET LIST

WEAPONS POSITIONS - Designate type if rossible, including AA, AW,artillery, mortar, etc. and indicate whether ornot occupied.

BUNKERS - Buildings of heavy construction or otherwise for-tified, not to include rooms excavated intohillsides or fortified weapons positions.

TUNNEL ENTRANCES - Includes rooms dug into hillsides.

TRENCHES - Include only military trenches (firing or commu-S nication), not canals or irrigation ditches.

FOOT/BICYCLE BRIDGES - Do not report vehicle bridges.

SROAD BLOCKS - Built-up barricades only; do not include road cuts.

STRONG POINTS - Indicate number of weapons positions and whetheror not occupied.

TRIANGULAR FORT - Indicate condition--occupied, unoccupied, aban-i doned,, etc.

PERSONNEL - Indicate quantity and what they are doing.

SAMPANS - Designate motorized or non-motorized, if possible.

VEHICLha - Indicate type--ox-carts, trucks, pedicabs, etc.

SUPPLY POINT - Includes both storage and distribution/trans-shipment types. Indicate occupied/unoccupied.

Fill in the following information on the bottom portion of your answersheet. It is not necessary to make annotations for these items,

FOXHOLES - Estimate the number on the entire frame, includ-

ing overlaps with previous frames.

ROAD CUTS - Count the number of sections of road containingroad cuts of any type. Do not count individualpits or ditches.

TRAILS - Estimate the number of trails and indicate theirapparent usage--to and from fields, between vil-lages, no apparent reason for existence.

CONFIDENCE - C - CertainProb - ProbablePoss - Possible

"-27 j

•_ I

APPENDIX D

INTERPRETATION REPORT FORM

MISSION NO.

FRAME NO. II NAE•E

AI2 5O4 T26 7ANNOT.

NO. IDENTIFICATION gTY CONF REMARKS,

Roadcuts

Trails

_______ Foxholes________

PREVIOUS EDITIONS OF THIS FORM ARE OBSOLETE

-29-

APPENDIX E

ANALYSIS OF VARIANCE FOR THREE DEPENDENTVARIABLES IN EXPERIMENT

Source of Variation Sum of Squares DF MSS F

RIGHTS

Between Groups 610.63 1 610.63 8.104*Person Within Group 9041.80 120 75.35

9652.43 121

Within:Keys 1475.41 1 1475.41 41.549*Missions 844.73 1 84 4 .75 23.789*Error 4261.36 120 35.51

6581.50 122

Total 16,233.92 243 IWRONGS

Between Groups 579.41 1 579.41 1.614Person Within Group 43,069.05 120

4,, 648.46 121

Within:Keys 2124.59 1 2124:59 17.906*Missions 2124.59 1 21A4.59 17.906*Error 14,257.82 120 118.65

18,487.00 122

Total 62,135.46 243

ACCURACY

Between Groups 443.61 1 443.ql 0.741Person Within Group 71,855.18 120 598.79

72,298.79 121

Within:Keys 12,393.92 1 12,393.92 69.656*Missions 626.56 1 626.56 3.521Error 21,351.51 120 177.93

34,372.00 122

Total 106,670.79 243

"F.05(I,120) = 3.S2.

-31