JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012) 167

rganic Persistent Intelligence, Surveillance, and Reconnais-sance (OPISR) is a visionary, game-changing approach to

intelligence, surveillance, and reconnaissance. OPISR is able to significantly reduce the time required to obtain and distribute rel-

evant intelligence to the frontline war fighter. OPISR is a novel combination of distributed image processing, information management, and control algorithms that enable real-time, autonomous coordination between ad hoc coalitions of autonomous unmanned vehicles, unattended ground sensors, and frontline users. In September 2011, a proto-type OPISR system was demonstrated with more than 12 unmanned vehicles and unattended ground sensors supporting three users.

Organic Persistent Intelligence, Surveillance, and Reconnaissance

David H. Scheidt

war­fighters­ to­directly­coordinate­on­ tactical­decisions­by­ using­ modern­ communications­ equipment­ (during­World­War­II,­this­was­radio).­Allied­forces­were­forbid-den­to­use­radio­because­ it­“wasn’t­ secure,”­and­Allied­maneuver­decisions­were­made­by­generals­at­headquar-ters­and­based­on­hand-couriered­reports.­German­deci-sions­ were­ made­ on-the-fly­ by­ Panzer­ III­ commanders­and­ Ju­87­ (Stuka)­pilots­ conversing­over­ the­ radio.­By­the­time­the­French­commanders­met­to­decide­what­to­do­about­the­German­advance,­Rommel­and­Guderian’s­Panzers­had­traveled­more­than­200­miles­and­reached­the­English­Channel.­

As­ demonstrated­ repeatedly­ in­ military­ history,­including­ the­ German­ advance­ in­ 1940,­ the­ speed­ at­

INTRODUCTIONIn­the­spring­of­1940,­the­combined­French,­British,­

Dutch,­and­Belgian­ forces­outnumbered­ their­German­counterparts­ in­ troops,­ mechanized­ equipment,­ tanks,­fighter­ planes,­ and­ bombers.­ The­ Me­ 109E­ German­fighter­ aircraft­ was­ roughly­ equivalent­ to­ the­ British­Spitfire,­and­the­French­Char­B1­tank­was­superior­to­the­German­Panzer­III.­In­addition,­the­Allies­were­fight-ing­ on­ their­ home­ soil,­ which­ greatly­ simplified­ their­logistics.­Yet­in­less­than­6­weeks,­the­Belgians,­Dutch,­and­French­surrendered­to­the­Germans,­and­the­British­retreated­across­the­English­Channel.­Even­though­the­Allies­had­superior­equipment­and­larger­forces,­they­were­defeated­by­the­Germans,­who­employed­Auftragstaktik,­a­command­and­control­technique­that­enabled­“edge”­

D. H. SCHEIDT

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012)168

such­ as­ the­ iRobot­ Packbot­ and­ the­ AeroVironment­Raven­ are­ essential­ equipment­ for­ the­ modern­ war-fighter.­The­agility­provided­by­field-deployable­vehicles­comes­ at­ a­ cost—the­ use­ of­ field-deployable­ units­increases­ logistics­ and­ workload­ demands­ on­ frontline­forces.­When­compared­with­larger­unmanned­vehicles,­field­deployable­units­offer­limited­sensing­and­time-on-target­ capabilities.­ Medium-sized­ vehicles,­ such­ as­ the­Insitu­ ScanEagle­ and­ AAI­ Shadow,­ offer­ longer­ time­on­ station­ and­ more­ capable­ payloads.­ Medium-sized­vehicles­also­do­not­make­logistics­or­workload­demands­on­the­edge­war­fighter.­Large­unmanned­vehicles­such­as­the­General­Atomics­Predator­and­Northrop­Grumman­Global­ Hawk­ offer­ still­ more­ capable­ payloads­ and­increased­time­on­station­and­also­do­not­increase­edge­war­fighter­logistics­or­workload.­However,­providing­the­edge­ war­fighter­ timely­ access­ to­ intelligence­ products­produced­by­medium­and­ large­vehicles­ is­ a­ challenge­because­ medium-­ and­ large-sized­ unmanned­ vehicles­produce­ massive­ amounts­ of­ data­ that­ are­ difficult­ to­process­ and­ disseminate­ from­ centralized­ command­posts.­In­fact,­Ariel­Bleicher­reported:

In­2009­alone,­the­U.S.­Air­Force­shot­24­years’­worth­of­video­ over­ Iraq­ and­ Afghanistan­ using­ spy­ drones.­ The­trouble­is,­there­aren’t­enough­human­eyes­to­watch­it­all.­The­deluge­of­video­data­from­these­unmanned­aerial­vehi-cles,­or­UAVs,­is­likely­to­get­worse.­By­next­year,­a­single­new­Reaper­drone­will­record­10­video­feeds­at­once,­and­the­Air­Force­plans­to­eventually­upgrade­that­number­to­65.­John­Rush,­chief­of­the­Intelligence,­Surveillance­and­Reconnaissance­Division­of­the­U.S.­National­Geospatial-Intelligence­Agency,­projects­that­it­would­take­an­unten-able­16,000­analysts­to­study­the­video­footage­from­UAVs­and­other­airborne­surveillance­systems.”3­

If­intelligence,­surveillance,­and­reconnaissance­(ISR)­information­ from­ medium-­ and­ large-scale­ unmanned­vehicles­ could­ be­ processed­ and­ distributed­ in­ time,­these­vehicles­could­potentially­provide­significant­ISR­capability­to­the­edge­war­fighter.­For­the­edge­war­fighter­to­ take­advantage­of­ the­ ISR­capability­ represented­by­these­ assets,­ information­ relevant­ to­ that­ specific­ war-fighter­ must­ be­ gleaned­ from­ the­ mass­ of­ information­available­and­then­must­be­presented­to­the­war­fighter­in­a­ timely­manner.­This­presents­a­challenge­because­crews­analyzing­UAV­payload­data­(far­fewer­than­Rush’s­16,000­analysts)­are­not­apprised­of­the­changing­tactical­needs­of­all­war­fighters,­nor­do­the­war­fighters­have­the­access­or­the­time­required­to­select­and­access­data­from­UAV­sources.­Currently,­operations­centers­are­used­to­gather­and­disseminate­information­from­persistent­ISR­assets.­ This­ centralized­ information­ management­ pro-cess­ introduces­ a­ delay­ between­ the­ observation­ and­transmission­to­the­war­fighter,­which­reduces­force­agil-ity­and­operational­effectiveness.­Although­U.S.­soldiers­are­empowered­to­operate­on­command­by­intent,­their­ISR­ systems­ are­ all­ too­ frequently­ centralized­ systems­reminiscent­of­the­French­command­structure.­For­U.S.­

which­ battlefield­ decisions­ are­ made­ can­ be­ a­ decid-ing­factor­in­the­battle.­A­process­model­that­describes­military­ command­ and­ control­ is­ the­ observe,­ orient,­decide,­ act­ (OODA)­ loop­ described­ by­ Boyd­ (Fig.­ 1).1­Boyd­ shows­ that,­ in­ military­ engagements,­ the­ side­that­ can­ “get­ inside­ the­ opponent’s­ OODA­ loop”­ by­more­rapidly­completing­the­OODA­cycle­has­a­distinct­advantage.­In­their­influential­study­“Power­to­the­Edge:­Command­ .­ .­ .­ Control­ .­ .­ .­ in­ the­ Information­ Age,”­Alberts­and­Hayes2­use­ the­ term­agility­ to­describe­an­organization’s­ability­to­rapidly­respond­to­changing­bat-tlefield­ conditions.­ Modern­ warfare­ case­ studies,­ such­as­ studies­ of­ the­ Chechens­ against­ the­ Russians,­ and­studies­of­not-so-modern­warfare,­ such­as­Napoleon­at­Ulm,­ indicate­ that­agile­organizations­enjoy­a­decisive­military­advantage.­Alberts­and­Hayes­point­out­that­a­common­ feature­of­ agile­organizations­ is­ an­empower-ment­of­frontline­forces,­referred­to­as­edge­war­fighters.­Commanders­facilitate­organizational­agility­by­exercis-ing­“command­by­intent,”­through­which­commanders­provide­abstract­goals­and­objectives­to­edge­war­fighters­who­ then­make­ independent­decisions­based­on­ these­goals­and­their­own­battlefield­awareness.­This­empow-erment­of­edge­war­fighters­reduces­the­OODA­loop­at­the­point­of­attack,­providing­the­desired­agility.

A­ distinguishing­ characteristic­ of­ the­ conflicts­ in­Afghanistan­and­Iraq­is­the­explosive­growth­in­the­use­of­ unmanned­ vehicles.­ Between­ the­ first­ and­ second­Gulf­ wars,­ unmanned­ vehicles­ transitioned­ from­ a­novelty­item­to­an­indispensable­component­of­the­U.S.­military.­ Field-deployable­ organic­ unmanned­ vehicles­

Observe

OrientAct

Decide

Figure 1. Boyd’s OODA cycle models the military decision-making process. Military organizations that perform their OODA cycle more rapidly than opponents gain a substantial competitive advantage.

ORGANIC PERSISTENT INTELLIGENCE, SURVEILLANCE, AND RECONNAISSANCE

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012) 169

to­ impacted­ soldiers­ in­ real­ time.­OPISR­ is­ capable­of­rapidly­ managing­ large,­ complex,­ dynamic­ situations­because­ it­ uses­ a­ decentralized,­ ad­ hoc­ organizational­structure.­Systems­that­use­decentralized­structures­such­as­OPISR­are­known­to­be­more­effective­at­the­timely­coordination­of­complex­systems.­An­explanation­of­why­decentralized­command­and­control­systems­can­provide­more­rapid­response­than­centralized­systems­is­found­in­“On­Optimizing­Command­and­Control­Structures”4­by­Scheidt­and­Schultz.­OPISR­tracks­the­location­and­ISR­needs­of­all­Blue­forces,­maintaining­a­contextual­aware-ness­of­the­war­fighter’s­current­tactical­needs.­

As­ relevant­ tactical­ information­becomes­ available,­OPISR­presents­it­directly­to­the­war­fighter­through­an­intuitive­ handheld­ device.­ The­ information­ require-ments­that­are­used­to­determine­information­relevance­are­defined­by­ the­war­fighter­ through­ the­ same­hand-held­ interface.­This­ interface­ supports­ abstract­queries­such­as­ (i)­patrol­ these­ roads;­ (ii)­ search­ this­ area;­ (iii)­provide­imagery­of­a­specific­location;­(iv)­track­all­tar-gets­of­a­specific­class­on­a­specific­route­or­location;­or­(v)­alert­me­whenever­a­threat­is­identified­within­a­cer-tain­distance­of­my­location.­Information­that­matches­these­ queries­ is­ sent­ by­ the­ system­ to­ the­ handheld­

forces­to­become­fully­agile,­the­ISR­systems­supporting­U.S.­soldiers­must­be­as­agile­as­the­soldiers­they­support,­and­ APL­ has­ developed­ a­ system­ to­ do­ just­ this.­ This­system­ is­ Organic­ Persistent­ Intelligence,­ Surveillance,­and­Reconnaissance­(OPISR),­an­ISR­infrastructure­that­provides­the­rapid­response­of­organic­ISR­systems­with­the­breadth­and­staying­power­of­persistent­ISR­systems.

THE OPISR SYSTEMOPISR­is­a­software­and­communications­subsystem­

that,­when­added­to­an­ISR­asset­such­as­an­unmanned­vehicle­or­unattended­sensor,­supports­the­rapid,­autono-mous­movement­of­ information­across­a­ tactical­ force.­As­shown­in­Fig.­2,­war­fighters­interact­with­OPISR­as­a­system.­When­using­OPISR,­war­fighters­connect­into­the­OPISR­“cloud,”­task­OPISR­with­mission-level­ISR­needs,­ and­ are­ subsequently­ provided­ with­ the­ intelli-gence­ they­ need.­ This­ capability­ provides­ intelligence­directly­ to­ the­ war­fighter­ without­ requiring­ the­ war-fighter­ to­ personally­ direct,­ or­ even­ know­ about,­ the­OPISR­ assets­ gathering­ the­ information.­ OPISR­ is­autonomous.­ As­ a­ system,­ OPISR­ seeks­ out­ relevant­information,­ pushing­ key­ tactical­ information­ directly­

Figure 2. OPISR’s concept of operation allows UAVs of various sizes to communicate with each other, with users, and with commanders through an ad hoc, asynchronous cloud. The cloud communicates goals from users to vehicles and sensor observations from users to vehicles. Comms, communications.

D. H. SCHEIDT

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012)170

pilot­ capable­ of­ providing­ stable­ flight­ can­ be­ modi-fied­to­become­autonomous­vehicles­by­connecting­the­autopilot­to­the­OPISR­processor.­When­the­vehicle­is­operating­ autonomously,­ the­ autopilot­ sends­ guidance­and­control­(GNC)­telemetry­ to­ the­OPISR­processor.­The­processor­uses­the­GNC­data­to­devise­a­continual­stream­ of­ waypoints­ that­ are­ sent­ to­ the­ autopilot­ to­follow.­The­OPISR­processor­also­uses­the­GNC­telem-etry­ to­ produce­ metadata­ that­ are­ associated­ with­ the­sensor­data.­The­combined­sensor­data­and­metadata­are­then­used­by­the­OPISR­system­as­a­whole.­

In­ service,­ unmanned­ vehicles­ frequently­ use­ sepa-rate­communications­channels­for­control­and­imagery.­Because­OPISR­devices­perform­both­image­processing­and­control­on­board­the­device,­these­communications­channels,­ and­ the­ traditional­pilot­ ground­ station,­ are­no­longer­required.­In­effect,­OPISR­devices­are­capable­of­operating­fully­independently­of­direct­human­super-vision.­Note­that­OPISR­devices­are­still­responding­to­war­fighter­ requests;­however,­ these­devices­accomplish­this­without­requiring­continuous­communications­with­the­ war­fighter­ being­ serviced.­ Although­ OPISR­ does­not­ require­ traditional­ control­ and­ payload­ communi-cations,­ OPISR­ devices­ do­ support­ these­ legacy­ capa-bilities.­Because­the­OPISR­nodes­communicate­over­a­separate­channel,­OPISR­functionality­may­be­provided­in­ tandem­with­ traditional­control.­This­ is­ in­keeping­with­the­OPISR­dictum­that­OPISR­be­an­entirely­addi-tive­capability;­unmanned­vehicle­owners­lose­no­func-tionality­ by­ adding­ OPISR.­ However,­ OPISR­ vehicles­are­responsive­to­commands­from­human­operators­and­will,­ at­any­ time,­allow­an­authorized­human­operator­to­override­OPISR­processor­decisions.­Likewise,­legacy­consumers­of­information­will­still­receive­their­analog­data­streams.­Note­that­even­when­the­OPISR­processor­is­denied­control­by­the­UAV­pilot,­the­OPISR­system­will­ continue­ to­ share­ information­ directly­ with­ edge­war­fighters­as­appropriate.­

device.­ The­ handheld­ interface­provides­ a­ map­ of­ the­ surround-ing­ area­ that­ displays­ real-time­tracks­ and­ detections­ and­ imag-ery­metadata.­The­ imagery­meta-data­ describes,­ at­ a­ glance,­ the­imagery­ available­ from­ the­ sur-rounding­ area.­ OPISR-enabled­vehicles­ are­ autonomous—if­ the­information­ required­ by­ the­ war-fighter­ is­ not­ available­ at­ the­time­ the­ query­ is­ made,­ OPISR­unmanned­vehicles­autonomously­relocate­ so­ that­ their­ sensors­can­obtain­ the­ required­ information.­OPISR-enabled­ unmanned­ vehi-cles­ support­ multiple­ war­fighters­simultaneously,­with­vehicles­self-organizing­to­define­joint­courses­of­action­that­satisfy­the­information­requirements­of­all­war­fighters.­

Because­war­fighters­are­required­to­operate­in­harsh,­failure-prone­ conditions,­ OPISR­ was­ designed­ to­ be­extremely­ robust­ and­ fault­ tolerant.­ OPISR’s­ designers­viewed­ communications­ opportunistically,­ designing­the­system­to­take­advantage­of­communications­chan-nels­ when­ available­ but­ making­ sure­ to­ avoid­ any/all­dependencies­on­continuous­high-quality­ service­com-munications.­Accordingly,­all­OPISR­devices­are­capa-ble­of­operating­independently­as­standalone­systems­or­as­ad­hoc­coalitions­of­devices.­When­an­OPISR­device­is­capable­of­communicating­with­other­devices,­it­will­exchange­information­through­networked­communica-tions­and­thereby­improve­the­effectiveness­of­the­system­as­a­whole.­However,­if­communications­are­unavailable,­each­device­will­continue­to­perform­previously­identi-fied­tasks.­When­multiple­devices­are­operating­in­the­same­area,­they­will­self-organize­to­efficiently­perform­whatever­ tasks­war­fighters­have­ requested­ (devices­ are­capable­ of­ coordinating­ their­ efforts­ even­ when­ they­cannot­communicate­over­a­network­if­they­are­capable­of­observing­each­other­through­their­organic­sensors).­Task­prioritization­is­based­on­war­fighter­authority­and­war­fighter-assigned­importance.­

OPISR HardwareOff-the-shelf­ unmanned­ vehicles­ and­ unattended­

sensors­ can­ be­ incorporated­ into­ the­ OPISR­ system­by­adding­the­OPISR­payload.­As­shown­in­Fig.­3,­the­OPISR­payload­consists­of­three­hardware­components:­an­ OPISR­ processor­ that­ executes­ the­ OPISR­ soft-ware,­an­OPISR­radio­that­provides­communications­to­other­OPISR­nodes­ including­OPISR’s­handheld­inter-face­ devices,­ and­ an­ analog-to-digital­ converter­ that­is­ used­ to­ convert­ payload­ sensor­ signals­ into­ digital­form.­Unmanned­vehicles­ that­have­an­onboard­auto-

OPISRpayloadAnalog-to-digital

converterPayload sensors

GNC sensors

Actuators

Payload comms

Traditionalground station

Control comms

Autopilot

OPISRground station

Other unmannedvehicle

OPISRprocessor

OPISRmodem

Stockunmannedvehicle

Legacy(not required)

Figure 3. OPISR’s hardware architecture is based on a modular payload that can be fitted onto different types of unmanned vehicles.

ORGANIC PERSISTENT INTELLIGENCE, SURVEILLANCE, AND RECONNAISSANCE

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012) 171

sensors­on­board­the­agent’s­device,­from­other­agents,­or­ from­ pattern­ recognition/data­ fusion­ software­ con-tained­within­the­agent.­The­integrity­of­the­data­stored­on­the­blackboard­is­maintained­by­a­truth­maintenance­system­(TMS).­The­TMS­performs­two­functions.­First,­the­TMS­resolves­conflicts­between­beliefs.­The­simplest­form­ of­ conflict­ resolution­ is­ accomplished­ by­ storing­the­belief­with­the­more­recent­time­stamp.­For­example,­one­belief­might­posit­that­there­is­a­target­at­grid­[x,­y]­at­time­t0,­and­a­second­belief­might­posit­that­there­is­no­target­at­grid­[x,­y]­at­time­t1.­More­sophisticated­con-flict-resolution­algorithms­are­scheduled­to­be­integrated­into­OPISR­in­2012.­The­second­TMS­function­is­ the­efficient­ storage­of­ information­within­the­blackboard.­When­performing­this­task,­the­TMS­caches­the­most­relevant,­timely­information­for­rapid­access­and,­when­long-lived­systems­generate­more­data­than­can­be­man-aged­within­ the­ system,­ the­TMS­ removes­ less­ impor-tant­information­from­the­blackboard.­For­caching­and­removal,­the­importance­of­information­is­defined­by­the­age,­proximity,­uniqueness,­and­operational­relevance.­

Agent Communications ManagerCoordination­ between­ agents­ is­ asynchronous,­

unscheduled,­ and­ completely­ decentralized,­ as­ it­ has­to­ be,­ because­ any­ centralized­ arbiter­ or­ scheduled­communications­ would­ introduce­ dependencies­that­ reduce­ the­ robustness­ and­ fault­ tolerance­ that­is­ paramount­ in­ the­ OPISR­ design.­ Because­ agent­communication­ is­ asynchronous­ and­ unscheduled,­there­ is­ no­ guarantee­ that­ any­ two­ agents­ will­ have­matching­ beliefs­ at­ an­ instance­ of­ time.­ Fortunately,­

OPISR SoftwareOPISR­is­based­on­a­distributed­multiagent­software­

architecture.­Each­software­agent­ serves­as­a­proxy­ for­the­device­on­which­it­is­located,­and­all­devices­within­OPISR­ have­ their­ own­ agents­ including­ unmanned­vehicles,­ unattended­ sensors,­ and­ user­ interfaces.­ As­shown­in­Fig.­4,­each­agent­ is­composed­of­ four­major­software­components:­ a­payload­manager,­which­man-ages­ the­ sensor­ information­ from­ the­ device’s­ organic­sensors;­ a­ distributed­ blackboard,­ which­ serves­ as­ a­repository­ for­ the­ shared­ situational­ awareness­ within­the­ agent­ system;­ an­ agent­ communications­ manager,­which­manages­the­flow­of­information­between­agents;­and­a­cSwarm­controller,­which­determines­a­course­of­action­for­those­devices­that­are­capable­of­autonomous­movement.­All­devices­within­the­system,­including­the­war­fighter’s­handheld­device,­are­peers­within­OPISR.­

Distributed BlackboardIn­ the­ 1980s,­ Nii5,­6­ described­ a­ method­ for­ multi-

agent­ systems­ to­ communicate­ with­ each­ other­ in­ an­asynchronous­manner­called­a­blackboard­system.­Like­their­namesake­in­the­physical­world,­blackboard­systems­allow­agents­to­post­messages­for­peer­agent­consumption­at­an­indeterminate­time.­Each­OPISR­agent­contains­a­personal­blackboard­ system­ that­maintains­ a­model­of­the­agent’s­environment.­Three­types­of­information­are­stored­on­each­agent’s­blackboard:­beliefs,­metadata,­and­raw­data.­Raw­data­are­unprocessed­sensor­data­from­a­sensor­within­the­OPISR­system.­Metadata­is­informa-tion­that­provides­context­to­a­set­of­raw­data­including­sensor­position,­pose,­and­time­of­collection.­Beliefs­are­abstract­ “facts”­ about­ the­current­ situation.­ Beliefs­include­ geospatial­ arti-facts­ such­as­ targets,­Blue­force­ locations,­ or­ search­areas.­ Beliefs­ can­ be­developed­ autonomously­from­ onboard­ pattern­recognition­ software­ or­data­ fusion­ algorithms­ or­asserted­ by­ humans.­ Mis-sion-level­ objectives,­ the­goals­ that­ drive­ OPISR,­are­a­special­class­of­belief­that­must­be­produced­by­a­human.­The­storing­and­retrieval­ of­ information­to­ and­ from­ agent­ black-boards­ is­ performed­ by­the­ blackboard­ manager.­The­ blackboard­ man-ager­ accepts,­ stores,­ and­retrieves­information­from­

Agent communicationsmanager Other agents

Network

Metadataobject

Imageryobject

Payload manager

GNC data

Sensor data

BlackboardmanagerBlackboard

Beliefs

Metadata

Raw data

Truth maintenancesystem

Distributed blackboard cSwarm controller

Fielddatabase

Subbehaviors

Traf�c monitor

Interface controlcomponent Autopilot

DCF

Figure 4. OPISR software architecture contains four major elements: the payload manager gener-ates new “beliefs” from sensor observations, the distributed blackboard maintains situational aware-ness by aggregating beliefs from the on- and offboard observations, the agent communications manager exchanges beliefs between other vehicles and users, and the cSwarm controller devises a course of action based on beliefs. DCF, Dynamic co-fields.

D. H. SCHEIDT

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012)172

• Searching­contiguous­areas­defined­by­war­fighters

• Searching­linear­networks­such­as­roads

• Transiting­to­a­waypoint

• Blue­force­over-watch

• Target­tracking

• Perimeter­patrol

• Information­ exchange­ infrastructure,­ in­ which­unmanned­ vehicles­ maneuver­ to­ form­ a­ network­connection­between­an­information­source,­such­as­an­unattended­ sensor,­ and­war­fighters­ that­ require­information­on­the­source.­Note­that­the­war­fighter­is­ not­ required­ to­ specify­ this­ behavior;­ the­ war-fighter­needs­only­to­specify­the­information­need,­and­the­vehicle(s)­utilize(s)­this­behavior­as­a­means­to­satisfy­the­need.­

• Active­diagnosis,­in­which­vehicles­reduce­uncertain­or­ incomplete­ observations­ through­ their­ organic­sensing­ capabilities.­ For­ example,­ a­ UAV­ with­ a­sensing­capability­that­is­able­to­classify­targets­will­automatically­move­to­and­classify­unclassified­tar-gets­being­tracked­by­a­cooperating­radar.­

In­addition­to­the­mission-level­behaviors­described­above,­OPISR­vehicles­exhibit­certain­attributes­within­all­behaviors.­These­universal­attributes­are:­

• Avoiding­ obstacles­ or­ user-defined­ out-of-bounds­areas

• Responding­ to­ direct­ human­ commands.­ OPISR­unmanned­vehicles­are­designed­to­function­auton-omously­ in­ response­ to­ mission-level­ objectives;­however,­ when­ operators­ provide­ explicit­ flight­instructions,­OPISR­vehicles­always­respond­to­the­human­ commands­ instead­ of­ to­ the­ autonomous­commands.

OPISR EXPERIMENTATIONThe­ current­ OPISR­ system­ is­ the­ culmination­ of­ a­

decade-long­APL­exploration­of­autonomous­unmanned­vehicles.­ Key­ technical­ elements­ of­ the­ OPISR­ system­were­demonstrated­in­hardware-in-the-loop­experiments­as­ early­ as­ 2003.­ More­ than­ 20­ hardware­ demonstra-tions­ have­ included­ the­ following:­ DCF,9­ the­ distrib-uted­ blackboard,10­ delay-tolerant­ communications,11­and­ simultaneous­ support­ for­ multiple­ end­ users.12­ As­successful­ as­ these­experiments­have­been,­APL,­prior­to­ 2011,­ had­ not­ integrated­ the­ full­ suite­ of­ OPISR­capabilities­ described­ in­ this­ article­ on­ a­ large,­ dispa-rate­ set­ of­ vehicles.­ In­ September­ 2011,­ APL­ demon-

the­ control­ algorithms­ used­ by­ OPISR­ are­ robust­ to­belief­ inconsistencies.­ Cross-agent­ truth­ maintenance­is­ designed­ to­ the­ same­ criteria­ as­ agent-to-agent­communications:­ Information­ exchanges­ between­agents­ seek­ to­ maximize­ the­ consistency­ of­ the­ most­important­ information­ but­ do­ not­ require­ absolute­consistency­ between­ agent­ belief­ systems.­ Information­exchange­ between­ agents­ is­ performed­ by­ the­ agent­communications­ manager.­ When­ communications­are­ established­ between­ agents,­ the­ respective­ agent­communications­ managers­ facilitate­ an­ exchange­ of­information­ between­ their­ respective­ blackboards.­When­ limited­bandwidth­and/or­brief­ exchanges­ limit­the­amount­of­information­exchanged­between­agents,­each­agent­communications­manager­uses­an­interface­control­ component­ to­prioritize­ the­ information­ to­be­transmitted.­ Information­ is­ transmitted­ in­ priority­order,­ with­ priority­ being­ determined­ by­ information­class­ (beliefs­ being­ the­ most­ important,­ followed­ by­metadata),­ goal­ association­ (e.g.,­ if­ a­ war­fighter­ has­requested­specific­information,­then­that­information­is­given­priority),­timeliness,­and­uniqueness.

Autonomous Control: cSwarmOPISR’s­ autonomous­ unmanned­ vehicles­ use­ an­

approach­ called­ dynamic­ co-fields­ (DCF),­ also­ known­as­stigmergic­potential­fields,­to­generate­movement­and­control­actions.­DCF­is­a­form­of­potential­field­control.­Potential­ field­ control­ techniques­ generate­ movement­or­trigger­actions­by­associating­an­artificial­field­func-tion­with­geospatial­objects.­In­OPISR,­the­objects­that­are­used­to­derive­fields­are­beliefs.­Fields­represent­some­combination­of­attraction­and/or­repulsion.­By­evaluating­the­fields­for­all­known­beliefs­at­a­vehicle’s­current­loca-tion,­a­gradient­vector­is­produced.­This­gradient­vector­is­then­used­to­dictate­a­movement­decision.­Developed­at­APL­in­2003,7­DCF­extends­an­earlier­potential­field­approach­called­co-fields8­by­making­the­potential­fields­dynamic­with­respect­to­time­and­also­making­vehicle­fields­self-referential.­Self-referential­fields­are­fields­that­induce­vehicle­decisions­that­are­generated­by­the­vehi-cle’s­ own­ presence.­ Adding­ these­ dynamic­ qualities­ is­key­to­managing­two­well-known­problems­with­poten-tial­ field­ approaches:­ namely,­ the­ tendency­ of­ vehicles­to­become­stuck­in­local­minima­and­the­propensity­to­exhibit­undesired­oscillatory­behavior.­As­implemented­in­OPISR,­DCF­is­used­to­cause­specific­behaviors­such­as­ search,­ transit,­or­ track,­as­well­as­behavioral­ selec-tion.­ The­ DCF­ algorithm­ is­ encoded­ in­ the­ cSwarm­software­module.­All­unmanned­vehicles­in­OPISR­exe-cute­cSwarm.­DCF­behaviors­specific­to­unique­classes­of­ vehicle­ are­ produced­ by­ tailoring­ the­ field­ formula,­which­ is­ stored­ in­ a­ database­ within­ cSwarm.­ OPISR­autonomous­unmanned­vehicles­are­capable­of­a­variety­of­behaviors­including:

ORGANIC PERSISTENT INTELLIGENCE, SURVEILLANCE, AND RECONNAISSANCE

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012) 173

(Fig.­5b),­a­Segway­ground­vehicle­(Fig.­6),­custom­APL-developed­surface­vehicles­(Fig.­5c),­and­an­OceanServer­Iver2­undersea­vehicle­ (Fig.­5d).­These­vehicles­used­a­wide­ range­of­payload­ sensors—including­ electro-opti-cal,­passive­acoustic,­side-scan­sonar,­and­lidar­sensors—to­detect,­ classify,­ and­ track­waterborne­vehicles,­ land­vehicles,­dismounts,­and­mine-like­objects.­ISR­tasking­was­generated­by­three­proxy­war­fighters,­two­of­which­were­on­land­and­one­of­which­was­on­the­water.­The­requested­ISR­tasks­required­the­use­of­all­of­the­vehicle­behaviors­previously­described.

FUTURE WORKIn­FY2012,­APL­is­planning­additional­improvements­

to­ the­ OPISR­ system.­ Specific­ OPISR­ improvements­include­ (i)­ the­ integration­ of­ more­ sophisticated­ data­fusion­techniques­ into­the­distributed­blackboard,­spe-cifically­upstream­data­ fusion­and­closed-loop­ ISR,­ (ii)­flight­testing­of­autonomous­behaviors­that­allow­UAVs­to­ form­ network­ bridges­ between­ remote­ sensors­ and­users,­ (iii)­ introduction­ of­ advanced­ simulation-based­test­and­evaluation­techniques,­and­(iv)­the­integration­of­Exec-Spec­into­the­OPISR­framework­and­Exec-Spec­flight­ testing.­Exec-Spec­ is­an­autonomy­system­devel-oped­by­APL­for­spacecraft­control.­In­the­OPISR–Exec-Spec­ integration,­ Exec-Spec­ will­ manage­ vehicle­ fault­management­and­safety­override­functions.­

CONCLUSIONThe­ OPISR­ system­ is­ a­ framework­ that­ provides­

a­ capability­ through­ which­ numerous­ unmanned­

strated­10­OPISR-enabled­vehicles­working­with­ three­unattended­ ground­ sensors­ in­ support­ of­ three­ users.­The­demonstration­ employed­ air,­ ground,­ surface,­ and­undersea­ISR­needs­with­surveillance­being­conducted­under­ the­ water,­ on­ the­ water,­ and­ on­ and­ over­ land.­The­ autonomous­ unmanned­ vehicles­ included­ three­Boeing­ ScanEagles­ (Fig.­ 5a),­ three­ Procerus­ Unicorns­

Figure 5. OPISR vehicles featuring (a) ScanEagle, (b) Unicorn, (c) surface vehicles, and (d) Iver2 undersea vehicle.

Figure 6. OPISR’s ground vehicles were Segway RMP 200s that were fitted with lidar, electro-optic, and passive acoustic sensors.

D. H. SCHEIDT

JOHNS HOPKINS APL TECHNICAL DIGEST, VOLUME 31, NUMBER 2 (2012)174

­ 3Bleicher,­A.,­“The­UAV­Data­Glut,”­IEEE Spectrum,­http://spectrum.ieee.org/robotics/military-robots/the-uav-data-glut­(Oct­2010).

­ 4Scheidt,­D.,­and­Schultz,­K.,­“On­Optimizing­Command­and­Control­Structures,”­in­Proc. 16th­International Command and Control Research and Technology Symp. (ICCRTS),­Quebec­City,­Quebec,­Canada,­pp.­1–12­(2011).­

­ 5Nii,­H.,­“The­Blackboard­Model­of­Problem­Solving­and­the­Evolu-tion­of­Blackboard­Architectures,”­AI Magazine­7(2),­38–53­(1986).

­ 6Nii,­H.,­“Blackboard­Application­Systems,­Blackboard­Systems­and­a­Knowledge­Engineering­Perspective,”­AI Magazine­7(3),­82–107­(1986).

­ 7Scheidt,­ D.,­ Stipes,­ J.,­ and­ Neighoff,­ T.,­ “Cooperating­ Unmanned­Vehicles,”­IEEE International Conf. on Networking, Sensing and Con-trol,­Tucson,­AZ,­pp.­326–331­(2005).

­ 8Mamei,­ M.,­ Zambonelli,­ F.,­ and­ Leonardi,­ L.,“Co-Fields:­ Towards­ a­Unifying­ Approach­ to­ the­ Engineering­ of­ Swarm­ Intelligent­ Sys-tems,”­in­Third International Workshop on Engineering Societies in the Agents World III,­Madrid,­Spain,­pp.­68–81­(2002).

­ 9Chalmers,­R.­W.,­Scheidt,­D.­H.,­Neighoff,­T.­M.,­Witwicki,­S.­J.,­and­Bamberger,­R.­J.,­“Cooperating­Unmanned­Vehicles,”­Proc. AIAA 3rd “Unmanned Unlimited” Technical Conf.,­Chicago,­IL,­paper­no.­AIAA­2004-6252­(2004).

10Hawthorne,­R.­C.,­Neighoff,­T.­M.,­Patrone,­D.­S.,­and­Scheidt,­D.­H.,­“Dynamic­ World­ Modeling­ in­ a­ Swarm­ of­ Heterogeneous­ Autono-mous­Vehicles,”­Proc. Association of Unmanned Vehicle Systems Inter-national,­Anaheim,­CA­(Aug­2004).

11Bamberger,­R.,­Hawthorne,­R.,­and­Farrag,­O.,­“A­Communications­Architecture­ for­ a­ Swarm­ of­ Small­ Unmanned,­ Autonomous­ Air­Vehicles,”­in­Proc. AUVSI’s Unmanned Systems North America Symp.,­Anaheim,­CA­(2004).

12Stipes,­J.,­Scheidt,­D.,­and­Hawthorne,­C.,­“Cooperating­Unmanned­Vehicles,”­ in­ Proc. International Conf. on Robotics and Automation,­Rome,­Italy­(2007).

13U.S.­ Department­ of­ Defense,­ Quadrennial­ Defense­ Review,­ http://www.defense.gov/qdr/­(2010).

platforms­ simultaneously­ provide­ real-time­ actionable­intelligence­ to­ tactical­ units;­ abstract,­ manageable­situational­awareness­to­theater­commanders;­and­high-quality­forensic­data­to­analysts.­APL­has­demonstrated­an­ OPISR­ system­ that­ includes­ a­ distributed,­ self-localizing­ camera­ payload­ that­ provides­ imagery­ and­positional­ metadata­ necessary­ to­ stitch­ information­from­multiple­sources;­a­distributed­collaboration­system­that­is­based­on­robust­ad­hoc­wireless­communications­and­agent-based­data­management;­and­a­user­interface­that­ allows­ users­ to­ receive­ real-time­ stitched­ imagery­from­unmanned­vehicles­and­that­does­not­require­users­to­directly­ control­ (or­ even­expressly­be­ aware­of)­ the­unmanned­vehicles­producing­the­imagery.­OPISR­is­a­bold­vision­that­presents­an­innovative­approach­to­ISR,­an­ important­ enabler­ emphasized­ in­ the­ Quadrennial Defense Review13­and­other­key­policy­documents,­and­gives­the­Laboratory­an­enhanced­ability­to­help­sponsors­address­future­capability­gaps­in­this­critical­area.­

REFERENCES­ 1Boyd,­ J.,­ The Essence of Winning and Losing,­ http://www.belisarius.

com/modern_business_strategy/boyd/essence/eowl_frameset.htm.­ 2Alberts,­D.,­and­Hayes,­R.,­“Power­to­the­Edge:­Command­.­.­.­Con-

trol­ .­ .­ .­ in­ the­ Information­ Age,”­ Information Age Transformation Series,­Washington,­DC:­Command­and­Control­Research­Program­Press­(2003).

David H. Scheidt­is­a­member­of­the­APL­Principal­Professional­Staff­in­the­Research­and­Exploratory­Development­Department.­His­current­work­concentrates­on­the­research­and­development­of­distributed­intelligent­control­systems.­Before­coming­to­APL,­he­spent­14­years­in­industry­performing­and­supervising­the­development­of­information­and­control­systems­including­a­metadatabase­containing­approximately­1000­heterogeneous­databases,­statewide­immunol-ogy­tracking,­locomotive­control,­railroad­dispatching,­and­distributed­multilevel­secure­information­systems.­His­e-mail­address­is­david.scheidt@jhuapl.edu.

The Author

The­Johns Hopkins APL Technical Digest­can­be­accessed­electronically­at­www.jhuapl.edu/techdigest.