devs agents to support conformance testing of emerging defense information standards
DESCRIPTION
DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards. Bernard P. Zeigler, Arizona Center for Integrative Modeling and Simulation Tucson Arizona www.acims.arizona.edu and Joint Interoperability Test Command (JITC) Fort Huachuca, Arizona. Outline. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/1.jpg)
DEVS Agents to Support Conformance Testing of Emerging
Defense Information Standards
Bernard P. Zeigler,Arizona Center for Integrative Modeling and Simulation
Tucson Arizonawww.acims.arizona.edu
andJoint Interoperability Test Command
(JITC)Fort Huachuca, Arizona
![Page 2: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/2.jpg)
Outline• “Agent-supported simulation deals with the use of agents as a support
facility to enable computer assistance in problem solving or enhancing cognitive capabilities.” (ADS’06 definition)
• The agent-supported simulation metaphor applies to testing the conformance of multi-agent systems to complex defense information standards
• Structure control agents induce structural change in themselves or others to effectuate different behaviors under different circumstances
• Structure control implemented in Dynamic Structure DEVS enables automation of standards conformance testing
• DEVS formalism is capable of capturing the information-processing complexities, including the dynamics, underlying the MIL-STD-6016C standard
• DEVS modeling and simulation methodology is attaining core-technology for automated testing of military tactical data link standards
![Page 3: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/3.jpg)
Background: Simulation-based development implies the need for simulation-based testing
-- distributed simulation allows testing a system that is first formulated as an abstract model -- both the System under Test (SUT) and the test device are coupled by an common interface
Network
Test Device
send/receive messages
System Under Test (SUT)
send/receivemessages
Raises the question: How to develop the Test Device in an authoritative manner?
Specified as abstract
model, e.g. UML
Connecting middle ware, e.g. HLA
![Page 4: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/4.jpg)
Problem: Conflicting Requirements
To deal with the increasing complexity and advanced decision capabilities of C4ISR systems
=>testing methodology has to become more rigorous, in-depth and thorough
To keep up with the rapid change and short development life cycles expected from the system builders
=> tests have to be ready to conduct in time scales compatible with the agile development strategies of new systems.
Solution: employ DEVS-based M&S
• to increase capabilities for simulation-based testing and •as a basis to increase the automation of testing processes.
![Page 5: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/5.jpg)
Testing of interface standards is a focus area for automated simulation-based testing.
Link-16 is required in all Joint and multi-national operations.
The Joint Interoperability Test Command (JITC) is developing an automated test generation methodology as its core technology for testing conformance of systems to Link-16 This methodology is fundamentally enabled by the DEVS formalized modeling and simulation approach
AWACS
TheaterWarning
ABL
DSP/SBIRS
F-15
JLENS
THAAD
PATRIOT
MEADS
ATACMSAVENGER
TEL
AEGIS (CEP)
SIS(MSCS)SIS(MSCS) Link-16specification
![Page 6: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/6.jpg)
DEVS Representation of Link-16
Constraints(Exception)Rules
Stop
Modify C2Record for TN
1 2 3
RuleProcessing
Stop, Do Nothing,Alerts, Or jump to other
Transaction
TrackDisplay
Operatordecisions
Validity checking
TransmitMsg
Other ConsequentProcessing
Jumps (stimuli) to other
Transactions of specification
Transaction Level - example P.1.2 = Drop Track Transmit
Preparation Processing
Timeouts
PeriodicMsg
Input to
systemDEVS
Output from
system
t1
t2 t
3t4
Level
3 CoupledSystem
2 I/O System
1 I/O Function
0 I/O Frame
System Theory Provides Levels of Structure/Behavior
![Page 7: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/7.jpg)
The JITC employed such testing for the initial major milestone evaluation of the Integrated Architecture Behavior Model (IABM) developed by the Joint Single Integrated Air Picture (SIAP) System Engineering Organization (JSSEO) in 2005.
The test exercise produced significant results that uncovered flaws in the model design and added acknowledged value to the model development.
Recent Successful Application
![Page 8: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/8.jpg)
Types of Distributed Simulation Testing
Testing Description Example
standards conformance test whether system conforms to standardsupports interoperability
Tactical Data LinkStandards, Link-16, VMF, USMTF,
interoperability test whether systems can interoperate (at the syntactic, higher, levels)
Joint Translator Forwarder (JXF) – air to/from land exchange of tactical data
mission/capabilities test whether system of systems have capabilities required for mission
Joint Close Air Support (JCAS)
![Page 9: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/9.jpg)
Test Driver controls the scenario
Multiplatform Distributed Simulation – controlled testing
Platform(System,
Component)
Platform(System,
Component)
Platform(System,
Component)
Test Driver
![Page 10: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/10.jpg)
Distributed Observers look for opportunities to test
Multiplatform Distributed Simulation - uncontrolled testing
Platform(System,
Component)
Platform(System,
Component)
Platform(System,
Component)
Observer Observer Observer
Test Coordinator
![Page 11: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/11.jpg)
Test Driver for Controlled Testing
Middleware
Coupled Test ModelCoupled Test Model
Jx1,data1Jx2,data2Jx3,data3
Jx4,data4 Jx1,data1Jx2,data2Jx3,data3
Jx4,data4 Jx1,data1Jx2,data2Jx3,data3
Jx4,data4
Component Test Model
1
Component Test Model
2
Component Test Model
3
SUT
![Page 12: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/12.jpg)
Test Model Generation for Controlled Testing
Mirroring (flipping) the transactions of a SUT model (system model behavior selected as a test case) allows automated creation of a test model
holdSend(Jx1,data1,t1) holdSend (Jx2,data2,t2)
holdSend (Jx3,data3,t3) waitReceive(Jx4,data4)
receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)
receiveAndProcess(Jx3,data3) transmit(Jx4,data4)
Jx1,data1Jx2,data2Jx3,data3
Jx4,data4
t1 t2 t3 t4time
Test ModelTest Model
SUT Model
![Page 13: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/13.jpg)
Test Manager for Opportunistic Testing• Replace Test Models by Test Detectors• Deploy Test Detectors in parallel, fed by the Observer• Test Detector activates a test when its conditions are met• Test results are sent to a Collector for further processing
Jx1,data1Jx2,data2Jx3,data3Jx4,data4
Test Detector 1
Test Detector 2
Test Detector 3
SUO ObserverOtherFederates
ResultsCollector
Test Manager
![Page 14: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/14.jpg)
The Test Detector watches for the arrival of the given subsequence of messages to the SUO and then watches for the corresponding system output
• Define a new primitive, processDetect, that replaces holdSend• Test Detector
– Tries to match the initial subsequence of messages received by the SUO– When the initial subsequence is successfully matched, it enables waitReceive (or
waitNotReceive) to complete the test
processDetect(Jx1,data1,t1) processDetect(Jx2,data2,t2)
processDetect(Jx3,data3,t3) waitReceive(Jx4,data4)
receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)
receiveAndProcess(Jx3,data3) transmit(Jx4,data4)
Jx1,data1Jx2,data2Jx3,data3
Jx4,data4
t1 t2 t3 t4time
Test Test DetectorDetector
SUO
Test Detector Generation for Opportunistic Testing
![Page 15: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/15.jpg)
Observer & System Under Observation (SUO)
System(e.g. DEVS)
inports outports
ObserverForSystem
inports outports
Observeroutports
Tap into inputs and outputs of SUO
Gather input/output dataand forward for testing
![Page 16: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/16.jpg)
Example: Joint Close Air Support (JCAS) Scenario
Natural Language Specification
JTAC works with ODA!JTAC is supported by a Predator!JTAC requests ImmediateCAS to AWACS !AWACS passes requestImmediateCAS to CAOC! CAOC assigns USMCAircraft to JTAC!CAOC sends readyOrder to USMCAircraft !USMCAircraft sends sitBriefRequest to AWACS !AWACS sends sitBrief to USMCAircraft !USMCAircraft sends requestForTAC to JTAC !JTAC sends TACCommand to USMCAircraft !USMCAircraft sends deconflictRequest to UAV!USMCAircraft gets targetLocation from UAV!!
![Page 17: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/17.jpg)
Observer of AWACS with JCAS
Data gathered by
Observer
addObserver(USMCAircraft, JCASNUM1);
Observer is connected to SUO andmonitors itsI/O traffic
![Page 18: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/18.jpg)
Test Detector Prototype:Sequence Matcher
processDetect(J2.2,data1,t1)
processDetect(J3.2,data2,t2)
waitReceive(J7.0,data3,t3)
Sequential triggering, same as test models
![Page 19: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/19.jpg)
Example of Effect of State: AWACS
Rules
R1: if phase = “passive” & receive= "ImmediateCASIn“then output = "CASResourcesSpec" & state = "doSurveillance“
R2: if state = "doSurveillance“ & receive= "sitBriefRequestIn“then output = "sitBriefOut“ & phase = “passive”
matchsequence 1:initial state = passiveprocessDetect(ImmediateCASIn,””,1)waitReceive(CASResourcesSpec,””)
matchsequence 2:initial state = doSurveillanceprocessDetect(sitBriefRequestIn,””,1)waitReceive(sitBriefOut,””)
state = doSurveillance
need to know the state to enable this sequence
i1 i2
o2o1
state = passive
![Page 20: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/20.jpg)
Solution: make activation of matchsequence2 conditional on matchsequence1
matchsequence2 can only start when matchsequence1 has successfully been performed
![Page 21: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/21.jpg)
Observation Test Of AWACS
Observer ofAWACS
AWACSTestManager
![Page 22: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/22.jpg)
Problem with Fixed Set of Test Detectors
• after a test detector has been started up, a message may arrive that requires it to be re-initialized
• Parallel search and processing required by fixed presence of multiple test detectors under the test manager may limit the processing and/or number of monitor points
• does not allow for changing from one test focus to another in real-time, e.g. going from format testing to correlation testing once format the first has been satisfied
Solution
• on-demand inclusion of test detector instances• remove detector when known to be “finished”• employ DEVS variable structure capabilities• requires intelligence to decide inclusion and removal
![Page 23: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/23.jpg)
Dynamic Test Suite: Features
• Test Detectors are inserted into Test Suite by Test Control
• Test Control selects Detectors based on incoming message
• Test Control passes on just received message and starts up Test Detector
• Each Detector stage starts up next stage and removes itself from Test Suite as soon as the result of its test is known– If the outcome is a pass (test is successful) then next stage is
started up
![Page 24: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/24.jpg)
Dynamic Inclusion/Removal of Test Detectors
message arrives
add induced test detectors into test set
test detector subcomponent removes its enclosing test detector when test case result is known (either pass or fail)
Test Manager Active Test Suite
Test Control
removeAncestorBrotherOf(“TestControl");
addModel(‘test detector”);addCoupling2(" Test Manager ",“Jmessage",“test detector", “Jmessage");
![Page 25: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/25.jpg)
Adding Ports and Coupling by Relation Methods can be issued by any atomic model:
• add my ports and coupling to ancestor relation of another model – E.g., find my ancestor that is a brother of the given model, and add my ports and associated internal and external coupling to this ancestor
A
B
C D
E
A
addMyPortsToAncestorBrotherOf(B)
Issued by E, adds E’s ports to C, which is ancestor brother of B, and does associated coupling
B
A
BC
D E
C D
E
![Page 26: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/26.jpg)
Adding Ports and Coupling (cont’d) Methods can be issued by any atomic model: • add output ports and coupling from source to destination – adds outports of source as input ports of destination and couples them• add input ports and coupling from source to destination – adds inports of source as outputs of destination and couples them
A
addOutportsNcoupling(C,B))
Issued by E, allows E to communicate with B
B
C D
E
A
B
C D
E
addInportsNcoupling(C,B))
in1 out1
In1 out1
![Page 27: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/27.jpg)
Removing Models by Name and by Relation Methods can be issued by any atomic model:
•remove model by name – accepts the unique name of any atomic or coupled model within hierarchy and removes it and its immediate coupling
• remove model by relation to another model – e.g., remove ancestor that is a brother of the given model, can be used to avoid providing atomic models with names of other models
A
B
C D
E
A
removeModel2(C)
removeAncestorBrotherOf(B)
Issued by D, removes its parent which in this case is its ancestor that is a brother of B
B
A
BC
D E
![Page 28: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/28.jpg)
AWACS Opportunistic Testing in JCAS
Test Control
CAS Model with AWACSobservation
Initially empty Test Suite
![Page 29: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/29.jpg)
AWACS Opportunistic Testing in JCAS (cont’d)
Test Control adds appropriate Test Detector and connects it in to interface,
Test Control observes CAS request message to AWACS
![Page 30: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/30.jpg)
AWACS Opportunistic Testing in JCAS (cont’d)
Test Control passes on start signal and request message
First stage detector verifies request message receipt and prepares to start up second stage
![Page 31: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/31.jpg)
AWACS Opportunistic Testing in JCAS (cont’d)
second stage waits for expected response from AWACS to request
First stage detector removes self from test suite
![Page 32: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/32.jpg)
AWACS Opportunistic Testing in JCAS (cont’d)
Second stage observes correct AWACS response and removes itself and starts up second part
![Page 33: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/33.jpg)
AWACS Opportunistic Testing in JCAS (cont’d)
At some later time, second part of Test Detector observes situation brief request message to AWACS First stage removes itself and starts up second stage
![Page 34: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/34.jpg)
AWACS Opportunistic Testing in JCAS (cont’d)
Second stage observes situation brief output from AWACS thus passing test, It removes itself and enclosing Test Detector
![Page 35: DEVS Agents to Support Conformance Testing of Emerging Defense Information Standards](https://reader036.vdocuments.site/reader036/viewer/2022070405/56814017550346895dab63c3/html5/thumbnails/35.jpg)
Summary• DEVS formalism is capable of capturing the information-processing
complexities, including the dynamics, underlying the MIL-STD-6016C standard
• DEVS modeling and simulation methodology is attaining core-technology for automated testing of military tactical data link standards
• Structure control implemented in Dynamic Structure DEVS enables automation of standards conformance testing
• Structure control agents induce structural change in themselves or others to effectuate different behaviors under different circumstances
• The agent-supported simulation metaphor applies to testing the conformance of multi-agent systems to complex defense information standards