test in multi-agent systems © les/puc-rio. background the www and the mobile devices new...

Test in Multi-Agent Systems

© LES/PUC-Rio

© LES/PUC-Rio

Background

• The WWW and the mobile devices new applications:

– collection of distributed components

– input from a variety of sources

– adapt according to users preferences.

• Agent technology:

– prominent way to implement such systems

– uses abstractions of a higher level.

© LES/PUC-Rio

R1 R3

R2 R4

Problem Domain

R6 R8

R5 R7

functions

files

modules

Procedural

How?

Object Oriented

Object Attribute Method

Class

What?

Agent Oriented

Agent Roles Messages

Organization Protocols

Which agents?

Agents – A new Level of Abstraction

© LES/PUC-Rio

Garcia, A., Lucena, C., Cowan D. Agents in Object-Oriented Software Engineering. Software Practice & Experience, Elsevier, 34 (5), pp. 489-521, 2004.

Agent Definition

• Agent:

– is an autonomous entity

– that executes a plan

– to achieve a goal

– based on its beliefs (attributes)

– communicates asynchronously (message based)

• Autonomy

– Different set of actions may achieve the same goal

System

© LES/PUC-Rio

Agent Technology: Benefits and Challenges

• On the one hand:– Facilitates to address current application

requirements.

• On the other hand:– new challenges to software testability and

consequently, to maintainability.

• Tests:– aims at uncovering regression bugs.

– serve as live documentation.

© LES/PUC-Rio

Obstacles to Software Testability

• Testability:

– Controlling the test input (controllability)

– Observing the output of the unit under test (observability)

• Obstacles:

– More than one “expected” behavior for an agent

– Test input: data + messages from concurrent conversations.

Voas, J., and Miller, K.. Software Testability: The New Verication. IEEE Software, 12 3:1728, 1995.

© LES/PUC-Rio

MAS Development Approaches

• Traditionally adopted:– prototyping process

– ad-hoc approaches to assure system quality.

• Few works define a testing process with supporting tool:– Agile

– Agile Passi

– SUnit

– JAT

Ali Murat Tiryaki, Sibel Oztuna, Oguz Dikenelli, R. Cenk Erdur

SUnit: Test Driven Development of Multi-Agent Systems

© LES/PUC-Rio

Approach

• Test driven based development approach that supports iterative and incremental MAS construction

• It implements a JUnit-based framework for writing automated tests for agent behaviors and interactions between agents.

– Mock agents to model the organizational aspects of the MAS

– It was implemented on to of Seagent (Hierarchical

Task Network)

© LES/PUC-Rio

Test Driven Development

• Test driven development (TDD)

– Style of software development in which development is driven by automated tests.

– Cycle: developer firstly writes functional tests for a unit/task, writes the code that passed the written tests, and then, the code is refactored to improve the design

• Supports the iterative and incremental construction of task

© LES/PUC-Rio

Agent Oriented Test Driven Development

• Scenario: It defines an elementary process within MASs which produce a net value for the initiator.

• Unit: Agent. Agents cooperatively execute the functionalities of the scenario at hand. Each agent has its own tasks based on the responsibilities within the scenario. Agents interact using an agent communication language which has a higher level semantics.

© LES/PUC-Rio

A Sample HTN Structure

Iterative cycle of AOTDD

© LES/PUC-Rio

SUnit Testing Framework

• It provides a framework for test driven plan development

• It supports the developer to create and execute tests in a uniform and an automatic way during the plan implementation

• It consists of three sub modules

– Structural Test Module

– Action Test Module

– Flow Test Module

© LES/PUC-Rio

Structural Test Module

• This module assists the developer by defining subtasks and by determining the inter-relationship of the subtasks which compose the HTN plan.

• For every single complex task in a plan, a structural test case should be implemented

© LES/PUC-Rio

Assert methods

Action Test Module

• Action is a primitive tasks that generates an outcome or a message to another agent in the platform.

• This module is designed to examine outcomes or messages in respect to the user defined provision sets for the primitive tasks.

© LES/PUC-Rio

Assert methods

Flow Test Module

• This module is designed to evaluate outcomes or outgoing messages of the tasks in respect to the incoming messages or predefined message sets

© LES/PUC-Rio

The Class Diagram of the Flow Test Module

Assert methods

Creating a Unit Test

• SeagentStructuralTestCase should be extended to validate the plan structure.

• SeagentActionTestCase have be extended to be able to write assert methods for primitive task(s).

– check the results of the primitive task(s) such as outcomes and/or generated messages.

• SeagentFlowTestCase check the correctness of OWL concept’s type and value with the expected ones.

© LES/PUC-Rio

Creating a Unit Test - Example

• Traveler tries to organize a holiday plan which includes hotel booking and transportation details. The aim of the scenario is to arrange the cheapest holiday plan by selecting the proper accommodation & transportation options based on the traveler preferences.

© LES/PUC-Rio

Creating a Unit Test - Structural Test

public void testLevelOne(){

assertDisinheritanceSender(disinheritOK,arrangeArrival);

assertProvisionSender(arrangeAccomodation,

provisionResInfoForAcc,getReservationInfo;

assertProvisionSender(arrangeArrival,

provisionResInfoForArr,arrangeAccomodation);

}

© LES/PUC-Rio

Creating a Unit Test – Flow Test

public class FullIsolatedTest extends SeagentFlowTestCase {

…

public MessageSet assignMessageSet() {

MessageSet mySet= new MessageSet();

FIPAMsg aMessage = new FIPAMsg();

…

mySet.addMessage(aMessage);

…

return mySet;

}

public void testForFirstLevel(){

assertActionOutcome(arrangeAccomodation, reservationInfo);

assertActionExecuted(arrangeArrival);

assertMessageArrived(aMessage);

}

© LES/PUC-Rio

JAT: A Test Automation Framework for Multi-Agent Systems

Roberta Coelho, Elder Cirilo, Uirá Kulesza

Arndt von Staa, Awais Rashid, Carlos Lucena

© LES/PUC-Rio

Approach

• It defines an approach for building and running MAS tests.

• It relies on Aspect Oriented Programming to:

– Control the test input

– Observe the output units under test

• It was implemented on top of JADE:

– Middleware for agent-based applications developed in Java.

© LES/PUC-Rio

JADE Agent Testing (JAT) Framework

• JAT enables:

– the creation of JUnit-style tests for JADE systems.

• JAT is used:

– as a generic testing framework for JADE developers

– same as JUnit is for Java developers

© LES/PUC-Rio

Fault Model

• Message ordering

• Message content

• Increases on message delay

• Faults in agents beliefs

• Faults in internal procedures

Agent-related new abstractions new sources of bugs

© LES/PUC-Rio

Supporting Ideas

1. Using Agents to test Agents:

• Mock Agent:

• adaptation of the mock object concept

• a fake implementation of a real agent

• sends messages to an agent under test (AUT) and

• checks its response:

• content, delay, type, ordering.

message response

send messagemethod call

method output

Tester

Test Class

Agent A

(A) (B)

© LES/PUC-Rio

Supporting Ideas

2. Monitoring Agents:

– Agents are autonomous

– The developer need to know agents state

• ex: to answer: output == expected output?

– Monitor Aspect observes and stores information about agents state transitions (ex: running, dead).

© LES/PUC-Rio

Supporting Ideas

2. Monitoring agents: (cont.)

• Agent state machine is a crosscutting abstraction;

• spread over the application and platform modules;

• can be represented as an Aspect (XPI);

public aspect AgentStateMachine { pointcut created():call(public ContainerController.createNewAgent(..));

pointcut from_created_to_running(Agent o): execution(protected void Agent.setup())

&& target(o);

pointcut from_running_to_dead (Agent o): execution(public void Agent.doDelete()) && target(o); ...}

running

created waiting

dead

© LES/PUC-Rio

Supporting Ideas

2. Monitoring agents: (cont.)

privileged public aspect Monitor {

...

before(Agent o): AgentStateMachine.from_created_to_running(o){

includeInRunningAgents(o);

}

public void waitUntilAgentIsDead (String mockAgentID){

while (!isDead(mockAgentID) && (MAXTIMEOUT > timeSpent)){

wait(20000);

...

}

}

...

}

© LES/PUC-Rio © LES/PUC-Rio

Supporting Ideas

3. Synchronizing Mock Agents

– Mock agents partially implement a test scenario

– We need to define the order Mocks Agents should interact with the AUT.

– Synchronization crosscutting concern

– Synchronizer Aspect: orchestrates the test scenario.

Synchronizer

Mock Agents

© LES/PUC-Rio

JAT Framework: The Architecture

Aspects

class JATDIAGRAM

EA 6.5 Unregistered Trial Version EA 6.5 Unregistered Trial Version EA 6.5 Unregistered Trial Version EA 6.5 Unregistered Trial Version EA 6.5 Unregistered Trial Version



































TestRunner

TestResult

TestCase TestSuite

TestAgent

AgentController ContainerController

TestReporter

+ getTestResult() : String+ setTestResult() : void

«aspect»Monitor

+ storeAgentState() : void+ waitUnitlAgentsDie() : void+ waitUnitlTestHasFinished() : void

Synchronizer

+ checkTurn() : boolean

MockAgentGenerator

+ generateMockAgent(File) : void

JadeTestCase

+ assertInteractionOK() : void+ startAUT() : void+ startMockAgent() : void

JadeMockAgent

+ receiveMessage() : void+ sendMessage() : void

AgentStateMachine

+ pointcut from_dead_to_started() : void+ pointcut from_runnig_to_dead() : void+ pointcut from_started_to_runnig() : void

class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


«aspect»Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


«xpi»AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


«interface»TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


JADE Middleware

<<crosscuts>>

Monitoring Concern MLegend:

Synchronization Concern Agent State Machine Concern

class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


JAT Framework

class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


JUnit Frameworkclass JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

«interface»Test

Agent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


class JATDIAGRAM




































TestRunner

TestResult

TestCase TestSuite

TestAgent


TestReporter


Monitor


Synchronizer


MockAgentGenerator


JadeTestCase


JadeMockAgent


AgentStateMachine


A S

S

A

M

<<uses>>

<<crosscuts>><<crosscuts>>

<<crosscuts>>

© LES/PUC-Rio

JADETestCase: Code Example

public class BookTradingTesteCase extends JADETestCase {

public void testBookTradingScena1(){ //Load Data Repositories … startAUT("seller","BookSeller"); startMockAgent("buyer1","BookBuyerMockAgent",FIRST); startMockAgent("buyer2","BookBuyerMockAgent",SECOND);

//Blocks until interaction finishes Monitor.aspectOf(). waitUntilTestHasFinished("buyer1"); Monitor.aspectOf(). waitUntilTestHasFinished("buyer2");

//Check Expected Result: assertInteractionOK("buyer1"); assertInteractionOK("buyer2");

//Check Expected Result: Object belief= getBelief(“seller”,“catalogue”);

assertEquals(expectedBelief,belief); }...}

© LES/PUC-Rio


:Monitor

10.check(msg)

AUT: JADEAtent

6. add(state)5. state = crosscuts( )

Legend:

join point intercepted by Monitor aspect

join point interceptedby Synchronizer aspect

objectaspect

1. testMethod()

:JADETestCase

3. start(params)

:JADEMockAgent

4. start(params)

8. send(msg)

9. send(msg)

11. setResult( )

7. waitUntilTestFinishes ( )

14. notifyTestFinished ()

16. assertions( )

:Synchronizer

2. loadInteractionOrder( )

13. add(state)12. state = crosscuts( )

MonitoringDataRepository

© LES/PUC-Rio

Case Studies Results: Detected faults

Fault TypeDetected Faults

BT EC AC All

On MAS Constraints - 2 - 2

Unimplemented MAS constraint - 2 - 2

On Interaction Protocol 3 6 4 13

Mismatching Protocols 1 - - 1

Incomplete Main Flow - 2 1 3

Missing Alternative Flows 2 4 3 9

On Internal Procedures 1 1 1 3

Boundary values - - 1 1

Logic Mistakes 1 1 - 2

TOTAL 4 9 5 19

© LES/PUC-Rio



BT EC AC All










TOTAL 4 9 5 19

Interaction protocols only define the main execution flow. Thus, the developer needs to decide: what should an agent do when it receives an unexpected message format?

© LES/PUC-Rio


• Mocks are useful to examine how agents behave under exceptional scenarios.


BT EC AC All










TOTAL 4 9 5 19

Interaction protocols only define the main execution flow. Thus, the developer needs to decide: what should an agent do when it receives an unexpected message format?

© LES/PUC-Rio

Case Studies Results: Effectiveness of test cases

• Justifies the need to complement JAT Tests with conventional OO unit tests (agent internal procedures).

Fault TypeFaults Injected/Detected

BT EC AC All

On Interaction Protocol 16/16 23/23 21/21 60/60

Message Ordering 6/6 10/10 8/8 24/24

Message Timing 6/6 10/10 7/7 23/23

Message Content 4/4 3/3 6/6 13/13

On Beliefs 4/4 1/1 2/2 7/7

Corrupt Belief 4/4 1/1 2/2 7/7

On Internal Procedures 5/5 7/2 4/4 15/10

Corrupt Return and Parameter values

2/2 6/2 2/2 8/4

Logic Mistakes 3/3 1/0 3/3 7/6

TOTAL 24/25 31/26 27/27 83/78

test in multi-agent systems © les/puc-rio. background the www and the mobile devices new...

Documents

lespucrio lespucrioobstacles

test driven plan development

agent behaviors

agent definitionagent

based development approach

automated tests

ieee software

test observabilityobstacles