methodologies evaluation
Post on 13-Jan-2016
44 Views
Preview:
DESCRIPTION
TRANSCRIPT
Methodologies evaluation
Agentlink IIIAOSE TFG
Budapest, 17 sep. 2005
AOSE TFG Budapest meeting, 17/9/2005 2
Evaluation framework for AOSEM
Towards an evaluation framework for AOSEM Previous approaches Questionnaire results Review Outline and plan for document on AOSEM evaluation
framework
AOSE TFG Budapest meeting, 17/9/2005 3
An evaluation framework for AOSEM
Context Diverse scope of application of methodologies
• Several aspects: analysis, design, implementation, deployment, validation, verification, etc.
• Several application domains: from closed systems to open systems, web support, etc.
Tool support• Tools for modelling and code generation• Some methodologies have no tool support at all (or in a very
experimental state) Development process not always defined Different notations Different agent concepts Standardization efforts Several approaches for integration:
• A common standard agent specification language: which one?• Fragments: method engineering
AOSE TFG Budapest meeting, 17/9/2005 4
An evaluation framework for AOSEM
Evaluation of AOSEM can help towards the success of AOSE Clarification of concepts => towards some standardization Integration of fragments Definition of AOSE processes: heavy to light approaches Promotion of tools
AOSE TFG Budapest meeting, 17/9/2005 5
Inputs for AOSEM evaluation
A. Sturm, O. Shehory, D. Dori (2004). Evaluation of Agent-Oriented Methodologies. In: AL3 TF1-AOSE TFG
Q.N. Tran, G. Low (2005). Comparison of ten agent-oriented methodologies. In: Henderson-Sellers, B. and Giorgini, P., editors (2005). Agent-Oriented Methodologies. Idea Group Publishing. Chapter XII, pp. 341-367.
C. Bernon, et al. (2004). A Study of some Multi-Agent Meta-Models. Proc. AOSE 2004 (to appear in LNCS, Springer-Verlag).
L. Cernuzzi, G. Rossi (2004). On the evaluation of agent oriented methodologies. In: Proc. of the OOPSLA 2002 Workshop on Agent-Oriented Methodologies.
L. Cernuzzi, M. Cossentino, F. Zambonelli (2005). Process Models for Agent-Based Development. International Journal on Engineering Applications of Artificial Intelligence (EAAI). Elsevier. (in edition?)
AOSE TFG Budapest meeting, 17/9/2005 6
Questionnaire
Originally from Mickael Winikoff and modified by Massimo Cossentino
Aim: assess an AOSE methodology against a range of criteria. The criteria fall into a number of areas. Concepts/properties: The ideas that the methodology deals
with, basically the ontology Modelling: The models that are constructed and the
notations used to express the models. Process: The phases and steps that are followed as part of
the methodology. Pragmatics: Practical issues that are concerns when
adopting a methodology (e.g., the availability of training materials and courses, the existence and cost of tools, etc.)
AOSE TFG Budapest meeting, 17/9/2005 7
Questionnaire
Answers from: ADELFE (Carole Bernon/creator) INGENIAS (Jorge Gómez-Sanz & Juan Pavón/creators) OPEN Process Framework (OPF) (Brian
Henderson-Sellers/creator) Prometheus-ROADMAP (Lin Padgham/creator) Gaia (Giancarlo Fortino/Alfredo Garro: users!!!) PASSI (M. Cossentino:creator, L. Sabatucci, V. Seidita/PhD
Students: users/doing research on it, 8 graduating students: users)
TROPOS (3 students)
• Others are always welcome!!!• Answers from users (not creators) can provide a better
critical view of methodologies
AOSE TFG Budapest meeting, 17/9/2005 8
Questionnaire
Looking at the results of the questionnaire
It can be useful to consider changes in the questionnaire Subjective interpretation of questions and answers Not applicable Missing questions Useful? Clarifying? Identification of methodology challenges
Let’s see what are the results and discuss…
AOSE TFG Budapest meeting, 17/9/2005 9N: None L: Low M: Medium H: High
Questionnaire – Concepts & PropertiesConcept/Property Adelfe Gaia Ingenias OPF PASSI Prome-
theusTROPOS
Autonomy H H H H H/H/M H L
Mental attitudes L N H H L/L/M M M
Proactiveness M L H H H/M/H H N
Reactiveness H L H H H/H/H H N
Concurrency H M H L H/H/M H L
Teamwork and roles
L H H H M/H/H L M
Cooperation model AMAS th. Teamwork ALL ALL Task del./ Teamwork
none Negotiation/ Task del.
Protocols support H H H H H/M/H H N
Communication modes
ALL Async mess. ALL ALL Direct N
Communication language
ALL ACL like ALL ALL Speech acts
messages
Situatedness H H H H H/M/M H H
Environment type All episodic DynamicContinuous
All discrete ALL ALL ALL Inacc., Non episodic, Dynam.
Creator/PhD Students/Grad. Stud.
AOSE TFG Budapest meeting, 17/9/2005 10SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly Agree
Questionnaire – Concepts & Properties
Concept/Property Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Other agent features
— Opennes Opennes Opennes Mobility, openness, security
Plans, agent decisions
Security, Trust, Delegation, Ownership,
Dependency, Provision
Non supported features
— — Security & Mobility (not
explicitly)
Security & Mobility (on going work)
Complex design-time
social organiza-
tions
Security & Mobility
Dynamic Behavior of
Agent
Clear concepts A A SA SA SA/N/N A A
Overloaded concepts
N D D SD D/D/N D N
More Agent-oriented than OO
A SA SA both SA/A/A SA SA
(Main) Supported agents
Cooperative BDI (mainly) BDI (mainly)
Mainly: State-based,
rational, reactive
ALL BDI, Rational
Society of agents modelling
No SA SA (on going work)
A/-/- No A
Society structure - - Groups/WF - p2p, simple hierarchies,
holons
- Agent Society Pattern, such
as Broker, Mediated,
Matchmaker
AOSE TFG Budapest meeting, 17/9/2005 11
Questionnaire – Modelling & Notation
Notation Adelfe Gaia Ingenias OPF PASSI Prome-theus
TROPOS
Support for static (structure) and dynamic (processing) aspects
SA A SA — SA/A/A SA D
Symbols and syntax well defined
A N SA — A/A/D A N
Well defined semantics A D SA — A/N/D A AClear notation A A N — A/A/N A NEasy to use notation A A SA — A/A/N A SAEasy to learn notation N SA A — N/N/N NA N
SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly AgreeNA: Not Applicable
A methodology is really notation independent. Yes, there is a need for a modelling language and in the FAME project we have FAML (FAME modelling language) although not yet a notation. So we can’t really answer these notation specific questions (i.e. 21-26)
Creator/PhD Students/Grad. Stud.
AOSE TFG Budapest meeting, 17/9/2005 12
Questionnaire – Modelling & Notation
Modelling Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Multiple views A N SA SA SA/A/A — A
Adequate and expressive
A N SA SA A/A/N — N
Traceability between models and between models and code
A D SA SA SA/N/A — D
Guidelines and techniques for consistency checking
A SD N N N/N/N — D
Supports refinement SA N SA N SA/A/A — A
Supports modularity SA D A SA SA/A/A — N
Supports component reusability
SA SD SA SA SA/A/A — SD
Extensible SA SD SA SA A/-/- SA A
Supports hierarchical modelling and abstraction
SA D SA SA SA/N/A SA A
Other issues
SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly Agree
AOSE TFG Budapest meeting, 17/9/2005 13
Questionnaire – Process
Lifecycle coverage Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Planning CE CEH CEHRequirements analysis CE CE CEH CEH CEH/-/- CEH CEArchitectural (or agent society) design
CE CE CEH CEH CEH/-/- CEH CE
Detailed (agent) design CE CE CEH CEH CEH/-/- CEH CEImplementation CEH CEH CEH/-/- E PTesting/Debugging H P CEH H PCEHDeployment P CEH CE
Maintenance CEH PDeath CEH
C: Clear definition of activitiesE: Examples givenH: Heuristics givenP: Partial
AOSE TFG Budapest meeting, 17/9/2005 14
Questionnaire – Process
Process Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Addresses Quality Assurance D SD N SA N/N/A A SDEstimating guidelines (cost, …)
— SD A N D/N/A N N
Support for decision making (e.g. when to move between phases)
A SD SA A N/A/A N D
Development approach Iterative/
incremental
Top-down
Iterative/incremen
talTransformation & architect
ural based
ANY Iterative/Incre-
mental
Iterative/Incre-
mental/Spiral
Top Down
Supports patterns or reusability
A SD D SA SA/-/- N N
Degree of user implication (i.e. it does requires user-designer communication ?)
Medium
— Medium Strong Weak — M
SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly Agree
AOSE TFG Budapest meeting, 17/9/2005 15
Questionnaire – Pragmatics
SD: Strongly Disagree D: Disagree N: Neutral A: Agree SA: Strongly Agree
Software tools Adelfe Gaia Ingenias OPF PASSI Prome- theus
TROPOS
Diagram editor OpenTool IDK editor PTK GR-Tool, ST-Tool, TAOM4E
Code generator IDK code g. Agent Factory
Design consistency checker
IDK ATA Prototype PTK GR-Tool, ST-Tool
Project Management AdelfeToolkit
Rapid prototyping
Reverse engineering Agent Factory
Automatic testing
Commercial or research product
OT: comm.AT: free
Research Research Research
Research Research
Adequate level of functionalities
A A A/-/- A? N
Quick and easy to learn N A N/-/- A? A
Support in raising the quality
A A SA/-/- SA? N
Reduces time to design/implem.
A SA SA/-/- SA? A
Other comments GPL license, UML/ Ingenias
notation
Considering other tools
AOSE TFG Budapest meeting, 17/9/2005 16
Questionnaire – Pragmatics
Pragmatics Adelfe Gaia Ingenias OPF PASSI Prome-theus
TROPOS
Audience All All All All All/-/- All Grad. st., experts,
researcher
Complexity compared to UML/RUP
About the same
About the same
About the same
A lot simpler About the same/-/-
About the
same
Simpler
Resources: Papers
X X X X X X
Text books X X X
Tutorial notes X X X X
Consulting services X X
Training services X X
Nr.applications built with meth.
1-5 21+ 6-20 21+ in OO/ME 21+/-/- 21+ 1-5
Were applications real? Yes No No All Y/Y/N Yes Yes
Any developed by other users?
No Yes Yes Yes Y/Y/N Yes No
Target any specific domain Complex systems
No No All but RT No/-/- No No
Support scalability Yes No Yes Yes A/N/- — N
Supports distributed systems
Yes — Yes Yes SA/A/SA — N
AOSE TFG Budapest meeting, 17/9/2005 17
Evaluation framework revisited
Taking the experience of this questionnaire Review evaluation framework criteria and their organization Review method for evaluation: questionnaire, case studies
development, ...• Refine questionnaire• Define case studies
Review metrics• How to avoid subjectivity
AOSE TFG Budapest meeting, 17/9/2005 18
Pragmatics
Evaluation framework revisited Criteria for AOSEM evaluation
Features
Domain
Complexity
Process
DeliverablesActivities
Team work
Domain specific methods
Tools
Modelling
Autonomy, society, …
AbstractionModularity
Domain specific concepts
Knowledge skillsScalability
AOSE TFG Budapest meeting, 17/9/2005 19
Towards an AOSEM evaluation framework
The evaluation framework should allow: Criteria refinement and extensions Criteria metrics depending on the domain
• E.g. agents in a web service or in robotics Definition of standard case studies for evaluation
• Evaluation of documentation and filling questionnaires is not enough
…
AOSE TFG Budapest meeting, 17/9/2005 20
Towards an AOSEM evaluation framework
The framework can be based on the definition and use of evaluation models Case studies for putting the methodologies to work Organized by criteria
• For each criteria, define metrics• Criteria can be refined to get more insight or being more specific
• For instance, agent behaviour, depending on whether BDI, neural network, CBR, reactive, or whatever model is used
• New criteria can be added• Some criteria may be considered non applicable• Associate criteria to case studies
AOSE TFG Budapest meeting, 17/9/2005 21
Outline and plan for document on AOSEM evaluation framework
Outline Participants Plan
top related