Methodologies evaluation - PowerPoint PPT Presentation

About This Presentation

Methodologies evaluation


Outline and plan for document on AOSEM evaluation framework. AOSE TFG ... Agent Society Pattern, such as Broker, Mediated, Matchmaker. A. BDI, Rational. SA. N. A ... – PowerPoint PPT presentation

Number of Views:16
Avg rating:3.0/5.0
Slides: 22
Provided by: juan75


Transcript and Presenter's Notes

Title: Methodologies evaluation

Methodologies evaluation
  • Agentlink III
  • Budapest, 17 sep. 2005

Evaluation framework for AOSEM
  • Towards an evaluation framework for AOSEM
  • Previous approaches
  • Questionnaire results
  • Review
  • Outline and plan for document on AOSEM evaluation

An evaluation framework for AOSEM
  • Context
  • Diverse scope of application of methodologies
  • Several aspects analysis, design,
    implementation, deployment, validation,
    verification, etc.
  • Several application domains from closed systems
    to open systems, web support, etc.
  • Tool support
  • Tools for modelling and code generation
  • Some methodologies have no tool support at all
    (or in a very experimental state)
  • Development process not always defined
  • Different notations
  • Different agent concepts
  • Standardization efforts
  • Several approaches for integration
  • A common standard agent specification language
    which one?
  • Fragments method engineering

An evaluation framework for AOSEM
  • Evaluation of AOSEM can help towards the success
    of AOSE
  • Clarification of concepts gt towards some
  • Integration of fragments
  • Definition of AOSE processes heavy to light
  • Promotion of tools

Inputs for AOSEM evaluation
  • A. Sturm, O. Shehory, D. Dori (2004). Evaluation
    of Agent-Oriented Methodologies. In AL3 TF1-AOSE
  • Q.N. Tran, G. Low (2005). Comparison of ten
    agent-oriented methodologies. In
    Henderson-Sellers, B. and Giorgini, P., editors
    (2005). Agent-Oriented Methodologies. Idea Group
    Publishing. Chapter XII, pp. 341-367.
  • C. Bernon, et al. (2004). A Study of some
    Multi-Agent Meta-Models. Proc. AOSE 2004 (to
    appear in LNCS, Springer-Verlag).
  • L. Cernuzzi, G. Rossi (2004). On the evaluation
    of agent oriented methodologies. In Proc. of the
    OOPSLA 2002 Workshop on Agent-Oriented
  • L. Cernuzzi, M. Cossentino, F. Zambonelli (2005).
    Process Models for Agent-Based Development.
    International Journal on Engineering Applications
    of Artificial Intelligence (EAAI). Elsevier. (in

  • Originally from Mickael Winikoff and modified by
    Massimo Cossentino
  • Aim assess an AOSE methodology against a range
    of criteria. The criteria fall into a number of
  • Concepts/properties The ideas that the
    methodology deals with, basically the ontology
  • Modelling The models that are constructed and
    the notations used to express the models.
  • Process The phases and steps that are followed
    as part of the methodology.
  • Pragmatics Practical issues that are concerns
    when adopting a methodology (e.g., the
    availability of training materials and courses,
    the existence and cost of tools, etc.)

  • Answers from
  • ADELFE (Carole Bernon/creator)
  • INGENIAS (Jorge Gómez-Sanz Juan Pavón/creators)
  • OPEN Process Framework (OPF) (Brian
  • Prometheus-ROADMAP (Lin Padgham/creator)
  • Gaia (Giancarlo Fortino/Alfredo Garro users!!!)
  • PASSI (M. Cossentinocreator, L. Sabatucci, V.
    Seidita/PhD Students users/doing research on it,
    8 graduating students users)
  • TROPOS (3 students)
  • Others are always welcome!!!
  • Answers from users (not creators) can provide a
    better critical view of methodologies

  • Looking at the results of the questionnaire
  • It can be useful to consider changes in the
  • Subjective interpretation of questions and
  • Not applicable
  • Missing questions
  • Useful? Clarifying?
  • Identification of methodology challenges
  • Lets see what are the results and discuss

Questionnaire Concepts Properties
Creator/PhD Students/Grad. Stud.
Concept/Property Adelfe Gaia Ingenias OPF PASSI Prome-theus TROPOS
Autonomy H H H H H/H/M H L
Mental attitudes L N H H L/L/M M M
Proactiveness M L H H H/M/H H N
Reactiveness H L H H H/H/H H N
Concurrency H M H L H/H/M H L
Teamwork and roles L H H H M/H/H L M
Cooperation model AMAS th. Teamwork ALL ALL Task del./ Teamwork none Negotiation/ Task del.
Protocols support H H H H H/M/H H N
Communication modes ALL Async mess. ALL ALL Direct N
Communication language ALL ACL like ALL ALL Speech acts messages
Situatedness H H H H H/M/M H H
Environment type All episodic Dynamic Continuous All discrete ALL ALL ALL Inacc., Non episodic, Dynam.
N None L Low M Medium H High
Questionnaire Concepts Properties
Concept/Property Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Other agent features Opennes Opennes Opennes Mobility, openness, security Plans, agent decisions Security, Trust, Delegation, Ownership, Dependency, Provision
Non supported features Security Mobility (not explicitly) Security Mobility (on going work) Complex design-time social organiza-tions Security Mobility Dynamic Behavior of Agent
Clear concepts A A SA SA SA/N/N A A
Overloaded concepts N D D SD D/D/N D N
More Agent-oriented than OO A SA SA both SA/A/A SA SA
(Main) Supported agents Cooperative BDI (mainly) BDI (mainly) Mainly State-based, rational, reactive ALL BDI, Rational
Society of agents modelling No SA SA (on going work) A/-/- No A
Society structure - - Groups/WF - p2p, simple hierarchies, holons - Agent Society Pattern, such as Broker, Mediated, Matchmaker
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
Questionnaire Modelling Notation
Creator/PhD Students/Grad. Stud.
Notation Adelfe Gaia Ingenias OPF PASSI Prome-theus TROPOS
Support for static (structure) and dynamic (processing) aspects SA A SA SA/A/A SA D
Symbols and syntax well defined A N SA A/A/D A N
Well defined semantics A D SA A/N/D A A
Clear notation A A N A/A/N A N
Easy to use notation A A SA A/A/N A SA
Easy to learn notation N SA A N/N/N NA N
A methodology is really notation independent.
Yes, there is a need for a modelling language and
in the FAME project we have FAML (FAME modelling
language) although not yet a notation. So we
cant really answer these notation specific
questions (i.e. 21-26)
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree NA Not
Questionnaire Modelling Notation
Modelling Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Multiple views A N SA SA SA/A/A A
Adequate and expressive A N SA SA A/A/N N
Traceability between models and between models and code A D SA SA SA/N/A D
Guidelines and techniques for consistency checking A SD N N N/N/N D
Supports refinement SA N SA N SA/A/A A
Supports modularity SA D A SA SA/A/A N
Supports component reusability SA SD SA SA SA/A/A SD
Extensible SA SD SA SA A/-/- SA A
Supports hierarchical modelling and abstraction SA D SA SA SA/N/A SA A
Other issues
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
Questionnaire Process
Lifecycle coverage Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Planning CE CEH CEH
Requirements analysis CE CE CEH CEH CEH/-/- CEH CE
Architectural (or agent society) design CE CE CEH CEH CEH/-/- CEH CE
Detailed (agent) design CE CE CEH CEH CEH/-/- CEH CE
Implementation CEH CEH CEH/-/- E P
Testing/Debugging H P CEH H PCEH
Deployment P CEH CE
Maintenance CEH P
Death CEH
C Clear definition of activities E Examples
given H Heuristics given P Partial
Questionnaire Process
Process Adelfe Gaia Ingenias OPF PASSI Prometheus TROPOS
Addresses Quality Assurance D SD N SA N/N/A A SD
Estimating guidelines (cost, ) SD A N D/N/A N N
Support for decision making (e.g. when to move between phases) A SD SA A N/A/A N D
Development approach Iterative/ incremental Top-down Iterative/ incremental Transformation architectural based ANY Iterative/ Incre-mental Iterative/ Incre-mental/ Spiral Top Down
Supports patterns or reusability A SD D SA SA/-/- N N
Degree of user implication (i.e. it does requires user-designer communication ?) Medium Medium Strong Weak M
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
Questionnaire Pragmatics
Software tools Adelfe Gaia Ingenias OPF PASSI Prome- theus TROPOS
Diagram editor OpenTool IDK editor PTK GR-Tool, ST-Tool, TAOM4E
Code generator IDK code g. Agent Factory
Design consistency checker IDK ATA Prototype PTK GR-Tool, ST-Tool
Project Management AdelfeToolkit
Rapid prototyping
Reverse engineering Agent Factory
Automatic testing
Commercial or research product OT comm. AT free Research Research Research Research Research
Adequate level of functionalities A A A/-/- A? N
Quick and easy to learn N A N/-/- A? A
Support in raising the quality A A SA/-/- SA? N
Reduces time to design/implem. A SA SA/-/- SA? A
Other comments GPL license, UML/ Ingenias notation Considering other tools
SD Strongly Disagree    D Disagree    N
Neutral   A Agree    SA Strongly Agree
Questionnaire Pragmatics
Pragmatics Adelfe Gaia Ingenias OPF PASSI Prome-theus TROPOS
Audience All All All All All/-/- All Grad. st., experts, researcher
Complexity compared to UML/RUP About the same About the same About the same A lot simpler About the same/-/- About the same Simpler
Resources Papers X X X X X X
Text books X X X
Tutorial notes X X X X
Consulting services X X
Training services X X
Nr.applications built with meth. 1-5 21 6-20 21 in OO/ME 21/-/- 21 1-5
Were applications real? Yes No No All Y/Y/N Yes Yes
Any developed by other users? No Yes Yes Yes Y/Y/N Yes No
Target any specific domain Complex systems No No All but RT No/-/- No No
Support scalability Yes No Yes Yes A/N/- N
Supports distributed systems Yes Yes Yes SA/A/SA N
Evaluation framework revisited
  • Taking the experience of this questionnaire
  • Review evaluation framework criteria and their
  • Review method for evaluation questionnaire, case
    studies development, ...
  • Refine questionnaire
  • Define case studies
  • Review metrics
  • How to avoid subjectivity

Evaluation framework revisited Criteria for
AOSEM evaluation
Process DeliverablesActivities Team
work Domain specific methods Tools
Modelling Autonomy, society, AbstractionModula
rity Domain specific concepts Knowledge
skills Scalability
Towards an AOSEM evaluation framework
  • The evaluation framework should allow
  • Criteria refinement and extensions
  • Criteria metrics depending on the domain
  • E.g. agents in a web service or in robotics
  • Definition of standard case studies for
  • Evaluation of documentation and filling
    questionnaires is not enough

Towards an AOSEM evaluation framework
  • The framework can be based on the definition and
    use of evaluation models
  • Case studies for putting the methodologies to
  • Organized by criteria
  • For each criteria, define metrics
  • Criteria can be refined to get more insight or
    being more specific
  • For instance, agent behaviour, depending on
    whether BDI, neural network, CBR, reactive, or
    whatever model is used
  • New criteria can be added
  • Some criteria may be considered non applicable
  • Associate criteria to case studies

Outline and plan for document on AOSEM
evaluation framework
  • Outline
  • Participants
  • Plan
Write a Comment
User Comments (0)