111111 benchmarking in wp 2.1. sep 28th, 2004 © r. garcía-castro, a. gómez-pérez raúl...

1 1 1 1 1 1Benchmarking in WP 2.1. Sep 28th, 2004 © R. García-Castro, A. Gómez-Pérez

Raúl García-Castro Asunción Gómez-Pérez

<rgarcia,[email protected]>

September 28th, 2004

Benchmarking in WP 2.1


Index

1. Progress

2. Deliverable 2.1.4


0 6 12 18 24 30 36 42 48

D2.1.1:Benchmarking

SoA

D2.1.4:BenchmarkingMethodology,

criteria, test suites

D2.1.6:Benchmarkingbuilding tools

Benchmarkingquerying, reasoning,

annotation

Benchmarkingsemantic

web service

Finished Started Not startedProgress:

Benchmarking activities timeline


Ontology Technology/Methods

Evaluation

Benchm

arking

Desired attributesWeaknesses

Comparative analysis...

Continuous improvementBest practices

Measurement

Experimentation

What has been done?in D 2.1.1 Survey of Scalability Techniques for Reasoning with Ontologies

• Overview of benchmarking, experimentation, and measurement• State of the Art of Ontology-based Technology Evaluation

Recommendations


What are we doing?T 2.1.4 Benchmarking methodology, criteria, and test suites

General evaluation criteria:• Interoperability• Scalability• Robustness

Benchmark suites for:• Interoperability• Scalability• Robustness

Benchmarking supporting tools:• Workload generators• Test generators• Monitoring tools• Statistical packages•...

Benchmarking results:• Comparative analysis• Compliance with norms• Weaknesses• Recommendations on tools• Recommendations on practices

Benchmarking

Methodology

Ontology tools:• Ontology building tools• Annotation tools• Querying and reasoning services• Semantic Web Services technology

GOAL: Provide a framework for benchmarking activities in WP 2.1 (and maybe other WPs)


Experiment results:

• test 1• test 2• test 3• ...

Experiment results:


What will be done?T 2.1.6: Benchmarking of ontology building tools

Tools/Partners:

... .........

Benchmarking results:• Comparative analysis• Compliance with norms• Weaknesses• Recommendations on tools• Recommendations on practices

Benchmark suites:• RDF(S) Import capability• OWL Import capability• RDF(S) Export capability• OWL Export capability• ...

Interoperability• Do the tools import/export from/to RDF(S)/OWL?• Are the imported/exported ontologies the same?• Is there any knowledge loss during import/export?• ...

UPM Experiment results:


NOOKOK

Benchmarking ontology

building tools

Benchmarking supporting tools:• Workload generators• Test generators• Monitoring tools• Statistical packages• ...


Index

1. Progress

2. Deliverable 2.1.4


D 2.1.4: Deliverable outline

1. Introduction2. Benchmarking methodology3. Building test suites for ontology tools4. General supporting tools for benchmarking5. Benchmarking ontology development tools and tool suites6. Benchmarking ontology-based annotation tools7. Benchmarking ontology querying tools and inference engines8. Benchmarking semantic web service technology9. Conclusion10. Glossary

D 2.1.4: Specification of a methodology, general criteria, and test suites for benchmarking ontology tools


ProcessesProcesses

D 2.1.4: Benchmarking methodology

ProcessesInputs Outputs

Task 1 Task n...

Plan 1 Goals identification

2 Subject identification

3 Management involvement

4 Participant identification

5 Planning and resource allocation

6 Partner selection

Experiment 7 Experiment definition

8 Experiment execution

9 Experiment results analysis

Improve10 Report writing

11 Findings communication

12 Findings implementation

13 Recalibration

Methodology processes

Methodology:

Benchmarking process is:• Planned• Collaborative

More Semantic Web orientedMore KW oriented


D 2.1.4: Benchmarking methodologyPlan1.- Benchmarking goals identification• Goals depend on the organisation’s vision, objectives, and strategies.

2.- Benchmarking subject identification

3.- Management involvement• Inform the organisation's management about the benefits of the benchmarking study and its costs.

• Management support is needed to proceed and when implementing changes based on the benchmarking.

4.- Participant identification• Identify and contact the members of the organisation that are involved with the selected tool.

• Select and train the members of the benchmarking team.

5.- Benchmarking planning and resource allocation• The planning must consider time and resources.

• The planning must be integrated into the organisation's planning.

• Analyse the current tools in the organisation.

• Select, understand, and document the tool whose improvement would significantly benefit the organisation, according to: end user needs or expectations, organisational goals, etc.

6.- Benchmarking partner selection• Identify, collect, and analyze information about the tools that are considered the best.

• Select the tools to benchmark with and make contact with someone in their organisations.

• The partner organisations may not belong to KW. Not all ‘best in class’ tools are developed by KW partners.


D 2.1.4: Benchmarking methodologyExperiment

7.- Experiment definition

8.- Experiment execution

9.- Experiment results analysis

• Determine the experimentation plan and method.

• Define the experiment that will be performed. The experiment must collect not just the data on the performance of the tools but the reasons of this performance.

• Communicate the partners the experimentation plan and method and agree on it.

• Perform the experiment according to the experimentation plan and method.

• The collected data must be documented and prepared for analysis.

• Compare the results obtained from the experiments and the practices that lead to these results.

• Document findings in a report, including the best practices found (if any).


D 2.1.4: Benchmarking methodologyImprove10.- Benchmarking report writing

The benchmarking report must provide an understandable summary of the benchmarking study with:

• An explanation of the benchmarking process followed.

• The results and conclusions of the experiments.

• The recommendations on improving the tools.

11.- Benchmarking findings communication

• Findings must be communicated to all the organisation (including identified participants) and to the benchmarking partners.

• Collect and analyze any feedback received.

12.- Benchmarking findings implementation

• Define a planning for the implementation of the benchmarking findings.

• Implement the necessary changes in order to achieve the desired results.

• Periodically monitor the benchmarked tool.

13.- Recalibration

• Recalibrate the benchmarking process using the lessons learnt.

• The benchmarking process should be repeated forever in order to obtain a continuous improvement.


D 2.1.4: Building test suites for ontology tools

• How to develop a test suite.

• The desirable properties that a test suite should have.


D 2.1.4: General supporting tools for benchmarking

List of tools that can be useful when performing benchmarking activities, like:

• Test generators

• Workload generators

• Monitoring tools

• Statistical packages

• ...


D 2.1.4: Benchmarking ontology ... tools

1. Candidate tools

• List of candidate tools to be benchmarked:

• Description

• Reasons for inclusion

2. General evaluation criteria

• Ontology ... tools functionalities with the general evaluation criteria that can be used when evaluating or when benchmarking these functionalities. Related to WP 2.1 topics (scalability, robustness, and interoperability).

3. Test suites

• Test suites for ontology ... tools related to WP 2.1 topics (scalability, robustness, and interoperability).

4. Supporting tools

• Supporting tools specific to ontology ... tools.

5. Conclusion

• Development• Annotation• Querying/inference• Semantic Web Service


D 2.1.4: Glossary

Definitions of terms used in the deliverable:

• Benchmark

• Benchmarking

• Benchmarking partner

• Best practice

• Interoperability

• Robustness

• Scalability

• ...


D 2.1.4: Tasks and responsibilitiesD 2.1.4 Specification of a methodology, criteria, and test suites for benchmarking ontology tools

Raúl García-Castro (UPM)

1.- Introduction Raúl García-Castro (UPM)

2.- Benchmarking methodology Raúl García-Castro (UPM)

3.- Building test suites for ontology tools Raúl García-Castro (UPM)

4.- General supporting tools for benchmarking Raúl García-Castro (UPM)

5.- Benchmarking ontology development tools and tool suites

Raúl García-Castro (UPM)

6.- Benchmarking ontology-based annotation tools ? Raúl asks Sheffield.

7.- Benchmarking ontology querying tools and inference engines

Holger Wache

8.- Benchmarking semantic web service technology ? Holger asks WP2.4 leader

9.- Conclusion Raúl García-Castro (UPM)

10.- Glossary All contributors


<7.5 weeks>

• 19 Nov Contributions of the partners to Raúl

<1 week>

• 26 Nov Draft v0: compilation of the parts (before next meeting)

<3 weeks>

• 17 Dec Draft v1: complete to Quality Assessor (WP leader)

<3 weeks>

• 7 Jan Draft v2: reviewed by QA to Quality Controller (Holger asks Matteo Bonifacio or Roberta Cuel)

<3 weeks>

• 28 Jan Draft v3: reviewed by QC to Quality Assurance Coordinator

<2.5 weeks>

• 14 Feb Final version to European Commission

D 2.1.4: Time schedule


Raúl García-Castro Asunción Gómez-Pérez

<rgarcia,[email protected]>

September 28th, 2004

Benchmarking in WP 2.1

111111 benchmarking in wp 2.1. sep 28th, 2004 © r. garcía-castro, a. gómez-pérez raúl...

Documents