Actions

Ontolog Forum

This is one of the "OntologySummit2013_Hackathon_Clinics" Projects ...

(HC-03) Evaluation of OOPS!, OQuaRE and OntoQA for FIBO Ontologies

Project Champions: Mike Bennett, MariCarmenSuarezFigueroa, MariaPovedaVillalon, JesualdoTomasFernandezBreis, AstridDuqueRamos, Samir Tartir

Event Date / Time: (Day-3) Sat 2013.04.13 ... see: [ details]

Archived work output can be found here

Logistical Details

Details (Saturday and Sunday)

We have a one and three quarter hour kickoff session using the slides to present an overview of what we wanted to achieve. After this we moved onto Go To Meeting for three sessions of a little under two hours each.

1. Dial-in for the "Open Webcast" segment (Sat 9am - 10:30 Eastern US Time; 2pm - 3:30 London 3pm - 4:30 Europe)

  • a. Phone (US): +1 (206) 402-0100 ... Conference ID: 141184# ; or
  • b. Skype: joinconference ... Conference ID: 141184#

Details (Sunday only - we will use the chat room and OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE/GoToMeeting only for this)

We reconvened on Sunday 14 April at 8am Pacific / 11am Eastern US / 4pm London / 5pm Europe to review results and GoogleDoc table. We ued the OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE/GoToMeeting facility below but not the Conference Bridge.

2. We will use the chat room facility throughout this session - this is at http://webconf.soaphub.org/conf/room/hc-03

3. We will use OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE/GoToMeeting (click through for phone and web link) for this session - details below.

Working Document (Google Docs)

Throughout the two days and beyond, we are working on the following document:

https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit

Outcomes

Saturday 13 April

In the initial kick-off segment we went through the [ slides ] slide presentation.

Following this, we took a set of ontologies provided by the EDM Council from the FIBO Foundations proposed specification and applied these to the three sets of tools OOPS! OQuaRE and OntoQA. These are "Business Conceptual Ontology" models.

Here is a link to the (unedited) in-session chat room:

http://ontolog.cim3.net/file/work/OntologySummit2013/Hackathon-Clinics/HC-03_FIBO-OOPS-OQuaRE-Evaluation/OntologySummit2013_hc-03_chat-transcript_unedited_20130413a.txt

Attendees Day 1

Summary We had practical demonstrations of all 3 tools OOPS! OQuaRE and OntoQA on real FIBO OWL ontologies, and looked at what measures the tools were showing us. We explored a couple of the metrics in depth. We looked at the OQuaRE table of quality measures and considered some changes and additions. I've tried to capture some of these in edits and additions to the table. Others will also be working asynchronously on this table.

We had a couple of strong new ideas for quality measures: Jacobus Geluk suggested having a suite of SPARQL queries that can be used as regression tests or for test-driven agile development, along with example instance data. The OntoQA tool has some tests that can be applied separately to that test data. Simon Spero suggested that the ACE plug-in for Protégé can be used not only to provide business descriptions, but as a good quality measure, with a human in the loop, to test whether some of the assertions in the ontology really mean what we meant them to mean.

On OOPS!, we imported the whole set of the FIBO ontologies we had, into a single test ontology and ran the pitfall scanner on this, analyzing all the imported ontologies. We established that most of the metrics were ones we would want to apply to the FIBO Business Conceptual Ontologies, not just operational ontologies. We can determine the required values for some of these, so for instance in Foundations we want a higher level of confidence in reusability and changeability.

On OQuaRE we saw the full set of tests run against several of the FIBO Foundations ontologies. We looked at some of these in depth. These implement specific mathematical algorithms which are documented in the OQuaRE document, with cross reference to the business definitions of quality measures which they support. Some of these measures may have higher priorities in FIBO-Foundations than other ontologies, e.g. reusability.

We had a demonstration of the OntoQA tool, on the FIBO Foundations ontologies. This has metrics for individual knowledge bases as well as for the conceptual model (these may also be applied to test data). These are presented in a spreadsheet, and some of them correspond to items in the OQuaRE table.

Future versions of the tools may allow more automatic configuration of these settings. In OOPS! we would want to be able to replace their annotation measure (which looks at rdfs:label), with measures that look at SKOS definitions and the like. There are other configuration changes and new developments which would benefit FIBO, and / or the results can be filtered according to the specific requirements with what there is now.

We had an in depth discussion on some FIBO modeling parameters, including the "Archetype" concept (similar to ontology design patterns) and how this helps with ontology quality. It should be possible to create code which would validate the application of archtypes and related ontology patterns. We also learned more about how the FIBO Archetype concept can be implemented in OWL more formally than it is at present.

At the end of the session we agreed to work offline on the Google Docs document in preparation for the Day 2 session.

Sunday 14 April

On the Sunday we worked through the Google Docs document - all our workings are on that document:

https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit

Attendees

Summary

We went through the table one row at a time. For each entry we looked at the quality requirement defined for that entry, expanded this into specific measures to look at in the ontology. Then we reviewed the OQuaRE metrics in the table and added any other OQuaRE metrics that would apply.Then we added metrics from OOPS! and OntoQA which would measure some aspect of the ontology which would be relevant to the same overall quality requirement.

We got to the end of the "Functional Adequacy" section. The next section will be Maintainability.

We expect to have a follow-up call to continue where we left off with this document.

Abstract

This ontology clinic aims to explore the application of ontology quality measures to ontologies produced under the Financial Industry Business Ontology (FIBO) umbrella.

In this clinic we will explore the application of the OOPS! and OQuare methodologies and tools to two styles of ontology developed under the FIBO umbrella: Business Conceptual Ontologies (BCOs) which are the FIBO standards themselves; and example "Operational Ontologies" derived from these for deployment in semantic technology applications.

We would look to establish which types of measure should be applied to each type of ontology and apply the relevant tools and techniques to these. In the case of OQuaRE, these measures will be applied in two ways: 1) application of the complete quality model; 2) application of the OQuaRE subcharacteristics and metrics relevant for FIBO evaluation, with the possibility of modifying the existing associations subcharacteristics-metrics.

From this activity we hope to make the first steps towards defining a formal quality process for the future development of formal standards under the FIBO umbrella, a set of quality assurance parameters for users who need to extend the FIBO BCO locally for their own conceptual semantic modeling, and a set of guidance notes, validation and verification techniques etc. for developers of semantic technology applications based on the FIBO standards. We will evaluate to what extent OQuaRE could be a start point for this quality process.

see: [ Project document ] . [ slides ]

Collaborators

OOPS!

  • Mari Carmen Suarez-Figueroa, Maria Poveda-Villalon - Ontology Engineering Group. Departamento de Inteligencia Artificial. Facultad de Informática, Universidad Politécnica de Madrid, Spain.

OQuaRE

  • Jesualdo Tom's Fernandez-Breis, Astrid Duque-Ramos - Departamento de Informática y Sistemas, Universidad de Murcia, Spain.

OntoQA

  • Samir Tartir - Computer Information Systems, Philadelphia University Jordan

Others

  • We are open to working with any and all others who may have tools, techniques or methodological material which may be applied either to business conceptual ontologies, to operational OWL ontologies or both.

Ontologies Involved

There are two distinct kinds of FIBO ontology:

  • FIBO Business Conceptual Ontologies (the proposed FIBO standards)
  • FIBO Operational Ontologies (RDF/OWL ontologies for reasoner-based applications)

Of these, will specifically bring the Business Conceptual Ontologies to this Clinic, as the following:

  • FIBO Foundations (sample ontologoies from this)
  • FIBO Business Entities - possibly

Objectives / goals

FIBO Background

FIBO is being developed as a series of "Business Conceptual Ontologies" (BCO) for concepts in the financial industry, that is, ontologies which represent industry terms, definitions and relationships at the level of conceptual models. Conceptual models, by definition, should not reflect application constraints. From these, we anticipate that users would derive operational ontologies for specific use cases, which would of course be subject to the relevant application constraints.

An open question in the development of FIBO is what ontology quality measures should be applied to the "Conceptual" ontologies, and which of the established OWL modeling best practices are applicable to such an ontology. That is, which requirements of semantic technology applications should be applied to the conceptual ontologies without compromising their requirements as conceptual models.

To complicate this question further, the BCOs are intended to be presented to business domain subject matter experts for validation, and local extensions of the BCO are intended to be understood and maintained as a business domain asset. In order to support business-friendly presentation in the currently available modeling tools, some compromises have been made in the way that the OWL language is used. Some of those compromises could be reversed once there are better ways of presenting these ontologies to a business audience.

Meanwhile, we expect potential users of the standards to derive "operational ontologies" from the conceptual ontologies, just as a conventional application developer would develop logical designs from conceptual models such as requirements catalogs. These operational ontologies must of course be subject to the quality requirements of any application (validation and/or verification of the delivered item against the stated business requirements), and since they are OWL ontologies, must also be subject to the quality constraints that are applicable to operational OWL ontologies.

OQuaRE Background

OQuaRE is a framework for Ontology Quality Requirements and Evaluation based on ISO/IEC 25000:2005, the standard for Software Quality Requirements and Evaluation. OQuaRE defines intrinsic and extrinsic quality criteria in terms of quality sub-characteristics. OQuaRE aims to define all the elements required for ontology evaluation: evaluation support, evaluation process and metrics. The current version of OQuaRE includes, so far, the quality model and the quality metrics:

1. The quality model is composed of a set of quality characteristics such as structural, functional adequacy, maintainability etc. and its associated sub-characteristics such as reliability, reusability, availability, redundancy, consistency, etc.
2. The quality metrics have been taken from the state of the art in ontology, such as Depth of subsumption hierarchy, Class Richness, Tangledness etc.

Complete definition of OQuaRE is available at:

Objectives

The objectives of this clinic are as follows:

A: Evaluation of FIBO Business Conceptual ontologies

  • Identification of relevant quality metrics and aspects for FIBO Business Conceptual Ontologies
  • Use and evaluation of ontology quality tools for the evaluation of FIBO Business Conceptual Ontologies
  • Applying these measures to the the "FIBO-Foundations" ontologies using the available tools
  • Consider how this can inform the formal methodology for FIBO development

B: Consider criteria for future Operational Ontologies

  • Identify the relevant quality measures for a FIBO-derived Operational Ontology
  • Consider how the application use case can be shown to be satisfied by a given operational ontology
  • See whether this can be formalized in such a way that formal "Conformance Points" can be defined which are of a suitable level of clarity and repeatability to be included in the OMG specification as formal Conformance criteria
  • Even if these requirements and tests cannot be formalized, consider what application guidelines can be created around these tools and techniques, to guide users of FIBO in creating robust ontology based applications which conform to their stated user requirements

Deliverables

  • Elements of a formal methodology for development of FIBO Business Conceptual Ontologies
  • Elements of a formal methodology for local extension of FIBO BCOs by end users, to create their own ontologies at the same conceptual level (for onward use either in conventional technology model driven development, data integration or the development of operational ontologies for semantic processing)
  • Formal conformance points for operational ontologies (new textual material for future versions of the FIBO OMG specifications)
  • Notes and "how to" material for developers of semantic technology applications that use FIBO
  • Formal Findings on the ontology quality tools

Remarks

We see this clinic as a vital first step in our development both of the formal methodologies for FIBO standards development and of the conformance points and developer guidance necessary for end users to make practical use of FIBO in semantic technology-based applications. The tools and techniques which are applied in this clinic (OOPS!, OQuaRE and any others which come to light) will likely form a part of those formal processes going forward.

Resources / References

OOPS!

OQuaRE

OntoQA



Retrieved from "https://ontologforum.com/index.php?title=OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE&oldid=2217"