• All participants are encouraged to provide their insights here, regarding Track A: Intrinsic Aspects of Ontology Evaluation

Some raw material from SteveRay and LeoObrst:

This document has as scope the dimensions of ontology evaluation, methods, criteria, and the properties to measure to ensure better quality ontologies.

Intrinsic Aspects of Ontology Evaluation Ontologies are built to solve problems, and ultimately an ontology’s worth can be measured by the effectiveness with which it helps in solving a particular problem. Nevertheless, as a designed artifact, there are a number of intrinsic characteristics that can be measured for any ontology that give an indication of how “well-designed” it is. Examples include the proper use of various relations found within an ontology, proper separation of concepts and facts (sometimes referred to as classes vs. instance distinctions), proper handling of data type declarations, embedding of semantics in naming (sometimes called “optimistic naming”), inconsistent range or domain constraints, better class/subclass determination, the use of principles of ontological analysis, and many more.

We focus in the communiqué on the evaluation of ontologies under the following intrinsic aspects:

• • Is the ontology free of obvious inconsistencies and errors in modeling?
  • Is the ontology structurally sound? How do we gauge that?
  • Is the ontology appropriately modular?
  • Is the ontology designed and implemented according to sound principles of logical, semantic, and ontological analysis?
  • Which intrinsic aspects of ontology evaluation are of greater value to downstream extrinsic ontology evaluation?

Section C (2) What are the desirable characteristics of ontologies? Do they depend on the intended use of the ontology? And how are they measured over the life cycle of the ontology?

1) Partitioning the Ontology Evaluation Space:

a. Intrinsic Evaluation Aspects: Intrinsic ontology evaluation, from our perspective, consists of two parts: Structural Intrinsic Evaluation and Domain Intrinsic Evaluation.

Structural Intrinsic Evaluation: Ontology evaluation that does not depend at all on knowledge of the domain being modeled, but does draw upon mathematical and logical properties such as graph-theoretic connectivity, logical consistency, model-theoretic interpretation issues, inter-modularity mappings and preservations, etc. Structural metrics such as branching factor, density, counts of ontology constructs, averages, and the like are intrinsic. Some meta-properties such as adherence to implications of transitivity, symmetry, reflexivity, and equivalence assertions may also figure in intrinsic notions.

In general, structural intrinsic criteria are focused only on domain-independent notions, mostly structural, and those based on the knowledge representation language.

Some examples of tools and methodologies that address intrinsic ontology evaluation:

I. Oops! Evaluation web site at http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp and described by MariaPovedaVillalon

II. OntoQA to develop metrics for any ontology based on structural properties and instance populations, described by SamirTartir

III. PatrickLambrix’s debugging of Isa-a taxonomic structures, especially with mappings between ontologies

Domain Intrinsic Evaluation: Evaluation where some understanding of the domain is needed in order to, for example, determine that a particular modeling construct is in alignment with the reality it is supposed to model. It may be that some meta-properties such as rigidity, identity, unity, etc., suggested by metaphysics, philosophical ontology, semantics, and philosophy of language are used to gauge the quality of the axioms of the ontology, including e.g., the subclass/isa taxonomic backbone of the ontology and other structural aspects of the ontology.

Most of the aspects of this category focus on ontological content methods such as better ontological and semantic analysis, including meta-property analysis (such as provided by methodologies like OntoClean, etc.)

Domain knowledge and better ways to represent that knowledge do come into play here, though divorced as much as possible from application-specific domain requirements that come more explicitly from extrinsic evaluation issues. At the extrinsic edge of domain intrinsic evaluation, the context-independent measures from Structural Intrinsic evaluation begin to blend into the very context-dependent, application issues of Extrinsic evaluation.

Some examples of tools and methodologies that address domain intrinsic ontology evaluation:

I. OQuaRE framework described by AstridDuqueRamos

II. OntoClean (Guarino and Welty)

III. MariaCopeland: Ontology Evolution and Regression Testing

IV. MelissaHaendel: Ontology Utility from a biological viewpoint

V. EdBarkmeyer: Recommended practices with mapping vocabularies (especially code-lists) to ontologies.

b. Extrinsic Evaluation Aspects: Ontology evaluation where the structure and design of the ontology is opaque to the tester, and the evaluation is determined by the correctness of answers to various interrogations of the model. In general, application requirements and domain requirements that are specifically needed by particular applications are the focus of extrinsic evaluation.

2) Evaluation Across the Ontology Lifecycle

Every criterion should be evaluated at each point in the ontology lifecycle, but with some criteria being more important (necessary/sufficient) at some points more than others. In other words, a better ontology evaluation methodology might define necessary and sufficient criteria (and their measures) derived from both intrinsic and extrinsic aspects that apply to different points in the ontology lifecycle.

In addition, the determination of these necessary or sufficient criteria may be subject to constraints: for example, though initially an intrinsic criterion of logical consistency of the ontology may be imposed as a necessary property at the beginning of the first phase of ontology development, it might be relaxed subsequently when it is determined that a different semantics will apply in how the ontology is interpreted within a given application (e.g., if the application-specific reasoning will not observe the full description logic Open World Assumption, but instead interpret the ontology under a locally Closed World Assumption).