Draft Ontology Summit 2007 Framework Assessment Criteria

Initial draft by Ken Baclawski / 2007.06.25

This document is an attempt to provide criteria for assessing the framework dimensions of an ontology. The criteria given here were based on the OntologySummit2007_Communique, the evaluations done during the Ontology Summit 2007, and other sources. All dimensions use a scale of 1 to 5 for the sake of uniformity.

• • Level 1: Glossary. Catalogs, controlled vocabularies, dictionaries and glossaries are all included. Tag ontologies are also on this level.
    • Sublevel A: No definitions
    • Sublevel B: Terms are defined using natural language.
  • Level 2: Thesaurus. Terms are related by synonymy and antonymy, and they may be organized in an "broader than"/"narrower than" hierarchy. Levels 1 and 2 are informal.
    • Sublevel A: No hierarchy
    • Sublevel B: Classification hierarchy
  • Level 3: Class and Structure Hierarchies. Terms represent formal classes. This is the lowest level that can be considered formal (in the sense of being defined using mathematical structures). XML schemas and relational database schemas are on this level, but an ER diagram would be on level 4.
    • Sublevel A: Ontology only has classes.
    • Sublevel B: Ontology includes instances.
  • Level 4: Properties. This level includes frame-based languages, RDF and ER diagrams.
    • Sublevel A: Value constraints are not supported.
    • Sublevel B: Value constraints are supported.
  • Level 5: Logic. OWL and FOL ontologies are all on this level.
    • Sublevel A: Less expressive than FOL.
    • Sublevel B: At least as expressive as FOL.
  • The levels were obtained by combining pairs of levels in the ontology continuum of DeborahMcGuinness.

• • Level 1: Informal; unstructured. For example, folksonomies.
  • Level 2: Low structure. For example, dictionaries, glossaries.
  • Level 3: Medium structure. For example, taxonomies based on broader/narrower rather than subclass.
  • Level 4: High structure. For example, faceted classification schemes.
  • Level 5: Formal structure. For example, directed graphs.
  • The difference between this dimension and expressiveness is how well organized the semantics is encoded. So the levels are the same as the expressiveness levels except that they were shifted by one level to fit with the description in the communique a little better. The evaluation for this dimension cannot be higher than the corresponding expressiveness level. The reason for having a second dimension is to deal with unstructured ontologies that are specified in a highly expressive language (e.g., a folksonomy specified using OWL).

• • Level 1: Very coarse; limited. Broad subject index with around 10 or 20 classifications.
  • Level 2: Coarse. For example, an upper ontology with about 100 classes.
  • Level 3: Medium. Ontologies with 1K to 10K classes.
  • Level 4: Fine. Ontologies with 10K to 100K classes.
  • Level 5: Very fine. Ontologies with 100K or more classes.
  • This dimension can be measured in many ways. The size and density may be the most useful:
    • The number of nodes, links and/or axioms (shown above). This is measuring the size of the ontology.
    • Relative size with respect to the scope or domain. This takes into account the size of the domain. However, it it not clear how one measures the size of the domain.
    • The average density (e.g., average number of axioms per term). For the less formal ontologies it is the average density of connections at each node or term. Thus a term catalog has density 0, a glossary or tag ontology has density 1, a simple taxonomy has density 2, and so on. It differs from expressiveness in that it measures the extent to which features are actually used rather than whether features are available.

• • Level 1: Multiple intended uses
  • Level 2: Two intended uses
  • Level 3: Classification; search; retrieval
  • Level 4: Interoperability; integration
  • Level 5: Mathematics; system specification
  • There is no meaning to the ordering of the levels. I ordered them in this way so that there is some correlation with the other dimensions.
    • Comments: (--PeterYim / 2007.06.26-04:19PDT)
      • (a) Ken, can you elaborate on your L-1 & L-2, why 'Multiple intended uses" and 'Two intended uses', instead of, say, 'Single intended use:______' and 'Multiple intended uses'?
      • (b) rather than 'Level(s)', may I suggest using the label 'Type(s)' instead (for this dimension, especially, but possibly for others, or even all dimensions too!)
    • Response: (--KenBaclawski / 2007.06.27-20:53:42EDT)
      • (a) The problem is that some other dimensions are linear while this dimension does not seem at first to have such an interpretation. However, perhaps it does. When the intended use is for a system specification, then the ontology is highly focused on that application. I set this as the highest level (or should it be the lowest??) Interoperability and integration are less focused and broader uses, but they still have a focus on a class of applications. So I set them at the next lower level. Search and retrieval are even less focused, so I set them at the next level. Finally, if the intended use includes several of these, then the purpose of the ontology is still less focused. So these were assigned to the lowest levels.
      • (b) The dimensions do seem to split between three that have a linear structure (expressiveness, structure and granularity), and four do not seem to fit very well as linear dimensions. Attempting to fit them all on the same scale is a bit procrustean, but any other uniform mechanism would also have this problem. At least the dimensions that do work as linear dimensions should be kept that way.

• • Level 1: No reasoning.
  • Level 2: Ad hoc reasoning.
  • Level 3: Some reasoning.
  • Level 4: Complex reasoning but not necessarily logical or rule-based. Reasoning is encoded with queries or procedures.
  • Level 5: Logical or rule-based reasoning.
  • A variety of automated reasoning strategies are currently being employed. The levels differ in the degree of complexity and sophistication.

• • Level 1: Descriptive.
  • Level 2:
  • Level 3:
  • Level 4:
  • Level 5: Prescriptive.
  • This dimension seems to be two-valued, but intermediate levels could be employed for ontologies in which correctness is a concern but not an overriding one.

• Governance
  • Level 1: Casual. No organization controls any aspect of the ontology.
  • Level 2:
  • Level 3: Controlled. While terms in the ontology (syntax) are controlled, the semantics is not. A controlled vocabulary would be at this level.
  • Level 4:
  • Level 5: Normative. Both the ontology and its semantics are tightly controlled.

--

This page is maintained by: Ken Baclawski

Please post any comments about the content as a subtopic within each framework dimension.