OntologySummit2013: (Track-B) "Extrinsic Aspects of Ontology Evaluation" Synthesis

Track Co-champions: TerryLongstreth & ToddSchneider

Scope: Requirements, evaluation framework, application domains, how evaluation helps

Mission Statement:

The intent is to explore, clarify, and identify gaps, practical and theoretical, in the of evaluation of ontology from a systems perspective using the paradigm of blackbox evaluation.

Extrinsic aspects of ontology evaluation includes subjective factors, measures or metrics, and the range of values of quantifiable attributes. In a systems context evaluations are derived from examination of inputs or stimuli (to the blackbox) and the outputs or externally measurable attributes or behaviors, where those behaviors are controlled or influenced by an ontology.

The ontology in question may be fully embedded/encapsulated within an entity or system, or may be externally accessible (and potentially shared) among multiple entities or systems. The separation of system or entity behaviors which are not governed by an ontology must be accounted for in any ontology evaluation process.

Extrinsic aspects to be considered include,

• interoperability among ontologies
• requirements and their verification
• how metrics can be derived from requirements
• how 'good' requirements relevant to ontology can be crafted
• fitness for purpose
• query performance
• relevant relational database evaluation methods, metrics and techniques
• differences in evaluation among an ontology and instance data
• how evaluation metrics can be derived from examination of test inputs or stimuli
• how evaluations can be used to revise requirements
• how evaluations can be used to correct an ontology

see also: OntologySummit2013_Extrinsic_Aspects_of_Ontology_Evaluation_CommunityInput

---

OntologySummit2013 Communiqué: Track B Contribution

Introduction – Track B

Track B of the 2013 Ontology Summit, Extrinsic Aspects of Ontology Evaluation, focused on aspects of ontology evaluation that did not require direct interaction with, or knowledge of, the ontology. A black-box approach to such evaluation was adopted. The blackbox paradigm properly represents the intent of testing or evaluating extrinsic attributes or qualities of an ontology.

Extrinsic ontology evaluation is a process or group of processes which evaluate ontological commitments in a blackbox mode against specifications.

Extrinsic aspects of ontology evaluation includes subjective factors, measures or metrics, and the range of values of quantifiable attributes that become available for evaluation over the course of the engineering and operations lifecycles. In a systems context, evaluations are derived from examination of inputs or stimuli (to the blackbox) and the outputs or externally measurable attributes or behaviors, where those behaviors are controlled or influenced by ontology. The ontology in question may be fully embedded or encapsulated within an entity or system, or may be externally accessible (and potentially shared) among multiple entities or systems. The separation of system or entity behaviors which are not governed by ontology must be accounted for in any ontology evaluation process. Much of extrinsic evaluation for ontology is indistinguishable from higher levels of system or enterprise testing models. When properly defined, the evaluating functions are not sensitive to specific ontological commitments or syntax.

As was noted during the summit, the boundary of the blackbox varies across lifecycle phases and evaluation processes; the boundary of the blackbox (for evaluation) expands as development moves to completion. This view allows many of the systems and software engineering processes, paradigms, and techniques which have been developed over the last several decades for evaluation across the engineering lifecycle, to be applied with little or no modification (e.g., unit, sub-system, integration, and [full] system testing). These systems and software paradigms are usually based on requirements or specifications (of different levels of detail).

Reuse

During the course of the presentations for Track B we learned that many of the systems and software paradigms and processes can be used for the evaluation of ontologies. Database implementations bear a strong similarity to ontologies and their related knowledge bases (i.e., instance data) and have an overlap in the fragment of first order logic applicable. So it's no surprise that techniques of database evaluation and testing, validating the contents, schema, and functionality within a database, can be applied to ontology evaluation. Among extrinsic factors in need of evaluation are error conditions, degraded mode operations, load and capacity testing/performance (critical for systems that provide services to a larger enterprise). Validation of the ontology and instance data integrity using system interfaces (e.g., create, read, update, and delete operations) should be performed.

Security

Of particular note are issues involving security vulnerabilities (of the system using ontologies) that may impact the validity of an embedded ontology. As in the case of the use of relational databases, systems using ontologies may have security vulnerabilities. Though ontologies by themselves may appear to pose no security risks, their infrastructure (e.g., reasoners, triple stores, etc.) or use in a system provides opportunities to introduce security vulnerabilities. As an example, consider vulnerabilities similar to an SQL injection attack. None of the evaluation methods or techniques discussed during the summit adequately (if at all) addressed aspects of ontology evaluation w.r.t. security. This is an open area for research.

Dynamics

There appears to be an implicit assumption that the ontology and its theory (of the domain it’s representing) is static over it course of use (in a system). Ontologies can provide a system with dynamic capabilities not possible with conventional database technologies. During operations the ontology can change, in addition to the instance data. These new possibilities of dynamic capabilities present a dilemma, for ontology and systems and software engineering, in that existing paradigms may not be applicable. How does evaluation change if the ontology (i.e., the T-box) is expected to change during operational use? Do such changes alter inferences or actions taken against the earlier (i.e. unchanged) ontological commitments? Do the evaluation models and methods in use for such an ontology include revisiting those prior actions to ensure consistency, accuracy and correctness goals are being met both before and after the changes have been applied.

The notions of ontology evaluation(s) presented did not explicitly address dynamics of an ontology (i.e., changes to the taxonomy, predicates or relations over the course of its use) nor the evaluations thereof. However, there are efforts underway to fill this gap with tools and paradigms as suggested from the NEMO project.

Automation

In software engineering there are tools that automate parts of the engineering lifecycle (e.g., regression analysis). For ontology evaluation such tools do not currently exist to the extent that they provide feedback directly about an ontology (used in the system under evaluation). However there are projects underway that may automate evaluation of some criteria.

Scope

Ontology evaluation takes place in some context and that context imparts a scope (of applicability or validity). This context and scope directly impacts the importance of evaluation criteria in achieving the purpose of an evaluation. The appropriate values or value ranges for particular evaluation criteria may differ depending on evaluation context and its scope. The evaluation criteria, or their importance, may also depend on the lifecycle phase(s) in which they are applied. Most importantly how an ontology is expected to be used and in what operational domains an ontology is intended to be used, will impact evaluation criteria decisions.

Requirements

Behind, and supporting any notion of, assessment or evaluation, from a systems perspective, are requirements. How should one create specifications that adequately represent semantic requirements, or that specify conditions for the intended interpretations or models of the ontology and that can be tested validated? How can tests be written that verify ontology requirements? While it should be expected that such requirements will be derived from more general systems or software requirements, the actual process that leads to appropriate requirements on the ontology and intended interpretations has not been studied extensively. Be that as it may, it is apparent that requirements development processes that meet the needs of ontology evaluation, as well as the needs of associated operational uses, must identify, configure and manage system level requirements with explicit descriptions of the expected use/intent, behavior, and results from the use of ontology.

Such procedures and processes must accommodate the potential complexity of systems employing ontologies, from an extrinsic or systems perspective, and in order to be able to relate testing of extrinsic aspects to overall ontology quality, and to be able to make use of existing evaluation paradigms. An initial approach to create sound ontology requirements derivation processes may be the paradigm of patterns. From the identification of appropriate patterns for deriving ontology requirements more specific patterns or processes may be developed.

At what point should a requirements development process be expected to produce full, formal logical expressions against which the theories or expressions in an ontology can be tested for satisfaction of the semantic expectations of the ontology?

Systems and software engineering disciplines have experimented since the inception of large scale computing with methods for automating tests to evaluate requirements satisfaction. To support such efforts for ontology, requirements will need to be developed and expressed in formal fashion, equivalent or better in expressiveness from that used to specify the ontology itself. It has been demonstrated that proof techniques can be employed for three cases of requirements specification evaluations. Presumably, once the requirements themselves have been validated using these techniques, they can become specifications for the ontology that is the ultimate product of the exercise.

Without adequately expressive requirements that focus on ontology, any evaluation will fail to meet expectations. Moreover, the applicability and overall value of requirements to ontology evaluation will be related to their level of detail and the lifecycle phase at which they may be evaluated.

While requirements development and management are mature aspects of systems and software engineering disciplines, establishing reliable methods and tools for requirements refinement with a goal of economically producing correct and useful ontologies would provide leverage for ontology development and use, more securely coupling them to the associated deliverable system(s).

-- maintained by the Track-B champions: TerryLongstreth & ToddSchneider ... please do not edit