OntologySummit2012: (Cross-Track-A1) "Ontology Quality and Large-Scale Systems" Synthesis

Track Co-Champions: Dr. AmandaVizedom & Mr. MikeBennett

Mission Statement:

This cross-track aspect will focus on the evaluation of ontologies within the context of Big Systems applications. Whether creating, developing, using, reusing, or searching for ontologies for use in big systems, engineers, architects, designers, developers and project owners will encounter questions about ontology evaluation and quality. How should those questions be answered? How do we know whether an ontology is fit for use in (or on) a large-scale engineered system or a large-scale systems engineering effort? This cross-track aspect ties together the evaluation-related discussions that arise within the Summit Tracks and individual sessions, providing a context in which to take up and address the issues generally. Specific focus will evolve with recurring themes, potentially including such topics as ontology quality characteristics, fitness for purpose, requirements, metrics, evaluation methodologies and resources.

see also: OntologySummit2012_Quality_CommunityInput

---

Quality in its most formal sense refers to the rigorous use of requirement specifications, requirements-centric design, multi-stage testing and revision, and other risk-management and quality assurance techniques. This is a hallmark of systems engineering, distinguishing it from less rigorous systems creation activities and essential to success in developing large-scale and complex systems and managing them throughout their life-cycles. Various sub-domains within systems engineering apply these risk- and complexity- management techniques to systems overall, to system components, to component interfaces, and engineering, interface, and other processes. Quality at any of these levels is defined in terms of the degree to which any one of the system, component, process, etc., meets the specified requirements. Analysis and specification of requirements and functions at each of these levels, along with identification and application of relevant quality measures, is an essential part of good systems engineering. In the ontology cross track we focused on this formal definition of quality as applied to ontologies, as well as considering more informal definitions of quality. In particular, we explored the management of ontology quality within large-scale systems engineering projects, looking for both lessons learned and areas needing better support.

It emerged that projects involving ontologies as part of engineered systems, or ontologies as part of systems engineering processes, tend to have little or no ontology quality assurance measures. Notably, even in systems engineering projects in which rigorous attention is spent on identification and specification of other components and aspects of systems, identification and specification of ontology requirements is given little to no attention.

Reasons for this exception to otherwise rigorous methodology vary, but can include a belief that ontologies are non-technical artifacts, not subject to engineering methodologies; a lack of necessary resources; an absence of concerns for related areas of accountability; a belief that variations in ontology do not affect end functionality or performance; or a belief that, however desirable quality assurance measures for ontologies might be, no such implementable, usable, reliable measures exist.

We considered two kinds of scenario in which ontologies may be used in a big systems project. The ontology may be an integral part of the solution (ontology as application), or it may be a part of the development of some system (ontology as business conceptual model). Many of the same quality assurance parameters may apply in both cases, but the requirements, and the use to which the ontology may put, will be very different.

The academic literature contains many approaches to formal quality of ontologies, many of them very mathematical. There less coverage in the literature of what it is that makes the terms in an ontology truly meaningful, or to link formal ontology requirements as such, to implementation. That is, there are many approaches to ontology quality but few quality assurance measures. This was addressed in a number of presentations at our cross-track session, with a strong focus both on quality factors for ontologies and on how these fit into the role of the ontology in various types of project.

The findings in the Federation and Integration tracks focuses on one of two usage scenarios for ontology: using ontology as common conceptual model).However, many of the available ontology quality measures focus on ontology as an application, with mathematical considerations like decidability and so on.

Where do techniques like ontology design patterns (Guarino ,Gangemi and others) fit in with this picture? Are these quality measures that run across both usages scenarios?, to the extent that the best application of these patterns leads to ontologies which respect semantics? Are there measures for meaning as well as measures for ontology as application? What measures apply to one or other or both? Is there research targeted at these distinct usage scenarios or are they just targeted at "ontologies" in the round?

This last may point to an interest direction for possible research: given an understanding of the different roles for ontologies as articulated in many of the tracks in this year’s Summit, we wonder if there are possible new directions for research into quality measures that are targeted to the intended purposes for which ontologies are developed?

Note also that at least one of the presentations in our cross-track session gave the impression that for the usage scenario in which the ontology is used as a formal conceptual model (and therefore has its role within the quality assurance process for other artifacts in a project) it was not appropriate to also apply quality measures to the ontologies themselves. This is an idea which is worth unpacking and challenging, with implications for large systems quality assurance either way.

To further explore and challenge these assumptions, a survey is being assembled which aims to identify the precise scenarios in which people are using ontologies in the context of big systems, and to get some idea of whether quality issues were considered and if so, how these were addressed.

The good news here is that these issues are comparatively easy to address with better documentation and dissemination of approaches to ontology quality management that are already used. There is work to be done, but a substantial improvement can be accomplished via wider attention to ontology quality and better documentation and dissemination of existing knowledge. Summit exchanges demonstrated that systems engineers are open to deploying that knowledge and ontologists are open to creating resources that systems engineers can use.

A greater challenge lies in understanding ontology requirements as they derive from usage characteristics. While the bulk of literature on ontology quality has historically been addressed to quality in a different sense, the trend is toward greater attention to factors related to fitness for purpose, and the relevance and usability of this research to systems applications is consistently increasing. To the extent that broadly-experienced ontologists develop a sense of what kind of ontology is needed for what kind of application, this sense remains largely in the head of the ontologists. Broadly-experienced ontologists can be difficult to identify, while less experienced ontologists may be skilled at developing certain kinds of ontologies but unaware of variations across application type.

Furthermore, to the extent that ontologists disagree, principles of variation in requirements should be not only explicit, but based on more than individual, anecdotal experience. This basis does not yet exist. To provide resources, including tools and methods, that reliably support ontology quality management in big systems engineering contexts, this basis is needed. That area is a forward challenge for ontology researchers. The survey developed for the summit is designed, in part, to provide some initial information to stimulate such research, and to suggest particular areas likely to be worth investigation.

-- maintained by the X-Track-A1 champions: AmandaVizedom & MikeBennett ... please do not edit