Actions

Ontolog Forum

Ontology Summit 2013: Panel Session-14 - Thu 2013-04-18

Summit Theme: "Ontology Evaluation Across the Ontology Lifecycle"

Session Topic: Ontology Summit 2013 Communique: Final Draft Review

Communique Co-Lead Editors & Session Co-chairs: ... intro slides (from the first review session)

  • Dr. FabianNeuhaus (NIST) and Dr. AmandaVizedom (Ind. Consultant)

Reference:

Archives

Abstract

OntologySummit2013 Session-14: Ontology Summit 2013 Communique: Final Draft Review - intro slides

This is our 8th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD with the support of our co-sponsors. The theme adopted for this Ontology Summit is: "Ontology Evaluation Across the Ontology Lifecycle."

Currently, there is no agreed methodology for development of ontologies, and there are no universally agreed metrics for ontology evaluation. At the same time, everybody agrees that there are a lot of badly engineered ontologies out there, thus people use -- at least implicitly -- some criteria for the evaluation of ontologies.

During this Ontology Summit, we seek to identify best practices for ontology development and evaluation. We will consider the entire lifecycle of an ontology -- from requirements gathering and analysis, through to design and implementation. In this endeavor, the Summit will seek collaboration with the software engineering and knowledge acquisition communities. Research in these fields has led to several mature models for the software lifecycle and the design of knowledge-based systems, and we expect that fruitful interaction among all participants will lead to a consensus for a methodology within ontological engineering. Following earlier Ontology Summit practice, the synthesized results of this season's discourse will be published as a Communique.

As a continuation from last week's session, in this 14th virtual session of the Summit, the co-lead editors will present the final draft of the summit communique. It should be noted that, while suggestions on smaller, more bounded items will still be accepted by the Communique co-lead editors until April 25, this (April 18th) is the last day for additional comments/suggestions of substance. During the session, the session chairs will moderate an open discussion among contributing editors and the community, in an effort to get this collaboratively authored work completed before the OntologySummit2013_Symposium, where this Communique will be presented and adopted.

More details about this Ontology Summit is available at: OntologySummit2013 (homepage for this summit)

Agenda

OntologySummit2013 - Virtual Session-14

  • Session Format: this is a virtual session conducted over an augmented conference call
  • 1. Opening (co-chairs) ... [ slides & material ]
  • 2. Presentation of plans and first draft of the communique (co-lead editors)
  • 3. Review by section and open discussion [Co-lead editors, co-editors & All] ... please refer to process above
  • 4. Wrap-up / Announcements (co-chairs) - [5 min.]

Proceedings

Please refer to the above

IM Chat Transcript captured during the session

see raw transcript here.

(for better clarity, the version below is a re-organized and lightly edited chat-transcript.)

Participants are welcome to make light edits to their own contributions as they see fit.

-- begin in-session chat-transcript --

[9:09] Peter P. Yim: Welcome to the

Ontology Summit 2013: Virtual Session-14 - Thu 2013-04-18

Summit Theme: Ontology Evaluation Across the Ontology Lifecycle

Session Topic: Ontology Summit 2013 Communique: Final Draft Review

  • Session Co-chairs

- Dr. Fabian Neuhaus (NIST) and Dr. Amanda Vizedom (Ind. Consultant)

Agenda:

1. Opening (co-chairs)

2. Presentation of plans and second draft of the communique (co-lead editors)

3. Review by section and open discussion [Co-lead editors, co-editors & All]

4. Wrap-up / Announcements (co-chairs)

Logistics:

  • (if you haven't already done so) please click on "settings" (top center) and morph from "anonymous" to your RealName (in WikiWord format)
  • Mute control: *7 to un-mute ... *6 to mute
  • Can't find Skype Dial pad?
    • for Windows Skype users: it may be under the "Call" dropdown menu as "Show Dial Pad"
    • for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later or the earlier Skype versions 2.x,)

if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it.

  • Note: ... it has come to our attention that our conference bridge provider is running into some problems with the "joinconference" skype connections.

In case anyone gets in trouble, please try to call the phone numbers instead (e.g. from your phone, skype-out, google-voice, etc.)

Attendees: Fabian Neuhaus (co-chair), Amanda Vizedom (co-chair), AstridDuqueRamos, Bob Smith,

Bobbin Teegarden, Doug Foxvog, Jack Ring, Joel Bender, JohnMcClure, Julien Corman, Ken Baclawski,

Lamar Henderson, Matthew West, Michael Grüninger, Mike Dean, Peter P. Yim, Steve Ray, Terry Longstreth

Proceedings:

[9:22] Joel Bender: howdy!

[9:32] Fabian Neuhaus: working draft is at:

https://docs.google.com/document/d/1jeU5DL9JuNiyxv7BDOMr0VgFIWmSKVHXcnLlcHF-Zg0/edit#

[9:35] anonymous morphed into Doug Foxvog

[9:43] JohnMcClure: hi DougFoxvog!

[9:48] Terry Longstreth: [in agreement with SteveRay's suggestion that we should include some

pointers to tools on the communique] A URL pointing to a Tool Summary page should also be in the

(TBS) Executive summary

[9:50] Terry Longstreth: Amanda / Fabian - Is the Purpose supposed to be part of the Exec summary, or

a separate section?

[9:57] Terry Longstreth: There are often two reasons for an executive summary: 1) to allow someone to

quickly glean the gist of the summarized document 2) to entice them to read the rest. It's not easy

to strike a balance between the two

[10:00] Amanda Vizedom: Terry, "purpose" is not meant to be a separate section. It is now discussed

in the Introduction, but we had comments last week asking for it to be addressed in the Exec

Summary.

[10:27] Terry Longstreth: @Amanda - re: Purpose - I suggest it include quoted or paraphrastic text

referring to the original intent of this summit ("The goal for Ontology Summit 2013 is to identify

best practices for ontology development and evaluation." That phrasing suggests absolutes ("best"

practices) when we really ended up with a survey of current practices, the associated tools, and

possible avenues for future research on practices and tools.

[9:51] Michael Grüninger: [in response to Peter's remark that there are only a few responses to the

software survey so far] I think that even five tools is good enough for a start. However, we should

try to engage more tool developers and users over time so that this page remains alive rather than

collect cyberdust when the Summit is over.

[10:15] anonymous morphed into Lamar Henderson

[10:33] JohnMcClure: 1) waterfall v agile: "systemic" requirements lead, by themselves, to

significant ontology commitments that may be implemented apart from a specific domain 2) reuse v

reusuability: the paper seems to focus on evaluating reusability, little on reuse of published

ontologies to satisfy the needs of a domain and the impact of choosing one path over another (this

is the "not invented here' syndrome) 3) properties v classes v individuals - paper does not discuss

evaluating properties of an ontology v its classes - the differences in evaluation criteria that are

to be applied. 4) metrics - more discussion about the state of ontology metrics would be nice to see

[10:21] AstridDuqueRamos: [in response to JohnMcClure's remark that "metrics" is not covered] An

approach to metrics realated to OQuaRE is here

http://miuras.inf.um.es/evaluation/oquare/Contenido.html

[10:29] Peter P. Yim: ref. "metrics" the HC-03 FIBO Clinic group did some really good work, which I

think we might make reference to, they (FICO, OOPS!, OQuaRE & OntoQA folks) are collaboratively

working on a document - see:

http://ontolog.cim3.net/cgi-bin/wiki.pl?OntologySummit2013_Hackathon_Clinics_FIBO_OOPS_OQuaRE#nid3QX

F

[10:28] AstridDuqueRamos:

https://docs.google.com/document/d/1ErbZV0IFj890lHFcnygsw6n93dxub1AamOu9oBnHdOo/edit#heading=h.kqnhd

tvmz5vq

[10:26] Steve Ray: @John: No problem. I believe that this communique should really be viewed as a

portal to the large body of material present on the Ontolog wiki pages and the associated

presentations, discussions, and library, all archived and freely available.

[10:23] JohnMcClure: thanks steve ray

[10:28] Steve Ray: @Terry: To avoid the impression this is a "how to" guide for ontology development,

maybe we should emphasize the "evaluation" focus more, and less on the "development" aspect.

[10:30] Terry Longstreth: @Steve - I'm just quoting from the summit home page. I'd certainly agree

that improving/studying operations are the real goal of evaluation.

[10:30] Terry Longstreth: c/real/one real/

[10:33] Terry Longstreth: Requirements aren't necessarily an early point in the lifecycle, if it's

truly a cycle.

[10:35] Amanda Vizedom: We need to say something about the *general* relationship between

requirements and evaluation criteria, and through there to metrics, etc.. From this we can link to

the metrics document cited above. Where should this go?

[10:35] Amanda Vizedom: Note in diagram and text: everything feeds back into requirements!

[10:28] Michael Grüninger: @JohnMcClure: Can you give an example that illustrates your comment about

classes/properties?

[10:37] JohnMcClure: @MichaelGruninger: OWL attributes for properties are vastly different from

those for classes of course - specification of these seem a natural criteria. Which are the ones

most critical, which are important, which are nice-to-have? I'd hope this paper would explicitly

explore the *data model for ontologies*. I also believe you should eat your own dog food - using the

guidleines from the paper, EVALUATE RDF

[10:45] Michael Grüninger: @JohnMcClure: You explicitly refer to OWL and RDF, which is why Amanda

said that this issue refers to the Formal Modeling phase.

[10:50] JohnMcClure: Michael Grüninger, the section on Formal Modeling does not discuss reusing other

ontologies that I can see... it seems that the document presumes "not invented here' is at play in

designs. I disagree!

[10:51] Amanda Vizedom: @Michael, exactly. Also, it's important to have a broader perspective on the

vast range of ontologies that are developed, used, and reused, and the range of formal languages and

applications in play. I said that discussion of properties and classes in OWL is "in the weeds"

precisely because it is a fine-grained area of consideration within a particular implementation

language, and even then the requirements for such an OWL ontology will be partially dependent on the

intended usage, including operational environment. That's more detailed than we can get - doing this

for each aspect of each implementation language in common use would require a book-length document.

What we can do is refer to collected resources that help with such details.

[10:38] JohnMcClure: Does this document point to an evaluation that's been done in the manner such

as it recommends? No? Why not?

[10:38] JohnMcClure: otherise the paper sounds a bit academic

[10:40] Peter P. Yim: @JohnMcClure - the reference to the HC-03 FIBO Clinic will address this issue you

are bringing up

[10:40] JohnMcClure: looking for it, cannot find such reference

[10:41] Peter P. Yim: @JohnMcClure - ref. above [10:28] & [10:29]

[10:43] JohnMcClure: thanks peter yim, (doc still trying to load....!)

[10:40] Amanda Vizedom: I think that our mental models are closer to agile or continuous than to

waterfall or other (slower, less efficient, more unidirectional) life cycles. We want to support a

wide variety of entry points and paths, noting where there are dependencies and interactions between

phases, and emphasizing the role of evaluation everywhere.

[10:42] Matthew West: Apologies for being so late.

[10:42] JohnMcClure: no ontologies are being reused? what about all the W3 ontologies????

[10:44] JohnMcClure: there is no glossary in the doc

[10:44] Joel Bender: (as I'm in the thick of the weeds of developing my first ontology, I can't +1

what Amanda just said enough!)

[10:46] Peter P. Yim: @JoelBender - can you recap that ("what Amanda just said") to provide context for

your comment ... (for archival purposes)

[10:49] Joel Bender: @Peter - I will review the audio recording and extract the statement for you to

splice into the chat log

[10:50] Peter P. Yim: Thanks, Joel [you might want to start around time-point 1:07:40]

[10:49] Doug Foxvog: [refering to the poll on choice of terminology we were about to take] Please put

the text options in the chat

[10:51] Michael Grüninger initiated a vote - please click the Vote button to cast your ballot:

Selection of Terminology (1) Quality (2) Fidelity (3) Validity This is a single choice vote.

[10:51] Fabian Neuhaus voted for: Quality

[10:51] Michael Grüninger voted for: Fidelity

[10:51] Peter P. Yim voted for: Fidelity

[10:51] Joel Bender voted for: Fidelity

[10:51] Steve Ray voted for: Validity

[10:51] Bob Smith voted for: Validity

[10:52] Amanda Vizedom voted for: Fidelity

[10:52] Bobbin Teegarden voted for: Quality

[10:52] JohnMcClure: 2 & 3 are subclasses of Quality

[10:52] JohnMcClure voted for: Quality - base class of other two

[10:53] AstridDuqueRamos: @agree with John

[10:52] Doug Foxvog voted for: Fidelity - Quality seems too broad, & validity seems too narrow

[10:52] Terry Longstreth voted for: Validity - determine that an ontology is an accurate

representation of the target domain.

[10:53] Steve Ray: Fidelity, to me, focuses on alignment with the domain, but doesn't bring to mind

the model consistency/quality. Hence I voted Validity.

[10:58] Lamar Henderson abstains

[10:54] Ken Baclawski voted for: Validity - Meanings of fidelity in general use: Faithfulness to a

person, cause, or belief, demonstrated by continuing loyalty and support. Sexual faithfulness to a

spouse or partner. So validity would be better.

[10:55] Peter P. Yim: @Ken - indeed, faithfulness, as in Hi-Fi

[10:55] Doug Foxvog: @Ken, i take "fidelity" to mean "faithfulness to a standard"

[10:55] Michael Grüninger: @Steve: I thought that we were voting on the term for aspect (2), which

explicitly mentions the domain

[10:56] Steve Ray: @Michael: My problem is that (2) is the only one where Fabian/Amanda have included

model consistency, so I'm trying to bring that to the surface.

[10:56] JohnMcClure: i agree with the speaker

[10:57] Peter P. Yim: @John - that's Matthew West

[10:57] JohnMcClure: oh!

[10:57] Amanda Vizedom: I think Matthews point is a good one, that it isn't obvious why the things

gathered under (2) go together

[10:57] Doug Foxvog: Aren't we discussing a term for aspect 2 -- " Does the ontology represent the

domain appropriately"?

[11:00] JohnMcClure: Doug Foxvog, this use of the term "appropriately" bothers me.... I think

minimally is better

[10:58] Amanda Vizedom: @doug, yes, but Fabian's definition goes beyond that...

[10:58] Amanda Vizedom: see p. 8

[10:58] Steve Ray: @Doug: Yes, but also that's where model consistency is put.

[10:58] Steve Ray: I agree with Amanda that we should separate the domain stuff and the internal

consistency stuff.

[10:58] Doug Foxvog: If so, "fidelity" seems right. Validity and quality bring up other issues.

[10:59] Steve Ray: I think we should add a 5th high-level characteristic to capture internal quality.

[10:59] Doug Foxvog: @Steve: model consistency seems to refer to an internal property -- not on

whether it is consistent with the domain.

[11:00] Steve Ray: @Doug: I completely agree, hence my desire for the 5th focus.

[11:01] Michael Grüninger: Shall i close the ballot?

[11:01] Michael Grüninger ended the vote - results: Selection of Terminology

Tally Choice 3 Quality 5 Fidelity 4 Validity 1 Abstains

[11:01] Ken Baclawski: @Peter - Indeed, there are many meanings for fidelity and validity in

different domains. I was just showing the meanings that are commonly used. Since both fidelity and

validity could be used in this context, it is difficult to come up with a criterion to distinguish

them. Perhaps the scientific meaning is the best way to distinguish them. "In science and

statistics, validity is the extent to which a concept, conclusion or measurement is well-founded and

corresponds accurately to the real world."

[11:02] Doug Foxvog: Various aspects have to do with quality. I wouldn't restrict that word to a

single aspect.

[11:02] Terry Longstreth: @Ken - +1

[11:03] JohnMcClure: Ken Baclawski, that's great - let's have a glossary in the document?

[11:04] Ken Baclawski: In the fields of scientific modelling and simulation, fidelity refers to the

degree to which a model or simulation reproduces the state and behaviour of a real world object,

feature or condition. Fidelity is therefore a measure of the realism of a model or simulation.

[11:04] Peter P. Yim: @Ken - nice ... as Fabian said, we'll pick the term, and define it clearly

[11:04] Amanda Vizedom: Yes, @Ken, that is the notion of fidelity that I have in mind.

[11:04] Fabian Neuhaus: Domain fidelity.

[11:04] Doug Foxvog: I like "domain fidelity".

[11:04] Amanda Vizedom: Separate into two characteristics? One that has been suggested is "domain

fidelity" -- what about the other?

[11:05] JohnMcClure: "domain coverage" works better for me, relates to overlapping sets

[11:05] Steve Ray: The other one could be: Model quality, or Model well-builtedness (!)

[11:06] Ken Baclawski: Perhaps "model fidelity"?

[11:06] Amanda Vizedom: There is also a compatible notion of the fidelity of an information exchange

or translation: the degree to which the end version contains all and only the information in the

beginning version. applies well to modeling & representation.

[11:06] Steve Ray: @Michael, what do you call what your student checks for?

[11:09] Michael Grüninger: @Steve: we refer to "ontology verification", since we are determining

whether or not the ontology satisfies the requirements that are formalized as intended models

[11:10] Steve Ray: @Michael: So a noun capturing the property measured during "ontology verification"

would be a candidate for the 5th criterion.

[11:12] Michael Grüninger: @Steve: although now that you mention it, we often refer to the question

of whether or not we have the right requirements (intended models) for the domain as "ontology

validation", analogous to software validation. Perhaps I should change my vote to "validity" from

fidelity :-)

[11:11] JohnMcClure: ok, here is one: few "best practices" have been defined by the ontology

community, which can be leveraged here. There are some however, eg the W3 has identified some -

where do these fit in with this communique?

[11:13] JohnMcClure: If this were a peer-reviewed paper, I think it would be thrown back at ya

because there is no glossary, there are no references, it reads like a discussion

[11:13] JohnMcClure: this is a flawed paper at this point, sorry

[11:15] Michael Grüninger: @JohnMcClure: this isn't intended to be an academic paper, but rather the

synthesis of discussions that have taken place during the Summit.

[11:17] JohnMcClure: an "academic paper" === SCIENCE.... so there's little science here, sorry

[11:20] Michael Grüninger: @John: I think the better analogy here is that the Communique is a "white

paper" rather than a scientific paper, particularly since the intended audience is not academia

[11:23] JohnMcClure: suggest you look at w3 recs as a model

[11:14] Amanda Vizedom: Deadline for substantial comments and suggestions to be considered: Tuesday,

23rd April 2013.

[11:14] Amanda Vizedom: Prose clean-up, typos, and such, can go on a bit longer.

[11:24] Peter P. Yim: Join us again, same time next week, for Ontology Summit 2013 session-15:

Pre-symposium preparation and review - Co-chairs: Mike Dean & Ram D. Sriram - developing session

details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2013_04_25

[11:30] Peter P. Yim: ... For those who will be presenting at the symposium, please note that slides are

due on Monday Apr-29 (not Tuesday, as we would normally have it!) ... please come to the session

next Thursday too, so we can tie down final details!

[11:30] Peter P. Yim: great session!

[11:31] Peter P. Yim: -- session ended 11:29am PDT --

-- end of in-session chat-transcript --

  • Further Question & Remarks - please post them to the [ ontology-summit ] listserv
    • all subscribers to the previous summit discussion, and all who responded to today's call will automatically be subscribed to the [ ontology-summit ] listserv
    • if you are already subscribed, post to <ontology-summit [at] ontolog.cim3.net>
    • (if you are not yet subscribed) you may subscribe yourself to the [ ontology-summit ] listserv, by sending a blank email to <ontology-summit-join [at] ontolog.cim3.net> from your subscribing email address, and then follow the instructions you receive back from the mailing list system.
    • (in case you aren't already a member) you may also want to join the ONTOLOG community and be subscribed to the [ ontolog-forum ] listserv, when general ontology-related topics (not specific to this year's Summit theme) are discussed. Please refer to Ontolog membership details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?WikiHomePage#nid1J
      • kindly email <peter.yim@cim3.com> if you have any question.

Additional Resources

For the record ...

How To Join (while the session is in progress)

Conference Call Details

  • Date: Thursday, 18-April-2013
  • Start Time: 9:30am PDT / 12:30pm EDT / 1:30pm ART / 6:30pm CEST / 5:30pm BST / 16:30 UTC
  • Expected Call Duration: 1.5~2.0 hours
  • Dial-in:
    • Phone (US): +1 (206) 402-0100 ... (long distance cost may apply)
      • ... [ backup nbr: (415) 671-4335 ]
      • when prompted enter Conference ID: 141184#
    • Skype: joinconference (i.e. make a skype call to the contact with skypeID="joinconference") ... (generally free-of-charge, when connecting from your computer)
      • when prompted enter Conference ID: 141184#
      • Unfamiliar with how to do this on Skype? ...
        • Add the contact "joinconference" to your skype contact list first. To participate in the teleconference, make a skype call to "joinconference", then open the dial pad (see platform-specific instructions below) and enter the Conference ID: 141184# when prompted.
      • Can't find Skype Dial pad? ...
        • for Windows Skype users: Can't find Skype Dial pad? ... it's under the "Call" dropdown menu as "Show Dial pad"
        • for Linux Skype users: please note that the dial-pad is only available on v4.1 (or later; or on the earlier Skype versions 2.x,) if the dialpad button is not shown in the call window you need to press the "d" hotkey to enable it. ... (ref.)
  • Shared-screen support (VNC session), if applicable, will be started 5 minutes before the call at: http://vnc2.cim3.net:5800/
    • view-only password: "ontolog"
    • if you plan to be logging into this shared-screen option (which the speaker may be navigating), and you are not familiar with the process, please try to call in 5 minutes before the start of the session so that we can work out the connection logistics. Help on this will generally not be available once the presentation starts.
    • people behind corporate firewalls may have difficulty accessing this. If that is the case, please download the slides and materials above (where applicable) and running them locally. The speaker(s) will prompt you to advance the slides during the talk.
  • In-session chat-room url: http://webconf.soaphub.org/conf/room/summit_20130418
    • instructions: once you got access to the page, click on the "settings" button, and identify yourself (by modifying the Name field from "anonymous" to your real name, like "JaneDoe").
    • You can indicate that you want to ask a question verbally by clicking on the "hand" button, and wait for the moderator to call on you; or, type and send your question into the chat window at the bottom of the screen.
    • thanks to the soaphub.org folks, one can now use a jabber/xmpp client (e.g. gtalk) to join this chatroom. Just add the room as a buddy - (in our case here) summit_20130418@soaphub.org ... Handy for mobile devices!
  • Discussions and Q & A:
    • Nominally, when a presentation is in progress, the moderator will mute everyone, except for the speaker.
    • To un-mute, press "*7" ... To mute, press "*6" (please mute your phone, especially if you are in a noisy surrounding, or if you are introducing noise, echoes, etc. into the conference line.)
    • we will usually save all questions and discussions till after all presentations are through. You are encouraged to jot down questions onto the chat-area in the mean time (that way, they get documented; and you might even get some answers in the interim, through the chat.)
    • During the Q&A / discussion segment (when everyone is muted), If you want to speak or have questions or remarks to make, please raise your hand (virtually) by clicking on the "hand button" (lower right) on the chat session page. You may speak when acknowledged by the session moderator (again, press "*7" on your phone to un-mute). Test your voice and introduce yourself first before proceeding with your remarks, please. (Please remember to click on the "hand button" again (to lower your hand) and press "*6" on your phone to mute yourself after you are done speaking.)
  • RSVP to peter.yim@cim3.com with your affiliation appreciated, ... or simply just by adding yourself to the "Expected Attendee" list below (if you are a member of the community already.)
  • Please note that this session may be recorded, and if so, the audio archive is expected to be made available as open content, along with the proceedings of the call to our community membership and the public at-large under our prevailing open IPR policy.

Attendees



Retrieved from "https://ontologforum.com/index.php?title=ConferenceCall_2013_04_18&oldid=1297"