HTTPS SSH
Read-me for the ORE 2014 Testing Framework.

Purpose
=======

The ORE 2014 Testing Framework should be used by reasoner developers to
verify that their reasoners comply the specification of the ORE 2014
Competition (see <http://vsl2014.at/pages/ORE-index.html>). In
particular, the framework allows for checking whether the reasoner
correctly processes the input and writes the expected output.

License
=======

The ORE 2014 Testing Framework is released as free software: you can
redistribute it and/or modify it under the terms of the GNU Lesser
General Public License as published by the Free Software Foundation,
either version 3 of the License, or (at your option) any later version.

The ORE 2014 Testing Framework is distributed in the hope that it will
be useful, but WITHOUT ANY WARRANTY; without even the implied warranty
of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.

Copies of the GNU General Public License and the GNU Lesser General
Public License have been included with this distribution in the file
’gpl.txt’ and ’lgpl-3.0.txt’, respectively. An online version is
available at <http://www.gnu.org/licenses/>.

The ORE 2014 Testing Framework uses the following libraries in
unmodified form:

1.  Simple Logging Facade for Java (SLF4J), <http://www.slf4j.org/>,
    released under the MIT license.

2.  Apache Logging Library for Java (log4j),
    <http://logging.apache.org/log4j/1.2/>, released under The Apache
    Software License, Version 2.0.

3.  Apache Commons Exec,
    <http://commons.apache.org/proper/commons-exec/>, released under The
    Apache Software License, Version 2.0.

4.  OWL API, <http://owlapi.sourceforge.net>, released under LGPL 3.0.

5.  HermiT, <http://hermit-reasoner.com/>, released under LGPL 3.0.

Overview and Usage
==================

The testing framework is completely realised with Java and, therefore,
it should be runnable on all Java supported platforms. However, it might
be necessary to provide different reasoner configurations for different
platforms. In particular, the wrapper scripts that trigger the reasoners
must be executable on the platform on which the evaluation is executed
(e.g., shell scripts on Unix/Linux, batch files on Windows). The
reasoner set-up is explained below in more detail. Please note that
reasoners must eventually be executable on Linux (see ORE 2014
Specification), otherwise the participation in the ORE 2014 Competition
is not possible.

The testing framework works in principle as follows (the examples use
the syntax for Linux and should also work for Mac OS X, minor
syntactical adjustments are, however, necessary for Windows, e.g., the
syntactical replacement of ’.sh’ and ’linux’ with ’.bat’ and ’windows’,
respectively):

1.  The evaluation of a category for a reasoner is started by executing
    the corresponding starter script, e.g., ’./test-classification.sh
    hermit-linux’.

2.  The testing framework tries to load the reasoner configuration and
    the corresponding queries.

3.  The queries are executed, the results of the reasoner are parsed,
    normalised, and compared to expected results (only hash codes are
    compared).

4.  The testing framework generates an evaluation report, identifies
    potential problems of the reasoner, and saves the gathered data into
    the output directory.

The organisation of the testing framework is as follows:

-   The ’data/reasoners/’ and ’data/ontologies/’ directories contain the
    reasoners and the ontologies, respectively.

-   The ’data/queries/’ directory contains classification, consistency,
    and realisation queries that refer to the ontologies in the
    ’data/ontologies/’ directory.

-   The responses and results of the reasoners as well as the evaluation
    summaries are written into a sub-directories of the
    ’data/responses/’ folder (the sub-directories are named by the
    reasoner and the execution date of the evaluation).

-   Expected results are stored in the ’data/expectations/’ directory.

-   The ’data/configs/’ directory contains configurations for the
    execution of the evaluations (e.g., timeouts). The
    ’default-config.dat’ configuration is loaded by default, other
    configurations can be used by passing their file name as additional
    argument to the testing framework (e.g., ’./test-classification.sh
    hermit-linux other-config.dat’).

-   Logs of the execution of the testing framework are appended to the
    ’log.txt’ file in the ’data/logs/’ directory.

-   The ’data/conversions/’ directory is used to cache converted
    ontologies in the corresponding serialisation format if this is
    requested by a reasoner.

Reasoner Set-up Instructions
============================

For testing a reasoner with the framework, the following steps are
necessary:

1.  Create a new folder in the ’data/reasoners/’ directory, named by the
    reasoner/the reasoner configuration (e.g., ’elk-linux’).

2.  Specify the settings for the new reasoner in the ’reasoner.dat’ file
    within the reasoner directory (e.g.,
    ’data/reasoners/elk-linux/reasoner.dat’).

    -   A simple way to archive this is to copy an existing reasoner
        configuration file (e.g.,
        ’data/reasoners/hermit-linux/reasoner.dat’) and modify some of
        the values (e.g., ’OutputPathName’ to ’elk-linux’,
        ’ReasonerName’ to ’ELK’, ’ProfileSupport’ to ’EL’, and
        ’DatatypeSupport’ to ’FALSE’). Also see comments in the existing
        configuration files (e.g.,
        ’data/reasoners/hermit-linux/reasoner.dat’).

3.  Create a starter/wrapper script (e.g., ’execReasoner.sh’) in the
    reasoner folder that triggers the reasoner (also see ORE 2014
    Specification).

    -   The wrapper script must be referenced in the reasoner
        configuration file (’reasoner.dat’) by the ’StarterScript’
        setting, (e.g., ’./execReasoner.sh’).

    -   The wrapper script is executed by the evaluation framework,
        i.e., it must be executable (e.g., apply ’chmod a+x
        execReasoner.sh’).

    -   For OWL API-based reasoners, the wrapper script can directly
        execute the OREv2ReasonerWrapper (’OREv2ReasonerWrapper.jar’),
        which already implements the input and output specifications for
        the ORE 2014 Competition. Note, the OREv2ReasonerWrapper
        requires as first argument the corresponding OWLReasonerFactory
        class (e.g., ’org.semanticweb.elk.owlapi.ElkReasonerFactory’)
        that has to be loaded with the Java Reflection API in order to
        create the OWLReasoner from the factory. Also note that the
        OREv2ReasonerWrapper may have to be recompiled such that all
        libraries of the reasoner are in the Classpath of the
        ’OREv2ReasonerWrapper.jar’ file. The OREv2ReasonerWrapper can be
        build by the Ant script ’build-wrapper-v2.xml’ (the libs of the
        reasoner must be copied into the ’lib’ directory).

4.  Start the testing framework by calling the corresponding starter
    script for a reasoning task with the reasoner folder as first
    argument (for example, ’./test-classification.sh elk-linux’).

    -   If the reasoner configuration file is not named ’reasoner.dat’
        within the reasoner folder, then the path to this file must be
        used as first argument.

    -   Optionally, the evaluation configuration file can be listed as
        second argument.