Read-me for the ORE 2014 Testing Framework. Purpose ======= The ORE 2014 Testing Framework should be used by reasoner developers to verify that their reasoners comply the specification of the ORE 2014 Competition (see <http://vsl2014.at/pages/ORE-index.html>). In particular, the framework allows for checking whether the reasoner correctly processes the input and writes the expected output. License ======= The ORE 2014 Testing Framework is released as free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. The ORE 2014 Testing Framework is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details. Copies of the GNU General Public License and the GNU Lesser General Public License have been included with this distribution in the file ’gpl.txt’ and ’lgpl-3.0.txt’, respectively. An online version is available at <http://www.gnu.org/licenses/>. The ORE 2014 Testing Framework uses the following libraries in unmodified form: 1. Simple Logging Facade for Java (SLF4J), <http://www.slf4j.org/>, released under the MIT license. 2. Apache Logging Library for Java (log4j), <http://logging.apache.org/log4j/1.2/>, released under The Apache Software License, Version 2.0. 3. Apache Commons Exec, <http://commons.apache.org/proper/commons-exec/>, released under The Apache Software License, Version 2.0. 4. OWL API, <http://owlapi.sourceforge.net>, released under LGPL 3.0. 5. HermiT, <http://hermit-reasoner.com/>, released under LGPL 3.0. Overview and Usage ================== The testing framework is completely realised with Java and, therefore, it should be runnable on all Java supported platforms. However, it might be necessary to provide different reasoner configurations for different platforms. In particular, the wrapper scripts that trigger the reasoners must be executable on the platform on which the evaluation is executed (e.g., shell scripts on Unix/Linux, batch files on Windows). The reasoner set-up is explained below in more detail. Please note that reasoners must eventually be executable on Linux (see ORE 2014 Specification), otherwise the participation in the ORE 2014 Competition is not possible. The testing framework works in principle as follows (the examples use the syntax for Linux and should also work for Mac OS X, minor syntactical adjustments are, however, necessary for Windows, e.g., the syntactical replacement of ’.sh’ and ’linux’ with ’.bat’ and ’windows’, respectively): 1. The evaluation of a category for a reasoner is started by executing the corresponding starter script, e.g., ’./test-classification.sh hermit-linux’. 2. The testing framework tries to load the reasoner configuration and the corresponding queries. 3. The queries are executed, the results of the reasoner are parsed, normalised, and compared to expected results (only hash codes are compared). 4. The testing framework generates an evaluation report, identifies potential problems of the reasoner, and saves the gathered data into the output directory. The organisation of the testing framework is as follows: - The ’data/reasoners/’ and ’data/ontologies/’ directories contain the reasoners and the ontologies, respectively. - The ’data/queries/’ directory contains classification, consistency, and realisation queries that refer to the ontologies in the ’data/ontologies/’ directory. - The responses and results of the reasoners as well as the evaluation summaries are written into a sub-directories of the ’data/responses/’ folder (the sub-directories are named by the reasoner and the execution date of the evaluation). - Expected results are stored in the ’data/expectations/’ directory. - The ’data/configs/’ directory contains configurations for the execution of the evaluations (e.g., timeouts). The ’default-config.dat’ configuration is loaded by default, other configurations can be used by passing their file name as additional argument to the testing framework (e.g., ’./test-classification.sh hermit-linux other-config.dat’). - Logs of the execution of the testing framework are appended to the ’log.txt’ file in the ’data/logs/’ directory. - The ’data/conversions/’ directory is used to cache converted ontologies in the corresponding serialisation format if this is requested by a reasoner. Reasoner Set-up Instructions ============================ For testing a reasoner with the framework, the following steps are necessary: 1. Create a new folder in the ’data/reasoners/’ directory, named by the reasoner/the reasoner configuration (e.g., ’elk-linux’). 2. Specify the settings for the new reasoner in the ’reasoner.dat’ file within the reasoner directory (e.g., ’data/reasoners/elk-linux/reasoner.dat’). - A simple way to archive this is to copy an existing reasoner configuration file (e.g., ’data/reasoners/hermit-linux/reasoner.dat’) and modify some of the values (e.g., ’OutputPathName’ to ’elk-linux’, ’ReasonerName’ to ’ELK’, ’ProfileSupport’ to ’EL’, and ’DatatypeSupport’ to ’FALSE’). Also see comments in the existing configuration files (e.g., ’data/reasoners/hermit-linux/reasoner.dat’). 3. Create a starter/wrapper script (e.g., ’execReasoner.sh’) in the reasoner folder that triggers the reasoner (also see ORE 2014 Specification). - The wrapper script must be referenced in the reasoner configuration file (’reasoner.dat’) by the ’StarterScript’ setting, (e.g., ’./execReasoner.sh’). - The wrapper script is executed by the evaluation framework, i.e., it must be executable (e.g., apply ’chmod a+x execReasoner.sh’). - For OWL API-based reasoners, the wrapper script can directly execute the OREv2ReasonerWrapper (’OREv2ReasonerWrapper.jar’), which already implements the input and output specifications for the ORE 2014 Competition. Note, the OREv2ReasonerWrapper requires as first argument the corresponding OWLReasonerFactory class (e.g., ’org.semanticweb.elk.owlapi.ElkReasonerFactory’) that has to be loaded with the Java Reflection API in order to create the OWLReasoner from the factory. Also note that the OREv2ReasonerWrapper may have to be recompiled such that all libraries of the reasoner are in the Classpath of the ’OREv2ReasonerWrapper.jar’ file. The OREv2ReasonerWrapper can be build by the Ant script ’build-wrapper-v2.xml’ (the libs of the reasoner must be copied into the ’lib’ directory). 4. Start the testing framework by calling the corresponding starter script for a reasoning task with the reasoner folder as first argument (for example, ’./test-classification.sh elk-linux’). - If the reasoner configuration file is not named ’reasoner.dat’ within the reasoner folder, then the path to this file must be used as first argument. - Optionally, the evaluation configuration file can be listed as second argument.