Commits

Miran Levar committed 53f9b79

Added basic documentation, fixed some bugs.

  • Participants
  • Parent commits 154d34e

Comments (0)

Files changed (13)

 ==================
 
 Orange Multitarget is an add-on for Orange_ data mining software package. It
-extends Orange by providing functionality to allow for classification of datasets
+extends Orange by providing methods that allow for classification of datasets
 with multiple classes.
 
 Currently supported techniques: clustering trees, PLS, binary relevance, chain classification and neural networks (for multi-label data only).

File _multitarget/__init__.py

 # Other algorithms which also work with multitarget data
 from Orange.regression import pls
 from Orange.regression import earth
+from pkg_resources import resource_filename
 
 import tree
 import chain
 import binary
 import neural
 
+
+def datasets():
+    yield ('multitarget', resource_filename(__name__, 'datasets'))
+
 class MultitargetLearner(Orange.classification.Learner):
     """
     Wrapper for multitarget problems that constructs independent models

File _multitarget/binary.py

+"""
+.. index:: Multi-target Binary Relevance Learner
+
+***************************************
+Multi-target Binary Relevance Learner
+***************************************
+
+
+.. index:: Multi-target Binary Relevance Learner
+.. autoclass:: Orange.multitarget.binary.BinaryRelevanceLearner
+    :members:
+    :show-inheritance:
+
+.. index:: Multi-target Binary Relevance Classifier
+.. autoclass:: Orange.multitarget.binary.BinaryRelevanceClassifier
+    :members:
+    :show-inheritance:
+
+"""
+
 import Orange.core as orange
 import Orange
-import random
 import copy
 from operator import add
 
 class BinaryRelevanceLearner(orange.Learner):
     """
-    Expands single class classification techniques into multi-target classification techniques by chaining the classification
-    data. A learner is constructed for each of the class variables in a random or given order. The data for each learner are
-    the features extended by all previously classified variables. This chaining passes the class informationd between
-    classifiers and allows class correlations to be taken into account.
-    TODO: cite weka source?
+    Creates a standard single class learner for each class variable. Binary relevance assumes independance of class variables.
 
-    :param learner: A single class learner that will be extended by chaining.
+    :param learner: A single class learner.
     :type learner: :class:`Orange.core.Learner`
 
-    :param rand: random generator used in bootstrap sampling. If None (default), 
-        then ``random.Random(42)`` is used.
-
     :param callback: a function to be called after every iteration of
             induction of classifier. The call returns a parameter
             (from 0.0 to 1.0) that provides an estimate
     :param name: learner name.
     :type name: string
 
-    :rtype: :class:`Orange.ensemble.forest.RandomForestClassifier` or 
-            :class:`Orange.ensemble.forest.RandomForestLearner`
+    :rtype: :class:`Orange.multitarget.BinaryRelevanceLearner` or 
+            :class:`Orange.multitarget.BinaryRelevanceCLassifier`
 
     """
 
         else:
             return self
 
-    def __init__(self, learner=None, name="Binary Relevance", rand=None, callback=None):
+    def __init__(self, learner=None, name="Binary Relevance", callback=None):
         self.name = name
-        self.rand = rand
         self.callback = callback
 
         if not learner:
             raise TypeError("Wrong specification, learner not defined")
         else:
-            self.learner = learner
-
-        if not self.rand:
-            self.rand = random.Random(42)
-
-        self.randstate = self.rand.getstate()
-           
+            self.learner = learner           
 
     def __call__(self, instances, weight=0):
         """
-        Learn from the given table of data instances.
+        Construct learners from the given table of data instances.
         
         :param instances: data for learning.
         :type instances: class:`Orange.data.Table`
         :param weight: weight.
         :type weight: int
 
-        :rtype: :class:`Orange.ensemble.chain.ClassifierChain`
+        :rtype: :class:`Orange.multitarget.BinaryRelevanceClassifier`
         """
 
         instances = Orange.data.Table(instances.domain, instances) # bypasses ownership
 
-        self.rand.setstate(self.randstate) 
         n = len(instances)
         m = len(instances.domain.class_vars)
         progress = 0.0
                 progress+=1
                 self.callback(progress / m)
 
-        return BinaryRelevanceClassifier(classifiers=classifiers, class_order=class_order, domains=domains, name=self.name, orig_domain=orig_domain)
+        return BinaryRelevanceClassifier(classifiers=classifiers, domains=domains, name=self.name)
 
 
 class BinaryRelevanceClassifier(orange.Classifier):
     """
-    Uses the classifiers induced by the :obj:`ClassifierChainLearner`. An input
+    Uses the classifiers induced by the :obj:`BinaryRelevanceLearner`. An input
     instance is classified into the class with the most frequent vote.
     However, this implementation returns the averaged probabilities from
     each of the trees if class probability is requested.
-
-    It should not be constructed manually. TODO: ask about this
     
-    :param classifiers: a list of classifiers to be used.
-    :type classifiers: list
+    :param classifiers: a list of classifiers.
+    :type classifiers: list of  :class:`Orange.core.Learner`
+        
+    :param domains: the domains used by learners.
+    :type domain: list of :class:`Orange.data.Domain`
     
-    
-    
-    :param domains: the domain of the learning set.
-    :type domain: :class:`Orange.data.Domain`
-    
-    :param class_var: the class feature.
-    :type class_var: :class:`Orange.feature.Descriptor`
-
-    :param class_vars: the multi-target class features.
-    :type class_vars: list of :class:`Orange.feature.Descriptor`
-
-    :param name: name of the resulting classifier.
-    :type name: str
+    :param name: name of the classifier.
+    :type name: string
 
     """
 
-    def __init__(self, classifiers, class_order, domains, name, orig_domain):
+    def __init__(self, classifiers, domains, name):
         self.classifiers = classifiers
-        self.class_order = class_order
         self.name = name
         self.domains = domains
-        self.orig_domain = orig_domain
 
     def __call__(self, instance, result_type = orange.GetValue):
         """
         :rtype: :class:`Orange.data.Value`, 
               :class:`Orange.statistics.Distribution` or a tuple with both
         """
-        m = len(self.class_order)
+        m = len(instance.domain.class_vars)
         values = [None for _ in range(m)] 
         probs = [None for _ in range(m)] 
 
             return [tuple(values),tuple(probs)]
 
     def __reduce__(self):
-        return type(self), (self.classifiers, self.class_order, self.domains, self.name, self.orig_domain), dict(self.__dict__)
+        return type(self), (self.classifiers, self.domains, self.name), dict(self.__dict__)
 
 if __name__ == '__main__':
     import time
     global_timer = time.time()
 
     data = Orange.data.Table('bridges.v2.nm')
-    #data = Orange.data.Table('ntp.fp3.nm')
-    
     
     l1 = BinaryRelevanceLearner(learner = Orange.classification.tree.SimpleTreeLearner)
     l2 = BinaryRelevanceLearner(learner = Orange.classification.bayes.NaiveLearner)
     l3 = BinaryRelevanceLearner(learner = Orange.classification.majority.MajorityLearner)
     l4 = Orange.multitarget.tree.MultiTreeLearner()
 
-
-    #l =  EnsembleClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, n_chains=3, name="ECC T")
-    #l = Orange.multitarget.tree.MultiTreeLearner()
-    #l = EnsembleClassifierChainLearner(learner = Orange.classification.majority.MajorityLearner, n_chains=3, name="ECC M")
-    # cross_validation not working
-    res = Orange.evaluation.testing.cross_validation([l1,l2,l3,l4],data, folds = 2)
-    #res = Orange.evaluation.testing.cross_validation([l],data,folds=5)
+    res = Orange.evaluation.testing.cross_validation([l1,l2,l3,l4],data)
 
     scores = Orange.evaluation.scoring.mt_average_score(res,Orange.evaluation.scoring.RMSE)
-   
-    print scores
-    print res.classifierNames
+
     for i in range(len(scores)):
         print res.classifierNames[i], scores[i]
 
-
     print "--DONE %.2f --" % (time.time()-global_timer)

File _multitarget/chain.py

+"""
+.. index:: Multi-target Classifier Chain Learner
+
+***************************************
+Multi-target Classifier Chain Learner
+***************************************
+
+
+.. index:: Multi-target Classifier Chain Learner
+.. autoclass:: Orange.multitarget.chain.ClassifierChainLearner
+    :members:
+    :show-inheritance:
+
+.. index:: Multi-target Classifier Chain Classifier
+.. autoclass:: Orange.multitarget.chain.ClassifierChain
+    :members:
+    :show-inheritance:
+
+.. index:: Multi-target Ensemble Classifier Chain Learner
+.. autoclass:: Orange.multitarget.chain.EnsembleClassifierChainLearner
+    :members:
+    :show-inheritance:
+
+.. index:: Multi-target Ensemble Classifier Chain Classifier
+.. autoclass:: Orange.multitarget.chain.EnsembleClassifierChain
+    :members:
+    :show-inheritance:
+
+"""
+
 import Orange.core as orange
 import Orange
 import random
     :param name: learner name.
     :type name: string
 
-    :rtype: :class:`Orange.ensemble.forest.RandomForestClassifier` or 
-            :class:`Orange.ensemble.forest.RandomForestLearner`
+    :rtype: :class:`Orange.multitarget.chain.RandomForestClassifier` or 
+            :class:`Orange.multitarget.chain.RandomForestLearner`
 
     """
 
         :param class_order: list of descriptors of class variables
         :type class_order: list of :class:`Orange.feature.Descriptor`
 
-        :rtype: :class:`Orange.ensemble.chain.ClassifierChain`
+        :rtype: :class:`Orange.multitarget.chain.ClassifierChain`
         """
 
         instances = Orange.data.Table(instances.domain, instances) # bypasses ownership
         elif not class_order:
             class_order = [cv for cv in instances.domain.class_vars]
             self.rand.shuffle(class_order)
-        print [cv.name for cv in class_order]
+
         learner = self.learner
 
         for i in range(m):
     However, this implementation returns the averaged probabilities from
     each of the trees if class probability is requested.
 
-    It should not be constructed manually. TODO: ask about this
+    It should not be constructed manually.
     
-    :param classifiers: a list of classifiers to be used.
-    :type classifiers: list
+    :param classifiers: a list of classifiers.
+    :type classifiers: list of  :class:`Orange.core.Learner`
+        
+    :param domains: the domains used by learners.
+    :type domain: list of :class:`Orange.data.Domain`
     
-    
-    
-    :param domains: the domain of the learning set.
+    :param orig_domain: the domain of the learning set.
     :type domain: :class:`Orange.data.Domain`
-    
-    :param class_var: the class feature.
-    :type class_var: :class:`Orange.feature.Descriptor`
 
-    :param class_vars: the multi-target class features.
-    :type class_vars: list of :class:`Orange.feature.Descriptor`
-
-    :param name: name of the resulting classifier.
-    :type name: str
+    :param name: name of the classifier.
+    :type name: string
 
     """
 
     classifiers and allows class correlations to be taken into account.
     TODO: cite weka source?
 
+    :param n_chains: Number of chains to be constructed.
+    :type n_chains: integer
+
+    :param sample_size: Size (percentage) of the subset taken from original dataset.
+    :type sample_size: float
+
     :param learner: A single class learner that will be extended by chaining.
     :type learner: :class:`Orange.core.Learner`
 
     :param rand: random generator used in bootstrap sampling. If None (default), 
         then ``random.Random(42)`` is used.
 
+
     :param callback: a function to be called after every iteration of
             induction of classifier. The call returns a parameter
             (from 0.0 to 1.0) that provides an estimate
     :param name: learner name.
     :type name: string
 
-    :rtype: :class:`Orange.ensemble.forest.RandomForestClassifier` or 
-            :class:`Orange.ensemble.forest.RandomForestLearner`
+    :rtype: :class:`Orange.multitarget.chain.EnsembleClassifierChainLearner` or 
+            :class:`Orange.multitarget.chain.EnsembleClassifierChain`
     """
 
     def __new__(cls, data=None, weight = 0, **kwargs):
 
     def __call__(self, instances, weight=0):
         """
+        Learn from the given table of data instances.
+        
+        :param instances: data for learning.
+        :type instances: class:`Orange.data.Table`
+
+        :param weight: weight.
+        :type weight: int
+
+        :param class_order: list of descriptors of class variables
+        :type class_order: list of :class:`Orange.feature.Descriptor`
+
+        :rtype: :class:`Orange.multitarget.chain.ClassifierChain`
         """
 
         ind = Orange.data.sample.SubsetIndices2(p0=1-self.sample_size)
     instance is classified into the class with the most frequent vote.
     However, this implementation returns the averaged probabilities from
     each of the trees if class probability is requested.
-
-    It should not be constructed manually. TODO: ask about this
-    
+   
     When constructed manually, the following parameters have to
     be passed:
 
-    :param classifiers: a list of classifiers to be used.
+    :param classifiers: a list of chain classifiers to be used.
     :type classifiers: list
     
-    
-    
-    :param domains: the domain of the learning set.
-    :type domain: :class:`Orange.data.Domain`
-    
-    :param class_var: the class feature.
-    :type class_var: :class:`Orange.feature.Descriptor`
-
-    :param class_vars: the multi-target class features.
-    :type class_vars: list of :class:`Orange.feature.Descriptor`
-
     :param name: name of the resulting classifier.
     :type name: str
 
 
 
 if __name__ == '__main__':
-    data = Orange.data.Table('test4')
-    l=ClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, name="CC T")
-    res = Orange.evaluation.testing.cross_validation([l],data,folds=2)
-
-    exit()
 
     print "STARTED"
     import time
     global_timer = time.time()
     data = Orange.data.Table('bridges.v2.nm')
-    #data = Orange.data.Table('bridges.v2.nm')
-    cl3 = EnsembleClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, n_chains=50, sample_size=0.25, name="ECC T", rand = random.seed(time.time()))
-    cl4 = EnsembleClassifierChainLearner(learner = Orange.classification.majority.MajorityLearner, n_chains=50, sample_size=0.25, name="ECC M", rand = random.seed(time.time()))
-    cl1 = ClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, name="CC T")
-    cl2 = ClassifierChainLearner(learner = Orange.classification.majority.MajorityLearner, name="CC M")
-    l1 = Orange.multitarget.tree.MultiTreeLearner()
-    l2 = Orange.multitarget.pls.PLSRegressionLearner()
-    l3 = Orange.ensemble.forest.RandomForestLearner(base_learner=Orange.multitarget.tree.MultiTreeLearner(), trees=50, name="RF MT" )
 
-    #l =  EnsembleClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, n_chains=3, name="ECC T")
-    #l = Orange.multitarget.tree.MultiTreeLearner()
-    #l = EnsembleClassifierChainLearner(learner = Orange.classification.majority.MajorityLearner, n_chains=3, name="ECC M")
-    # cross_validation not working
-    res = Orange.evaluation.testing.cross_validation([cl1,cl2,cl3,cl4,l1,l2,l3],data,folds=2)
-    #res = Orange.evaluation.testing.cross_validation([l],data,folds=5)
+    cl1 = EnsembleClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, n_chains=50, sample_size=0.25, name="ECC T", rand = random.seed(time.time()))
+    cl2 = EnsembleClassifierChainLearner(learner = Orange.classification.majority.MajorityLearner, n_chains=50, sample_size=0.25, name="ECC M", rand = random.seed(time.time()))
+    cl3 = ClassifierChainLearner(learner = Orange.classification.tree.SimpleTreeLearner, name="CC T")
+    cl4 = ClassifierChainLearner(learner = Orange.classification.majority.MajorityLearner, name="CC M")
+
+    res = Orange.evaluation.testing.cross_validation([cl1,cl2,cl3,cl4],data)
 
     scores = Orange.evaluation.scoring.mt_average_score(res,Orange.evaluation.scoring.RMSE)
    
-    print scores
-    print res.classifierNames
     for i in range(len(scores)):
         print res.classifierNames[i], scores[i]
 
-    print "CA"
-    scores = Orange.evaluation.scoring.mt_average_score(res,Orange.evaluation.scoring.CA)
-    print scores
-    print res.classifierNames
-    for i in range(len(scores)):
-        print res.classifierNames[i], scores[i]
-
-
-    #indices = Orange.data.sample.SubsetIndices2(data,0.25)
-    #data_learn = data.select(indices,1)
-    #data_test = data.select(indices,0)
-
-    #c1=cl3(data_learn)
-    #c2=l1(data_learn)
-    #
-    #res = Orange.evaluation.testing.test_on_data([c1, c2],data_test)
-    #print Orange.evaluation.scoring.mt_average_score(res,Orange.evaluation.scoring.RMSE)
-
     print "--DONE %.2f --" % (time.time()-global_timer)

File _multitarget/neural.py

-import Orange
+"""
+.. index:: Multi-target Neural Network Learner
+
+***************************************
+Multi-target Neural Network Learner
+***************************************
+
+
+.. index:: Multi-target Neural Network Learner
+.. autoclass:: Orange.multitarget.neural.NeuralNetworkLearner
+    :members:
+    :show-inheritance:
+
+.. index:: Multi-target Neural Network Classifier
+.. autoclass:: Orange.multitarget.neural.NeuralNetworkClassifier
+    :members:
+    :show-inheritance:
+
+"""
+
+
+import Orange
 import numpy as np
 np.seterr('ignore') # set to ignore to disable overflow errors
 np.random.seed(42) # TODO: check with Jure
 class NeuralNetworkLearner(Orange.classification.Learner):
     """
     NeuralNetworkLearner uses jzbontar's implementation of neural networks and wraps it in
-    an Orange compatible learner.
+    an Orange compatible learner. 
+    TODO: explain neural networks
+
+    :param name: learner name.
+    :type name: string
+
+    :param n_mid: Number of nodes in the hidden layer
+    :type n_mid: integer
+
+    :param reg_fact: Regularization factor.
+    :type reg_fact: float
+
+    :param max_iter: Maximum number of iterations.
+    :type max_iter: integer
+
+    :rtype: :class:`Orange.multitarget.neural.neuralNetworkLearner` or 
+            :class:`Orange.multitarget.chain.NeuralNetworkClassifier`
     """
 
     def __new__(cls, data=None, weight = 0, **kwargs):
             self.__init__(**kwargs)
             return self(data,weight)
 
-    def __init__(self, name="NeuralNetwork",n_mid=10,reg_fact=1,max_iter=1000,treshold=0.5):
+    def __init__(self, name="NeuralNetwork", n_mid=10, reg_fact=1, max_iter=1000):
         """
         Current default values are the same as in the original implementation (neural_networks.py)
         Currently supports only multi-label data.
         """
+
         self.name = name
         self.n_mid = n_mid
-        self.reg_fact=reg_fact
-        self.max_iter=max_iter
-        self.treshold = treshold
+        self.reg_fact = reg_fact
+        self.max_iter = max_iter
 
     def __call__(self,data,weight=0):
+        """
+        Learn from the given table of data instances.
         
+        :param instances: data for learning.
+        :type instances: class:`Orange.data.Table`
+
+        :param weight: weight.
+        :type weight: int
+
+        :param class_order: list of descriptors of class variables
+        :type class_order: list of :class:`Orange.feature.Descriptor`
+
+        :rtype: :class:`Orange.multitarget.chain.NeuralNetworkClassifier`
+        """
+
         #converts attribute data
         X = data.to_numpy()[0] 
 
         
         self.nn.fit(X,Y)
                
-        return NeuralNetworkClassifier(classifier=self.nn.predict, domain = data.domain, treshold=self.treshold)
+        return NeuralNetworkClassifier(classifier=self.nn.predict, domain = data.domain)
 
 
 class NeuralNetworkClassifier():
-    """Classifier returned by NeuralNetworkLearner"""
+    """    
+    Uses the classifier induced by the :obj:`NeuralNetworkLearner`.
+  
+    :param name: name of the classifier.
+    :type name: string
+    """
+
     def __init__(self,**kwargs):
         self.__dict__.update(**kwargs)
 
     def __call__(self,example, result_type=Orange.core.GetValue):
+        """
+        :param instance: instance to be classified.
+        :type instance: :class:`Orange.data.Instance`
+        
+        :param result_type: :class:`Orange.classification.Classifier.GetValue` or \
+              :class:`Orange.classification.Classifier.GetProbabilities` or
+              :class:`Orange.classification.Classifier.GetBoth`
+        
+        :rtype: :class:`Orange.data.Value`, 
+              :class:`Orange.statistics.Distribution` or a tuple with both
+        """
+
         # transform example to numpy
         input = np.array([[float(e) for e in example]])
         # transform results from numpy
         elif result_type == Orange.classification.Classifier.GetProbabilities: return tuple(mt_prob)
         else: 
             return [tuple(mt_value),tuple(mt_prob)]
+
+if __name__ == '__main__':
+    import time
+    print "STARTED"
+    global_timer = time.time()
+
+    data = Orange.data.Table('scene')
+    
+    l = NeuralNetworkLearner()
+
+    res = Orange.evaluation.testing.cross_validation([l],data)
+
+    scores = Orange.evaluation.scoring.mt_average_score(res,Orange.evaluation.scoring.RMSE)
+
+    for i in range(len(scores)):
+        print res.classifierNames[i], scores[i]
+
+    print "--DONE %.2f --" % (time.time()-global_timer)

File docs/rst/Orange.evaluation.reliability.rst

-.. automodule:: Orange.evaluation.reliability
-
-.. index:: Reliability Estimation
-
-.. index::
-   single: reliability; Reliability Estimation for Regression
-
-##########################################################
-Reliability estimation (``Orange.evaluation.reliability``)
-##########################################################
-
-*************************************
-Reliability Estimation for Regression
-*************************************
-
-Reliability assessment statistically predicts reliability of single
-predictions. Most of implemented algorithms are taken from Comparison of
-approaches for estimating reliability of individual regression predictions,
-Zoran Bosnić, 2008.
-
-The following example shows basic usage of reliability estimation methods:
-
-.. literalinclude:: code/reliability-basic.py
-    :lines: 7-
-
-The important points of this example are:
- * construction of reliability estimators using classes,
-   implemented in this module,
- * construction of a reliability learner that bonds a regular learner
-   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
-   reliability estimators,
- * calling the constructed classifier with
-   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
-   probabilities; :obj:`probability` is the object that gets appended the
-   :obj:`reliability_estimate` attribute, an instance of
-   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.
-
-It is also possible to do reliability estimation on whole data
-table, not only on single instance. Next example demonstrates usage of a
-cross-validation technique for reliability estimation. Reliability estimations
-for first 10 instances get printed:
-
-.. literalinclude:: code/reliability-run.py
-    :lines: 11-
-
-Reliability Methods
-===================
-
-Sensitivity Analysis (SAvar and SAbias)
----------------------------------------
-.. autoclass:: SensitivityAnalysis
-
-Variance of bagged models (BAGV)
---------------------------------
-.. autoclass:: BaggingVariance
-
-Local cross validation reliability estimate (LCV)
--------------------------------------------------
-.. autoclass:: LocalCrossValidation
-
-Local modeling of prediction error (CNK)
-----------------------------------------
-.. autoclass:: CNeighbours
-
-Bagging variance c-neighbours (BVCK)
-------------------------------------
-
-.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
-
-Mahalanobis distance
---------------------
-
-.. autoclass:: Mahalanobis
-
-Mahalanobis to center
----------------------
-
-.. autoclass:: MahalanobisToCenter
-
-Reliability estimation wrappers
-===============================
-
-.. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds)
-    :members:
-
-.. autoclass:: Classifier
-    :members:
-
-Reliability estimation results
-==============================
-
-.. autoclass:: Estimate
-    :members:
-    :show-inheritance:
-
-There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
-method IDs (ints) to method names (strings).
-
-In this module, there are also two constants for distinguishing signed and
-absolute reliability estimation measures::
-
-  SIGNED = 0
-  ABSOLUTE = 1
-
-Reliability estimation scoring methods
-======================================
-
-.. autofunction:: get_pearson_r
-
-.. autofunction:: get_pearson_r_by_iterations
-
-.. autofunction:: get_spearman_r
-
-Example of usage
-================
-
-.. literalinclude:: code/reliability-long.py
-    :lines: 11-26
-
-This script prints out Pearson's R coefficient between reliability estimates
-and actual prediction errors, and a corresponding p-value, for each of the
-reliability estimation measures used by default. ::
-
-  Estimate               r       p
-  SAvar absolute        -0.077   0.454
-  SAbias signed         -0.165   0.105
-  SAbias absolute       -0.099   0.333
-  BAGV absolute          0.104   0.309
-  CNK signed             0.233   0.021
-  CNK absolute           0.057   0.579
-  LCV absolute           0.069   0.504
-  BVCK_absolute          0.092   0.368
-  Mahalanobis absolute   0.091   0.375
-
-
-References
-==========
-
-Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
-reliability using local sensitivity analysis. <http://www.springerlink
-.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
-
-Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
-reliability of individual regression predictions. <http://www.sciencedirect
-.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
-67(3), pp. 504-516.
-
-Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
-for individual regression predictions. <http://journals.cambridge
-.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
-pp. 27-47.
-

File docs/rst/Orange.multitarget.binary.rst

+.. automodule:: Orange.multitarget.binary

File docs/rst/Orange.multitarget.chain.rst

+.. automodule:: Orange.multitarget.chain

File docs/rst/Orange.multitarget.neural.rst

+.. automodule:: Orange.multitarget.neural

File docs/rst/Orange.multitarget.rst

+###########################################
+Multi-target prediction (``multitarget``)
+###########################################
+
+Multi-target prediction tries to achieve better prediction accuracy or speed
+through prediction of multiple dependent variables at once. It works on
+:ref:`multi-target data <multiple-classes>`, which is also supported by
+Orange's tab file format using :ref:`multiclass directive <tab-delimited>`.
+
+.. toctree::
+   :maxdepth: 1
+
+   Orange.multitarget.tree
+   Orange.multitarget.binary
+   Orange.multitarget.chain
+   Orange.multitarget.neural
+   Orange.regression.pls
+   Orange.regression.earth
+
+For evaluation of multi-target methods, see the corresponding section in 
+:ref:`Orange.evaluation.scoring <mt-scoring>`.
+
+
+.. automodule:: Orange.multitarget

File docs/rst/Orange.multitarget.tree.rst

+.. automodule:: Orange.multitarget.tree

File docs/rst/index.rst

-Orange Reliability documentation
+Orange Multitarget documentation
 ================================
 
-Orange Reliability is an add-on for Orange_ data mining software package. It
-extends Orange by providing functionality to estimate reliability of individual
-regression and classification predictions.
+Orange Multitarget is an add-on for Orange_ data mining software package. It
+extends Orange by providing methods that allow for classification of datasets
+with multiple classes.
 
 .. _Orange: http://orange.biolab.si/
 
-Widgets
--------
-
-This module contains a widget named Reliability, but unfortunately it has not
-yet been documented.
-
-.. toctree::
-   :maxdepth: 1
  
 Scripting Reference
 -------------------
 .. toctree::
    :maxdepth: 1
 
-   Orange.evaluation.reliability
+   Orange.multitarget
 
 Installation
 ------------
 
-To install Reliability add-on for Orange from PyPi_ run::
+To install Multitarget add-on for Orange from PyPi_ run::
 
-    pip install Orange-Reliability
+    pip install Orange-Multitarget
 
 To install it from source code run::
 
 
 Source code is available on Bitbucket_. For issues and wiki we use Trac_.
 
-.. _Bitbucket: https://bitbucket.org/biolab/orange-reliability
+.. _Bitbucket: https://bitbucket.org/mlevar/orange-multitarget
 .. _Trac: http://orange.biolab.si/trac/
 
 Indices and tables
 
 ENTRY_POINTS = {
     'orange.addons': (
-        'multitarget_addon = _multitarget',
+        'multitarget = _multitarget',
     ),
     #'orange.widgets': (
     #    'Evaluate = _reliability.widgets',
     #),
 	#TODO: datasets
 	#'orange.data.io.search_paths': ('multitask = _multitask:datasets',)},
+	'orange.data.io.search_paths': ('multitarget = _multitarget:datasets',),
 }
 
 if __name__ == '__main__':