Reliability Estimation for Regression and Classification

********************************************************

-Reliability assessment statistically predicts reliability of single

-predictions. Most of implemented algorithms for regression are taken from

-Comparison of approaches for estimating reliability of individual

-regression predictions, Zoran Bosnić, 2008. Implementations for

-classification follow descriptions in Evaluating Reliability of Single

-Classifications of Neural Networks, Darko Pevec, 2011.

+Reliability assessment aims to predict reliabilities of individual

-The following example shows basic usage of reliability estimation methods:

+Most of implemented algorithms for regression described in

+"Comparison of approaches for estimating reliability of individual

+regression predictions, Zoran Bosnić, 2008" for regression and in

+in "Evaluating Reliability of Single

+Classifications of Neural Networks, Darko Pevec, 2011" for classification.

+We can use reliability estimation with any Orange learners. The following example:

+ * Constructs reliability estimators (implemented in this module),

+ * Combines a regular learner.

+ (:class:`~Orange.classification.knn.kNNLearner` in this case) with

+ reliability estimators.

+ * Obtains prediction probabilities from the constructed classifier

+ (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting

+ probabilities have and additional attribute, :obj:`reliability_estimate`

+ attribute, :class:`Orange.evaluation.reliability.Estimate`.

.. literalinclude:: code/reliability-basic.py

-The important points of this example are:

- * construction of reliability estimators using classes,

- implemented in this module,

- * construction of a reliability learner that bonds a regular learner

- (:class:`~Orange.classification.knn.kNNLearner` in this case) with

- reliability estimators,

- * calling the constructed classifier with

- :obj:`Orange.classification.Classifier.GetBoth` option to obtain class

- probabilities; :obj:`probability` is the object that gets appended the

- :obj:`reliability_estimate` attribute, an instance of

- :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.

-It is also possible to do reliability estimation on whole data

-table, not only on single instance. Next example demonstrates usage of a

-cross-validation technique for reliability estimation. Reliability estimations

-for first 10 instances get printed:

+We could also evaluate more examples. The next example prints reliability estimates

+for first 10 instances (with cross-validation):

.. literalinclude:: code/reliability-run.py

-For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains

-are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`.

+For regression, you can use all the described measures except :math:`O_{ref}`. Classification is

+supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.

Sensitivity Analysis (SAvar and SAbias)

---------------------------------------

Reliability estimation results

==============================

+ These constants distinguish signed and

+ absolute reliability estimation measures.

+ A dictionary that that maps reliability estimation

+ method IDs (integerss) to method names (strings).

-There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation

-method IDs (ints) to method names (strings).

-In this module, there are also two constants for distinguishing signed and

-absolute reliability estimation measures::

Reliability estimation scoring

==============================

+The following script prints Pearson's correlation coefficient (r) between reliability

+estimates and actual prediction errors, and a corresponding p-value, for

+default reliability estimation measures.

.. literalinclude:: code/reliability-long.py

-This script prints out Pearson's R coefficient between reliability estimates

-and actual prediction errors, and a corresponding p-value, for each of the

-reliability estimation measures used by default. ::

SAvar absolute -0.077 0.454