Source

orange-reliability / docs / rst / Orange.evaluation.reliability.rst

Full commit
Matija Polajnar 7dfb468 










Matija Polajnar f98f041 


Matija Polajnar 7dfb468 
markotoplak d2bc01f 

Matija Polajnar 7dfb468 
markotoplak d2bc01f 















Matija Polajnar 7dfb468 



markotoplak d2bc01f 

Matija Polajnar 7dfb468 

markotoplak 5e0e4eb 
Matija Polajnar 7dfb468 



markotoplak d2bc01f 

Matija Polajnar f50b356 
Matija Polajnar 7dfb468 






























Matija Polajnar f50b356 




markotoplak d1cc6a0 










Matija Polajnar f98f041 




Matija Polajnar 7dfb468 











markotoplak d2bc01f 











Matija Polajnar 7dfb468 





markotoplak 5e0e4eb 

Matija Polajnar 7dfb468 






markotoplak 5e0e4eb 

Matija Polajnar 7dfb468 
markotoplak d2bc01f 



Matija Polajnar 7dfb468 
markotoplak 5e0e4eb 
Matija Polajnar 7dfb468 
markotoplak d2bc01f 
markotoplak 5e0e4eb 
Matija Polajnar 7dfb468 


markotoplak 5e0e4eb 



Matija Polajnar 7dfb468 
markotoplak 5e0e4eb 
Matija Polajnar 7dfb468 
markotoplak 5e0e4eb 
Matija Polajnar 7dfb468 

















Matija Polajnar f50b356 


markotoplak 5e0e4eb 
.. automodule:: Orange.evaluation.reliability

.. index:: Reliability Estimation

.. index::
   single: reliability; Reliability Estimation for Regression

##########################################################
Reliability estimation (``Orange.evaluation.reliability``)
##########################################################

********************************************************
Reliability Estimation for Regression and Classification
********************************************************

Reliability assessment aims to predict reliabilities of individual
predictions. 

Most of implemented algorithms for regression described in
"Comparison of approaches for estimating reliability of individual
regression predictions, Zoran Bosnić, 2008" for regression and in
in "Evaluating Reliability of Single
Classifications of Neural Networks, Darko Pevec, 2011" for classification.

We can use reliability estimation with any Orange learners. The following example:

 * Constructs reliability estimators (implemented in this module),
 * Combines a regular learner.
   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
   reliability estimators.
 * Obtains prediction probabilities from the constructed classifier
   (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting
   probabilities have and additional attribute, :obj:`reliability_estimate`
   attribute, :class:`Orange.evaluation.reliability.Estimate`.

.. literalinclude:: code/reliability-basic.py
    :lines: 7-

We could also evaluate more examples. The next example prints reliability estimates
for first 10 instances (with cross-validation):

.. literalinclude:: code/reliability-run.py
    :lines: 7-

Reliability Methods
===================

For regression, you can use all the described measures except :math:`O_{ref}`. Classification is
supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.

Sensitivity Analysis (SAvar and SAbias)
---------------------------------------
.. autoclass:: SensitivityAnalysis

Variance of bagged models (BAGV)
--------------------------------
.. autoclass:: BaggingVariance

Local cross validation reliability estimate (LCV)
-------------------------------------------------
.. autoclass:: LocalCrossValidation

Local modeling of prediction error (CNK)
----------------------------------------
.. autoclass:: CNeighbours

Bagging variance c-neighbours (BVCK)
------------------------------------

.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())

Mahalanobis distance
--------------------

.. autoclass:: Mahalanobis

Mahalanobis to center
---------------------

.. autoclass:: MahalanobisToCenter

Density estimation using Parzen window (DENS)
---------------------------------------------

.. autoclass:: ParzenWindowDensityBased

Internal cross validation (ICV)
-------------------------------

.. autoclass:: ICV


Stacked generalization (Stacking)
-------------------------------

.. autoclass:: Stacking

Reference Estimate for Classification (:math:`O_{ref}`)
-------------------------------------------------------

.. autoclass:: ReferenceExpectedError

Reliability estimation wrappers
===============================

.. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds)
    :members:

.. autoclass:: Classifier
    :members:

Reliability estimation results
==============================

.. data:: SIGNED
    
.. data:: ABSOLUTE

    These constants distinguish signed and
    absolute reliability estimation measures.

.. data:: METHOD_NAME

    A dictionary that that maps reliability estimation
    method IDs (integerss) to method names (strings).

.. autoclass:: Estimate
    :members:
    :show-inheritance:



Reliability estimation scoring
==============================

.. autofunction:: get_pearson_r

.. autofunction:: get_pearson_r_by_iterations

.. autofunction:: get_spearman_r

Example
=======

The following script prints Pearson's correlation coefficient (r) between reliability 
estimates and actual prediction errors, and a corresponding p-value, for 
default reliability estimation measures. 

.. literalinclude:: code/reliability-long.py
    :lines: 7-22

Results::
  
  Estimate               r       p
  SAvar absolute        -0.077   0.454
  SAbias signed         -0.165   0.105
  SAbias absolute        0.095   0.352
  LCV absolute           0.069   0.504
  BVCK absolute          0.060   0.562
  BAGV absolute          0.078   0.448
  CNK signed             0.233   0.021
  CNK absolute           0.058   0.574
  Mahalanobis absolute   0.091   0.375
  Mahalanobis to center  0.096   0.349

References
==========

Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
reliability using local sensitivity analysis. <http://www.springerlink
.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.

Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
reliability of individual regression predictions. <http://www.sciencedirect
.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
67(3), pp. 504-516.

Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
for individual regression predictions. <http://journals.cambridge
.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
pp. 27-47.

Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of
Single Classifications of Neural Networks. <http://www.springerlink.com
/content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing
Algorithms*, 2011, pp. 22-30.