Commits

committed d2bc01f

• Participants
• Parent commits d1cc6a0

File docs/rst/Orange.evaluation.reliability.rst

` Reliability Estimation for Regression and Classification`
` ********************************************************`
` `
`-Reliability assessment statistically predicts reliability of single`
`-predictions. Most of implemented algorithms for regression are taken from`
`-Comparison of approaches for estimating reliability of individual`
`-regression predictions, Zoran Bosnić, 2008. Implementations for`
`-classification follow descriptions in Evaluating Reliability of Single`
`-Classifications of Neural Networks, Darko Pevec, 2011.`
`+Reliability assessment aims to predict reliabilities of individual`
`+predictions. `
` `
`-The following example shows basic usage of reliability estimation methods:`
`+Most of implemented algorithms for regression described in`
`+"Comparison of approaches for estimating reliability of individual`
`+regression predictions, Zoran Bosnić, 2008" for regression and in`
`+in "Evaluating Reliability of Single`
`+Classifications of Neural Networks, Darko Pevec, 2011" for classification.`
`+`
`+We can use reliability estimation with any Orange learners. The following example:`
`+`
`+ * Constructs reliability estimators (implemented in this module),`
`+ * Combines a regular learner.`
`+   (:class:`~Orange.classification.knn.kNNLearner` in this case) with`
`+   reliability estimators.`
`+ * Obtains prediction probabilities from the constructed classifier`
`+   (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting`
`+   probabilities have and additional attribute, :obj:`reliability_estimate``
`+   attribute, :class:`Orange.evaluation.reliability.Estimate`.`
` `
` .. literalinclude:: code/reliability-basic.py`
`     :lines: 7-`
` `
`-The important points of this example are:`
`- * construction of reliability estimators using classes,`
`-   implemented in this module,`
`- * construction of a reliability learner that bonds a regular learner`
`-   (:class:`~Orange.classification.knn.kNNLearner` in this case) with`
`-   reliability estimators,`
`- * calling the constructed classifier with`
`-   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class`
`-   probabilities; :obj:`probability` is the object that gets appended the`
`-   :obj:`reliability_estimate` attribute, an instance of`
`-   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.`
`-`
`-It is also possible to do reliability estimation on whole data`
`-table, not only on single instance. Next example demonstrates usage of a`
`-cross-validation technique for reliability estimation. Reliability estimations`
`-for first 10 instances get printed:`
`+We could also evaluate more examples. The next example prints reliability estimates`
`+for first 10 instances (with cross-validation):`
` `
` .. literalinclude:: code/reliability-run.py`
`     :lines: 7-`
` Reliability Methods`
` ===================`
` `
`-For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains`
`-are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`.`
`+For regression, you can use all the described measures except :math:`O_{ref}`. Classification is`
`+supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.`
` `
` Sensitivity Analysis (SAvar and SAbias)`
` ---------------------------------------`
` Reliability estimation results`
` ==============================`
` `
`+.. data:: SIGNED`
`+    `
`+.. data:: ABSOLUTE`
`+`
`+    These constants distinguish signed and`
`+    absolute reliability estimation measures.`
`+`
`+.. data:: METHOD_NAME`
`+`
`+    A dictionary that that maps reliability estimation`
`+    method IDs (integerss) to method names (strings).`
`+`
` .. autoclass:: Estimate`
`     :members:`
`     :show-inheritance:`
` `
`-There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation`
`-method IDs (ints) to method names (strings).`
` `
`-In this module, there are also two constants for distinguishing signed and`
`-absolute reliability estimation measures::`
`-`
`-  SIGNED = 0`
`-  ABSOLUTE = 1`
` `
` Reliability estimation scoring`
` ==============================`
` Example`
` =======`
` `
`+The following script prints Pearson's correlation coefficient (r) between reliability `
`+estimates and actual prediction errors, and a corresponding p-value, for `
`+default reliability estimation measures. `
`+`
` .. literalinclude:: code/reliability-long.py`
`     :lines: 7-22`
` `
`-This script prints out Pearson's R coefficient between reliability estimates`
`-and actual prediction errors, and a corresponding p-value, for each of the`
`-reliability estimation measures used by default. ::`
`+Results::`
`   `
`   Estimate               r       p`
`   SAvar absolute        -0.077   0.454`