orange-reliability / docs / rst / Orange.evaluation.reliability.rst

Reliability estimation (Orange.evaluation.reliability)

Reliability assessment statistically predicts reliability of single predictions. Most of implemented algorithms are taken from Comparison of approaches for estimating reliability of individual regression predictions, Zoran Bosnić, 2008.

The following example shows basic usage of reliability estimation methods:

The important points of this example are:

It is also possible to do reliability estimation on whole data table, not only on single instance. Next example demonstrates usage of a cross-validation technique for reliability estimation. Reliability estimations for first 10 instances get printed:

Reliability Methods

Sensitivity Analysis (SAvar and SAbias)

Variance of bagged models (BAGV)

Local cross validation reliability estimate (LCV)

Local modeling of prediction error (CNK)

Bagging variance c-neighbours (BVCK)

Mahalanobis distance

Mahalanobis to center

Reliability estimation wrappers

Reliability estimation results

There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation method IDs (ints) to method names (strings).

In this module, there are also two constants for distinguishing signed and absolute reliability estimation measures:

SIGNED = 0
ABSOLUTE = 1

Reliability estimation scoring methods

Example of usage

This script prints out Pearson's R coefficient between reliability estimates and actual prediction errors, and a corresponding p-value, for each of the reliability estimation measures used by default.

Estimate               r       p
SAvar absolute        -0.077   0.454
SAbias signed         -0.165   0.105
SAbias absolute       -0.099   0.333
BAGV absolute          0.104   0.309
CNK signed             0.233   0.021
CNK absolute           0.057   0.579
LCV absolute           0.069   0.504
BVCK_absolute          0.092   0.368
Mahalanobis absolute   0.091   0.375

References

Bosnić, Z., Kononenko, I. (2007) Estimation of individual prediction reliability using local sensitivity analysis. Applied Intelligence 29(3), pp. 187-203.

Bosnić, Z., Kononenko, I. (2008) Comparison of approaches for estimating reliability of individual regression predictions. Data & Knowledge Engineering 67(3), pp. 504-516.

Bosnić, Z., Kononenko, I. (2010) Automatic selection of reliability estimates for individual regression predictions. The Knowledge Engineering Review 25(1), pp. 27-47.

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.