Source

orange-reliability / docs / rst / Orange.evaluation.reliability.rst

Reliability estimation (Orange.evaluation.reliability)

Reliability assessment aims to predict reliabilities of individual predictions. Most of implemented algorithms for regression described in [Bosnic2008] and in [Pevec2011] for classification.

We can use reliability estimation with any Orange learners. The following example:

We could also evaluate more examples. The next example prints reliability estimates for first 10 instances (with cross-validation):

Reliability estimation wrappers

Reliability Methods

All measures except \(O_{ref}\) work with regression. Classification is supported by BAGV, LCV, CNK and DENS, \(O_{ref}\) .

Sensitivity Analysis (SAvar and SAbias)

Variance of bagged models (BAGV)

Local cross validation reliability estimate (LCV)

Local modeling of prediction error (CNK)

Bagging variance c-neighbours (BVCK)

Mahalanobis distance

Mahalanobis to center

Density estimation using Parzen window (DENS)

Internal cross validation (ICV)

Stacked generalization (Stacking)

Reference Estimate for Classification (\(O_{ref}\) )

Reliability estimation results

Reliability estimation scoring

Example

The following script prints Pearson's correlation coefficient (r) between reliability estimates and actual prediction errors, and a corresponding p-value, for default reliability estimation measures.

Results:

Estimate               r       p
SAvar absolute        -0.077   0.454
SAbias signed         -0.165   0.105
SAbias absolute        0.095   0.352
LCV absolute           0.069   0.504
BVCK absolute          0.060   0.562
BAGV absolute          0.078   0.448
CNK signed             0.233   0.021
CNK absolute           0.058   0.574
Mahalanobis absolute   0.091   0.375
Mahalanobis to center  0.096   0.349

References

[Bosnic2007]Bosnić, Z., Kononenko, I. (2007) Estimation of individual prediction reliability using local sensitivity analysis. Applied Intelligence 29(3), pp. 187-203.
[Bosnic2008]Bosnić, Z., Kononenko, I. (2008) Comparison of approaches for estimating reliability of individual regression predictions. Data & Knowledge Engineering 67(3), pp. 504-516.
[Bosnic2010]Bosnić, Z., Kononenko, I. (2010) Automatic selection of reliability estimates for individual regression predictions. The Knowledge Engineering Review 25(1), pp. 27-47.
[Pevec2011]Pevec, D., Štrumbelj, E., Kononenko, I. (2011) Evaluating Reliability of Single Classifications of Neural Networks. Adaptive and Natural Computing Algorithms, 2011, pp. 22-30.
[Wolpert1992]Wolpert, David H. (1992) Stacked generalization. Neural Networks, Vol. 5, 1992, pp. 241-259.
[Dzeroski2004]Dzeroski, S. and Zenko, B. (2004) Is combining classifiers with stacking better than selecting the best one? Machine Learning, Vol. 54, 2004, pp. 255-273.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.