# Source

# orange-reliability / docs / rst / Orange.evaluation.reliability.rst

# Reliability estimation (`Orange.evaluation.reliability`)

Reliability assessment aims to predict reliabilities of individual predictions.

Most of implemented algorithms for regression described in "Comparison of approaches for estimating reliability of individual regression predictions, Zoran Bosnić, 2008" for regression and in in "Evaluating Reliability of Single Classifications of Neural Networks, Darko Pevec, 2011" for classification.

We can use reliability estimation with any Orange learners. The following example:

- Constructs reliability estimators (implemented in this module),
- Combines a regular learner. (:class:`~Orange.classification.knn.kNNLearner` in this case) with reliability estimators.
- Obtains prediction probabilities from the constructed classifier (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting probabilities have and additional attribute, :obj:`reliability_estimate` attribute, :class:`Orange.evaluation.reliability.Estimate`.

We could also evaluate more examples. The next example prints reliability estimates for first 10 instances (with cross-validation):

## Reliability Methods

For regression, you can use all the described measures except \(O_{ref}\) . Classification is supported by BAGV, LCV, CNK and DENS, \(O_{ref}\) .

### Sensitivity Analysis (SAvar and SAbias)

### Variance of bagged models (BAGV)

### Local cross validation reliability estimate (LCV)

### Local modeling of prediction error (CNK)

### Bagging variance c-neighbours (BVCK)

### Mahalanobis distance

### Mahalanobis to center

### Density estimation using Parzen window (DENS)

### Internal cross validation (ICV)

### Stacked generalization (Stacking)

### Reference Estimate for Classification (\(O_{ref}\) )

## Reliability estimation wrappers

## Reliability estimation results

## Reliability estimation scoring

## Example

The following script prints Pearson's correlation coefficient (r) between reliability estimates and actual prediction errors, and a corresponding p-value, for default reliability estimation measures.

Results:

Estimate r p SAvar absolute -0.077 0.454 SAbias signed -0.165 0.105 SAbias absolute 0.095 0.352 LCV absolute 0.069 0.504 BVCK absolute 0.060 0.562 BAGV absolute 0.078 0.448 CNK signed 0.233 0.021 CNK absolute 0.058 0.574 Mahalanobis absolute 0.091 0.375 Mahalanobis to center 0.096 0.349

## References

Bosnić, Z., Kononenko, I. (2007) Estimation of individual prediction
reliability using local sensitivity analysis. *Applied Intelligence* 29(3), pp. 187-203.

Bosnić, Z., Kononenko, I. (2008) Comparison of approaches for estimating
reliability of individual regression predictions. *Data & Knowledge Engineering*
67(3), pp. 504-516.

Bosnić, Z., Kononenko, I. (2010) Automatic selection of reliability estimates
for individual regression predictions. *The Knowledge Engineering Review* 25(1),
pp. 27-47.

Pevec, D., Štrumbelj, E., Kononenko, I. (2011) Evaluating Reliability of
Single Classifications of Neural Networks. *Adaptive and Natural Computing
Algorithms*, 2011, pp. 22-30.