- :param res: results of evaluation, done using learners,

- wrapped into :class:`Orange.evaluation.reliability.Classifier`.

+ :param res: Evaluation results with :obj:`reliability_estimate`.

:type res: :class:`Orange.evaluation.testing.ExperimentResults`

- Return Pearson's coefficient between the prediction error and each of the

- used reliability estimates. Also, return the p-value of each of

+ Pearson's coefficients between the prediction error and

+ reliability estimates with p-values.

prediction_error = get_prediction_error_list(res)

- :param res: results of evaluation, done using learners,

- wrapped into :class:`Orange.evaluation.reliability.Classifier`.

+ :param res: Evaluation results with :obj:`reliability_estimate`.

:type res: :class:`Orange.evaluation.testing.ExperimentResults`

- Return Spearman's coefficient between the prediction error and each of the

- used reliability estimates. Also, return the p-value of each of

+ Spearman's coefficients between the prediction error and

+ reliability estimates with p-values.

prediction_error = get_prediction_error_list(res)

def get_pearson_r_by_iterations(res):

- :param res: results of evaluation, done using learners,

- wrapped into :class:`Orange.evaluation.reliability.Classifier`.

+ :param res: Evaluation results with :obj:`reliability_estimate`.

:type res: :class:`Orange.evaluation.testing.ExperimentResults`

- Return average Pearson's coefficient over all folds between prediction error

- and each of the used estimates.

+ Pearson's coefficients between prediction error

+ and reliability estimates averaged over all folds.

results_by_fold = Orange.evaluation.scoring.split_by_iterations(res)

number_of_estimates = len(res.results[0].probabilities[0].reliability_estimate)

number_of_instances = len(res.results)

number_of_folds = len(results_by_fold)

results = [0 for _ in xrange(number_of_estimates)]

sig = [0 for _ in xrange(number_of_estimates)]

method_list = [0 for _ in xrange(number_of_estimates)]

- Reliability estimate. Contains attributes that describe the results of

- reliability estimation.

+ Describes a reliability estimate.

- ~~A numerical~~ reliability~~ estimate~~.

.. attribute:: signed_or_absolute

- Determines whether the method ~~used gives~~ a signed or absolute result.

+ Determines whether the method returned a signed or absolute result.

Has a value of either :obj:`SIGNED` or :obj:`ABSOLUTE`.

- An integer ID of reliability estimation method used.

+ An integer ID of the reliability estimation method used.

.. attribute:: method_name

- Name (string) of reliability estimation method used.

+ Name (string) of the reliability estimation method used.

def __init__(self, estimate, signed_or_absolute, method):

:rtype: :class:`Orange.evaluation.reliability.SensitivityAnalysisClassifier`

- To estimate the reliability of prediction for a given instance,

- the learning set is extended with that instance with the label changes to

+ The learning set is extended with that instancem, where the label is changed to

:math:`K + \epsilon (l_{max} - l_{min})` (:math:`K` is the initial prediction,

:math:`\epsilon` a sensitivity parameter, and :math:`l_{min}` and

- :math:`l_{max}` the lower and upper bounds of labels on training data)

+ :math:`l_{max}` the lower and upper bounds of labels on training data).

Results for multiple values of :math:`\epsilon` are combined

- into SAvar and SAbias. SAbias ~~can be used either in~~ a signed or absolute form.

+ into SAvar and SAbias. SAbias has a signed or absolute form.

:math:`SAvar = \\frac{\sum_{\epsilon \in E}(K_{\epsilon} - K_{-\epsilon})}{|E|}`

:math:`SAbias = \\frac{\sum_{\epsilon \in E} (K_{\epsilon} - K ) + (K_{-\epsilon} - K)}{2 |E|}`

- :param m: Number of bagg~~ing~~ models~~ to be used with BAGV estimate~~

+ :param m: Number of bagged models. Default: 50.

- :param for~~ ~~instances: Optional. If test instances

+ :param for_instances: Optional. If test instances

are given as a parameter, this class can compute their reliabilities

on the fly, which saves memory.

:rtype: :class:`Orange.evaluation.reliability.BaggingVarianceClassifier`

- :math:`m` different bagging models are used to estimate

- the value of dependent variable for a given instance. For regression,

- the variance of predictions is a reliability

+ For regression, BAGV is the variance of predictions:

:math:`BAGV = \\frac{1}{m} \sum_{i=1}^{m} (K_i - K)^2`, where

:math:`K = \\frac{\sum_{i=1}^{m} K_i}{m}` and :math:`K_i` are

predictions of individual models.

- For classification, 1 minus the average Euclidean distance between class

- probability distributions predicted by the model, and distributions

- predicted by the individual bagged models, is the BAGV reliability

- measure. For classification, a greater value implies a better

+ For classification, BAGV is 1 minus the average Euclidean

+ distance between class probability distributions predicted by the

+ model, and distributions predicted by the individual bagged model;

+ a greater value implies a better prediction.

This reliability measure can run out of memory if individual classifiers themselves

use a lot of memory; it needs :math:`m` times memory

- :param distance_weighted: ~~F~~or classification~~,~~

+ :param distance_weighted: Relevant only for classification;

use an average distance between distributions, weighted by :math:`e^{-d}`,

where :math:`d` is the distance between predicted instance and the

:rtype: :class:`Orange.evaluation.reliability.CNeighboursClassifier`

- For regression, CNK is ~~defined ~~a difference

+ For regression, CNK is a difference

between average label of its nearest neighbours and the prediction. CNK

can be either signed or absolute. A greater value implies greater prediction error.

- Adds reliability estimation to any learner: multiple reliability estimation

- algorithms can be used simultaneously.

- This learner can be used as any other learner,

+ Adds reliability estimation to any prediction method.

+ This class can be used as any other Orange learner,

but returns the classifier wrapped into an instance of

:class:`Orange.evaluation.reliability.Classifier`.

:param box_learner: Learner to wrap into a reliability estimation

:type box_learner: :obj:`~Orange.classification.Learner`

def __call__(self, instances, weight=None, **kwds):

- """~~Learn from the given table of data instances~~.

+ """Construct a classifier.

- :param instances: ~~Data to learn from~~.

+ :param instances: Learning data.

:type instances: Orange.data.Table

:param weight: Id of meta attribute with weights of instances

def __call__(self, instance, result_type=Orange.core.GetValue):

- Classify and estimate reliability ~~of estimation ~~for a new instance.

+ Classify and estimate reliability for a new instance.

When :obj:`result_type` is set to

:obj:`Orange.classification.Classifier.GetBoth` or

:obj:`Orange.classification.Classifier.GetProbabilities`,