Commits

Zoltán Szabó  committed 541d2ac

Chi-square mutual information estimation based on Pearson chi-square divergence: added. Shannon entropy estimation based on an alternative linearly corrected spacing method: added. See 'IChiSquare_DChiSquare_initialization', 'IChiSquare_DChiSquare_estimation.m', 'HShannon_spacing_Vplin2_initialization.m', 'HShannon_spacing_Vplin2_estimation.m'. Quick tests updated with the new estimators, see 'quick_test_HShannon.m', 'quick_test_Iimreg.m', 'quick_test_Iindependence.m'.

  • Participants
  • Parent commits e97e852
  • Tags release-0.47

Comments (0)

Files changed (12)

File CHANGELOG.txt

+v0.47 (Nov 1, 2013):
+-Chi-square mutual information estimation based on Pearson chi-square divergence: added; see 'IChiSquare_DChiSquare_initialization', 'IChiSquare_DChiSquare_estimation.m'.
+-Shannon entropy estimation based on an alternative linearly corrected spacing method: added; see 'HShannon_spacing_Vplin2_initialization.m', 'HShannon_spacing_Vplin2_estimation.m'.
+-Quick tests updated with the new estimators, see 'quick_test_HShannon.m', 'quick_test_Iimreg.m', 'quick_test_Iindependence.m'.
+
 v0.46 (Oct 21, 2013):
 -Phi-entropy (f-entropy) estimation based on the spacing method: added; see 'HPhi_spacing_initialization.m', 'HPhi_spacing_estimation.m'.
 -Pearson chi square divergence (chi square distance) estimation based on k-nearest neighbors: added; see 'DChiSquare_kNN_k_initialization.m', 'DChiSquare_kNN_k_estimation.m'.
 
 News:
 
-- ITE has been accepted at [NIPS-2013: MLOSS workshop](http://mloss.org/workshop/nips13/), [PDF](http://www.gatsby.ucl.ac.uk/~szabo/publications/szabo13information.pdf).
+- ITE has been accepted for presentation at [NIPS-2013: MLOSS workshop](http://mloss.org/workshop/nips13/), [PDF](http://www.gatsby.ucl.ac.uk/~szabo/publications/szabo13information.pdf).
 
 * * *
 
 ITE can estimate 
 
 - `entropy (H)`: Shannon entropy, R�nyi entropy, Tsallis entropy (Havrda and Charv�t entropy), complex entropy, Phi-entropy (f-entropy),
-- `mutual information (I)`: generalized variance, kernel canonical correlation analysis, kernel generalized variance, Hilbert-Schmidt independence criterion, Shannon mutual information (total correlation, multi-information), L2 mutual information, R�nyi mutual information, Tsallis mutual information, copula-based kernel dependency, multivariate version of Hoeffding's Phi, Schweizer-Wolff's sigma and kappa, complex mutual information, Cauchy-Schwartz quadratic mutual information, Euclidean distance based quadratic mutual information, distance covariance, distance correlation, approximate correntropy independence measure,
+- `mutual information (I)`: generalized variance, kernel canonical correlation analysis, kernel generalized variance, Hilbert-Schmidt independence criterion, Shannon mutual information (total correlation, multi-information), L2 mutual information, R�nyi mutual information, Tsallis mutual information, copula-based kernel dependency, multivariate version of Hoeffding's Phi, Schweizer-Wolff's sigma and kappa, complex mutual information, Cauchy-Schwartz quadratic mutual information, Euclidean distance based quadratic mutual information, distance covariance, distance correlation, approximate correntropy independence measure, chi-square mutual information (Hilbert-Schmidt norm of the normalized cross-covariance operator, squared-loss mutual information,  mean square contingency), 
 - `divergence (D)`: Kullback-Leibler divergence (relative entropy, I directed divergence), L2 divergence, R�nyi divergence, Tsallis divergence, Hellinger distance, Bhattacharyya distance, maximum mean discrepancy (kernel distance), J-distance (symmetrised Kullback-Leibler divergence, J divergence), Cauchy-Schwartz divergence, Euclidean distance based divergence, energy distance (specially the Cramer-Von Mises distance), Jensen-Shannon divergence, Jensen-R�nyi divergence, K divergence, L divergence, certain f-divergences (Csisz�r-Morimoto divergence, Ali-Silvey distance), non-symmetric Bregman distance (Bregman divergence), Jensen-Tsallis divergence, symmetric Bregman distance, Pearson chi square divergence (chi square distance),
 - `association measures (A)`, including `measures of concordance`: multivariate extensions of Spearman's rho (Spearman's rank correlation coefficient, grade correlation coefficient), correntropy, centered correntropy, correntropy coefficient, correntropy induced metric, centered correntropy induced metric, multivariate extension of Blomqvist's beta (medial correlation coefficient), multivariate conditional version of Spearman's rho, lower/upper tail dependence via conditional Spearman's rho,
 - `cross quantities (C)`: cross-entropy,
 
 **Download** the latest release: 
 
-- code: [zip](https://bitbucket.org/szzoli/ite/downloads/ITE-0.46_code.zip), [tar.bz2](https://bitbucket.org/szzoli/ite/downloads/ITE-0.46_code.tar.bz2), 
-- [documentation (pdf)](https://bitbucket.org/szzoli/ite/downloads/ITE-0.46_documentation.pdf).
+- code: [zip](https://bitbucket.org/szzoli/ite/downloads/ITE-0.47_code.zip), [tar.bz2](https://bitbucket.org/szzoli/ite/downloads/ITE-0.47_code.tar.bz2), 
+- [documentation (pdf)](https://bitbucket.org/szzoli/ite/downloads/ITE-0.47_documentation.pdf).
 
 

File code/estimators/base_estimators/DChiSquare_kNN_k_estimation.m

 function [D] = DChiSquare_kNN_k_estimation(Y1,Y2,co)
 %function [D] = DChiSquare_kNN_k_estimation(Y1,Y2,co)
-%Estimates the Pearson chi square divergence (D) of Y1 and Y2 using the kNN method (S={k}). 
+%Estimates the Pearson chi-square divergence (D) of Y1 and Y2 using the kNN method (S={k}). 
 %
 %We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
 %

File code/estimators/base_estimators/DChiSquare_kNN_k_initialization.m

 function [co] = DChiSquare_kNN_k_initialization(mult,post_init)
 %function [co] = DChiSquare_kNN_k_initialization(mult)
 %function [co] = DChiSquare_kNN_k_initialization(mult,post_init)
-%Initialization of the kNN (k-nearest neighbor, S={k}) based Pearson chi square divergence estimator.
+%Initialization of the kNN (k-nearest neighbor, S={k}) based Pearson chi-square divergence estimator.
 %
 %Note:
 %   1)The estimator is treated as a cost object (co).

File code/estimators/base_estimators/DMMD_Ustat_estimation.m

 function [D] = DMMD_Ustat_estimation(Y1,Y2,co)
 %function [D] = DMMD_Ustat_estimation(Y1,Y2,co)
-%Estimates divergence (D) of Y1 and Y2 using the MMD (maximum mean discrepancy) method, applying V-statistics. 
+%Estimates divergence (D) of Y1 and Y2 using the MMD (maximum mean discrepancy) method, applying U-statistics. 
 %
 %We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
 %
 term2 = sum(sum(kY2Y2)) / (num_of_samplesY2^2);
 term3 = -2 * sum(sum(kY1Y2)) / (num_of_samplesY1*num_of_samplesY2);
 
-D = sqrt(abs(term1+term2+term3)); %abs(): to avoid 'sqrt(negative)' values
+D = sqrt(abs(term1+term2+term3)); %abs(): to avoid 'sqrt(negative)' values

File code/estimators/base_estimators/HShannon_spacing_Vplin2_estimation.m

+function [H] = HShannon_spacing_Vplin2_estimation(Y,co)
+%function [H] = HShannon_spacing_Vplin2_estimation(Y,co)
+%Estimates the Shannon entropy (H) of Y using Vasicek's spacing method with piecewise linear correction-2. 
+%
+%We use the naming convention 'H<name>_estimation' to ease embedding new entropy estimation methods.
+%
+%INPUT:
+%   Y: Y(:,t) is the t^th sample.
+%  co: entropy estimator object.
+%
+%REFERENCE: 
+%   Nader Ebrahimi, Kurt Pflughoeft and Ehsan S. Soofi. Two measures of sample entropy. Statistics and Probability Letters, 20(3):225-234, 1994.
+
+%Copyright (C) 2013 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK. The information theoretical quantity of interest can be (and is!) estimated exactly [co.mult=1]; the computational complexity of the estimation is essentially the same as that of the 'up to multiplicative constant' case [co.mult=0].
+
+[d,num_of_samples] = size(Y);
+
+%verification:
+    if d~=1
+        error('The samples must be one-dimensional for this estimator.');
+    end
+
+m = floor(sqrt(num_of_samples));%m/num_of_samples->0, m,num_of_samples->infty; m<num_of_samples/2
+Y_sorted = sort(Y);
+
+%padding:
+    %I:
+        Y_sorted = [repmat(Y_sorted(1),1,m),Y_sorted,repmat(Y_sorted(end),1,m)];%with the smallest (left) and largest (right) element
+    %II [a,b such that  P(Y\in[a,b])=1 ]:
+        %a = Y_sorted(1);  
+        %b = Y_sorted(end);
+        %Y_sorted = [a + ([1:m]-1)/m*(Y_sorted(1)-a), Y_sorted, b-(num_of_samples-[num_of_samples-m:num_of_samples])/m * (b-Y_sorted(end))]; %a->Y_sorted(1) linearly, Y_sorted, Y_sorted(end)->b linearly
+
+diffs = Y_sorted(2*m+1:num_of_samples+2*m) - Y_sorted(1:num_of_samples);
+c = [1+([1:m]-1)/m,2*ones(1,num_of_samples-2*m),1+(num_of_samples-[num_of_samples-m+1:num_of_samples])/m]; %piecewise linear correction-2
+%c:
+    c1 = [1 + ([1:m]+1)/m - [1:m]/m^2];
+    c2 = [2*ones(1,num_of_samples-2*m-1)];
+    c3 = [1 + (num_of_samples-[num_of_samples-m:num_of_samples])/(m+1)];
+    c = [c1,c2,c3];
+H = mean(log (num_of_samples / m * diffs./c));
+
+
+  
+

File code/estimators/base_estimators/HShannon_spacing_Vplin2_initialization.m

+function [co] = HShannon_spacing_Vplin2_initialization(mult)
+%function [co] = HShannon_spacing_Vplin2_initialization(mult)
+%Initialization of the Shannon differential entropy (H) estimator of Vasicek's spacing technique with piecewise linear correction-2.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'H<name>_initialization' to ease embedding new entropy estimation methods.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+
+%Copyright (C) 2013 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields (following the template structure of the estimators to make uniform usage of the estimators possible):
+    co.name = 'Shannon_spacing_Vplin2';
+    co.mult = mult;   
+

File code/estimators/meta_estimators/IChiSquare_DChiSquare_estimation.m

+function [I] = IChiSquare_DChiSquare_estimation(Y,ds,co)
+%function [I] = IChiSquare_DChiSquare_estimation(Y,ds,co)
+%Estimates chi-square mutual information (I) based on Pearson chi-square divergence. The estimation is carried out according to the relation: I(y^1,...,y^M) = D(f_y,\prod_{m=1}^M f_{y^m}).
+%
+%Note:
+%   1)We use the naming convention 'I<name>_estimation' to ease embedding new mutual information estimation methods.
+%   2)This is a meta method: the Pearson chi-square divergence estimator can be arbitrary. 
+%
+%INPUT:
+%   Y: Y(:,t) is the t^th sample.
+%  ds: subspace dimensions. ds(m) = dimension of the m^th subspace, m=1,...,M (M=length(ds)).
+%  co: mutual information estimator object.
+
+%Copyright (C) 2013 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK. The information theoretical quantity of interest can be (and is!) estimated exactly [co.mult=1]; the computational complexity of the estimation is essentially the same as that of the 'up to multiplicative constant' case [co.mult=0].
+
+%verification:
+    if sum(ds) ~= size(Y,1);
+        error('The subspace dimensions are not compatible with Y.');
+    end
+
+[Y1,Y2] = div_sample_generation(Y,ds);
+I = D_estimation(Y1,Y2,co.member_co);

File code/estimators/meta_estimators/IChiSquare_DChiSquare_initialization.m

+function [co] = IChiSquare_DChiSquare_initialization(mult,post_init)
+%function [co] = IChiSquare_DChiSquare_initialization(mult)
+%function [co] = IChiSquare_DChiSquare_initialization(mult,post_init)
+%Initialization of the "meta" chi-square mutual information based on Pearson chi-square divergence.
+%Mutual information is estimated using the relation: I(y^1,...,y^M) = D(f_y,\prod_{m=1}^M f_{y^m}).
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'I<name>_initialization' to ease embedding new mutual information estimation methods.
+%   3)This is a meta method: the Pearson chi-square divergence estimator can be arbitrary.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%   post_init: {field_name1,field_value1,field_name2,field_value2,...}; cell array containing the names and the values of the cost object fields that are to be used
+%   (instead of their default values). For further details, see 'post_initialization.m'.
+%OUTPUT:
+%   co: cost object (structure).
+
+%Copyright (C) 2013 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields (following the template structure of the estimators to make uniform usage of the estimators possible):
+    co.name = 'ChiSquare_DChiSquare';
+    co.mult = mult;
+	
+%other fields:    
+    co.member_name = 'ChiSquare_kNN_k'; %you can change it to any Pearson chi-square divergence estimator
+
+%post initialization (put it _before_ initialization of the members in case of a meta estimator):    
+    if nargin==2 %there are given (name,value) cost object fields
+        co = post_initialization(co,post_init);
+    end  
+    
+%initialization of the member(s):
+    co.member_co = D_initialization(co.member_name,mult); 
+   

File code/estimators/quick_tests/quick_test_HShannon.m

             %cost_name = 'Shannon_spacing_Vb'; %d=1
             %cost_name = 'Shannon_spacing_Vpconst';%d=1
             %cost_name = 'Shannon_spacing_Vplin';  %d=1
+            %cost_name = 'Shannon_spacing_Vplin2';  %d=1
             %cost_name = 'Shannon_spacing_LL';     %d=1
             %cost_name = 'Shannon_KDP';            %d>=1
             %cost_name = 'Shannon_MaxEnt1'; %d=1; approximation around the normal distribution...

File code/estimators/quick_tests/quick_test_Iimreg.m

             %cost_name = 'Tsallis_DTsallis'; %h>=0
             %cost_name = 'dCov_IHSIC';       %h>=0
             %cost_name = 'ApprCorrEntr';     %h=0; computationally intensive
+            %cost_name = 'ChiSquare_DChiSquare'; %h>=0
          
 %initialization: 
     ds = (2*h+1)^2 *ones(2,1);

File code/estimators/quick_tests/quick_test_Iindependence.m

             %cost_name = 'Tsallis_DTsallis'; %dm>=1,M>=2
             %cost_name = 'dCov_IHSIC';       %dm>=1,M =2
             %cost_name = 'ApprCorrEntr';     %dm =1,M =2
+            %cost_name = 'ChiSquare_DChiSquare'; %dm>=1,M>=2
         
 %initialization:
     num_of_samples_max = num_of_samples_v(end);