Commits

Zoltán Szabó committed 1ef93d3

Kullback-Leibler divergence estimation based on MLE + analytical formula in the chosen exponential family, a new sampling based entropy estimator with KDE correction on the left/right sides: added 'DKL_expF_initialization.m', 'DKL_expF_estimation.m', 'HShannon_spacing_VKDE_initialization.m', 'HShannon_spacing_VKDE_estimation.m'). Quick tests: updated with the new estimators ('quick_test_HShannon.m', 'quick_test_DKL.m', 'quick_test_Dequality.m').

Comments (0)

Files changed (11)

+v0.57 (Apr 10, 2014):
+-A new sampling based entropy estimator with KDE (kernel density estimation) correction on the left/right sides: added; see 'HShannon_spacing_VKDE_initialization.m', 'HShannon_spacing_VKDE_estimation.m'. 
+-Kullback-Leibler divergence estimation based on maximum likelihood estimation + analytical formula in the chosen exponential family: added see 'DKL_expF_initialization.m' and 'DKL_expF_estimation.m'.
+-Quick tests: updated with the new estimators, see 'quick_test_HShannon.m', 'quick_test_DKL.m', 'quick_test_Dequality.m'.
+
 v0.56 (Marc 27, 2014):
 -Distribution regression (supervised entropy learning, aerosol optical depth prediction based on satellite images): added; see the new directory 'tests_distribution_regression' in the 'quick_tests' folder.
 -MMD distance computation based on U-statistics, expected kernel: upgraded to cover new kernels (exponential, Cauchy, Matern, polynomial, rational quadratic, inverse multiquadratic), see 'DMMD_Ustat_initialization.m', 'DMMD_Ustat_estimation.m', 'Kexpected_initialization.m', 'Kexpected_estimation.m'.
 
 **Download** the latest release: 
 
-- code: [zip](https://bitbucket.org/szzoli/ite/downloads/ITE-0.56_code.zip), [tar.bz2](https://bitbucket.org/szzoli/ite/downloads/ITE-0.56_code.tar.bz2), 
-- documentation: [pdf](https://bitbucket.org/szzoli/ite/downloads/ITE-0.56_documentation.pdf).
+- code: [zip](https://bitbucket.org/szzoli/ite/downloads/ITE-0.57_code.zip), [tar.bz2](https://bitbucket.org/szzoli/ite/downloads/ITE-0.57_code.tar.bz2), 
+- documentation: [pdf](https://bitbucket.org/szzoli/ite/downloads/ITE-0.57_documentation.pdf).

code/estimators/base_estimators/DKL_expF_estimation.m

+function [D] = DKL_expF_estimation(Y1,Y2,co)
+%function [D] = DKL_expF_estimation(Y1,Y2,co)
+%Estimates the Kullback-Leibler divergence (D) of Y1 and Y2 using maximum likelihood estimation (MLE) + analytical formula associated to the chosen exponential family.
+%
+%We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
+%
+%INPUT:
+%  Y1: Y1(:,t) is the t^th sample from the first distribution.
+%  Y2: Y2(:,t) is the t^th sample from the second distribution. Note: the number of samples in Y1 [=size(Y1,2)] and Y2 [=size(Y2,2)] can be different.
+%  co: divergence estimator object.
+%
+%REFERENCE: 
+%    Frank Nielsen and Sylvain Boltz. The Burbea-Rao and Bhattacharyya centroids. IEEE Transaction on Information Theory, 57:5455–5466, 2011.
+
+%Copyright (C) 2012-2014 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK. The information theoretical quantity of interest can be (and is!) estimated exactly [co.mult=1]; the computational complexity of the estimation is essentially the same as that of the 'up to multiplicative constant' case [co.mult=0]. In other words, the estimation is carried out 'exactly' (instead of up to 'proportionality').
+
+%verification:
+    if size(Y1,1) ~= size(Y2,1)
+        error('The dimension of the samples in Y1 and Y2 must be equal.');
+    end
+    
+%MLE:
+  np1 = expF_MLE(Y1,co.distr);
+  np2 = expF_MLE(Y2,co.distr);
+  
+D = expF_Bregman_distance(co.distr,np2,np1);

code/estimators/base_estimators/DKL_expF_initialization.m

+function [co] = DKL_expF_initialization(mult,post_init)
+%function [co] = DKL_expF_initialization(mult)
+%function [co] = DKL_expF_initialization(mult,post_init)
+%Initialization of the exponential family based Kullback-Leibler divergence estimator (maximum likelihood + analytical formula associated to the chosen exponential family).
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'D<name>_initialization' to ease embedding new divergence estimation methods.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes (='exact' estimation), '=0' no (=estimation up to 'proportionality').
+%   post_init: {field_name1,field_value1,field_name2,field_value2,...}; cell array containing the names and the values of the cost object fields that are to be used
+%   (instead of their default values). For further details, see 'post_initialization.m'.
+%OUTPUT:
+%   co: cost object (structure).
+
+%Copyright (C) 2012-2014 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields (following the template structure of the estimators to make uniform usage of the estimators possible):
+    co.name = 'KL_expF';
+    co.mult = mult;
+    
+%other fields:
+    co.distr = 'normal'; %exponential family used for estimation; fixed
+    
+%post initialization (put it _before_ initialization of the members in case of a meta estimator):    
+    if nargin==2 %there are given (name,value) cost object fields
+        co = post_initialization(co,post_init);
+    end    

code/estimators/base_estimators/HShannon_spacing_VKDE_estimation.m

+function [H] = HShannon_spacing_VKDE_estimation(Y,co)
+%function [H] = HShannon_spacing_VKDE_estimation(Y,co)
+%Estimates the Shannon entropy (H) of Y using the Vasicek's spacing method corrected with KDE (kernel density estimation) at the left and right sides.
+%
+%We use the naming convention 'H<name>_estimation' to ease embedding new entropy estimation methods.
+%
+%INPUT:
+%   Y: Y(:,t) is the t^th sample.
+%  co: entropy estimator object.
+%
+%REFERENCE: Havva Alizadeh Noughabi and Reza Alizadeh Noughabi. On the entropy estimators. Journal of Statistical Computatiion and Simulation, 83:784-792, 2013.
+
+%Copyright (C) 2013 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK. The information theoretical quantity of interest can be (and is!) estimated exactly [co.mult=1]; the computational complexity of the estimation is essentially the same as that of the 'up to multiplicative constant' case [co.mult=0].
+
+[d,num_of_samples] = size(Y);
+
+%verification:
+    if d~=1
+        error('The samples must be one-dimensional for this estimator.');
+    end
+
+m = floor(sqrt(num_of_samples));%m/num_of_samples->0, m,num_of_samples->infty; m<num_of_samples/2
+Y_sorted = sort(Y);
+stdY = std(Y); %sample standard deviation
+h = 1.06 * stdY * num_of_samples^(-1/5);
+
+%s:
+   sD = sqdistance(Y_sorted([1:m,end-m+1:end])/h,Y_sorted/h);%pairwise squared distances to KDE
+   s13 = mean(exp(-sD/2),2) / (sqrt(2*pi)*h); %column vector => ".'" in "H=..."
+   s2 = (2*m/num_of_samples) ./ (Y_sorted(2*m+1:end) - Y_sorted(1:end-2*m));
+   
+H = -mean(log([s13.',s2]));
+   
+
+
+  
+

code/estimators/base_estimators/HShannon_spacing_VKDE_initialization.m

+function [co] = HShannon_spacing_VKDE_initialization(mult)
+%function [co] = HShannon_spacing_VKDE_initialization(mult)
+%Initialization of the Shannon differential entropy (H) estimator based on Vasicek's spacing method corrected with KDE (kernel density estimation) at the left and right sides.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'H<name>_initialization' to ease embedding new entropy estimation methods.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+
+%Copyright (C) 2013 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields (following the template structure of the estimators to make uniform usage of the estimators possible):
+    co.name = 'Shannon_spacing_VKDE';
+    co.mult = mult;   
+            

code/estimators/quick_tests/tests_analytical_vs_estimation/quick_test_DKL.m

             cost_name = 'KL_kNN_k';       %d>=1
             %cost_name = 'KL_kNN_kiTi';   %d>=1  
             %cost_name = 'KL_PSD_SzegoT'; % d=1
+            %cost_name = 'KL_expF';       %d>=1
+
         %meta:
             %cost_name = 'KL_CCE_HShannon'; %d>=1
         

code/estimators/quick_tests/tests_analytical_vs_estimation/quick_test_HShannon.m

             %cost_name = 'Shannon_MaxEnt2';    %d=1; approximation around the normal distribution...
             %cost_name = 'Shannon_PSD_SzegoT'; %d=1
             %cost_name = 'Shannon_expF';       %d>=1; distr = 'normal'
-            
+            %cost_name = 'Shannon_spacing_VKDE'; %d=1
+
         %meta:
             %cost_name = 'Shannon_DKL_N';  %d>=1
             %cost_name = 'Shannon_DKL_U';  %d>=1          

code/estimators/quick_tests/tests_other_consistency/quick_test_Dequality.m

              %cost_name = 'MMD_Ustat_iChol'; %d>=1
              %cost_name = 'MMD_Vstat_iChol'; %d>=1
              %cost_name = 'KL_PSD_SzegoT';   % d=1
+             %cost_name = 'KL_expF';         %d>=1
+
         %meta:
              %cost_name = 'Jdistance';       %d>=1 
              %cost_name = 'KL_CCE_HShannon'; %d>=1

code/estimators/utilities/exp_family/expF_Bregman_distance.m

+function [B] = expF_Bregman_distance(distr,np1,np2)
+%function [B] = expF_Bregman_distance(distr,np1,np2)
+%Computes the Bregman distance (B) of the natural parameters np1 and np2. The Bregman distance is defined by the log-normalizer of the given exponential family (distr).
+
+%Copyright (C) 2012-2014 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+diff_nps = expF_np1_np2_subtract(np1,np2); %theta1 - theta2
+B = expF_F(distr,np1) - expF_F(distr,np2) - expF_np1_np2_mult(diff_nps,expF_gradF(distr,np2)); %F(theta1) - F(theta2) - <theta1 - theta2, nabla F(theta2)>
+
+

code/estimators/utilities/exp_family/expF_np1_np2_subtract.m

+function [np] = expF_np1_np2_subtract(np1,np2)
+%function [np] = expF_np1_np2_subtract(np1,np2)
+%Subtracts the natural parameter np2 from np1, the result is np. (np1, np2: structures with the same fields; example: np1.t1, np1.t2, np2.t1, np2.t2).
+
+%Copyright (C) 2012-2014 Zoltan Szabo ("http://www.gatsby.ucl.ac.uk/~szabo/", "zoltan (dot) szabo (at) gatsby (dot) ucl (dot) ac (dot) uk")
+%
+%This file is part of the ITE (Information Theoretical Estimators) toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+np = np1;
+F1 = fieldnames(np1);
+for n = 1 : length(F1)
+    aF = F1{n}; %field-name of F1 = field-name of F2 ('=' <== assumption) 
+    np.(aF) = np.(aF) - np2.(aF); 
+end