Commits

Zoltan Szabo committed 63da916

Symmetric Bregman distance estimation based on (i) nonsymmetric Bregman distance, (ii) k-nearest neighbors: added; see 'DsymBregman_DBregman_initialization.m', 'DsymBregman_DBregman_estimation.m', 'DsymBregman_kNN_k_initialization.m', 'DsymBregman_kNN_k_estimation.m'.

  • Participants
  • Parent commits 7873a1f
  • Tags release-0.39

Comments (0)

Files changed (7)

File CHANGELOG.txt

+v0.39 (June 12, 2013):
+-Symmetric Bregman distance estimation based on nonsymmetric Bregman distance: added; see 'DsymBregman_DBregman_initialization.m', 'DsymBregman_DBregman_estimation.m'.
+-Symmetric Bregman distance estimation based on k-nearest neighbors: added; see 'DsymBregman_kNN_k_initialization.m', 'DsymBregman_kNN_k_estimation.m'. 
+
 v0.38 (June 1, 2013):
 -Jensen-Tsallis divergence estimation: added; see 'DJensenTsallis_HTsallis_initialization.m' and 'DJensenTsallis_HTsallis_estimation.m'.
 -Bregman distance estimation: added; see 'DBregman_kNN_k_initialization.m' and 'DBregman_kNN_k_estimation.m'.
 
 - `entropy (H)`: Shannon entropy, Rényi entropy, Tsallis entropy (Havrda and Charvát entropy), complex entropy,
 - `mutual information (I)`: generalized variance, kernel canonical correlation analysis, kernel generalized variance, Hilbert-Schmidt independence criterion, Shannon mutual information, L2 mutual information, Rényi mutual information, Tsallis mutual information, copula-based kernel dependency, multivariate version of Hoeffding's Phi, Schweizer-Wolff's sigma and kappa, complex mutual information, Cauchy-Schwartz quadratic mutual information, Euclidean distance based quadratic mutual information, distance covariance, distance correlation, approximate correntropy independence measure,
-- `divergence (D)`: Kullback-Leibler divergence (relative entropy, I directed divergence), L2 divergence, Rényi divergence, Tsallis divergence, Hellinger distance, Bhattacharyya distance, maximum mean discrepancy (kernel distance), J-distance (symmetrised Kullback-Leibler divergence, J divergence), Cauchy-Schwartz divergence, Euclidean distance based divergence, energy distance (specially the Cramer-Von Mises distance), Jensen-Shannon divergence, Jensen-Rényi divergence, K divergence, L divergence, certain f-divergences (Csiszár-Morimoto divergence, Ali-Silvey distance), non-symmetric Bregman distance (Bregman divergence), Jensen-Tsallis divergence,
+- `divergence (D)`: Kullback-Leibler divergence (relative entropy, I directed divergence), L2 divergence, Rényi divergence, Tsallis divergence, Hellinger distance, Bhattacharyya distance, maximum mean discrepancy (kernel distance), J-distance (symmetrised Kullback-Leibler divergence, J divergence), Cauchy-Schwartz divergence, Euclidean distance based divergence, energy distance (specially the Cramer-Von Mises distance), Jensen-Shannon divergence, Jensen-Rényi divergence, K divergence, L divergence, certain f-divergences (Csiszár-Morimoto divergence, Ali-Silvey distance), non-symmetric Bregman distance (Bregman divergence), Jensen-Tsallis divergence, symmetric Bregman distance,
 - `association measures (A)`, including `measures of concordance`: multivariate extensions of Spearman's rho (Spearman's rank correlation coefficient, grade correlation coefficient), correntropy, centered correntropy, correntropy coefficient, correntropy induced metric, centered correntropy induced metric, multivariate extension of Blomqvist's beta (medial correlation coefficient), multivariate conditional version of Spearman's rho, lower/upper tail dependence via conditional Spearman's rho,
 - `cross quantities (C)`: cross-entropy.
 

File code/H_I_D_A_C/base_estimators/DsymBregman_kNN_k_estimation.m

+function [D] = DsymBregman_kNN_k_estimation(Y1,Y2,co)
+%Estimates the symmetric Bregman distance (D) of Y1 and Y2 using the kNN method (S={k}). 
+%
+%We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
+%
+%INPUT:
+%  Y1: Y1(:,t) is the t^th sample from the first distribution.
+%  Y2: Y2(:,t) is the t^th sample from the second distribution. Note: the number of samples in Y1 [=size(Y1,2)] and Y2 [=size(Y2,2)] can be different.
+%  co: divergence estimator object.
+%
+%REFERENCE: 
+%   Nikolai Leonenko, Luc Pronzato, and Vippal Savani. A class of Renyi information estimators for multidimensional densities. Annals of Statistics, 36(5):2153-2182, 2008.
+%   Imre Csiszar. Generalized projections for non-negative functions. Acta Mathematica Hungarica, 68:161-185, 1995.
+%   Lev M. Bregman. The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7:200-217, 1967.
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK.
+
+[dY1,num_of_samplesY1] = size(Y1);
+[dY2,num_of_samplesY2] = size(Y2);
+
+%verification:
+    if dY1~=dY2
+        error('The dimension of the samples in Y1 and Y2 must be equal.');
+    end
+
+I_alpha_Y1 = estimate_Ialpha(Y1,co);
+I_alpha_Y2 = estimate_Ialpha(Y2,co);
+D_temp3_Y1Y2 = estimate_Dtemp3(Y1,Y2,co);
+D_temp3_Y2Y1 = estimate_Dtemp3(Y2,Y1,co);
+    
+D = (I_alpha_Y1 + I_alpha_Y2 - D_temp3_Y1Y2 - D_temp3_Y2Y1) / (co.alpha-1);
+

File code/H_I_D_A_C/base_estimators/DsymBregman_kNN_k_initialization.m

+function [co] = DsymBregman_kNN_k_initialization(mult)
+%Initialization of the kNN (k-nearest neighbor, S={k}) based symmetric Bregman distance estimator.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'D<name>_initialization' to ease embedding new divergence estimation methods.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields:
+    co.name = 'symBregman_kNN_k';
+    co.mult = mult;
+    
+%other fields:
+    %Possibilities for 'co.kNNmethod' (see 'kNN_squared_distances.m'): 
+        %I: 'knnFP1': fast pairwise distance computation and C++ partial sort; parameter: co.k.                
+        %II: 'knnFP2': fast pairwise distance computation; parameter: co.k. 						
+        %III: 'knnsearch' (Matlab Statistics Toolbox): parameters: co.k, co.NSmethod ('kdtree' or 'exhaustive').
+        %IV: 'ANN' (approximate nearest neighbor); parameters: co.k, co.epsi. 
+		%I:
+            co.kNNmethod = 'knnFP1';
+            co.k = 3;%k-nearest neighbors				
+		%II:
+            %co.kNNmethod = 'knnFP2';
+            %co.k = 3;%k-nearest neighbors				
+        %III:
+            %co.kNNmethod = 'knnsearch';
+            %co.k = 3;%k-nearest neighbors
+            %co.NSmethod = 'kdtree';
+        %IV:
+            %co.kNNmethod = 'ANN';
+            %co.k = 3;%k-nearest neighbors
+            %co.epsi = 0; %=0: exact kNN; >0: approximate kNN, the true (not squared) distances can not exceed the real distance more than a factor of (1+epsi).
+    co.alpha = 0.7; %assumption: not equal to 1
+    
+%initialize the ann wrapper in Octave, if needed:
+    initialize_Octave_ann_wrapper_if_needed(co.kNNmethod);

File code/H_I_D_A_C/meta_estimators/DsymBregman_DBregman_estimation.m

+function [D_S] = DsymBregman_DBregman_estimation(Y1,Y2,co)
+%Estimates the symmetric Bregman distance (D) of Y1 and Y2 using the relation: D_S = (D_NS(f1,f2) + D_NS (f2,f1)) / alpha, where D_NS is the nonsymmetric Bregman distance. 
+%
+%Note:
+%   1)We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
+%   2)This is a meta method: the (nonsymmetric) Bregman distance estimator can arbitrary.
+%
+%INPUT:
+%  Y1: Y1(:,t) is the t^th sample from the first distribution (f1).
+%  Y2: Y2(:,t) is the t^th sample from the second distribution (f2). Note: the number of samples in Y1 [=size(Y1,2)] and Y2 [=size(Y2,2)] can be different.
+%  co: divergence estimator object.
+%
+%REFERENCE: 
+%   Nikolai Leonenko, Luc Pronzato, and Vippal Savani. A class of Renyi information estimators for multidimensional densities. Annals of Statistics, 36(5):2153-2182, 2008.
+%   Imre Csiszar. Generalized projections for non-negative functions. Acta Mathematica Hungarica, 68:161-185, 1995.
+%   Lev M. Bregman. The relaxation method of finding the common points of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7:200-217, 1967.
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK.
+
+%verification:
+    if size(Y1,1)~=size(Y2,1)
+        error('The dimension of the samples in Y1 and Y2 must be equal.');
+    end
+
+D_S = (D_estimation(Y1,Y2,co.member_co) + D_estimation(Y2,Y1,co.member_co)) / co.alpha;
+

File code/H_I_D_A_C/meta_estimators/DsymBregman_DBregman_initialization.m

+function [co] = DsymBregman_DBregman_initialization(mult)
+%Initialization of the symmetric Bregman distance estimator using the relation: D_S = (D_NS(f1,f2) + D_NS (f2,f1)) / alpha, where D_NS is the nonsymmetric Bregman distance.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'D<name>_initialization' to ease embedding new divergence estimation methods.
+%   3)This is a meta method: the (nonsymmetric) Bregman distance estimator can arbitrary.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields:
+    co.name = 'symBregman_DBregman';
+    co.mult = mult;
+    
+%other fields:
+    co.alpha = 0.7; %assumption: not equal to 1
+    
+    co.member_name = 'Bregman_kNN_k'; %you can change it to any (nonsymmetric) Bregman distance estimator
+    co.member_co = D_initialization(co.member_name,mult);
+    
+    co.member_co.alpha = co.alpha; %automatism for setting the parameters (co.alpha) of member_co (co_2) in a <=2 deep meta construction (co_1 -> co_2); otherwise, please set the parameters (co.alpha) in the member_co-s.
+    

File doc/ITE_documentation.txt

-From release 0.20, "https://bitbucket.org/szzoli/ite/downloads": Downloads:
-1)'ITE-<release>_documentation.pdf': documentation of a given release of the toolbox.
-2)'ITE-latest_documentation.pdf': temporary documentation, compatible with the actual source. 
+From v0.20, the documentation of a given release is available at 'https://bitbucket.org/szzoli/ite/downloads': Downloads tab: 'ITE-<release>_documentation.pdf'.
 
-Note: 2) exists only if 1) does not cover the latest source.
 
 
+