Commits

Zoltan Szabo committed d708ee0

Jensen-Renyi divergence, Jensen-Shannon divergence estimation: added; see 'DJensenRenyi_HRenyi_initialization.m', 'DJensenRenyi_HRenyi_estimation.m', 'DJensenShannon_HShannon_initialization.m' and 'DJensenShannon_HShannon_estimation.m'.

Comments (0)

Files changed (7)

+v0.36 (Apr 26, 2013):
+-Jensen-Renyi divergence estimation: added; see 'DJensenRenyi_HRenyi_initialization.m' and 'DJensenRenyi_HRenyi_estimation.m'.
+-Jensen-Shannon divergence estimation: added; see 'DJensenShannon_HShannon_initialization.m' and 'DJensenShannon_HShannon_estimation.m'.
+
 v0.35 (Apr 1, 2013):
 -An alternative Jacobi optimization based ICA solution with general entropy/mutual information estimators: added; see 'estimate_ICA.m', 'estimate_ICA_Jacobi2.m'. The method is using a different fining scheme in the rotation angle search compared to 'estimate_ICA_Jacobi1.m'.
 
 
 - `entropy (H)`: Shannon entropy, Rényi entropy, Tsallis entropy (Havrda and Charvát entropy), complex entropy,
 - `mutual information (I)`: generalized variance, kernel canonical correlation analysis, kernel generalized variance, Hilbert-Schmidt independence criterion, Shannon mutual information, L2 mutual information, Rényi mutual information, Tsallis mutual information, copula-based kernel dependency, multivariate version of Hoeffding's Phi, Schweizer-Wolff's sigma and kappa, complex mutual information, Cauchy-Schwartz quadratic mutual information, Euclidean distance based quadratic mutual information, distance covariance, distance correlation, approximate correntropy independence measure,
-- `divergence (D)`: Kullback-Leibler divergence (relative entropy), L2 divergence, Rényi divergence, Tsallis divergence, Hellinger distance, Bhattacharyya distance, maximum mean discrepancy (kernel distance, an integral probability metric), J-distance (symmetrised Kullback-Leibler divergence), Cauchy-Schwartz divergence, Euclidean distance based divergence, energy distance (specially the Cramer-Von Mises distance),
+- `divergence (D)`: Kullback-Leibler divergence (relative entropy; I directed divergence), L2 divergence, Rényi divergence, Tsallis divergence, Hellinger distance, Bhattacharyya distance, maximum mean discrepancy (kernel distance, an integral probability metric), J-distance (symmetrised Kullback-Leibler divergence), Cauchy-Schwartz divergence, Euclidean distance based divergence, energy distance (specially the Cramer-Von Mises distance), Jensen-Shannon divergence, Jensen-Rényi divergence,
 - `association measures (A)`, including `measures of concordance`: multivariate extensions of Spearman's rho (Spearman's rank correlation coefficient, grade correlation coefficient), correntropy, centered correntropy, correntropy coefficient, correntropy induced metric, centered correntropy induced metric, multivariate extension of Blomqvist's beta (medial correlation coefficient), multivariate conditional version of Spearman's rho, lower/upper tail dependence via conditional Spearman's rho,
 - `cross quantities (C)`: cross-entropy.
 
 
 **Download** the latest release: 
 
-- code: [zip](https://bitbucket.org/szzoli/ite/downloads/ITE-0.35_code.zip), [tar.bz2](https://bitbucket.org/szzoli/ite/downloads/ITE-0.35_code.tar.bz2), 
-- [documentation (pdf)](https://bitbucket.org/szzoli/ite/downloads/ITE-0.35_documentation.pdf).
+- code: [zip](https://bitbucket.org/szzoli/ite/downloads/ITE-0.36_code.zip), [tar.bz2](https://bitbucket.org/szzoli/ite/downloads/ITE-0.36_code.tar.bz2), 
+- [documentation (pdf)](https://bitbucket.org/szzoli/ite/downloads/ITE-0.36_documentation.pdf).
 
 

code/H_I_D_A_C/meta_estimators/DJensenRenyi_HRenyi_estimation.m

+function [D_JR] = DJensenRenyi_HRenyi_estimation(Y1,Y2,co)
+%Estimates the Jensen-Renyi divergence of Y1 and Y2 using the relation: 
+%D_JR(f_1,f_2) = H_{R,alpha}(w1*y^1+w2*y^2) - [w1*H_{R,alpha}(y^1) + w2*H_{R,alpha}(y^2)], where y^i has density f_i (i=1,2), w1*y^1+w2*y^2 is the mixture distribution of y^1 and y^2 with w1,w2 positive weights, and H_{R,alpha} denotes the Renyi entropy.
+%
+%Note:
+%   1)We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
+%   2)This is a meta method: the Renyi entropy estimator can be arbitrary.
+%
+%INPUT:
+%  Y1: Y1(:,t) is the t^th sample from the first distribution.
+%  Y2: Y2(:,t) is the t^th sample from the second distribution.
+%  co: divergence estimator object.
+%
+%REFERENCE:
+%  A.B. Hamza and H. Krim. Jensen-Renyi divergence measure: theoretical and computational perspectives. In IEEE International Symposium on Information Theory (ISIT), page 257, 2003.
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK.
+
+w = co.w;
+mixtureY = mixture_distribution(Y1,Y2,w);
+D_JR =  H_estimation(mixtureY,co.member_co) - (w(1)*H_estimation(Y1,co.member_co) + w(2)*H_estimation(Y2,co.member_co));

code/H_I_D_A_C/meta_estimators/DJensenRenyi_HRenyi_initialization.m

+function [co] = DJensenRenyi_HRenyi_initialization(mult)
+%Initialization of the Jensen-Renyi divergence estimator, defined according to the relation:
+%D_JR(f_1,f_2) = H_{R,alpha}(w1*y^1+w2*y^2) - [w1*H_{R,alpha}(y^1) + w2*H_{R,alpha}(y^2)], where y^i has density f_i (i=1,2), w1*y^1+w2*y^2 is the mixture distribution of y^1 and y^2 with w1,w2 positive weights, and H_{R,alpha} denotes the Renyi entropy.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'D<name>_initialization' to ease embedding new divergence estimation methods.
+%   3)This is a meta method: the Renyi entropy estimator can arbitrary.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields:
+    co.name = 'JensenRenyi_HRenyi';
+    co.mult = mult;
+    
+%other fields:
+    co.alpha = 0.95; 
+    
+    co.w = [1/2,1/2]; %assumption: co.w(i)>0, sum(co.w)=1
+    co.member_name = 'Renyi_kNN_k'; %you can change it to any Renyi entropy estimator
+    co.member_co = H_initialization(co.member_name,mult);
+    
+    co.member_co.alpha = co.alpha; %automatism for setting the parameters (co.alpha) of member_co (co_2) in a <=2 deep meta construction (co_1 -> co_2); otherwise, please set the parameters (co.alpha) in the member_co-s.

code/H_I_D_A_C/meta_estimators/DJensenShannon_HShannon_estimation.m

+function [D_JS] = DJensenShannon_HShannon_estimation(Y1,Y2,co)
+%Estimates the Jensen-Shannon divergence of Y1 and Y2 using the relation: 
+%D_JS(f_1,f_2) = H(w1*y^1+w2*y^2) - [w1*H(y^1) + w2*H(y^2)], where y^i has density f_i (i=1,2), w1*y^1+w2*y^2 is the mixture distribution of y^1 and y^2 with w1,w2 positive weights, and H denotes the Shannon entropy.
+%
+%Note:
+%   1)We use the naming convention 'D<name>_estimation' to ease embedding new divergence estimation methods.
+%   2)This is a meta method: the Shannon entropy estimator can be arbitrary.
+%
+%INPUT:
+%  Y1: Y1(:,t) is the t^th sample from the first distribution.
+%  Y2: Y2(:,t) is the t^th sample from the second distribution.
+%  co: divergence estimator object.
+%
+%REFERENCE:
+%  Jianhua Lin. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37:145-151, 1991.
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK.
+
+w = co.w;
+mixtureY = mixture_distribution(Y1,Y2,w);
+D_JS =  H_estimation(mixtureY,co.member_co) - (w(1)*H_estimation(Y1,co.member_co) + w(2)*H_estimation(Y2,co.member_co));

code/H_I_D_A_C/meta_estimators/DJensenShannon_HShannon_initialization.m

+function [co] = DJensenShannon_HShannon_initialization(mult)
+%Initialization of the Jensen-Shannon divergence estimator, defined according to the relation:
+%D_JS(f_1,f_2) = H(w1*y^1+w2*y^2) - [w1*H(y^1) + w2*H(y^2)], where y^i has density f_i (i=1,2), w1*y^1+w2*y^2 is the mixture distribution of y^1 and y^2 with w1,w2 positive weights, and H denotes the Shannon entropy.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We use the naming convention 'D<name>_initialization' to ease embedding new divergence estimation methods.
+%   3)This is a meta method: the Shannon entropy estimator can arbitrary.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields:
+    co.name = 'JensenShannon_HShannon';
+    co.mult = mult;
+    
+%other fields:
+    co.w = [1/2,1/2]; %assumption: co.w(i)>0, sum(co.w)=1
+    co.member_name = 'Shannon_kNN_k'; %you can change it to any Shannon entropy estimator
+    co.member_co = H_initialization(co.member_name,mult);

code/H_I_D_A_C/utilities/mixture_distribution.m

+function [mixture] = mixture_distribution(varargin)
+%function [mixture] = mixture_distribution(X1,...,XM,w)
+%Sampling from the mixture distribution, where samples from the individual distributions and the mixing weights are given.
+%
+%INPUT:
+%   Xm: samples from the m^{th} distribution; Xm(:,t) is the t^th sample (m=1,...,M).
+%   w: mixing weights. Assumption: length(w)=M, w>0, sum(w)=1. 
+%OUTPUT:
+%   mixture: samples from the mixture distribution. mixture(:,i) ~ w(1) * p_x1 + ...+ w(M) * p_xM, where p_xm denotes the distribution of Xm (m=1,...,M).
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%varargin -> num_of_comps, Xs, ds, Ts, ws:
+    num_of_comps = nargin-1;
+    Xs = cell(num_of_comps,1); %{X1,...,XM}
+    ds = zeros(num_of_comps,1); %[d1,...,dM], dimensions
+    Ts = zeros(num_of_comps,1); %[T1,...,TM], number of samples
+    for k = 1 : num_of_comps
+        Xs{k} = varargin{k};
+        ds(k) = size(Xs{k},1);
+        Ts(k) = size(Xs{k},2);
+    end
+    ws = varargin{nargin}(:);%column vector
+
+%verification:
+    if ~all(ds(1)==ds) %needed: d1=...=dM
+        error('The distributions must be of the same dimension.');
+    end
+    if ~all(ws>0)
+         error('The mixture coefficients must be positive.');
+    end
+    if sum(ws)~=1
+        error('The mixture coefficients must sum to one.');
+    end
+	if length(ws)~=num_of_comps
+		error('The number of the weights and the number of Xm-s must be equal.');
+    end
+    
+%take the maximal number of samples (T) for which 'T*w1<=T1, ..., T*wM<=TM', then Tm:=floor(T*wm), i.e. compute the trimmed number of samples:
+    T = min(Ts./ws);
+    Tws = floor(T.*ws);
+    
+%mix Xm-s:    
+    num_of_samples = sum(Tws);
+    cum_Tws = cumsum([1;Tws(1:end-1)]);%1,T_1+1,T_1+T_2+1,... = starting indices of the Xm-s
+    mixture = zeros(ds(1),num_of_samples);
+    for k = 1 : num_of_comps
+        T = Tws(k);
+        mixture(:,cum_Tws(k):cum_Tws(k)+T-1) = Xs{k}(:,1:T); %trim the 'irrelevant' part, the result is added to the mixture
+    end
+    mixture = mixture(:,randperm(num_of_samples)); %permute the samples to obtain the mixture (the weights have been taken into account in the trimming part)
+