Commits

Zoltan Szabo committed 210e7ee

Edgeworth expansion based Shannon entropy estimator: added; see 'HShannon_Edgeworth_initialization.m', 'HShannon_Edgeworth_estimation.m'.

Comments (0)

Files changed (6)

--Lookup table: added (see ITE_documentation.pdf: Section C).
+v0.16 (Nov 2, 2012):
+-Edgeworth expansion based Shannon entropy estimator: added; see 'HShannon_Edgeworth_initialization.m', 'HShannon_Edgeworth_estimation.m'.
+-Lookup table for the underlying H/I/D estimation formulas: added (see ITE_documentation.pdf: Section C).
 -estimate_HRenyi_constant.m: simplified; gam: deleted.
 
 v0.15 (Oct 29, 2012):
 -The Hellinger and Bhattacharyya distances are now available in ITE. They can be estimated via k-nearest neighbor methods; see 'DHellinger_kNN_k_initialization.m', 'DHellinger_kNN_k_estimation.m', 'DBhattacharyya_kNN_k_initialization.m', and 'DBhattacharyya_kNN_k_estimation.m'.
 -volume_of_the_unit_ball.m: added. 
--DL2_kNN_k_estimation.m: a '/'->'*' typo corrected.
+-DL2_kNN_k_estimation.m: a '/'->'*' typo corrected (c: volume of the unit ball).
 
 v0.14 (Oct 29, 2012):
 -Monte-Carlo simulation to compute the additive constants in Renyi entropy estimation: added; see 'estimate_HRenyi_constant.m'.
 
 v0.13 (Oct 27, 2012):
 -Tsallis entropy is now available in ITE; it can be estimated via k-nearest neighbors, see 'HTsallis_kNN_k_initialization.m', 'HTsallis_kNN_k_estimation.m'.
--A '/'->'*' typo corrected in 'HRenyi_kNN_k_estimation.m'; see 'estimate_Ialpha.m' (V).
+-A '/'->'*' typo corrected in 'HRenyi_kNN_k_estimation.m'; see 'estimate_Ialpha.m' (V: volume of the unit ball).
 
 v0.12 (Oct 27, 2012):
 -Schweizer-Wolff's sigma and kappa: added; see 'ISW1_initialization.m', 'ISW1_estimation.m', 'ISWinf_initialization.m', 'ISWinf_estimation.m'.

code/H_I_D/base_estimators/HShannon_Edgeworth_estimation.m

+function [H] = HShannon_Edgeworth_estimation(Y,co)
+%Estimates the Shannon entropy (H) of Y (Y(:,t) is the t^th sample) using Edgeworth expansion. Cost parameters are provided in the cost object co.
+%
+%We make use of the naming convention 'H<name>_estimation', to ease embedding new entropy estimation methods.
+%
+%REFERENCE: Marc Van Hulle. Edgeworth approximation of multivariate differential entropy. Neural Computation, 17(9), 1903-1910, 2005.
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%co.mult:OK.
+
+[d,num_of_samples] = size(Y);
+
+%normalize Y to have zero mean and unit std:
+    Y = whiten_E0(Y);%E=0, this step does not change the Shannon entropy of the variable
+    %std(Y(i,:))=1:
+        s = sqrt(sum(Y.^2,2)/(num_of_samples-1));%= std(Y,[],2)
+        Y = Y./repmat(s,1,num_of_samples);
+        H_whiten = log(prod(s));%we will take this scaling into account via the entropy transformation rule [ H(Wz) = H(z)+log(|det(W)|) ] at the end
+    
+H_normal = log(det(cov(Y.')))/2 + d/2 * log(2*pi) + d/2;%Shannon entropy of a normal variable with cov(Y.') covariance.
+[t1,t2,t3] = Edgeworth_t1_t2_t3(Y);
+
+H = (H_normal - (t1+t2+t3) / 12) + H_whiten;

code/H_I_D/base_estimators/HShannon_Edgeworth_initialization.m

+function [co] = HShannon_Edgeworth_initialization(mult)
+%Initialization of the Edgeworth expansion based Shannon entropy estimator. The expansion is carried out around the normal distribution.
+%
+%Note:
+%   1)The estimator is treated as a cost object (co).
+%   2)We make use of the naming convention 'H<name>_initialization', to ease embedding new entropy estimation methods.
+%
+%INPUT:
+%   mult: is a multiplicative constant relevant (needed) in the estimation; '=1' means yes, '=0' no.
+%OUTPUT:
+%   co: cost object (structure).
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+%mandatory fields:
+    co.name = 'Shannon_Edgeworth';
+    co.mult = mult;   

code/H_I_D/utilities/Edgeworth_t1_t2_t3.m

+function [t1,t2,t3] = Edgeworth_t1_t2_t3(Y)
+%Computes the three kappa_ijk := E[x_i x_j x_k] based terms (t1,t2,t3) in the Edgeworth expansion based entropy estimator, see 'HShannon_Edgeworth_estimation.m'.
+%
+%INPUT:
+%  Y: Y(:,t) is the t^th sample
+%
+%Copyright (C) 2012 Zoltan Szabo ("http://nipg.inf.elte.hu/szzoli", "szzoli (at) cs (dot) elte (dot) hu")
+%
+%This file is part of the ITE (Information Theoretical Estimators) Matlab/Octave toolbox.
+%
+%ITE is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
+%the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
+%
+%This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
+%MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+%
+%You should have received a copy of the GNU General Public License along with ITE. If not, see <http://www.gnu.org/licenses/>.
+
+d = size(Y,1);%dimension
+
+%t1:
+    t1 = 0;
+    for i = 1 : d %d terms
+        kappa_iii = mean(Y(i,:).^3); 
+        t1 = t1 + kappa_iii^2;
+    end
+    
+%t2:
+    t2 = 0;
+    for i = 1:d
+    for j = [1:i-1,i+1:d] %j\ne i; 2*nchoosek(d,2) terms
+        kappa_iij = mean(Y(i,:).^2 .* Y(j,:));
+        t2 = t2 + kappa_iij^2;
+    end
+    end
+    t2 = 3 * t2;
+    
+%t3:
+    t3 = 0;
+    for i = [1:d-2]%i<j<k; nchoosek(d,3) terms
+    for j = [i+1:d-1]
+    for k = [j+1:d]
+        kappa_ijk = mean(Y(i,:).*Y(j,:).*Y(k,:));
+        t3 = t3 + (kappa_ijk)^2;
+    end
+    end
+    end
+    t3 = t3 / 6;

code/IPA/optimization/clustering_UD0.m

 %       ii)Special cost types/schemes allow for more efficient optimization procedures. 
 %   cost_name: depending on entropy/mutual information based ISA formulation [see below A)-F)], it can take the values:
 %   i)entropy:
-%       a)base methods: 'Shannon_kNN_k', 'Renyi_kNN_k', 'Renyi_kNN_1tok', 'Renyi_kNN_S', 'Renyi_weightedkNN', 'Renyi_MST', 'Renyi_GSF'.
+%       a)base methods: 'Shannon_kNN_k', 'Renyi_kNN_k', 'Renyi_kNN_1tok', 'Renyi_kNN_S', 'Renyi_weightedkNN', 'Renyi_MST', 'Renyi_GSF', 'Tsallis_kNN_k', 'Shannon_Edgeworth'.
 %       b)meta methods: 'complex', 'ensemble', 'RPensemble'.
 %   ii)mutual information:
 %       a)base methods: 'GV', 'KCCA', 'KGV', 'HSIC', 'Hoeffding', 'SW1', 'SWinf'.
         end
     otherwise 
         disp('Error: optimization type=?');
-end
+end
Add a comment to this file

doc/ITE_documentation.pdf

Binary file modified.

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.