limitation of dimension

Issue #10 resolved
shidehrzf
created an issue

I tried the quick test of MI. It seems to me when I try with a slightly higher dimension data, the error between analytical and estimation is very high. Is there any restriction about the dimension?

Thanks in advance.

Comments (3)

  1. Zoltán Szabó repo owner

    Hi,

    Thanks for your message.

    The primary goal of the quick tests is to enable fast checking of the estimators against some analytical quantities and to accelerate the development of new techniques. Section 6 of the documentation enlists the included analytical expressions. If the distribution and the information theoretical quantity in your application belongs to this list, then the task is 'easy', these expressions are directly applicable. The interesting situation arises when this is not the case.

    The requirements of the estimators (number of components, dimensions, ...) are detailed in Table 3 (page 22) and Table 9 (page 37) of the documentation in case of mutual information-type quantities.

    For the convergence rates (such as dependence on the dimenion, smoothness, ...), please see the included references in the documentation and the code, as a starting point. Estimating these quantities is by any means a non-trivial, challenging task; this is one of the reasons why ITE exists.

    From application point of view:

    • In many cases one can/might wish to reduce the problem to the estimation of entropy/mutual information(MI)/divergence... of smaller dimensional quantities; examples include feature selection or blind source separation. In ITE the ICA/ISA direction is elaborated.

    • Convergence rates might not shed light on the whole picture because typically one does not really care about estimating MI etc. itself, but using it in some context, for example in an objective function. In other words, how precise estimation of 'MI' is needed depends heavily on the application. 'Propagating' the estimation errors can require a bit of computation but can be doable, see for example the 'transfer' of MMD estimation error to kernel ridge regression; link.

    • Wiki (about successful ITE applications) can provide further insights/inspiration.

    Best,

    Zoltan

  2. Log in to comment