NaNs in softmax; patch available

[Daniel Baker]

Hi,

‌

The softmax implementation in blaze-lib is the naive approach of straightforward computation. Unfortunately, it is numerically unstable and results in NANs on normal data.

I have patched this here (https://bitbucket.org/dnbaker/blaze-softmax-patch/src/master/ , with the commit at https://bitbucket.org/dnbaker/blaze-softmax-patch/commits/1a1ffd9fb15d1aeff366d08985b8e6edcf05c208) in the above, but I have not been able to create a pull request due to arcane difficulties with Bitbucket and pull requests. I’ve been able to switch to this patched version for my own code, but it would be preferable to build off of the main repository.

‌

Thanks,

‌

Daniel

[Klaus Iglberger]

Hi Daniel!

Thanks a lot for providing an improved implementation for the softmax() operation. This is highly appreciated. We’ll integrate the patch as quickly as possible, which unfortunately will take a couple of days. Thanks again,

Best regards,

Klaus!

[Klaus Iglberger]

Hi Daniel!

Could you provide us with more information about the implementation you chose (e.g. a link to a paper, a detailed numerical analysis, etc.)? Thanks,

Best regards,

Klaus!

[Daniel Baker]

Sure.

I’ve mostly found the trick documented in blog posts (https://lingpipe-blog.com/2009/03/17/softmax-without-overflow/, https://timvieira.github.io/blog/post/2014/02/11/exp-normalize-trick/), but it’s also equation 1.4 in https://arxiv.org/abs/1909.03469.

‌

There’s the alternative formulation in 1.5, but this seemed simple to do efficiently with existing blaze functionality.

Something related I haven’t investigated it in detail: https://arxiv.org/abs/2001.04438 proposes a “2-pass” algorithm that’s particularly numerically stable.

[Klaus Iglberger]

Commit 0445e05 improves the numerical stability of the softmax() functions for dense vectors and dense matrices. The fix is immediately available via cloning the Blaze repository and will be officially released in Blaze 3.9.