NaNs in softmax; patch available

Issue #371 resolved
Daniel Baker created an issue


The softmax implementation in blaze-lib is the naive approach of straightforward computation. Unfortunately, it is numerically unstable and results in NANs on normal data.

I have patched this here ( , with the commit at in the above, but I have not been able to create a pull request due to arcane difficulties with Bitbucket and pull requests. I’ve been able to switch to this patched version for my own code, but it would be preferable to build off of the main repository.



Comments (8)

  1. Klaus Iglberger

    Hi Daniel!

    Thanks a lot for providing an improved implementation for the softmax() operation. This is highly appreciated. We’ll integrate the patch as quickly as possible, which unfortunately will take a couple of days. Thanks again,

    Best regards,


  2. Klaus Iglberger

    Hi Daniel!

    Could you provide us with more information about the implementation you chose (e.g. a link to a paper, a detailed numerical analysis, etc.)? Thanks,

    Best regards,


  3. Daniel Baker reporter


    I’ve mostly found the trick documented in blog posts (,, but it’s also equation 1.4 in

    There’s the alternative formulation in 1.5, but this seemed simple to do efficiently with existing blaze functionality.

    Something related I haven’t investigated it in detail: proposes a “2-pass” algorithm that’s particularly numerically stable.

  4. Klaus Iglberger

    Commit 0445e05 improves the numerical stability of the softmax() functions for dense vectors and dense matrices. The fix is immediately available via cloning the Blaze repository and will be officially released in Blaze 3.9.

  5. Log in to comment