Provide a softmax() overload with support for row- and columnwise evaluation

[Klaus Iglberger]

Description

The Blaze library provides an implementation of the softmax() function for dense matrices. Unfortunately, this function does not provide a choice how to evaluate softmax() (total, rowwise, or columnwise). There should be an overload for softmax() that allows to specify how softmax() is evaluated via the blaze::rowwise and blaze::columnwise flags.

Examples

// Creating the matrix
//    ( 1 2 3 )
//    ( 4 1 2 )
//    ( 3 4 1 )
blaze::StaticMatrix<double,3UL,3UL> A{ { 1.0, 2.0, 3.0 }
                                     , { 4.0, 1.0, 2.0 }
                                     , { 3.0, 4.0, 1.0 } };

// Evaluating the softmax function rowwise
blaze::StaticMatrix<double,3UL,3UL> B;
B = softmax<rowwise>( A );  // Results in ( 0.0900306 0.244728  0.665241 )
                            //            ( 0.843795  0.0420101 0.114195 )
                            //            ( 0.259496  0.705385  0.035119 )

blaze::StaticMatrix<double,3UL,3UL> C;
C = softmax<columnwise>( A );  // Results in ( 0.035119 0.114195  0.665241  )
                               //            ( 0.705385 0.0420101 0.244728  )
                               //            ( 0.259496 0.843795  0.0900306 )

Tasks

• provide an overload for the softmax() function that enables row-/columnwise evaluation
• provide a full documentation of the softmax() overload
• update and improve the documentation of the existing softmax() function
• ensure compatibility with all existing dense matrix classes
• ensure compatibility with all existing dense matrix expressions
• guarantee maximum performance for the operations
• add test cases for the softmax() overload

[Klaus Iglberger]

Summary

The feature has been implemented, tested, optimized, and documented as required. It is immediately available via cloning the Blaze repository and will be officially released in Blaze 3.5.

Row-/Columnwise softmax()

The softmax function, also called the normalized exponential function, of a given dense matrix can be computed via softmax(). The resulting dense matrix consists of real values in the range (0..1], which add up to 1.

blaze::StaticMatrix<double,3UL,3UL> A{ { 1.0, 2.0, 3.0 }
                                     , { 4.0, 1.0, 2.0 }
                                     , { 3.0, 4.0, 1.0 } };
blaze::StaticMatrix<double,3UL,3UL> B;

// Evaluating the softmax function
B = softmax( A );  // Results in ( 0.0157764  0.0428847  0.116573  )
                   //            ( 0.316878   0.0157764  0.0428847 )
                   //            ( 0.116573   0.316878   0.0157764 )

double b = sum( B );  // Results in 1

Alternatively it is possible to compute a row- or column wise softmax() function. The resulting dense matrix consists of real values in the range (0..1], which add up to the number of rows or columns, respectively.

blaze::StaticMatrix<double,3UL,3UL> C, D;

// Evaluating the rowwise softmax function
C = softmax<rowwise>( A );  // Results in ( 0.0900306  0.244728   0.665241 )
                            //            ( 0.843795   0.0420101  0.114195 )
                            //            ( 0.259496   0.705385   0.035119 )

double c = sum( C );  // Results in 3 (the number of rows of A)

// Evaluating the columnwise softmax function
D = softmax<columnwise>( A );  // Results in ( 0.035119  0.114195   0.665241  )
                               //            ( 0.705385  0.0420101  0.244728  )
                               //            ( 0.259496  0.843795   0.0900306 )

double d = sum( D );  // Results in 3 (the number of columns of A)