Provide support for the softmax function

[Klaus Iglberger]

Description

The Blaze library should provide a softmax function for both dense vectors and dense matrices. More specifically, it should be possible to ...

• ... evaluate softmax() on dense vectors;
• ... evaluate softmax() on dense matrices.

Conceptually, the softmax() should work as follows:

Vector Example

// Creating the vector ( 1 2 3 4 1 2 3 )
blaze::StaticVector<double,7UL> x{ 1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0 };
blaze::StaticVector<double,7UL> y;

// Evaluating the softmax function
y = softmax( x );  // Results in ( 0.024 0.064 0.175 0.475 0.024 0.064 0.175 )

Matrix Example

// Creating the matrix
//    ( 1 2 3 )
//    ( 4 1 2 )
//    ( 3 4 1 )
blaze::StaticMatrix<double,3UL,3UL> A{ { 1.0, 2.0, 3.0 }
                                     , { 4.0, 1.0, 2.0 }
                                     , { 3.0, 4.0, 1.0 } };
blaze::StaticMatrix<double,3UL,3UL> B;

// Evaluating the softmax function
B = softmax( A );  // Results in ( 0.0157764 0.0428847 0.116573  )
                   //            ( 0.316878  0.0157764 0.0428847 )
                   //            ( 0.116573  0.316878  0.0157764 )

Tasks

• implement the softmax function for dense vectors
• implement the softmax function for dense matrices
• provide a full documentation of the softmax function
• ensure compatibility with all existing dense vector and matrix classes
• ensure compatibility with all existing dense vector and matrix expressions
• guarantee maximum performance for the operations
• add test cases for the entire softmax functionality

[Klaus Iglberger]

The following two functions implement the softmax() function for both vectors and matrices:

template< typename VT, bool TF >
VT softmax( const blaze::Vector<VT,TF>& v )
{
   VT tmp( exp( ~v ) );
   const auto scalar( sum( ~tmp ) );
   tmp /= scalar;
   return tmp;
}

template< typename MT, bool SO >
MT softmax( const blaze::Matrix<MT,SO>& m )
{
   MT tmp( exp( ~m ) );
   const auto scalar( sum( ~tmp ) );
   tmp /= scalar;
   return tmp;
}

Please note that both functions make use of the sum() function, which is available on master and will be officially released in Blaze 3.4.

Please feel free to use these functions until they have been integrated into Blaze and are officially supported.

[Klaus Iglberger]

Summary

The feature has been implemented, tested, optimized, and documented as required. It is immediately available via cloning the Blaze repository and will be officially released in Blaze 3.4.

softmax()

The softmax function, also called the normalized exponential function, of a given dense vector or matrix can be computed via softmax(). The resulting dense vector or matrix consists of real values in the range (0..1], which add up to 1.

Vector Example

blaze::StaticVector<double,7UL,rowVector> x{ 1.0, 2.0, 3.0, 4.0, 1.0, 2.0, 3.0 };
blaze::StaticVector<double,7UL,rowVector> y;

// Evaluating the softmax function
y = softmax( x );     // Results in ( 0.024 0.064 0.175 0.475 0.024 0.064 0.175 )
double s = sum( y );  // Results in 1

Matrix Example

blaze::StaticMatrix<double,3UL,3UL> A{ { 1.0, 2.0, 3.0 }
                                     , { 4.0, 1.0, 2.0 }
                                     , { 3.0, 4.0, 1.0 } };
blaze::StaticMatrix<double,3UL,3UL> B;

// Evaluating the softmax function
B = softmax( A );     // Results in ( 0.0157764 0.0428847 0.116573  )
                      //            ( 0.316878  0.0157764 0.0428847 )
                      //            ( 0.116573  0.316878  0.0157764 )
double s = sum( B );  // Results in 1