The Blazemark is the benchmark suite of the Blaze math library. It provides benchmarks for a direct comparison of several (Smart) Expression Template based libraries for various arithmetic operations. Additionally, for some operations, it allows a comparison to "good old" C code and/or C++ operator overloading. The Blazemark It is located in the
./blazemark subdirectory of the Blaze library.
Currently the following libraries are included in the Blazemark:
- Blitz++; Minimum requirement: Version 0.10
- Boost uBLAS; Minimum requirement: Version 1.54
- GMM++; Minimum requirement: Version 4.1
- Armadillo; Minimum requirement: Version 2.4.2
- MTL4; Minimum requirement: Version 4.0
- Eigen3; Minimum requirement: Version 3.1
Configuration and Compilation
Just as the Blaze library itself, the Blazemark is fairly easy to configure and compile. Since currently there is no direct support for Windows available, the following instructions only summarize the necessary steps to configure and compile the Blazemark on Linux and MacOSX systems.
The first step is to adapt the
Configfile of the Blazemark home directory (
./blazemark). In the
Configfile the necessary parameters for all required libraries (such as Boost) as well as all parameters for the optional libraries (any BLAS library, Blitz++, GMM, Armadillo, MTL4, Eigen3) can be specified. Note that it is also necessary to specifically select a compiler along with a set of suitable compilation flags. It is of highest importance to chose a reasonable set of compilation flags in order to achieve maximum performance of the benchmarked libraries. Assuming a "Sandy Bridge" or "Ivy Bridge" CPU capable of AVX, for the GNU C++ compiler, the following set of compiler flags are recommended:
-Wall -Wextra -Werror -Wshadow -Woverloaded-virtual -ansi -O3 -mavx -DNDEBUG -fpermissive
In case of the Intel C++ compiler, the following flags are recommended:
-Werror -Wshadow -w1 -ansi -O3 -mavx -DNDEBUG -inline-level=2 -finline -fbuiltin
Any text editor can be used to adapt the
The next step is the execution of the
configure script. It uses the
Configfile to create a
Makefile and to adapt several header files required for the compilation process:
Alternatively it is also possible to specify a specific configuration file, which for instance enables different configurations:
In order to compile the complete benchmark suite, the
Makefile can be used:
Due to the large number of available benchmarks, this process will take several minutes. Alternatively, a specific benchmark can be build (for instance the sparse matrix/dense vector multiplication; for a complete list of all available benchmarks see the last section in this wiki):
The resulting executable benchmark is located in the
./blazemark/bin subdirectory. Before executing the benchmark, the size of the vectors and/or matrices involved in the benchmark can be specified via the according parameter file in the
./blazemark/params subdirectory. Again, any text editor can be used for this task:
Follow the instructions contained in the file to properly configure the parameters for the benchmark. After that, the benchmark can be started by simply calling the executable:
Command Line Parameters
Each benchmark executable offers the possibility to activate or deactivate specific kernels/libraries via the command line. Per default, all kernels/libraries for a particular benchmark are executed (which can be set in the
./blazemark/config/Config.h header file). However, via the following command line arguments the kernels can be activated very flexibly:
-clike: Activates the C-like kernel.
-classic: Activates the classic C++ kernel.
-blas: Activates the plain BLAS kernel.
-blaze: Activates the Blaze kernel.
-boost: Activates the Boost uBLAS kernel.
-blitz: Activates the Blitz++ kernel.
-gmm: Activates the GMM++ kernel.
-armadillo: Activates the Armadillo kernel.
-mtl: Activates the MTL kernel.
-eigen: Activates the Eigen kernel.
Additionally, these command line arguments can be prefixed with the
-no: Deactivates the according kernel.
-only: Deactivates all other kernels except the given one. Since the command line arguments are evaluated from left to right, all succeeding activation commands will activate the according kernels.
The following two examples illustrate the use of these command. The call
runs the sparse row-major matrix/dense column vector multiplication benchmark using all available libraries except the Boost uBLAS library, since the
-no-boost command specifically deactivates the Boost kernel. The call
./bin/smatdvecmult -only-blaze -eigen
runs the multiplication with only the Blaze and the Eigen libraries, since the
-only-blaze command deactivates all kernels except Blaze and the succeeding
-eigen command re-activates the Eigen kernel.
Note that not all libraries support every operation. If no kernel exists for a certain library for a particular benchmark the command line argument is without effect.
Interpreting the Output
The following example show the output of the dense vector/dense vector addition benchmark, which was run for vectors of size 100 and 10000000:
Dense Vector/Dense Vector Addition: C-like implementation [MFlop/s]: 100 1115.44 10000000 206.317 Classic operator overloading [MFlop/s]: 100 415.703 10000000 112.557 Blaze [MFlop/s]: 100 2602.56 10000000 292.569 Boost uBLAS [MFlop/s]: 100 1056.75 10000000 208.639 Blitz++ [MFlop/s]: 100 1011.1 10000000 207.855 GMM++ [MFlop/s]: 100 1115.42 10000000 207.699 Armadillo [MFlop/s]: 100 1095.86 10000000 208.658 MTL [MFlop/s]: 100 1018.47 10000000 209.065 Eigen [MFlop/s]: 100 2173.48 10000000 209.899 N=100, steps=55116257 C-like = 2.33322 (4.94123) Classic = 6.26062 (13.2586) Blaze = 1 (2.11777) Boost uBLAS = 2.4628 (5.21565) Blitz++ = 2.57398 (5.4511) GMM++ = 2.33325 (4.94129) Armadillo = 2.3749 (5.0295) MTL = 2.55537 (5.41168) Eigen = 1.19742 (2.53585) N=10000000, steps=8 C-like = 1.41805 (0.387753) Classic = 2.5993 (0.710753) Blaze = 1 (0.27344) Boost uBLAS = 1.40227 (0.383437) Blitz++ = 1.40756 (0.384884) GMM++ = 1.40862 (0.385172) Armadillo = 1.40215 (0.383403) MTL = 1.39941 (0.382656) Eigen = 1.39386 (0.381136)
The first section presents the individual results of each library in MFlop/s. The competitors in this benchmarks are a plain C-like implementation, classical C++ operator overloading, the Blaze library, the Boost uBLAS library, Blitz++, GMM++, Armadillo, MTL, and Eigen.
The second section compares the libraries for each specified vector size. The first line shows the size of the vectors and the number of iteration steps used for the benchmark. After that the libraries are listed. The first number represents a factor of how much slower the library was than the fastest competitor; a number of 1 therefore represents the fastest competitor. The number in brackets is the runtime spend in the benchmark.
Due to the high number of possible operations and thus also benchmarks, the names of the benchmarks and executables are assembled from abbreviations for the involved left and right operands:
- dvec: Dense column vector
- tdvec: Dense row vector
- vec3: Dense 3D column vector
- tvec3: Dense 3D row vector
- vec6: Dense 6D column vector
- tvec6: Dense 6D row vector
- svec: Sparse column vector
- tsvec: Sparse row vector
- dmat: Dense row-major matrix
- tdmat: Dense column-major matrix
- mat3: Dense 3x3 row-major matrix
- tmat3: Dense 3x3 column-major matrix
- mat6: Dense 6x6 row-major matrix
- tmat6: Dense 6x6 column-major matrix
- smat: Sparse row-major matrix
- tsmat: Sparse column-major matrix
For instance, the
tdvecsmatmult benchmark represents the multiplication between a dense row vector with a row-major sparse matrix.
The following is a complete list of all available benchmarks, listed in alphabetical order: