Adding vigra to benchmarks

[Mario Emmenlauer]

Dear all,

I'd like to add vigra (https://ukoethe.github.io/vigra/) to the benchmarks. It seems pretty straightforward, but please help me with one question: which files are auto-generated? Is blazemark main configure file auto-generated? Are the functions in blazemark/src/main/ auto-generated? Other sources?

I can pretty easily find my way around in the source, but I'm scared to waste time editing auto-generated files manually :-)

[Mario Emmenlauer]

Oh and how did you create the nice color plots for the wiki? Is there a script for that?

[Klaus Iglberger]

Hi Mario!

We gladly assist you in helping to integrate vigra to the Blazemark for your purposes. However, at this time we would not accept pull requests of this magnitude (you would have to modify a considerable portion of the Blazemark and add many files). Since the idea to add vigra to the Blazemark is still sound we will leave the issue open to be resolved at a later time.

Best regards,

Klaus!

P.S.: The plots are generated via xmgrace, which is a common Linux Tool, The output of the Blazemark can be used directly. However, the script for doing this is not part of the repository.

[Mario Emmenlauer]

Hi Klaus,

this is helpful to know. Just to be 150% certain I understand correctly: the benchmark compile scripts and sources are manually created and maintained, you are not auto-generating these files (except that the root "configure" script is generating the root "Makefile"). I am quite good in maintaining the consistent programming style, so my addition of vigra will fit natively into your code. However I'm pretty certain that I will not implement all possible arithmetic operations, it would be just too much work. So I can understand if you do not want to integrate these larger changes at this point in time. Do you foresee that this will change in the next months? I'd be less inclined to make a clean pull request if I would have to maintain it for a long time before its integrated.

Cheers and thanks, Mario

[Klaus Iglberger]

Hi Mario!

The configure script only creates some header files to activate the individual libraries (e.g. <blazemark/system/Armadillo.h>, <blazemark/system/Blitz.h>, ...) and the Makefile. The Makefile is generated such that it only takes the activated libraries into account. Also, it allows us to built each benchmark individually (e.g. make dvecdvecadd), since we usually need a few benchmarks and rarely build all benchmarks.

Unfortunately I cannot give you an accurate estimate for the the next months. But let's agree on the follow plan: You integrate the vigra operations you are interested in and compare the performance to other libraries (especially Blaze, Armadillo, Eigen, and if applicable some BLAS library). If you can convince me with some performance numbers that the performance of vigra is in general good enough to justify the effort of integrating it (especially for the operations where you don't call a BLAS library), then I will join the effort to extend the Blazemark. I hope you agree that this is a reasonable course of action.

Best regards,

Klaus!

[Mario Emmenlauer]

Dear Klaus,

Thanks a lot for this detailed explanation! I got already preliminary numbers from a few benchmarks with vigra and (if I did not overlook some required optimizations) it seems that at this point in time, vigras performance is rather below the other competitors like blaze and Eigen!

While this might sound like a disadvantage for adding vigra, I find this the biggest strength of your blazemark! As far as its feasible, I would love to gain insight into the pro's and con's of array handling libraries for C++. I found it very worthwhile to learn that vigras performance for "smaller" array sizes of up to 500² pixels compared to blaze and Eigen is rather poor. I provided this (preliminary) feedback also to vigra developers and I'm curious if something can be optimized that I might have naiively overlooked. And more importantly, it makes blaze all the more attractive to me :-)

Best regards and happy weekend,

Mario

[Mario Emmenlauer]

Its been a while that I benchmarked vigra for its linear algebra capabilities with blazemark. It turns out that with the right compiler settings, vigra can gain a bit of speed. It is still below the performance of blaze and eigen, but for certain operations and sizes the difference is rather minor. I will perform more testing and report back.

[Mario Emmenlauer]

We have added a significant set of Vigra functions to blazemark, I'm therefore closing this issue as resolved.

[Mario Emmenlauer]

PR will come.