support 64 bit numbers for SystemStatistic's malloc'ed memory size

[Former user]

The mallinfo(3) call used by SystemStatistics is limited to 32bit signed numbers and thus cannot record if more than 2GB of memory are used.

There is no direct 64bit equivalent, however there is a malloc_info function (http://man7.org/linux/man-pages/man3/malloc_info.3.html) which can be used, in combination with open_memstream(3) (http://man7.org/linux/man-pages/man3/open_memstream.3.html) to obtain an in-memory XML string containing the required information.

Looking at https://git.amper.me/open-source/glibc/blob/4d653a59ffeae0f46f76a40230e2cfa9587b7e7e/malloc/malloc.c#L5148 and how that same function computes uordblks the proper way to get a 64bit integer would seem to be to compute total_aspace - total_avail.

"Parsing" the XML string can be done using strstr(xml, "<aspace type=\"total\" size=\"") followed by an strtol.

[Ian Hinder]

Very interesting - thanks! I didn’t know there was an alternative to mallinfo. Note that this is a GNU extension, so we would have to check if it was available. Also, we should check what happens if a different malloc (e.g. jemalloc or tcmalloc) is being used.

I don’t have time to implement it, but it sounds like a very good idea to use this if available, and fall back to mallinfo if not.

‌

‌

[Roland Haas]

There's an implementation in branch rhaas/short_walltime of CactusUtils. Turns out Cactus already checks if malloc.h (though not explicitly for malloc_info or open_memstream) exists. I would be extremely surprised if this worked with tcmalloc or jemalloc. It also restrict is restricted to Linux, at least on OSX, even when compiling with gcc there is no malloc.h, though given that all clusters are running GNU/Linux these days I am not too worried about that.

The branch produces numbers identical to mallinfo’s uordblks for the test function in embedded in it (for informational purposes only).

[Roland Haas]

This went to the wrong branch. It should have been rhaas/mallinfo

‌

[Roland Haas]

Git hash 3a033501 "Cactus: check for malloc_info function" of cactus of branch rhaas/mallinfo of the flesh adds detection logic to configure.

[Roland Haas]

Pull requests are here:

• https://bitbucket.org/cactuscode/cactusutils/pull-requests/28/add-64bit-support-for-mmaped-memory/diff

• https://bitbucket.org/cactuscode/cactus/pull-requests/91/detect-malloc_info/diff

‌

[Roland Haas]

Please review.

[Roland Haas]

Unless objected I will apply this after 2020-06-01.

[Roland Haas]

Applied as git hash 39bb8fce "Cactus: regenerate configure" of cactus

Applied as git hash c2c4091 "SystemStatistics: avoid SEGFAULT in case of unexpected XML input" of cactusutils