support 64 bit numbers for SystemStatistic's malloc'ed memory size
The mallinfo(3)
call used by SystemStatistics
is limited to 32bit signed numbers and thus cannot record if more than 2GB of memory are used.
There is no direct 64bit equivalent, however there is a malloc_info
function (http://man7.org/linux/man-pages/man3/malloc_info.3.html) which can be used, in combination with open_memstream(3)
(http://man7.org/linux/man-pages/man3/open_memstream.3.html) to obtain an in-memory XML string containing the required information.
Looking at https://git.amper.me/open-source/glibc/blob/4d653a59ffeae0f46f76a40230e2cfa9587b7e7e/malloc/malloc.c#L5148 and how that same function computes uordblks
the proper way to get a 64bit integer would seem to be to compute total_aspace - total_avail
.
"Parsing" the XML string can be done using strstr(xml, "<aspace type=\"total\" size=\"")
followed by an strtol
.
Comments (9)
-
-
- changed status to open
-
There's an implementation in branch rhaas/short_walltime of CactusUtils. Turns out Cactus already checks if
malloc.h
(though not explicitly formalloc_info
oropen_memstream
) exists. I would be extremely surprised if this worked with tcmalloc or jemalloc. It also restrict is restricted to Linux, at least on OSX, even when compiling with gcc there is no malloc.h, though given that all clusters are running GNU/Linux these days I am not too worried about that.The branch produces numbers identical to
mallinfo
’suordblks
for the test function in embedded in it (for informational purposes only). -
This went to the wrong branch. It should have been rhaas/mallinfo
-
-
Pull requests are here:
-
Please review.
-
Unless objected I will apply this after 2020-06-01.
-
- changed status to resolved
Applied as git hash 39bb8fce "Cactus: regenerate configure" of cactus
Applied as git hash c2c4091 "SystemStatistics: avoid SEGFAULT in case of unexpected XML input" of cactusutils
- Log in to comment
Very interesting - thanks! I didn’t know there was an alternative to mallinfo. Note that this is a GNU extension, so we would have to check if it was available. Also, we should check what happens if a different malloc (e.g. jemalloc or tcmalloc) is being used.
I don’t have time to implement it, but it sounds like a very good idea to use this if available, and fall back to mallinfo if not.