support 64 bit numbers for SystemStatistic's malloc'ed memory size

Create issue
Issue #2352 resolved
Former user created an issue

The mallinfo(3) call used by SystemStatistics is limited to 32bit signed numbers and thus cannot record if more than 2GB of memory are used.

There is no direct 64bit equivalent, however there is a malloc_info function (http://man7.org/linux/man-pages/man3/malloc_info.3.html) which can be used, in combination with open_memstream(3) (http://man7.org/linux/man-pages/man3/open_memstream.3.html) to obtain an in-memory XML string containing the required information.

Looking at https://git.amper.me/open-source/glibc/blob/4d653a59ffeae0f46f76a40230e2cfa9587b7e7e/malloc/malloc.c#L5148 and how that same function computes uordblks the proper way to get a 64bit integer would seem to be to compute total_aspace - total_avail.

"Parsing" the XML string can be done using strstr(xml, "<aspace type=\"total\" size=\"") followed by an strtol.

Comments (9)

  1. Ian Hinder

    Very interesting - thanks! I didn’t know there was an alternative to mallinfo. Note that this is a GNU extension, so we would have to check if it was available. Also, we should check what happens if a different malloc (e.g. jemalloc or tcmalloc) is being used.

    I don’t have time to implement it, but it sounds like a very good idea to use this if available, and fall back to mallinfo if not.

  2. Roland Haas

    There's an implementation in branch rhaas/short_walltime of CactusUtils. Turns out Cactus already checks if malloc.h (though not explicitly for malloc_info or open_memstream) exists. I would be extremely surprised if this worked with tcmalloc or jemalloc. It also restrict is restricted to Linux, at least on OSX, even when compiling with gcc there is no malloc.h, though given that all clusters are running GNU/Linux these days I am not too worried about that.

    The branch produces numbers identical to mallinfo’s uordblks for the test function in embedded in it (for informational purposes only).

  3. Log in to comment