Use umm_malloc for Lua to reduce memory fragmentation

#724 Merged at 2e5d9bb
Repository
Branch
lua_umm_malloc
Repository
Branch
lua_fix
Author
  1. Alex
Reviewers
Description

Rather than letting Lua allocate a few thousands of tiny blocks with ML memory backend (which is slow, and memory fragmentation may cause Canon code to fail), in this experiment I tried pre-allocating a 512KB block and using umm_malloc on it.

What doesn't fit here (e.g. if you have some more scripts) is allocated with default ML routines.

Besides reduced fragmentation, scripts are loaded quite a bit faster now.

Disadvantages:

  • even if you have only a tiny script, this will allocate 512K.
  • on some cameras, this block may be from shoot_malloc, and there are some disadvantages with that (for example, in-camera raw development, or enabling certain image processing options, may fail with err70, because they expect the shoot_malloc buffer to be completely free - probably to change the memory management profile or whatever). Of course, that is true for any other memory-intensive module, until we find some other allocator that can be kept allocated without Canon code complaining about it.

Any suggestion for a memory allocator (external library) that can grow at runtime, as needed?

Comments (2)

  1. David Milligan

    No more err70 on 60D, even with all scripts (including extras) loaded. Yay!

    Free umm heap: 67K, Free Memory: 274K + 996K

    These cameras have boat loads of physical RAM (relative to their other specs), it's a shame that it has to be such a concern. It'd be nice if we could just trick Canon firmware into thinking there is less and permanently wire it for ourselves, then use our own allocator for it. What's a few MB out of hundreds?

    1. Alex author

      Right. The only problem - RscMgr is so complex and has so many hardcoded addresses (for example, look at memnavi routines) that I don't know how to solve it for the general case.

      On 70D, the RscMgr "allocates" (with hardcoded address) memory for IMGPLAY overlapping the first raw buffer right after a still photo was captured (which interferes with raw overlays). For that case, I'm seriously thinking to change the memory map somehow, so these two buffers no longer overlap. (discussion)

      The ML boot method shrinks one of the two malloc buffers from above, so Canon firmware really thinks there is less. But these two buffers are too small, and sometimes they are full, as with 1100D.

      There are a lot of small allocators based on AllocateMemory code (look for references to the function that actually does the work for memory allocation): svg, zlib, winsys, ddd.

      On 60D and 7D in particular, there appears to be an unused block of memory at the end, where ML is currently loading autoexec, and it's pretty large (7-8 MB) (source), so I'm thinking to switch back to classic boot method, and use a simple allocator (or maybe even Canon's) to manage that block and make all that memory available to ML.

      There is CONFIG_MARK_UNUSED_MEMORY_AT_STARTUP - this tool may help with guessing what memory blocks are really unused by Canon code. On 5D3 for example, there appears to be a lot of that in LiveView, in the SS_DEVELOP1/2 areas, but it's quite fragmented (Canon code allocates buffers with very large spacing between them). Outside LV, almost the entire memory is available with shoot_malloc + srm_malloc.