1. SCons
  2. Core
  3. SCons
  4. Pull requests

Pull requests

#97 Merged at b4bc497

Enhancement: reducing overall memory consumption...

  1. dirkbaechle

This patch aims at reducing the required memory for clean builds and update runs in large SCons projects (like more than several thousand files).

It adds a special method named release_target_info() to the Node base class, which is then responsible for releasing as many attributes as possible after the current target was built, or found to be up-to-date.

No existing interfaces had to be changed, all tests passed successfully on my side (tested under Linux only, can't tell what happens under Windows).

Comments (6)

  1. Tom Tanner

    Why do you set some of the lists to None rather than empty lists? It seems to add a lot special 'if this isn't None' code. I realise I'm probably being a bit dim here, as I'm no python expert, but surely the memory overhead of an empty list is fairly minimal, and it'd make the code a little cleaner

    1. dirkbaechle author

      Hi Tom,

      thanks a lot for your comments.

      I'm no Python expert either. ;) I didn't really measure how big the benefit of releasing lists on its own is exactly. I'm just trying to release as much as possible, without breaking any stuff. And when we talk about 10000+ files, these "fairly minimal" memory overheads per node start to count...

      With the (probably) upcoming changes that introduce the usage of "slots", the code will get even more entangled. But that's where we have to make the basic decision: Do we want to have a clean code base, like in speckle-free, that can't be used for large-scale projects, or do we give the user what he actually wants and needs for his/her daily work? I'm currently aiming at #2...

      1. Tom Tanner

        Well, having to have extra checks for whether something is None where it could be set up as an empty list makes the code more brittle and more difficult to maintain which isn't good for point #2.

        1. dirkbaechle author

          That's why I wrote "give the user what he wants", not "give the developer/maintainer what he wants". ;)

          I can perfectly understand that things like these matter to people like yourself and me, where both roles unite on the same person. But, let's face it, we're a minority... Most of the actual SCons users simply don't care about the "man behind the curtains", and they complain about it being too slow or too much of a memory-hog...not that the code isn't "clean enough".

          1. Tom Tanner

            Yes, but what I'm saying (not very well) is how does the memory overhead of an empty list balance against the time overhead of checking for a None list?