1. osunix
  2. ospkg

Source

ospkg / dev-notes / heapy.rst

How to use guppy/heapy for tracking down memory usage

This is a work in progress. It will grow a bit and it may not be entirely accurate everywhere.

Tutorial of sorts

All this was done on a checkout of marienz@gentoo.org-20060908111256-540d8fb3db5b337e, you should be able to check that out and follow along using something like:

bzr revert -rrevid:marienz@gentoo.org-20060908111256-540d8fb3db5b337e

in a pkgcore branch.

Heapy is powerful but has a learning curve. Problems are the documentation (http://guppy-pe.sourceforge.net/heapy_Use.html among others) is a bit unusual and there are various dynamic importing and other tricks in use that mean things like dir() are less helpful than they are on more "normal" python objects. This document's main purpose is to show you how to ask heapy various kinds of questions. It may or may not show a few cases where pkgcore uses more memory than it should too.

First, get an x86. Heapy currently does not like 64 bit archs much.

Emerge it:

emerge guppy

Fire up an interactive python prompt, set stuff up:

>>> from guppy import hpy
>>> from pkgcore.config import load_config
>>> c = load_config()
>>> hp = hpy()

Just to show how annoying heapy's internal tricks are:

>>> dir(hp)
['__doc__', '__getattr__', '__init__', '__module__', '__setattr__', '_hiding_tag_', '_import', '_name', '_owner', '_share']
>>> help(hp)
Help on class _GLUECLAMP_ in module guppy.etc.Glue:

_GLUECLAMP_ = <guppy.heapy.Use interface at 0x-484b8554>

This object is your "starting point", but as you can see the underlying machinery is not giving away any useful usage instructions.

Do everything that allocates some memory but is not the problem you are tracking down now. Then do:

>>> hp.setrelheap()

Everything allocated before this call will not be in the data sets you get later.

Now do your memory-intensive thing:

>>> l = list(x for x in c.repo["portdir"] if x.data)

Keep an eye on system memory consumption. You want to use up a lot but not all of your system ram for nicer statistics. The python process was eating about 109M res in top when the above stuff finished, which is pretty good (for my 512mb ram box).

>>> h = hp.heap()

The fun one. This object is basically a snapshot of what's reachable in ram (minus the stuff excluded through setrelheap earlier) which you can do various fun tricks with. Its str() is a summary:

>>> h
Partition of a set of 1449133 objects. Total size = 102766644 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 985931  68 46300932  45  46300932  45 str
     1  24681   2 22311624  22  68612556  67 dict of pkgcore.ebuild.ebuild_src.package
     2  49391   3 21311864  21  89924420  88 dict (no owner)
     3 115974   8  3776948   4  93701368  91 tuple
     4 152181  11  3043616   3  96744984  94 long
     5  36009   2  1584396   2  98329380  96 weakref.KeyedRef
     6  11328   1  1540608   1  99869988  97 dict of pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
     7  24702   2   889272   1 100759260  98 types.MethodType
     8  11424   1   851840   1 101611100  99 list
     9  24681   2   691068   1 102302168 100 pkgcore.ebuild.ebuild_src.package
<54 more rows. Type e.g. '_.more' to view.>

(You might want to keep an eye on ram usage: heapy made the process grow another dozen mb here. It gets painfully slow if it starts swapping, so if that happens reduce your data set).

Notice the "Total size" in the top right: about 100M. That's what we need to compare later numbers with.

So here we can see that (surprise!) we have a ton of strings in memory. We also have various kinds of dicts. Dicts are treated a bit specially: the "dict of pkgcore.ebuild.ebuild_src.package" simply means "all the dicts that are __dict__ attributes of instances of that class". "dict (no owner)" are all the dicts that are not used as __dict__ attribute.

You probably guessed what you can use "index" for:

>>> h[0]
Partition of a set of 985931 objects. Total size = 46300932 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 985931 100 46300932 100  46300932 100 str

Ok, that looks pretty useless, but it really is not. The "sets" heapy gives you (like "h" and "h[0]") are a bunch of objects, grouped together by an "equivalence relation". The default one (with the crazy name "Clodo" for "Class or dict owner") groups together all objects of the same class and dicts with the same owner. We can also partition the sets by a different equivalence relation. Let's do a silly example first:

>>> h.bytype
Partition of a set of 1449133 objects. Total size = 102766644 bytes.
 Index  Count   %     Size   % Cumulative  % Type
     0 985931  68 46300932  45  46300932  45 str
     1  85556   6 45226592  44  91527524  89 dict
     2 115974   8  3776948   4  95304472  93 tuple
     3 152181  11  3043616   3  98348088  96 long
     4  36009   2  1584396   2  99932484  97 weakref.KeyedRef
     5  24702   2   889272   1 100821756  98 types.MethodType
     6  11424   1   851840   1 101673596  99 list
     7  24681   2   691068   1 102364664 100 pkgcore.ebuild.ebuild_src.package
     8  11328   1   317184   0 102681848 100 pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
     9    408   0    26112   0 102707960 100 types.CodeType
<32 more rows. Type e.g. '_.more' to view.>

As you can see this is the same thing as the default view, but with all the dicts lumped together. A more useful one is:

>>> h.byrcs
Partition of a set of 1449133 objects. Total size = 102766644 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0 870779  60 43608088  42  43608088  42 dict (no owner)
     1  24681   2 22311624  22  65919712  64 pkgcore.ebuild.ebuild_src.package
     2 221936  15 20575932  20  86495644  84 dict of pkgcore.ebuild.ebuild_src.package
     3 242236  17  8588560   8  95084204  93 tuple
     4      6   0  1966736   2  97050940  94 dict of weakref.WeakValueDictionary
     5  36009   2  1773024   2  98823964  96 dict (no owner), dict of
                                             pkgcore.ebuild.ebuild_src.package, weakref.KeyedRef
     6  11328   1  1540608   1 100364572  98 pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
     7  26483   2   800432   1 101165004  98 list
     8  11328   1   724992   1 101889996  99 dict of pkgcore.ebuild.ebuild_src.ThrowAwayNameSpace
     9      3   0   393444   0 102283440 100 dict of pkgcore.repository.prototype.IterValLazyDict
<132 more rows. Type e.g. '_.more' to view.>

What this does is:

  • for every object, find all its referrers
  • Classify those referrers using the "Clodo" relation you saw earlier
  • Create a set of those classifiers of referrers. That means a set of things like "tuple, dict of someclass", not of actual referring objects.
  • Group together all the objects with the same set of classifiers of referrers.

So now we know that we have a lot of objects referenced only by one or more dicts (still not very useful) and also a lot of them referenced by one "normal" dict, referenced by the dict of (meaning "an attribute of") ebuild_src.package, and referenced by a WeakRef. Hmm, I wonder what those are. But let's store this view of the data first, since it took a while to generate ("_" is a feature of the python interpreter, it's always the last result):

>>> byrcs = _
>>> byrcs[5]
Partition of a set of 36009 objects. Total size = 1773024 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  36009 100  1773024 100   1773024 100 dict (no owner), dict of
                                             pkgcore.ebuild.ebuild_src.package, weakref.KeyedRef

Erm, yes, we knew that already. If you look in the top right of the table you can see it is still grouping the items by the kind of their referrer, which is not very useful here. To get more information we can change what they are grouped by:

>>> byrcs[5].byclodo
Partition of a set of 36009 objects. Total size = 1773024 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  36009 100  1773024 100   1773024 100 str
>>> byrcs[5].bysize
Partition of a set of 36009 objects. Total size = 1773024 bytes.
 Index  Count   %     Size   % Cumulative  % Individual Size
     0  10190  28   489120  28    489120  28        48
     1   7584  21   394368  22    883488  50        52
     2   7335  20   322740  18   1206228  68        44
     3   3947  11   221032  12   1427260  80        56
     4   3364   9   134560   8   1561820  88        40
     5   1903   5   114180   6   1676000  95        60
     6    877   2    56128   3   1732128  98        64
     7    285   1    19380   1   1751508  99        68
     8    451   1    16236   1   1767744 100        36
     9     57   0     4104   0   1771848 100        72

This took the set of objects with that odd set of referrers and redisplayed them grouped by "clodo". So now we know they're all strings. Most of them are pretty small too. To get some idea of what we're dealing with we can pull some random examples out:

>>> byrcs[5].byid
Set of 36009 <str> objects. Total size = 1773024 bytes.
 Index     Size   %   Cumulative  %   Representation (limited)
     0       80   0.0        80   0.0 'media-plugin...re20051219-r1'
     1       76   0.0       156   0.0 'app-emulatio...4.20041102-r1'
     2       76   0.0       232   0.0 'dev-php5/ezc...hemaTiein-1.0'
     3       76   0.0       308   0.0 'games-misc/f...wski-20030120'
     4       76   0.0       384   0.0 'mail-client/...pt-viewer-0.8'
     5       76   0.0       460   0.0 'media-fonts/...-100dpi-1.0.0'
     6       76   0.0       536   0.0 'media-plugin...gdemux-0.10.4'
     7       76   0.0       612   0.0 'media-plugin...3_pre20051219'
     8       76   0.0       688   0.0 'media-plugin...3_pre20051219'
     9       76   0.0       764   0.0 'media-plugin...3_pre20060502'
>>> byrcs[5].byid[0].theone
'media-plugins/vdr-streamdev-server-0.3.3_pre20051219-r1'

A pattern emerges! (sets with one item have a "theone" attribute with the actual item, all sets have a "nodes" attribute that returns an iterator yielding the items).

We could have used another heapy trick to get a better idea of what kind of string this was:

>>> byrcs[5].byvia
Partition of a set of 36009 objects. Total size = 1773024 bytes.
 Index  Count   %     Size   % Cumulative  % Referred Via:
     0      1   0       80   0        80   0 "['cpvstr']", '.key', '.keys()[23147]'
     1      1   0       76   0       156   0 "['cpvstr']", '.key', '.keys()[12285]'
     2      1   0       76   0       232   0 "['cpvstr']", '.key', '.keys()[12286]'
     3      1   0       76   0       308   0 "['cpvstr']", '.key', '.keys()[16327]'
     4      1   0       76   0       384   0 "['cpvstr']", '.key', '.keys()[17754]'
     5      1   0       76   0       460   0 "['cpvstr']", '.key', '.keys()[19079]'
     6      1   0       76   0       536   0 "['cpvstr']", '.key', '.keys()[21704]'
     7      1   0       76   0       612   0 "['cpvstr']", '.key', '.keys()[23473]'
     8      1   0       76   0       688   0 "['cpvstr']", '.key', '.keys()[24239]'
     9      1   0       76   0       764   0 "['cpvstr']", '.key', '.keys()[3070]'
<35999 more rows. Type e.g. '_.more' to view.>

Ouch, 36009 total rows for 36009 objects. What this did is similar to what "byrcs" did: for every object in the set it determined how they can be reached through their referrers, then groups objects that can be reached in the same ways together. Unfortunately it is grouping everything reachable as a dictionary key differently, so this is not very useful.

XXX WTF XXX

It is not likely this accomplishes anything, but let's assume we want to know if there are any objects in this set not reachable as the "key" attribute. Heapy can tell us (although this is very slow... there might be a better way but I do not know it yet):

>>> nonkeys = byrcs[5] & hp.Via('.key').alt('<')
>>> nonkeys.byrcs
hp.Nothing

(remember "hp" was our main entrance into heapy, the object that gave us the set of all objects we're interested in earlier).

What does this do? "hp.Via('.key')" creates a "symbolic set" of "all objects reachable only as the 'key' attribute of something" (it's a "symbolic set" because there are no actual objects in it). The "alt" method gives us a new symbolic set of everything reachable via "less than" this way. We then intersect this with our set and discover there is nothing left.

A similar construct that does not do what we want is:

>>> nonkeys = byrcs[5] & ~hp.Via('.key')

The "~" operator inverts the symbolic set, giving a set matching everything not reachable exactly as a "key" attribute. The key word here is "exactly": since everything in our set was also reachable in two other ways this intersection matches everything.

Ok, let's get back to the stuff actually eating memory:

>>> h[0].byrcs
Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
    0 670791  68 31716096  68  31716096  68 dict (no owner)
    1 139232  14  6525856  14  38241952  83 tuple
    2 136558  14  6042408  13  44284360  96 dict of pkgcore.ebuild.ebuild_src.package
    3  36009   4  1773024   4  46057384  99 dict (no owner), dict of
                                            pkgcore.ebuild.ebuild_src.package, weakref.KeyedRef
    4   1762   0   107772   0  46165156 100 list
    5    824   0    69476   0  46234632 100 types.CodeType
    6    140   0    31312   0  46265944 100 function, tuple
    7    194   0    11504   0  46277448 100 dict of module
    8     30   0     6284   0  46283732 100 dict of type
    9     55   0     1972   0  46285704 100 dict of module, tuple

Remember h[0] gave us all str objects, so this is all string objects grouped by the kind(s) of their referrers. Also notice index 3 here is the same set of stuff we saw earlier:

>>> h[0].byrcs[3] ^ byrcs[5]
hp.Nothing

Most operators do what you would expect, & intersects for example.

"We have a lot of strings in dicts" is not that useful either, let's see if we can narrow that down a little:

>>> h[0].byrcs[0].referrers.byrcs
Partition of a set of 44124 objects. Total size = 18636768 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  24681  56 12834120  69  12834120  69 dict of pkgcore.ebuild.ebuild_src.package
     1  19426  44  5371024  29  18205144  98 dict (no owner)
     2      1   0   393352   2  18598496 100 dict of pkgcore.repository.prototype.IterValLazyDict
     3      1   0     6280   0  18604776 100 __builtin__.set
     4      1   0     6280   0  18611056 100 dict of module, guppy.heapy.heapyc.RootStateType
     5      1   0     6280   0  18617336 100 dict of pkgcore.ebuild.eclass_cache.cache
     6      1   0     6280   0  18623616 100 dict of
                                             pkgcore.repository.prototype.PackageIterValLazyDict
     7      4   0     5536   0  18629152 100 type
     8      4   0     3616   0  18632768 100 dict of type
     9      1   0     1672   0  18634440 100 dict of module, dict of os._Environ

(Broken down: h[0].byrcs[0] is the set of all str objects referenced only by dicts, h[0].byrcs[0].referrers is the set of those dicts, and the final .byrcs displays those dicts grouped by their referrers)

Keep an eye on the size column. We have over 12M worth of just dicts (not counting the stuff in them) referenced only as attribute of ebuild_src.package. If we include the stuff kept alive by those dicts we're talking about a big chunk of the 100MB total here:

>>> t = _
>>> t[0].domisize
61269552

60M out of our 100M would be deallocated if we killed those dicts. So let's ask heapy what dicts that are:

>>> t[0].byvia
Partition of a set of 24681 objects. Total size = 12834120 bytes.
 Index  Count   %     Size   % Cumulative  % Referred Via:
     0  24681 100 12834120 100  12834120 100 "['data']"

(it is easy to get confused by the "byrcs" view of our "t". t[0] is not a bunch of "dict of ebuild_src.package". It is a bunch of dicts with strings in them, namely those that are referred to by the dict of ebuild_src.package, and not by anything else. So the byvia output means those dicts with strings in them are all "data" attributes of ebuild_src.package instances).

(sidenote: earlier we saw byvia say ".key", now it says "['data']". It's different because the previous type used __slots__ (so there was no "dict of" involved) and this type does not (so there is a "dict of" and our dicts are the "data" key in it).

So what is in the dicts:

>>> t[0].referents
Partition of a set of 605577 objects. Total size = 34289392 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 556215  92 27710068  81  27710068  81 str
     1  24681   4  6085704  18  33795772  99 dict (no owner)
     2  24681   4   493620   1  34289392 100 long
>>> _.byvia
Partition of a set of 605577 objects. Total size = 34289392 bytes.
 Index  Count   %     Size   % Cumulative  % Referred Via:
     0  24681   4  6085704  18   6085704  18 "['_eclasses_']"
     1  21954   4  3742976  11   9828680  29 "['DEPEND']"
     2  22511   4  3300052  10  13128732  38 "['RDEPEND']"
     3  24202   4  2631304   8  15760036  46 "['SRC_URI']"
     4  24681   4  1831668   5  17591704  51 "['DESCRIPTION']"
     5  24674   4  1476680   4  19068384  56 "['HOMEPAGE']"
     6  24681   4  1297680   4  20366064  59 "['KEYWORDS']"
     7  24681   4   888516   3  21254580  62 '.keys()[3]'
     8  24681   4   888516   3  22143096  65 '.keys()[9]'
     9  24681   4   810108   2  22953204  67 "['LICENSE']"
<32 more rows. Type e.g. '_.more' to view.>

Strings, nested dicts and longs, and most size eaten up by the "_eclasses_" values. There is also a significant amount eaten up by keys values, which is a bit odd, so let's investigate:

>>> refs = t[0].referents
>>> i=iter(refs.byvia[7].nodes)
>>> i.next()
'DESCRIPTION'
>>> i.next()
'DESCRIPTION'
>>> i.next()
'DESCRIPTION'
>>> i.next()
'DESCRIPTION'
>>> i.next()
'DESCRIPTION'

Eep!

>>> refs.byvia[7].bysize
Partition of a set of 24681 objects. Total size = 888516 bytes.
 Index  Count   %     Size   % Cumulative  % Individual Size
     0  24681 100   888516 100    888516 100        36

It looks like we have 24681 identical strings here, using up about 1M of memory. The other odd entry is the '_eclasses_' string apparently.

Extra stuff for c extension developers

To provide accurate statistics if your code uses extension types you must provide heapy with a way to get the following data for your custom types:

  • How large is a certain instance?
  • What objects does an instance contain?
  • How does the instance refer to a contained object?

You provide these through a NyHeapDef struct, defined in heapdef.h in the guppy source. This header is not installed, so you should just copy it into your source tree. It is a good idea to read this header file side by side with the following descriptions, since it contains details omitted here. The stdtypes.c file contains implementations for the basic python types which you can read for inspiration.

The NyHeapDef struct provides heapy with three function pointers:

SizeGetter

To answer "how large is an instance" you provide a NyHeapDef_SizeGetter function that is called with a PyObject* and returns an int: the number of bytes the object occupies. If you do not provide this function heapy uses a default that looks at the tp_basicsize and tp_itemsize fields of the type. This means that if you do not allocate any extra memory for non-python objects (e.g. for c strings) you do not need to provide this function.

Traverser

To answer "What objects does an instance contain" you provide a traversal function (NyHeapDef_Traverser). This is called with a pointer to a "visit procedure", an instance of your extension type and some other stuff. You should then call the visit procedure for every python object contained in your object.

This might sound familiar: to support the python garbage collector you provide a very similar function (tp_traverse). Actually heapy will use tp_traverse if you do not provide a heapy-specific traverse function. Doing this makes sense if you do not support the garbage collector for some reason, or if you contain objects that are irrelevant to the garbage collector.

An example would be a type that contains a single python string object (that no other code can get a reference to). If this object does not have references to other python objects it cannot be involved in cycles so supporting gc would be useless. However you do still want heapy to know about the memory occupied by the contained string. You could do that by adding that size in your NyHeapDef_SizeGetter function but it is probably easier to tell heapy about the string through the traversal function (so you do not have to calculate the memory occupied by the string).

If the above type would also have a reference to some arbitrary (non-private) python object it should support gc, but it does not need to tell gc about the contained string. So you would have two traversal functions, one for heapy that visits the string and one for gc that does not.

RelationGetter

The last function heapy wants tells it in what way your instance refers to some contained object. It is used to provide the "byvia" view. This calls a visit function once for each way your instance refers to a target object, telling it what kind of reference it is.

Providing the heapdef struct to heapy

Once you have the needed function pointers in a struct you need to pass this to heapy somehow. This is done through a standard cpython mechanism called "cobjects". From python these look like rather stupid objects you cannot do anything with, but from c you can pull out a void* that was put in when the object was constructed. You can wrap an arbitrary pointer in a CObject, make it available as attribute of your module, then import it from some other module, pull the void* back out and cast it to the original type.

heapy looks for a _NyHeapDefs_ attribute on all loaded modules. If this attribute exists and is a CObject the pointer in it is used as a pointer to an array of NyHeapDef struct (terminated with a struct with only nulls). Example code doing this is in sets.c in the guppy source.