cpython_sandbox / Doc / library / weakref.rst

:mod:`weakref` --- Weak references

Source code: :source:`Lib/weakref.py`


The :mod:`weakref` module allows the Python programmer to create :dfn:`weak references` to objects.

In the following, the term :dfn:`referent` means the object which is referred to by a weak reference.

A weak reference to an object is not enough to keep the object alive: when the only remaining references to a referent are weak references, :term:`garbage collection` is free to destroy the referent and reuse its memory for something else. However, until the object is actually destroyed the weak reference may return the object even if there are no strong references to it.

A primary use for weak references is to implement caches or mappings holding large objects, where it's desired that a large object not be kept alive solely because it appears in a cache or mapping.

For example, if you have a number of large binary image objects, you may wish to associate a name with each. If you used a Python dictionary to map names to images, or images to names, the image objects would remain alive just because they appeared as values or keys in the dictionaries. The :class:`WeakKeyDictionary` and :class:`WeakValueDictionary` classes supplied by the :mod:`weakref` module are an alternative, using weak references to construct mappings that don't keep objects alive solely because they appear in the mapping objects. If, for example, an image object is a value in a :class:`WeakValueDictionary`, then when the last remaining references to that image object are the weak references held by weak mappings, garbage collection can reclaim the object, and its corresponding entries in weak mappings are simply deleted.

:class:`WeakKeyDictionary` and :class:`WeakValueDictionary` use weak references in their implementation, setting up callback functions on the weak references that notify the weak dictionaries when a key or value has been reclaimed by garbage collection. :class:`WeakSet` implements the :class:`set` interface, but keeps weak references to its elements, just like a :class:`WeakKeyDictionary` does.

:class:`finalize` provides a straight forward way to register a cleanup function to be called when an object is garbage collected. This is simpler to use than setting up a callback function on a raw weak reference.

Most programs should find that using one of these weak container types or :class:`finalize` is all they need -- it's not usually necessary to create your own weak references directly. The low-level machinery is exposed by the :mod:`weakref` module for the benefit of advanced uses.

Not all objects can be weakly referenced; those objects which can include class instances, functions written in Python (but not in C), instance methods, sets, frozensets, some :term:`file objects <file object>`, :term:`generator`s, type objects, sockets, arrays, deques, regular expression pattern objects, and code objects.

Several built-in types such as :class:`list` and :class:`dict` do not directly support weak references but can add support through subclassing:

class Dict(dict):
    pass

obj = Dict(red=1, green=2, blue=3)   # this object is weak referenceable

Other built-in types such as :class:`tuple` and :class:`int` do not support weak references even when subclassed (This is an implementation detail and may be different across various Python implementations.).

Extension types can easily be made to support weak references; see :ref:`weakref-support`.

Return a weak reference to object. The original object can be retrieved by calling the reference object if the referent is still alive; if the referent is no longer alive, calling the reference object will cause :const:`None` to be returned. If callback is provided and not :const:`None`, and the returned weakref object is still alive, the callback will be called when the object is about to be finalized; the weak reference object will be passed as the only parameter to the callback; the referent will no longer be available.

It is allowable for many weak references to be constructed for the same object. Callbacks registered for each weak reference will be called from the most recently registered callback to the oldest registered callback.

Exceptions raised by the callback will be noted on the standard error output, but cannot be propagated; they are handled in exactly the same way as exceptions raised from an object's :meth:`__del__` method.

Weak references are :term:`hashable` if the object is hashable. They will maintain their hash value even after the object was deleted. If :func:`hash` is called the first time only after the object was deleted, the call will raise :exc:`TypeError`.

Weak references support tests for equality, but not ordering. If the referents are still alive, two references have the same equality relationship as their referents (regardless of the callback). If either referent has been deleted, the references are equal only if the reference objects are the same object.

This is a subclassable type rather than a factory function.

Note

Like :meth:`__del__` methods, weak reference callbacks can be called during interpreter shutdown when module globals have been overwritten with :const:`None`. This can make writing robust weak reference callbacks a challenge. Callbacks registered using :class:`finalize` do not have to worry about this issue because they will not be run after module teardown has begun.

Mapping class that references keys weakly. Entries in the dictionary will be discarded when there is no longer a strong reference to the key. This can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects. This can be especially useful with objects that override attribute accesses.

Note

Caution: Because a :class:`WeakKeyDictionary` is built on top of a Python dictionary, it must not change size when iterating over it. This can be difficult to ensure for a :class:`WeakKeyDictionary` because actions performed by the program during iteration may cause items in the dictionary to vanish "by magic" (as a side effect of garbage collection).

:class:`WeakKeyDictionary` objects have the following additional methods. These expose the internal references directly. The references are not guaranteed to be "live" at the time they are used, so the result of calling the references needs to be checked before being used. This can be used to avoid creating references that will cause the garbage collector to keep the keys around longer than needed.

Mapping class that references values weakly. Entries in the dictionary will be discarded when no strong reference to the value exists any more.

Note

Caution: Because a :class:`WeakValueDictionary` is built on top of a Python dictionary, it must not change size when iterating over it. This can be difficult to ensure for a :class:`WeakValueDictionary` because actions performed by the program during iteration may cause items in the dictionary to vanish "by magic" (as a side effect of garbage collection).

:class:`WeakValueDictionary` objects have the following additional methods. These method have the same issues as the and :meth:`keyrefs` method of :class:`WeakKeyDictionary` objects.

Set class that keeps weak references to its elements. An element will be discarded when no strong reference to it exists any more.

A custom :class:`ref` subclass which simulates a weak reference to a bound method (i.e., a method defined on a class and looked up on an instance). Since a bound method is ephemeral, a standard weak reference cannot keep hold of it. :class:`WeakMethod` has special code to recreate the bound method until either the object or the original function dies:

>>> class C:
...     def method(self):
...         print("method called!")
...
>>> c = C()
>>> r = weakref.ref(c.method)
>>> r()
>>> r = weakref.WeakMethod(c.method)
>>> r()
<bound method C.method of <__main__.C object at 0x7fc859830220>>
>>> r()()
method called!
>>> del c
>>> gc.collect()
0
>>> r()
>>>

Return a callable finalizer object which will be called when obj is garbage collected. A finalizer is alive until it is called (either explicitly or at garbage collection), and after that it is dead. Calling a live finalizer returns the result of evaluating func(*arg, **kwargs), whereas calling a dead finalizer returns :const:`None`.

Exceptions raised by finalizer callbacks during garbage collection will be shown on the standard error output, but cannot be propagated. They are handled in the same way as exceptions raised from an object's :meth:`__del__` method or a weak reference's callback.

When the program exits, each remaining live finalizer is called unless its :attr:`atexit` attribute has been set to false. They are called in reverse order of creation.

A finalizer will never invoke its callback during the later part of the interpreter shutdown when module globals are liable to have been replaced by :const:`None`.

Note

It is important to ensure that func, args and kwargs do not own any references to obj, either directly or indirectly, since otherwise obj will never be garbage collected. In particular, func should not be a bound method of obj.

Weak Reference Objects

Weak reference objects have no methods and no attributes besides :attr:`ref.__callback__`. A weak reference object allows the referent to be obtained, if it still exists, by calling it:

>>> import weakref
>>> class Object:
...     pass
...
>>> o = Object()
>>> r = weakref.ref(o)
>>> o2 = r()
>>> o is o2
True

If the referent no longer exists, calling the reference object returns :const:`None`:

>>> del o, o2
>>> print(r())
None

Testing that a weak reference object is still live should be done using the expression ref() is not None. Normally, application code that needs to use a reference object should follow this pattern:

# r is a weak reference object
o = r()
if o is None:
    # referent has been garbage collected
    print("Object has been deallocated; can't frobnicate.")
else:
    print("Object is still live!")
    o.do_something_useful()

Using a separate test for "liveness" creates race conditions in threaded applications; another thread can cause a weak reference to become invalidated before the weak reference is called; the idiom shown above is safe in threaded applications as well as single-threaded applications.

Specialized versions of :class:`ref` objects can be created through subclassing. This is used in the implementation of the :class:`WeakValueDictionary` to reduce the memory overhead for each entry in the mapping. This may be most useful to associate additional information with a reference, but could also be used to insert additional processing on calls to retrieve the referent.

This example shows how a subclass of :class:`ref` can be used to store additional information about an object and affect the value that's returned when the referent is accessed:

import weakref

class ExtendedRef(weakref.ref):
    def __init__(self, ob, callback=None, **annotations):
        super(ExtendedRef, self).__init__(ob, callback)
        self.__counter = 0
        for k, v in annotations.items():
            setattr(self, k, v)

    def __call__(self):
        """Return a pair containing the referent and the number of
        times the reference has been called.
        """
        ob = super(ExtendedRef, self).__call__()
        if ob is not None:
            self.__counter += 1
            ob = (ob, self.__counter)
        return ob

Example

This simple example shows how an application can use objects IDs to retrieve objects that it has seen before. The IDs of the objects can then be used in other data structures without forcing the objects to remain alive, but the objects can still be retrieved by ID if they do.

import weakref

_id2obj_dict = weakref.WeakValueDictionary()

def remember(obj):
    oid = id(obj)
    _id2obj_dict[oid] = obj
    return oid

def id2obj(oid):
    return _id2obj_dict[oid]

Finalizer Objects

Often one uses :class:`finalize` to register a callback without bothering to keep the returned finalizer object. For instance

>>> import weakref
>>> class Object:
...     pass
...
>>> kenny = Object()
>>> weakref.finalize(kenny, print, "You killed Kenny!")  #doctest:+ELLIPSIS
<finalize object at ...; for 'Object' at ...>
>>> del kenny
You killed Kenny!

The finalizer can be called directly as well. However the finalizer will invoke the callback at most once.

>>> def callback(x, y, z):
...     print("CALLBACK")
...     return x + y + z
...
>>> obj = Object()
>>> f = weakref.finalize(obj, callback, 1, 2, z=3)
>>> assert f.alive
>>> assert f() == 6
CALLBACK
>>> assert not f.alive
>>> f()                     # callback not called because finalizer dead
>>> del obj                 # callback not called because finalizer dead

You can unregister a finalizer using its :meth:`~finalize.detach` method. This kills the finalizer and returns the arguments passed to the constructor when it was created.

>>> obj = Object()
>>> f = weakref.finalize(obj, callback, 1, 2, z=3)
>>> f.detach()                                           #doctest:+ELLIPSIS
(<__main__.Object object ...>, <function callback ...>, (1, 2), {'z': 3})
>>> newobj, func, args, kwargs = _
>>> assert not f.alive
>>> assert newobj is obj
>>> assert func(*args, **kwargs) == 6
CALLBACK

Unless you set the :attr:`~finalize.atexit` attribute to :const:`False`, a finalizer will be called when the program exit if it is still alive. For instance

>>> obj = Object()
>>> weakref.finalize(obj, print, "obj dead or exiting")  #doctest:+ELLIPSIS
<finalize object at ...; for 'Object' at ...>
>>> exit()                                               #doctest:+SKIP
obj dead or exiting

Comparing finalizers with :meth:`__del__` methods

Suppose we want to create a class whose instances represent temporary directories. The directories should be deleted with their contents when the first of the following events occurs:

  • the object is garbage collected,
  • the object's :meth:`remove` method is called, or
  • the program exits.

We might try to implement the class using a :meth:`__del__` method as follows:

class TempDir:
    def __init__(self):
        self.name = tempfile.mkdtemp()

    def remove(self):
        if self.name is not None:
            shutil.rmtree(self.name)
            self.name = None

    @property
    def removed(self):
        return self.name is None

    def __del__(self):
        self.remove()

This solution has a couple of serious problems:

  • There is no guarantee that the object will be garbage collected before the program exists, so the directory might be left. This is because reference cycles containing an object with a :meth:`__del__` method can never be collected. And even if the :class:`TempDir` object is not itself part of a reference cycle, it may still be kept alive by some unkown uncollectable reference cycle.
  • The :meth:`__del__` method may be called at shutdown after the :mod:`shutil` module has been cleaned up, in which case :attr:`shutil.rmtree` will have been replaced by :const:`None`. This will cause the :meth:`__del__` method to fail and the directory will not be removed.

Using finalizers we can avoid these problems:

class TempDir:
    def __init__(self):
        self.name = tempfile.mkdtemp()
        self._finalizer = weakref.finalize(self, shutil.rmtree, self.name)

    def remove(self):
        self._finalizer()

    @property
    def removed(self):
        return not self._finalizer.alive

Defined like this, even if a :class:`TempDir` object is part of a reference cycle, that reference cycle can still be garbage collected. If the object never gets garbage collected the finalizer will still be called at exit.

Note

If you create a finalizer object in a daemonic thread just as the the program exits then there is the possibility that the finalizer does not get called at exit. However, in a daemonic thread :func:`atexit.register`, try: ... finally: ... and with: ... do not guarantee that cleanup occurs either.

Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.