Source

python-clinic / Doc / library / pickle.rst

Full commit

:mod:`pickle` --- Python object serialization

The :mod:`pickle` module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. "Pickling" is the process whereby a Python object hierarchy is converted into a byte stream, and "unpickling" is the inverse operation, whereby a byte stream is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as "serialization", "marshalling," [1] or "flattening", however, to avoid confusion, the terms used here are "pickling" and "unpickling"..

Warning

The :mod:`pickle` module is not intended to be secure against erroneous or maliciously constructed data. Never unpickle data received from an untrusted or unauthenticated source.

Relationship to other Python modules

The :mod:`pickle` module has an transparent optimizer (:mod:`_pickle`) written in C. It is used whenever available. Otherwise the pure Python implementation is used.

Python has a more primitive serialization module called :mod:`marshal`, but in general :mod:`pickle` should always be the preferred way to serialize Python objects. :mod:`marshal` exists primarily to support Python's :file:`.pyc` files.

The :mod:`pickle` module differs from :mod:`marshal` in several significant ways:

  • The :mod:`pickle` module keeps track of the objects it has already serialized, so that later references to the same object won't be serialized again. :mod:`marshal` doesn't do this.

    This has implications both for recursive objects and object sharing. Recursive objects are objects that contain references to themselves. These are not handled by marshal, and in fact, attempting to marshal recursive objects will crash your Python interpreter. Object sharing happens when there are multiple references to the same object in different places in the object hierarchy being serialized. :mod:`pickle` stores such objects only once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects.

  • :mod:`marshal` cannot be used to serialize user-defined classes and their instances. :mod:`pickle` can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.

  • The :mod:`marshal` serialization format is not guaranteed to be portable across Python versions. Because its primary job in life is to support :file:`.pyc` files, the Python implementers reserve the right to change the serialization format in non-backwards compatible ways should the need arise. The :mod:`pickle` serialization format is guaranteed to be backwards compatible across Python releases.

Note that serialization is a more primitive notion than persistence; although :mod:`pickle` reads and writes file objects, it does not handle the issue of naming persistent objects, nor the (even more complicated) issue of concurrent access to persistent objects. The :mod:`pickle` module can transform a complex object into a byte stream and it can transform the byte stream into an object with the same internal structure. Perhaps the most obvious thing to do with these byte streams is to write them onto a file, but it is also conceivable to send them across a network or store them in a database. The module :mod:`shelve` provides a simple interface to pickle and unpickle objects on DBM-style database files.

Data stream format

The data format used by :mod:`pickle` is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can't represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.

By default, the :mod:`pickle` data format uses a relatively compact binary representation. If you need optimal size characteristics, you can efficiently :doc:`compress <archiving>` pickled data.

The module :mod:`pickletools` contains tools for analyzing data streams generated by :mod:`pickle`. :mod:`pickletools` source code has extensive comments about opcodes used by pickle protocols.

There are currently 4 different protocols which can be used for pickling.

  • Protocol version 0 is the original "human-readable" protocol and is backwards compatible with earlier versions of Python.
  • Protocol version 1 is an old binary format which is also compatible with earlier versions of Python.
  • Protocol version 2 was introduced in Python 2.3. It provides much more efficient pickling of :term:`new-style class`es. Refer to PEP 307 for information about improvements brought by protocol 2.
  • Protocol version 3 was added in Python 3. It has explicit support for :class:`bytes` objects and cannot be unpickled by Python 2.x. This is the default as well as the current recommended protocol; use it whenever possible.

Module Interface

To serialize an object hierarchy, you simply call the :func:`dumps` function. Similarly, to de-serialize a data stream, you call the :func:`loads` function. However, if you want more control over serialization and de-serialization, you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively.

The :mod:`pickle` module provides the following constants:

The :mod:`pickle` module provides the following functions to make the pickling process more convenient:

The :mod:`pickle` module defines three exceptions:

The :mod:`pickle` module exports two classes, :class:`Pickler` and :class:`Unpickler`:

What can be pickled and unpickled?

The following types can be pickled:

  • None, True, and False
  • integers, floating point numbers, complex numbers
  • strings, bytes, bytearrays
  • tuples, lists, sets, and dictionaries containing only picklable objects
  • functions defined at the top level of a module
  • built-in functions defined at the top level of a module
  • classes that are defined at the top level of a module
  • instances of such classes whose :attr:`__dict__` or the result of calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for details).

Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` exception; when this happens, an unspecified number of bytes may have already been written to the underlying file. Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`.

Note that functions (built-in and user-defined) are pickled by "fully qualified" name reference, not by value. This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [2]

Similarly, classes are pickled by named reference, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute attr is not restored in the unpickling environment:

class Foo:
    attr = 'A class attribute'

picklestring = pickle.dumps(Foo)

These restrictions are why picklable functions and classes must be defined in the top level of a module.

Similarly, when class instances are pickled, their class's code and data are not pickled along with them. Only the instance data are pickled. This is done on purpose, so you can fix bugs in a class or add methods to the class and still load objects that were created with an earlier version of the class. If you plan to have long-lived objects that will see many versions of a class, it may be worthwhile to put a version number in the objects so that suitable conversions can be made by the class's :meth:`__setstate__` method.

Pickling Class Instances

In this section, we describe the general mechanisms available to you to define, customize, and control how class instances are pickled and unpickled.

In most cases, no additional code is needed to make instances picklable. By default, pickle will retrieve the class and the attributes of an instance via introspection. When a class instance is unpickled, its :meth:`__init__` method is usually not invoked. The default behaviour first creates an uninitialized instance and then restores the saved attributes. The following code shows an implementation of this behaviour:

def save(obj):
    return (obj.__class__, obj.__dict__)

def load(cls, attributes):
    obj = cls.__new__(cls)
    obj.__dict__.update(attributes)
    return obj

Classes can alter the default behaviour by providing one or several special methods:

Refer to the section :ref:`pickle-state` for more information about how to use the methods :meth:`__getstate__` and :meth:`__setstate__`.

Note

At unpickling time, some methods like :meth:`__getattr__`, :meth:`__getattribute__`, or :meth:`__setattr__` may be called upon the instance. In case those methods rely on some internal invariant being true, the type should implement :meth:`__getnewargs__` to establish such an invariant; otherwise, neither :meth:`__new__` nor :meth:`__init__` will be called.

As we shall see, pickle does not use directly the methods described above. In fact, these methods are part of the copy protocol which implements the :meth:`__reduce__` special method. The copy protocol provides a unified interface for retrieving the data necessary for pickling and copying objects. [3]

Although powerful, implementing :meth:`__reduce__` directly in your classes is error prone. For this reason, class designers should use the high-level interface (i.e., :meth:`__getnewargs__`, :meth:`__getstate__` and :meth:`__setstate__`) whenever possible. We will show, however, cases where using :meth:`__reduce__` is the only option or leads to more efficient pickling or both.

Persistence of External Objects

For the benefit of object persistence, the :mod:`pickle` module supports the notion of a reference to an object outside the pickled data stream. Such objects are referenced by a persistent ID, which should be either a string of alphanumeric characters (for protocol 0) [4] or just an arbitrary object (for any newer protocol).

The resolution of such persistent IDs is not defined by the :mod:`pickle` module; it will delegate this resolution to the user defined methods on the pickler and unpickler, :meth:`persistent_id` and :meth:`persistent_load` respectively.

To pickle objects that have an external persistent id, the pickler must have a custom :meth:`persistent_id` method that takes an object as an argument and returns either None or the persistent id for that object. When None is returned, the pickler simply pickles the object as normal. When a persistent ID string is returned, the pickler will pickle that object, along with a marker so that the unpickler will recognize it as a persistent ID.

To unpickle external objects, the unpickler must have a custom :meth:`persistent_load` method that takes a persistent ID object and returns the referenced object.

Here is a comprehensive example presenting how persistent ID can be used to pickle external objects by reference.

Dispatch Tables

If one wants to customize pickling of some classes without disturbing any other code which depends on pickling, then one can create a pickler with a private dispatch table.

The global dispatch table managed by the :mod:`copyreg` module is available as :data:`copyreg.dispatch_table`. Therefore, one may choose to use a modified copy of :data:`copyreg.dispatch_table` as a private dispatch table.

For example

f = io.BytesIO()
p = pickle.Pickler(f)
p.dispatch_table = copyreg.dispatch_table.copy()
p.dispatch_table[SomeClass] = reduce_SomeClass

creates an instance of :class:`pickle.Pickler` with a private dispatch table which handles the SomeClass class specially. Alternatively, the code

class MyPickler(pickle.Pickler):
    dispatch_table = copyreg.dispatch_table.copy()
    dispatch_table[SomeClass] = reduce_SomeClass
f = io.BytesIO()
p = MyPickler(f)

does the same, but all instances of MyPickler will by default share the same dispatch table. The equivalent code using the :mod:`copyreg` module is

copyreg.pickle(SomeClass, reduce_SomeClass)
f = io.BytesIO()
p = pickle.Pickler(f)

Handling Stateful Objects

Here's an example that shows how to modify pickling behavior for a class. The :class:`TextReader` class opens a text file, and returns the line number and line contents each time its :meth:`readline` method is called. If a :class:`TextReader` instance is pickled, all attributes except the file object member are saved. When the instance is unpickled, the file is reopened, and reading resumes from the last location. The :meth:`__setstate__` and :meth:`__getstate__` methods are used to implement this behavior.

class TextReader:
    """Print and number lines in a text file."""

    def __init__(self, filename):
        self.filename = filename
        self.file = open(filename)
        self.lineno = 0

    def readline(self):
        self.lineno += 1
        line = self.file.readline()
        if not line:
            return None
        if line.endswith('\n'):
            line = line[:-1]
        return "%i: %s" % (self.lineno, line)

    def __getstate__(self):
        # Copy the object's state from self.__dict__ which contains
        # all our instance attributes. Always use the dict.copy()
        # method to avoid modifying the original state.
        state = self.__dict__.copy()
        # Remove the unpicklable entries.
        del state['file']
        return state

    def __setstate__(self, state):
        # Restore instance attributes (i.e., filename and lineno).
        self.__dict__.update(state)
        # Restore the previously opened file's state. To do so, we need to
        # reopen it and read from it until the line count is restored.
        file = open(self.filename)
        for _ in range(self.lineno):
            file.readline()
        # Finally, save the file.
        self.file = file

A sample usage might be something like this:

>>> reader = TextReader("hello.txt")
>>> reader.readline()
'1: Hello world!'
>>> reader.readline()
'2: I am line number two.'
>>> new_reader = pickle.loads(pickle.dumps(reader))
>>> new_reader.readline()
'3: Goodbye!'

Restricting Globals

By default, unpickling will import any class or function that it finds in the pickle data. For many applications, this behaviour is unacceptable as it permits the unpickler to import and invoke arbitrary code. Just consider what this hand-crafted pickle data stream does when loaded:

>>> import pickle
>>> pickle.loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
hello world
0

In this example, the unpickler imports the :func:`os.system` function and then apply the string argument "echo hello world". Although this example is inoffensive, it is not difficult to imagine one that could damage your system.

For this reason, you may want to control what gets unpickled by customizing :meth:`Unpickler.find_class`. Unlike its name suggests, :meth:`find_class` is called whenever a global (i.e., a class or a function) is requested. Thus it is possible to either completely forbid globals or restrict them to a safe subset.

Here is an example of an unpickler allowing only few safe classes from the :mod:`builtins` module to be loaded:

import builtins
import io
import pickle

safe_builtins = {
    'range',
    'complex',
    'set',
    'frozenset',
    'slice',
}

class RestrictedUnpickler(pickle.Unpickler):

    def find_class(self, module, name):
        # Only allow safe classes from builtins.
        if module == "builtins" and name in safe_builtins:
            return getattr(builtins, name)
        # Forbid everything else.
        raise pickle.UnpicklingError("global '%s.%s' is forbidden" %
                                     (module, name))

def restricted_loads(s):
    """Helper function analogous to pickle.loads()."""
    return RestrictedUnpickler(io.BytesIO(s)).load()

A sample usage of our unpickler working has intended:

>>> restricted_loads(pickle.dumps([1, 2, range(15)]))
[1, 2, range(0, 15)]
>>> restricted_loads(b"cos\nsystem\n(S'echo hello world'\ntR.")
Traceback (most recent call last):
  ...
pickle.UnpicklingError: global 'os.system' is forbidden
>>> restricted_loads(b'cbuiltins\neval\n'
...                  b'(S\'getattr(__import__("os"), "system")'
...                  b'("echo hello world")\'\ntR.')
Traceback (most recent call last):
  ...
pickle.UnpicklingError: global 'builtins.eval' is forbidden

As our examples shows, you have to be careful with what you allow to be unpickled. Therefore if security is a concern, you may want to consider alternatives such as the marshalling API in :mod:`xmlrpc.client` or third-party solutions.

Examples

For the simplest code, use the :func:`dump` and :func:`load` functions.

import pickle

# An arbitrary collection of objects supported by pickle.
data = {
    'a': [1, 2.0, 3, 4+6j],
    'b': ("character string", b"byte string"),
    'c': set([None, True, False])
}

with open('data.pickle', 'wb') as f:
    # Pickle the 'data' dictionary using the highest protocol available.
    pickle.dump(data, f, pickle.HIGHEST_PROTOCOL)

The following example reads the resulting pickled data.

import pickle

with open('data.pickle', 'rb') as f:
    # The protocol version used is detected automatically, so we do not
    # have to specify it.
    data = pickle.load(f)

Footnotes

[1]Don't confuse this with the :mod:`marshal` module
[2]The exception raised will likely be an :exc:`ImportError` or an :exc:`AttributeError` but it could be something else.
[3]The :mod:`copy` module uses this protocol for shallow and deep copying operations.
[4]The limitation on alphanumeric characters is due to the fact the persistent IDs, in protocol 0, are delimited by the newline character. Therefore if any kind of newline characters occurs in persistent IDs, the resulting pickle will become unreadable.