Python 3 Patterns & Idioms / src / Metaclasses.rst


Objects are created by other objects: special objects called "classes" that we can set up to spit out objects that are configured to our liking. What creates these special "class" objects, though? Class objects are created by other special objects, called metaclasses.

The default metaclass is called type and in the vast majority of cases it does the right thing. In some situations, however, you can gain leverage by modifying the way that classes are produced -- typically by performing extra actions or injecting code. When this is the case, you can use metaclass programming to modify the way that some of your class objects are created.

It's worth re-emphasizing that in the vast majority of cases, you don't need metaclasses, because it's a fascinating toy and the temptation to use it everywhere can be overwhelming. Some of the examples in this chapter will show both metaclass and non-metaclass solutions to a problem, so you can see that there's usually another (often simpler) approach.

Some of the functionality that was previously only available with metaclasses is now available in a simpler form using class decorators. It is still useful, however, to understand metaclasses, and certain results can still be achieved only through metaclass programming.

Basic Metaclasses

So metaclasses create classes, and classes create instances. Normally when we write a class, the default metaclass type is automatically invoked to create that class, and we aren't even aware that it's happening.

It's possible to explicitly code the metaclass' creation of a class. type called with one argument produces the type information of an existing class; type called with three arguments creates a new class object. The arguments when invoking type are the name of the class, a list of base classes, and a dictionary giving the namespace for the class (all the fields and methods). So the equivalent of:

class C: pass


C = type('C', (), {})

Classes are often referred to as "types," so this reads fairly sensibly: you're calling a function that creates a new type based on its arguments.

We can also add base classes, fields and methods:

# Metaclasses/

def howdy(self, you):
    print("Howdy, " + you)

MyList = type('MyList', (list,), dict(x=42, howdy=howdy))

ml = MyList()

The ability to generate classes programmatically using type opens up some interesting possibilities. Consider the example in the Jython chapter -- all the subclasses in that case were written using repetetive code. We can automate the generation of the subclasses using type:

# Metaclasses/

class Event(object):
    events = [] # static

    def __init__(self, action, time):
        self.action = action
        self.time = time

    def __cmp__ (self, other):
        "So sort() will compare only on time."
        return cmp(self.time, other.time)

    def run(self):
        print("%.2f: %s" % (self.time, self.action))

    def run_events():;
        for e in

def create_mc(description):
    "Create subclass using the 'type' metaclass"
    class_name = "".join(x.capitalize() for x in description.split())
    def __init__(self, time):
        Event.__init__(self, description + " [mc]", time)
    globals()[class_name] = \
        type(class_name, (Event,), dict(__init__ = __init__))

def create_exec(description):
    "Create subclass by exec-ing a string"
    class_name = "".join(x.capitalize() for x in description.split())
    klass = """
class %s(Event):
    def __init__(self, time):
        Event.__init__(self, "%s [exec]", time)
""" % (class_name, description)
    exec klass in globals()

if __name__ == "__main__":
    descriptions = ["Light on", "Light off", "Water on", "Water off",
                    "Thermostat night", "Thermostat day", "Ring bell"]
    initializations = "ThermostatNight(5.00); LightOff(2.00); \
        WaterOn(3.30); WaterOff(4.45); LightOn(1.00); \
        RingBell(7.00); ThermostatDay(6.00)"
    [create_mc(dsc) for dsc in descriptions]
    exec initializations in globals()
    [create_exec(dsc) for dsc in descriptions]
    exec initializations in globals()

The Event base class is the same. The classes are created automatically using the create_mc() function, which takes its description argument and generates a class name from it. Then it defines an __init__() method, which it puts into the namespace dictionary for the type call, producing a new subclass of Event. Note that the resulting class must be inserted into the global namespace, otherwise it will not be seen.

This approach works fine, but then consider the subsequent create_exec() function, which accomplishes the same thing by calling exec on a string defining the class. This will be much easier to understand by the vast majority of the people reading your code: those who do not understand metaclasses.

The Metaclass Hook

So far, we've only used the type metaclass directly. Metaclass programming involves hooking our own operations into the creation of class objects. This is accomplished by:

  1. Writing a subclass of the metaclass type.
  2. Inserting the new metaclass into the class creation process using the metaclass hook.

In Python 2.x, the metaclass hook is a static field in the class called __metaclass__. In the ordinary case, this is not assigned so Python just uses type to create the class. But if you define __metaclass__ to point to a callable that takes four arguments, Python will call __metaclass__() after the initial creation of the class object, passing in the class object, the class name, the list of base classes and the namespace dictionary.

Thus, the basic process of metaclass programming looks like this:

# Metaclasses/
# Two-step metaclass creation in Python 2.x

class SimpleMeta1(type):
    def __init__(cls, name, bases, nmspc):
        super(SimpleMeta1, cls).__init__(name, bases, nmspc)
        cls.uses_metaclass = lambda self : "Yes!"

class Simple1(object):
    __metaclass__ = SimpleMeta1
    def foo(self): pass
    def bar(): pass

simple = Simple1()
print([m for m in dir(simple) if not m.startswith('__')])
# A new method has been injected by the metaclass:
print simple.uses_metaclass()

""" Output:
['bar', 'foo', 'uses_metaclass']

By convention, when defining metaclasses cls is used rather than self as the first argument to all methods except __new__() (which uses mcl, for reasons explained later). cls is the class object that is being modified.

Note that the practice of calling the base-class constructor first (via super()) in the derived-class constructor should be followed with metaclasses as well.

__metaclass__ only needs to be callable, so in Python 2.x it's possible to define __metaclass__ inline:

# Metaclasses/
# Combining the steps for metaclass creation in Python 2.x

class Simple2(object):
    class __metaclass__(type):
        def __init__(cls, name, bases, nmspc):
            # This won't work:
            # super(__metaclass__, cls).__init__(name, bases, nmspc)
            # Less-flexible specific call:
            type.__init__(cls, name, bases, nmspc)
            cls.uses_metaclass = lambda self : "Yes!"

class Simple3(Simple2): pass
simple = Simple3()
print simple.uses_metaclass()

""" Output:

The compiler won't accept the super() call because it says __metaclass__ hasn't been defined, forcing us to use the specific call to type.__init__().

Because it only needs to be callable, it's even possible to define __metaclass__ as a function:

# Metaclasses/
# A function for __metaclass__ in Python 2.x

class Simple4(object):
    def __metaclass__(name, bases, nmspc):
        cls = type(name, bases, nmspc)
        cls.uses_metaclass = lambda self : "Yes!"
        return cls

simple = Simple4()
print simple.uses_metaclass()

""" Output:

As you'll see, Python 3 doesn't allow the syntax of these last two examples. Even so, the above example makes it quite clear what's happening: the class object is created, then modified, then returned.


Or does it?

The Metaclass Hook in Python 3

Python 3 changes the metaclass hook. It doesn't disallow the __metaclass__ field, but it ignores it. Instead, you use a keyword argument in the base-class list:

This means that none of the (clever) alternative ways of defining __metaclass__ directly as a class or function are available in Python 3 [[check this]]. All metaclasses must be defined as separate classes. This is probably just as well, as it makes metaclass programs more consistent and thus easier to read and understand.

Example: Self-Registration of Subclasses

It is sometimes convienient to use inheritance as an organizing mechanism -- each sublclass becomes an element of a group that you work on. For example, the in the Comprehensions chapter, the subclasses of Language were all the languages that needed to be processed. Each Language subclass described specific processing traits for that language.

To solve this problem, consider a system that automatically keeps a list of all of it's "leaf" subclasses (only the classes that have no inheritors). This way we can easily enumerate through all the subtypes:

# Metaclasses/

class ClassSet(set):
    "Simplify printing a set of classes"
    def __str__(self):
        return "(" + ", ".join([c.__name__ for c in self]) + ")"

class RegisterLeafClasses(type):
    def __init__(cls, name, bases, nmspc):
        super(RegisterLeafClasses, cls).__init__(name, bases, nmspc)
        if not hasattr(cls, 'registry'):
            cls.registry = ClassSet()
        cls.registry -= set(bases) # Remove base classes

class Color(object):
    __metaclass__ = RegisterLeafClasses

class Blue(Color): pass
class Red(Color): pass
class Green(Color): pass
class Yellow(Color): pass
class PhthaloBlue(Blue): pass
class CeruleanBlue(Blue): pass

class Shape(object):
    __metaclass__ = RegisterLeafClasses

class Round(Shape): pass
class Square(Shape): pass
class Triangular(Shape): pass
class Boxy(Shape): pass
class Circle(Round): pass
class Ellipse(Round): pass

""" Output:
(Red, Blue, Yellow, Green)
(Red, CeruleanBlue, Yellow, PhthaloBlue, Green)
(Square, Round, Boxy, Triangular)
(Square, Ellipse, Boxy, Circle, Triangular)

Two separate tests are used to show that the registries are independent of each other. Each test shows what happens when another level of leaf classes are added -- the former leaf becomes a base class, and so is removed from the registry.

Using Class Decorators

Using the inspect module

(As in the Comprehensions chapter)

Using __init__ vs. __new__ in Metaclasses

It can be confusing when you see metaclass examples that appear to arbitrarily use __new__ or __init__ -- why choose one over the other?

__new__ is called for the creation of a new class, while __init__ is called after the class is created, to perform additional initialization before the class is handed to the caller:

# Metaclasses/
from pprint import pprint

class Tag1: pass
class Tag2: pass
class Tag3:
    def tag3_method(self): pass

class MetaBase(type):
    def __new__(mcl, name, bases, nmspc):
        return super(MetaBase, mcl).__new__(mcl, name, bases, nmspc)

    def __init__(cls, name, bases, nmspc):
        super(MetaBase, cls).__init__(name, bases, nmspc)

class MetaNewVSInit(MetaBase):
    def __new__(mcl, name, bases, nmspc):
        # First argument is the metaclass ``MetaNewVSInit``
        for x in (mcl, name, bases, nmspc): pprint(x)
        # These all work because the class hasn't been created yet:
        if 'foo' in nmspc: nmspc.pop('foo')
        name += '_x'
        bases += (Tag1,)
        nmspc['baz'] = 42
        return super(MetaNewVSInit, mcl).__new__(mcl, name, bases, nmspc)

    def __init__(cls, name, bases, nmspc):
        # First argument is the class being initialized
        for x in (cls, name, bases, nmspc): pprint(x)
        if 'bar' in nmspc: nmspc.pop('bar') # No effect
        name += '_y' # No effect
        bases += (Tag2,) # No effect
        nmspc['pi'] = 3.14159 # No effect
        super(MetaNewVSInit, cls).__init__(name, bases, nmspc)
        # These do work because they operate on the class object:
        cls.__name__ += '_z'
        cls.__bases__ += (Tag3,)
        cls.e = 2.718

class Test(object):
    __metaclass__ = MetaNewVSInit
    def __init__(self):
    def foo(self): print('foo still here')
    def bar(self): print('bar still here')

t = Test()
print('class name: ' + Test.__name__)
print('base classes: ', [c.__name__ for c in Test.__bases__])
print([m for m in dir(t) if not m.startswith("__")])

""" Output:
<class '__main__.MetaNewVSInit'>
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
 '__metaclass__': <class '__main__.MetaNewVSInit'>,
 '__module__': '__main__',
 'bar': <function bar at 0x7ed70>,
 'foo': <function foo at 0x7ed30>}


<class '__main__.Test_x'>
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
 '__metaclass__': <class '__main__.MetaNewVSInit'>,
 '__module__': '__main__',
 'bar': <function bar at 0x7ed70>,
 'baz': 42}


class name: Test_x_z
('base classes: ', ['object', 'Tag1', 'Tag3'])
['bar', 'baz', 'e', 'tag3_method']
bar still here

The primary difference is that when overriding __new__() you can change things like the 'name', 'bases' and 'namespace' arguments before you call the super constructor and it will have an effect, but doing the same thing in __init__() you won't get any results from the constructor call.

One special case in __new__() is that you can manipulate things like __slots__, but in __init__() you can't.

Note that, since the base-class version of __init__() doesn't make any modifications, it makes sense to call it first, then perform any additional operations. In C++ and Java, the base-class constructor must be called as the first operation in a derived-class constructor, which makes sense because derived-class constructions can then build upon base-class foundations.

In many cases, the choice of __new__() vs __init__() is a style issue and doesn't matter, but because __new__() can do everything and __init__() is slightly more limited, some people just start using __new__() and stick with it. This use can be confusing -- I tend to hunt for the reason that __init__() has been chosen, and if I can't find it wonder whether the author knew what they were doing. I prefer to only use __new__() when it has meaning -- when you must in order to change things that only __new__() can change.

Class Methods and Metamethods

A metamethod can be called from either the metaclass or from the class, but not from an instance. A classmethod can be called from either a class or its instances, but is not part of the metaclass.

(Is a similar relationship true with attributes, or is it different?)

Intercepting Class Creation

This example implements singleton using metaclasses, by overriding the __call__() metamethod, which is invoked when a new instance is created:

class Singleton(type):
    instance = None # I don't think this will work; needs to be
                    # attched to the class.
    def __call__(cls, *args, **kw):
        if not cls.instance:
             cls.instance = super(Singleton, cls).__call__(*args, **kw)
        return cls.instance

class ASingleton(object):
    __metaclass__ == Singleton

a = ASingleton()
b = ASingleton()
assert a is b

By overriding __call__() in the metaclass, the creation of instances are intercepted. Instance creation is bypassed if one already exists.

Further Reading

Excellent step-by-step introduction to metaclasses:
Metaclass intro and comparison of syntax between Python 2.x and 3.x:
David Mertz's metaclass primer:

Three-part in-depth coverage of metaclasses on IBM Developer Works. Quite useful and authoritative:

Michele Simionato's articles on Artima, with special emphasis on the difference between Python 2.x and 3.x metaclasses:

Once you understand the foundations, you can find lots of examples by searching for "metaclass" within the Python Cookbook:

The printed version of the Python Cookbook has far fewer examples than the online version, but the print version has been filtered and edited and so tends to be more authoritative.

Ian Bicking writes about metaclasses:

For more advanced study, the book Putting Metaclasses to Work.