Source

Python 3 Patterns & Idioms / html / _sources / Metaclasses.txt

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
.. index::
   Metaclasses
   class decorators

********************************************************************************
Metaclasses
********************************************************************************

Objects are created by other objects: special objects called "classes"
that we can set up to spit out objects that are configured to our
liking. What creates these special "class" objects, though? Class
objects are created by other special objects, called metaclasses.

The default metaclass is called ``type`` and in the vast majority of
cases it does the right thing. In some situations, however, you can
gain leverage by modifying the way that classes are produced --
typically by performing extra actions or injecting code. When this is
the case, you can use *metaclass* programming to modify the way that
some of your class objects are created.

It's worth re-emphasizing that in *the vast majority of cases, you
don't need metaclasses*, because it's a fascinating toy and the
temptation to use it everywhere can be overwhelming. Some of the
examples in this chapter will show both metaclass and non-metaclass
solutions to a problem, so you can see that there's usually another
(often simpler) approach.

Some of the functionality that was previously only available with
metaclasses is now available in a simpler form using class
decorators. It is still useful, however, to understand metaclasses,
and certain results can still be achieved only through metaclass
programming.

Basic Metaclasses
================================================================================

So metaclasses create classes, and classes create instances. Normally
when we write a class, the default metaclass ``type`` is automatically
invoked to create that class, and we aren't even aware that it's happening. 

It's possible to explicitly code the metaclass' creation of a
class. ``type`` called with one argument produces the type information
of an existing class; ``type`` called with three arguments creates a
new class object. The arguments when invoking ``type`` are the name of the class,
a list of base classes, and a dictionary giving the namespace for the
class (all the fields and methods). So the equivalent of::

    class C: pass

is::

    C = type('C', (), {})

Classes are often referred to as "types," so this reads fairly
sensibly: you're calling a function that creates a new type based on
its arguments.

We can also add base classes, fields and methods::

    # Metaclasses/MyList.py

    def howdy(self, you):
        print("Howdy, " + you)

    MyList = type('MyList', (list,), dict(x=42, howdy=howdy))

    ml = MyList()
    ml.append("Camembert")
    print(ml)
    print(ml.x)
    ml.howdy("John")

    """ Output:
    ['Camembert']
    42
    Howdy, John
    """

The ability to generate classes programmatically using ``type`` opens
up some interesting possibilities. Consider the GreenHouseLanguage.py
example in the Jython chapter -- all the subclasses in that case were
written using repetetive code. We can automate the generation of the
subclasses using ``type``::

    # Metaclasses/GreenHouse.py

    class Event(object):
        events = [] # static

        def __init__(self, action, time):
            self.action = action
            self.time = time
            Event.events.append(self)

        def __cmp__ (self, other):
            "So sort() will compare only on time."
            return cmp(self.time, other.time)

        def run(self):
            print("%.2f: %s" % (self.time, self.action))

        @staticmethod
        def run_events():
            Event.events.sort();
            for e in Event.events:
                e.run()

    def create_mc(description):
        "Create subclass using the 'type' metaclass"
        class_name = "".join(x.capitalize() for x in description.split())
        def __init__(self, time):
            Event.__init__(self, description + " [mc]", time)
        globals()[class_name] = \
            type(class_name, (Event,), dict(__init__ = __init__))

    def create_exec(description):
        "Create subclass by exec-ing a string"
        class_name = "".join(x.capitalize() for x in description.split())
        klass = """
    class %s(Event):
        def __init__(self, time):
            Event.__init__(self, "%s [exec]", time)
    """ % (class_name, description)
        exec klass in globals()

    if __name__ == "__main__":
        descriptions = ["Light on", "Light off", "Water on", "Water off", 
                        "Thermostat night", "Thermostat day", "Ring bell"]
        initializations = "ThermostatNight(5.00); LightOff(2.00); \
            WaterOn(3.30); WaterOff(4.45); LightOn(1.00); \
            RingBell(7.00); ThermostatDay(6.00)"
        [create_mc(dsc) for dsc in descriptions]
        exec initializations in globals()
        [create_exec(dsc) for dsc in descriptions]
        exec initializations in globals()
        Event.run_events()

    """ Output:
    1.00: Light on [mc]
    1.00: Light on [exec]
    2.00: Light off [mc]
    2.00: Light off [exec]
    3.30: Water on [mc]
    3.30: Water on [exec]
    4.45: Water off [mc]
    4.45: Water off [exec]
    5.00: Thermostat night [mc]
    5.00: Thermostat night [exec]
    6.00: Thermostat day [mc]
    6.00: Thermostat day [exec]
    7.00: Ring bell [mc]
    7.00: Ring bell [exec]
    """

The ``Event`` base class is the same. The classes are created
automatically using the ``create_mc()`` function, which takes its
``description`` argument and generates a class name from it. Then it
defines an ``__init__()`` method, which it puts into the namespace
dictionary for the ``type`` call, producing a new subclass of
``Event``. Note that the resulting class must be inserted into the
global namespace, otherwise it will not be seen.

This approach works fine, but then consider the subsequent
``create_exec()`` function, which accomplishes the same thing by
calling ``exec`` on a string defining the class. This will be much
easier to understand by the vast majority of the people reading your
code: those who do not understand metaclasses.

The Metaclass Hook
================================================================================

So far, we've only used the ``type`` metaclass directly. Metaclass
programming involves hooking our own operations into the creation of
class objects. This is accomplished by:

      1. Writing a subclass of the metaclass ``type``.
      2. Inserting the new metaclass into the class creation process
         using the *metaclass hook*.

In Python 2.x, the metaclass hook is a static field in the class
called ``__metaclass__``. In the ordinary case, this is not assigned
so Python just uses ``type`` to create the class. But if you define
``__metaclass__`` to point to a callable that takes four arguments,
Python will call ``__metaclass__()`` after the initial creation of the
class object, passing in the class object, the class name, the list of
base classes and the namespace dictionary.

Thus, the basic process of metaclass programming looks like this::

    # Metaclasses/SimpleMeta1.py
    # Two-step metaclass creation in Python 2.x

    class SimpleMeta1(type):
        def __init__(cls, name, bases, nmspc):
            super(SimpleMeta1, cls).__init__(name, bases, nmspc)
            cls.uses_metaclass = lambda self : "Yes!"

    class Simple1(object):
        __metaclass__ = SimpleMeta1
        def foo(self): pass
        @staticmethod
        def bar(): pass

    simple = Simple1()
    print([m for m in dir(simple) if not m.startswith('__')])
    # A new method has been injected by the metaclass:
    print simple.uses_metaclass()

    """ Output:
    ['bar', 'foo', 'uses_metaclass']
    Yes!
    """

By convention, when defining metaclasses ``cls`` is used rather than
``self`` as the first argument to all methods except ``__new__()``
(which uses ``mcl``, for reasons explained later). ``cls``
is the class object that is being modified.

Note that the practice of calling the base-class constructor first (via
``super()``) in the derived-class constructor should be followed with
metaclasses as well.

``__metaclass__`` only needs to be callable, so in Python
2.x it's possible to define ``__metaclass__`` inline::

    # Metaclasses/SimpleMeta2.py
    # Combining the steps for metaclass creation in Python 2.x

    class Simple2(object):
        class __metaclass__(type):
            def __init__(cls, name, bases, nmspc):
                # This won't work:
                # super(__metaclass__, cls).__init__(name, bases, nmspc)
                # Less-flexible specific call:
                type.__init__(cls, name, bases, nmspc)
                cls.uses_metaclass = lambda self : "Yes!"

    class Simple3(Simple2): pass
    simple = Simple3()
    print simple.uses_metaclass()

    """ Output:
    Yes!
    """

The compiler won't accept the ``super()`` call because it says
``__metaclass__`` hasn't been defined, forcing us to use the specific
call to ``type.__init__()``. 

Because it only needs to be callable, it's even possible to define
``__metaclass__`` as a function::

    # Metaclasses/SimpleMeta3.py
    # A function for __metaclass__ in Python 2.x

    class Simple4(object):
        def __metaclass__(name, bases, nmspc):
            cls = type(name, bases, nmspc)
            cls.uses_metaclass = lambda self : "Yes!"
            return cls

    simple = Simple4()
    print simple.uses_metaclass()

    """ Output:
    Yes!
    """

As you'll see, Python 3 doesn't allow the syntax of these last two
examples. Even so, the above example makes it quite clear what's
happening: the class object is created, then modified, then returned.

.. Note:: Or does it allow that syntax?


The Metaclass Hook in Python 3
----------------------------------------------------------------------

Python 3 changes the metaclass hook. It doesn't disallow the
``__metaclass__`` field, but it ignores it. Instead, you use a keyword
argument in the base-class list:

This means that none of the (clever) alternative ways of defining
``__metaclass__`` directly as a class or function are available in
Python 3 [[check this]]. All metaclasses must be defined as separate
classes. This is probably just as well, as it makes metaclass programs
more consistent and thus easier to read and understand.




.. Possible example: simplification of XML creation via operator
   overloading.


Example: Self-Registration of Subclasses
================================================================================

It is sometimes convienient to use inheritance as an organizing
mechanism -- each sublclass becomes an element of a group that you
work on. For example, in **CodeManager.py** in the **Comprehensions**
chapter, the subclasses of **Language** were all the languages that
needed to be processed. Each **Language** subclass described specific
processing traits for that language.

To solve this problem, consider a system that automatically keeps a
list of all of it's "leaf" subclasses (only the classes that have no
inheritors). This way we can easily enumerate through all the
subtypes::

    # Metaclasses/RegisterLeafClasses.py

    class ClassSet(set):
        "Simplify printing a set of classes"
        def __str__(self):
            return "(" + ", ".join([c.__name__ for c in self]) + ")"

    class RegisterLeafClasses(type):
        def __init__(cls, name, bases, nmspc):
            super(RegisterLeafClasses, cls).__init__(name, bases, nmspc)
            if not hasattr(cls, 'registry'):
                cls.registry = ClassSet()
            cls.registry.add(cls)
            cls.registry -= set(bases) # Remove base classes

    class Color(object):
        __metaclass__ = RegisterLeafClasses

    class Blue(Color): pass
    class Red(Color): pass
    class Green(Color): pass
    class Yellow(Color): pass
    print(Color.registry)
    class PhthaloBlue(Blue): pass
    class CeruleanBlue(Blue): pass
    print(Color.registry)

    class Shape(object):
        __metaclass__ = RegisterLeafClasses

    class Round(Shape): pass
    class Square(Shape): pass
    class Triangular(Shape): pass
    class Boxy(Shape): pass
    print(Shape.registry)
    class Circle(Round): pass
    class Ellipse(Round): pass
    print(Shape.registry)

    """ Output:
    (Red, Blue, Yellow, Green)
    (Red, CeruleanBlue, Yellow, PhthaloBlue, Green)
    (Square, Round, Boxy, Triangular)
    (Square, Ellipse, Boxy, Circle, Triangular)
    """

Two separate tests are used to show that the registries are
independent of each other. Each test shows what happens when another
level of leaf classes are added -- the former leaf becomes a base
class, and so is removed from the registry.

Using Class Decorators
--------------------------------------------------------------------------------

Using the **inspect** module
--------------------------------------------------------------------------------

(As in the Comprehensions chapter)

Example: Making a Class "Final"
================================================================================

It is sometimes convenient to prevent a class from being inherited::

    # Metaclasses/Final.py
    # Emulating Java's 'final'

    class final(type):
        def __init__(cls, name, bases, namespace):
            for klass in bases:
                if isinstance(klass, final):
                    raise TypeError(str(klass.__name__) + " is final")
            super(final, cls).__init__(name, bases, namespace)

    class A(object):
        pass

    class B(A):
        __metaclass__= final

    # Produces compile-time error:
    class C(B):
        pass

    """ Output:
    ...
    TypeError: B is final
    """

.. can this be done with decorators?


Using ``__init__`` vs. ``__new__`` in Metaclasses
================================================================================

It can be confusing when you see metaclass examples that appear to
arbitrarily use ``__new__`` or ``__init__`` -- why choose one over the other?

``__new__`` is called for the creation of a new class, while
``__init__`` is called after the class is created, to perform
additional initialization before the class is handed to the caller::

    # Metaclasses/NewVSInit.py
    from pprint import pprint

    class Tag1: pass
    class Tag2: pass
    class Tag3:
        def tag3_method(self): pass

    class MetaBase(type):
        def __new__(mcl, name, bases, nmspc):
            print('MetaBase.__new__\n')
            return super(MetaBase, mcl).__new__(mcl, name, bases, nmspc)

        def __init__(cls, name, bases, nmspc):
            print('MetaBase.__init__\n')
            super(MetaBase, cls).__init__(name, bases, nmspc)

    class MetaNewVSInit(MetaBase):
        def __new__(mcl, name, bases, nmspc):
            # First argument is the metaclass ``MetaNewVSInit``
            print('MetaNewVSInit.__new__')
            for x in (mcl, name, bases, nmspc): pprint(x)
            print('')
            # These all work because the class hasn't been created yet:
            if 'foo' in nmspc: nmspc.pop('foo')
            name += '_x'
            bases += (Tag1,)
            nmspc['baz'] = 42
            return super(MetaNewVSInit, mcl).__new__(mcl, name, bases, nmspc)

        def __init__(cls, name, bases, nmspc):
            # First argument is the class being initialized
            print('MetaNewVSInit.__init__')
            for x in (cls, name, bases, nmspc): pprint(x)
            print('')
            if 'bar' in nmspc: nmspc.pop('bar') # No effect
            name += '_y' # No effect
            bases += (Tag2,) # No effect
            nmspc['pi'] = 3.14159 # No effect
            super(MetaNewVSInit, cls).__init__(name, bases, nmspc)
            # These do work because they operate on the class object:
            cls.__name__ += '_z'
            cls.__bases__ += (Tag3,)
            cls.e = 2.718

    class Test(object):
        __metaclass__ = MetaNewVSInit
        def __init__(self):
            print('Test.__init__')
        def foo(self): print('foo still here')
        def bar(self): print('bar still here')

    t = Test()
    print('class name: ' + Test.__name__)
    print('base classes: ', [c.__name__ for c in Test.__bases__])
    print([m for m in dir(t) if not m.startswith("__")])
    t.bar()
    print(t.e)

    """ Output:
    MetaNewVSInit.__new__
    <class '__main__.MetaNewVSInit'>
    'Test'
    (<type 'object'>,)
    {'__init__': <function __init__ at 0x7ecf0>,
     '__metaclass__': <class '__main__.MetaNewVSInit'>,
     '__module__': '__main__',
     'bar': <function bar at 0x7ed70>,
     'foo': <function foo at 0x7ed30>}

    MetaBase.__new__

    MetaNewVSInit.__init__
    <class '__main__.Test_x'>
    'Test'
    (<type 'object'>,)
    {'__init__': <function __init__ at 0x7ecf0>,
     '__metaclass__': <class '__main__.MetaNewVSInit'>,
     '__module__': '__main__',
     'bar': <function bar at 0x7ed70>,
     'baz': 42}

    MetaBase.__init__

    Test.__init__
    class name: Test_x_z
    ('base classes: ', ['object', 'Tag1', 'Tag3'])
    ['bar', 'baz', 'e', 'tag3_method']
    bar still here
    2.718
    """


The primary difference is that when overriding ``__new__()`` you can change
things like the 'name', 'bases' and 'namespace' arguments before you
call the super constructor and it will have an effect, but doing the
same thing in ``__init__()`` you won't get any results from the constructor
call.

One special case in ``__new__()`` is that you can
manipulate things like ``__slots__``, but in ``__init__()`` you can't.

Note that, since the base-class version of ``__init__()`` doesn't make any
modifications, it makes sense to call it first, then perform any
additional operations. In C++ and Java, the base-class constructor
*must* be called as the first operation in a derived-class
constructor, which makes sense because derived-class constructions can
then build upon base-class foundations.

In many cases, the choice of ``__new__()`` vs ``__init__()`` is a style issue and
doesn't matter, but because ``__new__()`` can do everything and ``__init__()`` is
slightly more limited, some people just start using ``__new__()`` and stick with
it. This use can be confusing -- I tend to hunt for the reason that
``__init__()`` has been chosen, and if I can't find it wonder whether
the author knew what they were doing. I prefer to only use ``__new__()``
when it has meaning -- when you must in order to change things that
only ``__new__()`` can change. 

Class Methods and Metamethods
================================================================================

A metamethod can be called from either the metaclass or from the
class, but not from an instance. A classmethod can be called from
either a class or its instances, but is not part of the metaclass.

(Is a similar relationship true with attributes, or is it different?)

Intercepting Class Creation
--------------------------------------------------------------------------------

This example implements *Singleton* using metaclasses, by overriding the
``__call__()`` metamethod, which is invoked when a new instance is
created::

    # Metaclasses/Singleton.py

    class Singleton(type):
        instance = None
        def __call__(cls, *args, **kw):
            if not cls.instance:
                 cls.instance = super(Singleton, cls).__call__(*args, **kw)
            return cls.instance

    class ASingleton(object):
        __metaclass__ = Singleton

    a = ASingleton()
    b = ASingleton()
    assert a is b
    print(a.__class__.__name__, b.__class__.__name__)

    class BSingleton(object):
        __metaclass__ = Singleton

    c = BSingleton()
    d = BSingleton()
    assert c is d
    print(c.__class__.__name__, d.__class__.__name__)
    assert c is not a

    """ Output:
    ('ASingleton', 'ASingleton')
    ('BSingleton', 'BSingleton')
    """

By overriding ``__call__()`` in the metaclass, the creation of
instances are intercepted. Instance creation is bypassed if one
already exists.

Note the dependence upon the behavior of static class fields. When
``cls.instance`` is first read, it gets the static value of
``instance`` from the metaclass, which is ``None``. However, when the
assignment is made, Python creates a local version for the particular
class, and the next time ``cls.instance`` is read, it sees that local
version. Because of this behavior, each class ends up with its own
class-specific ``instance`` field (thus ``instance`` is not somehow
being "inherited" from the metaclass).

.. class decorator version that modifies __new__().

Metaclass Conflicts
================================================================================

http://code.activestate.com/recipes/204197/

Further Reading
================================================================================

    Excellent step-by-step introduction to metaclasses:
        http://cleverdevil.org/computing/78/

    Metaclass intro and comparison of syntax between Python 2.x and 3.x:
        http://mikewatkins.ca/2008/11/29/python-2-and-3-metaclasses/

    David Mertz's metaclass primer:
        http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html

    Three-part in-depth coverage of metaclasses on IBM Developer Works. Quite useful and authoritative:
      - http://www.ibm.com/developerworks/linux/library/l-pymeta.html
      - http://www.ibm.com/developerworks/linux/library/l-pymeta2/
      - http://www.ibm.com/developerworks/linux/library/l-pymeta3.html

    Michele Simionato's articles on Artima, with special emphasis on the difference between Python 2.x and 3.x metaclasses:
      - http://www.artima.com/weblogs/viewpost.jsp?thread=236234
      - http://www.artima.com/weblogs/viewpost.jsp?thread=236260

    Once you understand the foundations, you can find lots of examples
    by searching for "metaclass" within the Python Cookbook:
    http://code.activestate.com/recipes/langs/python/

    The printed version of the Python Cookbook has far fewer examples
    than the online version, but the print version has been filtered
    and edited and so tends to be more authoritative.

    Ian Bicking writes about metaclasses:
      - http://blog.ianbicking.org/a-conservative-metaclass.html
      - http://blog.ianbicking.org/metaclass-fun.html
      - http://blog.ianbicking.org/A-Declarative-Syntax-Extension.html
      - http://blog.ianbicking.org/self-take-two.html

    For more advanced study, the book `Putting Metaclasses to Work
    <http://www.pearsonhighered.com/educator/academic/product/0,,0201433052,00%2ben-USS_01DBC.html>`_.
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.