Source

astoptimizer / README

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
+++++++++++++
AST Optimizer
+++++++++++++

astoptimizer is an optimizer for Python code working on the Abstract Syntax
Tree (AST, high-level representration). It does as much work as possible at
compile time.

The compiler is static, it is not a just-in-time (JIT) compiler, and so don't
expect better performances than psyco or PyPy for example. Optimizations
depending on the type of functions arguments cannot be done for examples. Only
optimizations on immutable types (constants) are done.

Website: http://pypi.python.org/pypi/astoptimizer

Source code hosted at: https://bitbucket.org/haypo/astoptimizer


Optimizations
=============

 * Call builtin functions if arguments are constants (need "builtin_funcs"
   feature). Examples:

   - ``len("abc")`` => ``3``
   - ``ord("A")`` => ``65``

 * Call methods of builtin types if the object and arguments are constants.
   Examples:

   - ``u"h\\xe9ho".encode("utf-8")`` => ``b"h\\xc3\\xa9ho"``
   - ``"python2.7".startswith("python")`` => ``True``
   - ``(32).bit_length()`` => ``6``
   - ``float.fromhex("0x1.8p+0")`` => ``1.5``

 * Call functions of math and string modules for functions without
   border effect. Examples:

   - ``math.log(32) / math.log(2)`` => ``5.0``
   - ``string.atoi("5")`` => ``5``

 * Format strings for str%args and print(arg1, arg2, ...) if arguments
   are constants and the format string is valid.
   Examples:

   - ``"x=%s" % 5`` => ``"x=5"``
   - ``print(1.5)`` => ``print("1.5")``

 * Simplify expressions. Examples:

   - ``not(x in y)`` => ``x not in y``
   - ``4 and 5 and x and 6`` => ``x and 6``

 * Optimize loops (range => xrange needs "builtin_funcs" features). Examples:

   - ``while True: pass`` => ``while 1: pass``
   - ``for x in range(3): pass`` => ``for x in (0, 1, 2): pass``
   - ``for x in range(1000): pass`` => ``for x in xrange(1000): pass`` (Python 2)

 * Optimize iterators, list, set and dict comprehension, and generators (need
   "builtin_funcs" feature). Examples:

   - ``iter(set())`` => ``iter(())``
   - ``frozenset("")`` => ``frozenset()``
   - ``(x for x in "abc" if False)`` => ``(None for x in ())``
   - ``(x*2 for x in "abc" if True)`` => ``(x*2 for x in ("a", "b", "c"))``
   - ``list(x for x in iterable)`` => ``list(iterable)``
   - ``tuple(x for x in "abc")`` => ``("a", "b", "c")``
   - ``list(x for x in range(3))`` => ``[0, 1, 2]``
   - ``[x for x in ""]`` => ``[]``
   - ``[x for x in iterable]`` => ``list(iterable)``
   - ``set([x for x in "abc"])`` => ``{"a", "b", "c"}`` (Python 2.7+) or ``set(("a", "b", "c"))``

 * Replace list with tuple (need "builtin_funcs" feature). Examples:

   - ``for x in [1, 2, 3]: pass`` => ``for x in (1, 2, 3): pass``
   - ``x in [1, 2, 3]`` => ``x in (1, 2, 3)``
   - ``list([x, y, z])`` => ``[x, y, z]``
   - ``set([1, 2, 3])`` => ``{1, 2, 3}`` (Python 2.7+)

 * Evaluate unary and binary operators, subscript and comparaison if all
   arguments are constants. Examples:

   - ``1 + 2 * 3`` => ``7``
   - ``not True`` => ``False``
   - ``"abc" * 3`` => ``"abcabcabc"``
   - ``"abcdef"[:3]`` => ``"abc"``
   - ``(2, 7, 3)[1]`` => ``7``
   - ``frozenset("ab") | frozenset("bc")`` => ``frozenset("abc")``
   - ``None is None`` => ``True``
   - ``"2" in "python2.7"`` => ``True``
   - ``x in [1, 2, 3]`` => ``x in {1, 2, 3}`` (Python 3) or ``x in (1, 2, 3)`` (Python 2)
   - ``def f(): return 2 if 4 < 5 else 3`` => ``def f(): return 2``

 * Remove dead code. Examples:

   - ``def f(): return 1; return 2`` => ``def f(): return 1``
   - ``if DEBUG: print("debug")`` => ``pass`` with DEBUG declared as False
   - ``while 0: print("never executed")`` => ``pass``


Use astoptimizer in your project
================================

To enable astoptimizer globally on your project, add the following lines at the
very begining of your application::

    import astoptimizer
    config = astoptimizer.Config('builtin_funcs', 'pythonbin')
    # customize the config here
    astoptimizer.patch_compile(config)

On Python 3.3, imports will then use the patched compile() function and so
all modules will be optimized. With older versions, the compileall module
(ex: compileall.compile_dir()) can be used to compile an application
with optimizations enabled.


Example
=======

Example with the high-level function ``optimize_code``::

    from astoptimizer import optimize_code
    code = "print(1+1)"
    code = optimize_code(code)
    exec(code)

Example the low-level functions ``optimize_ast``::

    from astoptimizer import Config, parse_ast, optimize_ast, compile_ast
    config = Config('builtin_funcs', 'pythonbin')
    code = "print(1+1)"
    tree = parse_ast(code)
    tree = optimize_ast(tree, config)
    code = compile_ast(tree)
    exec(code)

See also ``demo.py`` script.


Configuration
=============

Unsafe optimizations are disabled by default. Use the Config() class to enable
more optimizations.

Features enabled by default:

 * ``"builtin_types"``: methods of bytes, str, unicode, tuple, frozenset, int
   and float types
 * ``"math"``, ``"string"``: constants and functions without border effects of
   the math / string module

Optional features:

 * ``"builtin_funcs"``: builtin functions like abs(), str(), len(), etc. Examples:

   - ``len("abc")`` => ``3``
   - ``ord("A")`` => ``65``
   - ``str(123)`` => ``"123"``

 * ``"pythonbin"``: Enable this feature if the optimized code will be executed by
   the same Python binary: so exactly the same Python version with the same
   build options. Allow to optimize non-BMP unicode strings on Python < 3.3.
   Enable the ``"platform"`` feature. Examples:

   - ``u"\\U0010ffff"[0]`` => ``u"\\udbff"`` or ``u"\\U0010ffff"`` (depending on
     build options, narrow or wide Unicode)
   - ``sys.version_info.major`` => ``2``
   - ``sys.maxunicode`` => ``0x10ffff``

 * ``"pythonenv"``: Enable this feature if you control the environment
   variables (like ``PYTHONOPTIMIZE``) and Python command line options (like
   ``-Qnew``).  On Python 2, allow to optimize int/int. Enable ``"platform"``
   and ``"pythonbin"`` features. Examples:

   - ``__debug__`` => ``True``
   - ``sys.flags.optimize`` => ``0``

 * ``"platform"``: optimizations specific to a platform. Examples:

   - ``sys.platform`` => ``"linux2"``
   - ``sys.byteorder`` => ``"little"``
   - ``sys.maxint`` => ``2147483647``
   - ``os.linesep`` => ``"\\n"``

 * ``"struct"``: struct module, calcsize(), pack() and unpack() functions.

 * ``"cpython_tests"``: disable some optimizations to workaround issues with
   the CPython test suite. Only use it for tests.

Use ``Config("builtin_funcs", "pythonbin")`` to enable most optimizations.  You
may also enable ``"pythonenv"`` to enable all optimizations, but then the
optimized code will depends on environment variables and Python command line
options.


Advices
=======

Advices to help the AST optimizer:

 * Declare your constants using config.add_constant()
 * Declare your pure functions (functions with no border effect) using
   config.add_func()
 * Don't use "from module import \*". If "import \*" is used, builtins
   functions are not optimized anymore for example.


Limitations
===========

 * Operations on mutable values are not optimized, ex: len([1, 2, 3]).
 * Unsafe optimizations are disabled by default. For example, len("\\U0010ffff") is not
   optimized because the result depends on the build options of Python. Enable
   "builtin_funcs" and "pythonenv" features to enable more optimizations.
 * len() is not optimized if the result is bigger than 2^31-1.
   Enable "pythonbin" configuration feature to optimize the call for bigger
   objects.
 * On Python 2, operators taking a bytes string and a unicode string are not
   optimized if the bytes string has to be decoded from the default encoding or
   if the unicode string has to be encoded to the default encoding. Exception:
   pure ASCII strings are optimized. For example, b"abc" + u"def" is replaced
   with u"abcdef", whereas u"x=%s" % b"\\xe9" is not optimized.
 * On Python 3, comparaison between bytes and Unicode strings are not optimized
   because the comparaison may emit a warning or raise a BytesWarning
   exception. Bytes string are not converted to Unicode string. For example,
   b"abc" < "abc" and str(b"abc") are not optimized. Converting a bytes string
   to Unicode is never optimized.


ChangeLog
=========

Version 0.4 (2012-12-10)
------------------------

Bugfixes:

 * Don't replace range() with xrange() if arguments cannot be converted to C
   long
 * Disable float.fromhex() optimization by default: float may be shadowed.
   Use "builtin_funcs" to enable this optimization.

Changes:

 * Add the "struct" configuration feature: functions of the struct module
 * Optimize print() on Python 2 with "from __future__ import print_function"
 * Optimize iterators, list, set and dict comprehension, and generators
 * Replace list with tuple

Version 0.3.1 (2012-09-12)
--------------------------

Bugfixes:

 * Disable optimizations on functions and constants if a variable with the same
   name is set. Example: "len=ord; print(len('A'))",
   "sys.version = 'abc'; print(sys.version)".
 * Don't optimize print() function, frozenset() nor range() functions if
   "builtin_funcs" feature is disabled
 * Don't remove code if it contains global or nonlocal.
   Example: "def f(): if 0: global x; x = 2".

Version 0.3 (2012-09-11)
------------------------

Major changes:

 * Add astoptimizer.patch_compile(config=None) function to simply hook the
   builtin compile() function.
 * Add "pythonbin" configuration feature.
 * Disable optimizations on builtin functions by default. Add "builtin_funcs"
   feature to the configuration to optimize builtin functions.
 * Remove dead code (optionnal optimization)
 * It is now posible to define a callback for warnings of the optimizer
 * Drop support of Python 2.5, it is unable to compile an AST tree to bytecode.
   AST objects of Python 2.5 don't accept arguments in constructors.

Bugfixes:

 * Handle "from math import \*" correctly
 * Don't optimize operations if arguments are bytes and unicode strings.
   Only optimize if string arguments have the same type.
 * Disable optimizations on non-BMP unicode strings by default. Optimizations
   enabled with "pythonbin" feature.

Other changes:

 * More functions, methods and constants:

   - bytes, str, unicode: add more methods.
   - math module: add most remaining functions
   - string module: add some functions and all constants

 * not(a in b) => a not in b, not(a is b) => a is not b
 * a if bool else b
 * for x in range(n) => for x in xrange(n) (only on Python 2)
 * Enable more optimizations if a function is not a generator
 * Add sys.flags.<attr> and sys.version_info.<attr> constants

Version 0.2 (2012-09-02)
------------------------

Major changes:

 * Check input arguments before calling an operator or a function, instead of
   catching errors.
 * New helper functions optimize_code() and optimize_ast() should be used
   instead of using directly the Optimizer class.
 * Support tuple and frozenset types

Changes:

 * FIX: add Config.max_size to check len(obj) result
 * FIX: disable non portable optimizations on non-BMP strings
 * Support Python 2.5-3.3
 * Refactor Optimizer: Optimizer.visit() now always visit children before
   calling the optimizer for a node, except for assignments
 * Float and complex numbers are no more restricted by the integer range of the
   configuration
 * More builtin functions. Examples: divmod(int, int), float(str), min(tuple),
   sum(tuple).
 * More method of builtin types. Examples: str.startswith(), str.find(),
   tuple.count(), float.is_integer().
 * math module: add math.ceil(), math.floor() and math.trunc().
 * More module constants. Examples: os.O_RDONLY, errno.EINVAL,
   socket.SOCK_STREAM.
 * More operators: a not in b, a is b, a is not b, +a.
 * Conversion to string: str(), str % args and print(arg1, arg2, ...).
 * Support import aliases. Examples: "import math as M; print(M.floor(1.5))"
   and "from math import floor as F; print(F(1.5))".
 * Experimental support of variables (disabled by default).

Version 0.1 (2012-08-12)
------------------------

 * First public version (to reserve the name on PyPI!)


See also
========

Faster Python implementations
-----------------------------

 * `PyPy <http://pypy.org/>`_

   - AST optimizer of PyPy:
     `astcompiler/optimize.py <https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/astcompiler/optimize.py>`_

 * `Hotpy <http://code.google.com/p/hotpy/>`_
   and `Hotpy 2 <https://bitbucket.org/markshannon/hotpy_2>`_,
   based on `GVMT <http://code.google.com/p/gvmt/>`_ (Glasgow Virtual
   Machine Toolkit)
 * `numba
   <https://github.com/numba/numba>`_
 * `pymothoa <http://code.google.com/p/pymothoa/>`_ uses LLVM
   ("don't support classes nor exceptions")
 * `WPython <http://code.google.com/p/wpython/>`_: 16-bit word-codes instead of byte-codes
 * `Cython <http://www.cython.org/>`_:

   * `Compiler/Optimize.py
     <https://github.com/cython/cython/blob/master/Cython/Compiler/Optimize.py>`_
   * `Compiler/ParseTreeTransforms.py
     <https://github.com/cython/cython/blob/master/Cython/Compiler/ParseTreeTransforms.py>`_
   * `Compiler/Builtin.py
     <https://github.com/cython/cython/blob/master/Cython/Compiler/Builtin.py>`_
   * `Compiler/Pipeline.py
     <https://github.com/cython/cython/blob/master/Cython/Compiler/Pipeline.py#L123>`_

CPython issues
--------------

 * `Issue #11549 <http://bugs.python.org/issue11549>`_:
   Build-out an AST optimizer, moving some functionality out of the peephole optimizer
 * `Issue #10399 <http://bugs.python.org/issue10399>`_:
   AST Optimization: inlining of function calls
 * `Issue #7682 <http://bugs.python.org/issue7682>`_:
   "Optimisation of if with constant expression":
 * `Issue #2181 <http://bugs.python.org/issue2181>`_:
   optimize out local variables at end of function
 * `Issue #1346238 <http://bugs.python.org/issue1346238>`_:
   A constant folding optimization pass for the AST

Old projects
------------

 * `Unladen Swallow <http://code.google.com/p/unladen-swallow/>`_, fork of
   CPython 2.6, use LLVM. No more maintained

   - `ProjectPlan
     <http://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_
   - `Unladen Swallow Retrospective
     <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_
   - `PEP 3146
     <http://python.org/dev/peps/pep-3146/>`_

 * `psyco <http://psyco.sourceforge.net/>`_: JIT. The author of pysco, Armin
   Rigo, co-created the PyPy project.

AST
---

 * `instrumenting_the_ast.html <http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html>`_
 * `the-internals-of-python-generator-functions-in-the-ast
   <http://tomlee.co/2008/04/the-internals-of-python-generator-functions-in-the-ast/>`_
 * `tlee-ast-optimize branch
   <http://svn.python.org/view/python/branches/tlee-ast-optimize/Python/optimize.c?view=log>`_
 * `ast-optimization-branch-elimination-in-generator-functions
   <http://grokbase.com/p/python/python-dev/0853rf4s1a/ast-optimization-branch-elimination-in-generator-functions>`_

Bytecode
--------

 * `byteplay <http://code.google.com/p/byteplay/>`_
 * `diving-into-byte-code-optimization-in-python
   <http://www.slideshare.net/cjgiridhar/diving-into-byte-code-optimization-in-python>`_
 * `BytecodeAssembler <http://pypi.python.org/pypi/BytecodeAssembler>`_
 * `ByteplayDoc <http://wiki.python.org/moin/ByteplayDoc>`_
 * `Hacking Python bytecode <http://geofft.mit.edu/blog/sipb/73>`_

Other
-----

 * `ASP <http://pypi.python.org/pypi/asp>`_:
   ASP is a SEJITS (specialized embedded just-in-time compiler) toolkit for Python.
 * `PerformanceTips <http://wiki.python.org/moin/PythonSpeed/PerformanceTips/>`_
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.