Source

pypy-effect-analysis / pypy / doc / getting-started.txt

Full commit
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
==================================
PyPy - Getting Started 
==================================

.. contents::
.. sectnum::

.. _howtopypy: 

What is PyPy ?
==============

PyPy is an implementation of the Python_ programming language written in
Python itself, flexible and easy to experiment with.  Our long-term goals are
to target a large variety of platforms, small and large, by providing a
compiler toolsuite that can produce custom Python versions.  Platform, memory
and threading models are to become aspects of the translation process - as
opposed to encoding low level details into the language implementation itself.
Eventually, dynamic optimization techniques - implemented as another
translation aspect - should become robust against language changes. `more...`_

.. _Python: http://docs.python.org/ref
.. _`more...`: architecture.html

Just the facts 
============== 

.. _gettingpypy: 

Downloading & running the PyPy 1.0 release 
--------------------------------------------

Download one of the following release files: 

*pypy-1.0*
    
    * `pypy-1.0.0.tar.bz2`_ (sources, unix line endings) or
    * `pypy-1.0.0.tar.gz`_ (sources, unix line endings) or
    * `pypy-1.0.0.zip`_ (sources, windows line-endings) or
    * `pypy-1.0.0-win32.zip`_ (precompiled executables for windows)
    * or get the `latest stable version via subversion`_.
 
After unpacking the source downloads you can change to the
``pypy-1.0.0`` directory and execute the following command line::

    python pypy/bin/py.py 

This will give you a PyPy prompt, i.e. a very compliant Python
interpreter implemented in Python.  PyPy passes around `98% of
CPythons core language regression tests`_.  Because this invocation of
PyPy still runs on top of CPython, it runs around 2000 times slower
than the original CPython.

However, since the 0.7.0 release it is possible to use PyPy to `translate
itself to lower level languages`_ after which it runs standalone, is not
dependant on CPython anymore and becomes faster.

If you are using the precompiled Windows executables, please look
at the included ``README.txt`` on how to start already translated
interpreters.  Note that the examples in the html documentation generally 
assume that you have a py.py; with precompiled binaries, you need to 
pick one with the matching features compiled in.

.. _`98% of CPythons core language regression tests`: http://codespeak.net/~hpk/pypy-testresult/ 
.. _`pypy-1.0.0.tar.bz2`: http://codespeak.net/download/pypy/pypy-1.0.0.tar.bz2
.. _`pypy-1.0.0.zip`: http://codespeak.net/download/pypy/pypy-1.0.0.zip
.. _`pypy-1.0.0.tar.gz`: http://codespeak.net/download/pypy/pypy-1.0.0.tar.gz
.. _`pypy-1.0.0-win32.zip`: http://codespeak.net/download/pypy/pypy-1.0.0-win32.zip

.. _`latest stable version via subversion`:   

Svn-check out & run the latest PyPy as a two-liner
--------------------------------------------------

If you want to play with the stable development PyPy version 
you can check it out from the repository using subversion. Download 
and install subversion_ if you don't already have it. Then you can
issue on the command line (DOS box or terminal)::

    svn co http://codespeak.net/svn/pypy/dist pypy-dist 

This will create a directory named ``pypy-dist``, and will get you the PyPy
source in ``pypy-dist/pypy`` and documentation files in
``pypy-dist/pypy/doc``.

After checkout you can get a PyPy interpreter via:: 

    python pypy-dist/pypy/bin/py.py

have fun :-) 

We have some `help on installing subversion`_ for PyPy.
Have a look at `interesting starting points`_ 
for some guidance on how to continue. 

.. _`help on installing subversion`: svn-help.html
.. _subversion: svn-help.html

Understanding PyPy's architecture
---------------------------------

For in-depth information about architecture and coding documentation 
head over to the `documentation section`_ where you'll find lots of 
interesting information.  Additionally, in true hacker spirit, you 
may just `start reading sources`_ . 

.. _`documentation section`: index.html 

Running all of PyPy's tests  
---------------------------

If you want to see `if PyPy works on your machine/platform`_
you can simply run PyPy's large test suite with:: 

    cd pypy
    python test_all.py directory-or-files

test_all.py is just another name for `py.test`_ which is the
testing tool that we are using and enhancing for PyPy.
Note that running all the tests takes a very long time, and
enormous amounts of memory if you are trying to run them all
in the same process; test_all.py is only suitable to run a
subset of them at a time.  To run them all we have an
`autotest driver`_ that executes the tests directory by
directory and produces pages like the following one:

    http://wyvern.cs.uni-duesseldorf.de/pypytest/summary.html

.. _`if PyPy works on your machine/platform`: index.html#status
.. _`autotest driver`: http://codespeak.net/pipermail/pypy-dev/2006q3/003273.html

Filing bugs or feature requests 
-------------------------------

You may file `bug reports`_ on our issue tracker which is
also accessible through the 'issues' top menu of 
the PyPy website.  `using the development tracker`_ has 
more detailed information on specific features of the tracker. 

.. _`using the development tracker`: coding-guide.html#using-development-tracker

.. _`interesting starting points`: 

Interesting Starting Points in PyPy 
===================================

The following assumes that you have successfully downloaded and extracted the
PyPy release or have checked out PyPy using svn. It assumes that you are in
the top level directory of the PyPy source tree, e.g. pypy-x.x (if you
got a release) or pypy-dist (if you checked out the most recent version using
subversion).

Main entry point  
------------------------------------------

The py.py interpreter
+++++++++++++++++++++

To start interpreting Python with PyPy, use Python 2.3 or greater::

    cd pypy
    python bin/py.py

After a few seconds (remember: this is running on top of CPython), 
you should be at the PyPy prompt, which is the same as the Python 
prompt, but with an extra ">".

Now you are ready to start running Python code.  Most Python
modules should work if they don't involve CPython extension 
modules.  Here is an example of determining PyPy's performance 
in pystones:: 

    >>>> from test import pystone 
    >>>> pystone.main(10)

The parameter is the number of loops to run through the test. The
default is 50000, which is far too many to run in a non-translated
PyPy version (i.e. when PyPy's interpreter itself is being interpreted 
by CPython). 

py.py options
+++++++++++++

To list the PyPy interpreter command line options, type::

    cd pypy
    python bin/py.py --help

py.py supports most of the options that CPython supports too (in addition to a
large amount of options that can be used to customize py.py).
As an example of using PyPy from the command line, you could type::

    python py.py -c "from test import pystone; pystone.main(10)"

Alternatively, as with regular Python, you can simply give a
script name on the command line::

    python py.py ../../lib-python/2.4.1/test/pystone.py 10

See our  `configuration sections`_ for details about what all the commandline
options do.

Special PyPy features 
-------------------------- 

Interpreter-level console
+++++++++++++++++++++++++

There are quite a few extra features of the PyPy console: If you press
<Ctrl-C> on the console you enter the interpreter-level console, a
usual CPython console.  You can then access internal objects of PyPy
(e.g. the object space) and any variables you have created on the PyPy
prompt with the prefix ``w_``:: 

    >>>> a = 123
    >>>> <Ctrl-C>
    *** Entering interpreter-level console ***
    >>> w_a
    W_IntObject(123)

Note that the prompt of the interpreter-level console is only '>>>' since
it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under
Linux) or <Ctrl-Z>, <Enter> (under Windows).

You may be interested in reading more about the distinction between
`interpreter-level and app-level`_.

.. _`interpreter-level and app-level`: coding-guide.html#interpreter-level

.. _`trace example`: 

Tracing bytecode and operations on objects
++++++++++++++++++++++++++++++++++++++++++ 

You can use the trace object space to monitor the interpretation
of bytecodes in connection with object space operations.  To enable 
it, set ``__pytrace__=1`` on the interactive PyPy console:: 

    >>>> __pytrace__ = 1
    Tracing enabled
    >>>> a = 1 + 2
    |- <<<< enter <inline>a = 1 + 2 @ 1 >>>>
    |- 0    LOAD_CONST    0 (W_IntObject(1))
    |- 3    LOAD_CONST    1 (W_IntObject(2))
    |- 6    BINARY_ADD
      |-    add(W_IntObject(1), W_IntObject(2))   -> W_IntObject(3)
    |- 7    STORE_NAME    0 (a)
      |-    hash(W_StringObject('a'))   -> W_IntObject(-468864544)
      |-    int_w(W_IntObject(-468864544))   -> -468864544
    |-10    LOAD_CONST    2 (<W_NoneObject()>)
    |-13    RETURN_VALUE
    |- <<<< leave <inline>a = 1 + 2 @ 1 >>>>


.. _ `lazily computed objects`:

Lazily computed objects 
+++++++++++++++++++++++

One of the original features provided by PyPy is the "thunk" 
object space, providing lazily-computed objects in a fully
transparent manner::

    cd pypy
    python bin/py.py -o thunk

    >>>> from __pypy__ import thunk
    >>>> def longcomputation(lst):
    ....     print "computing..."
    ....     return sum(lst)
    .... 
    >>>> x = thunk(longcomputation, range(5))
    >>>> y = thunk(longcomputation, range(10))

From the application perspective, ``x`` and ``y`` represent 
exactly the objects being returned by the ``longcomputation()``
invocations.  You can put these objects into a dictionary 
without triggering the computation:: 

    >>>> d = {5: x, 10: y}
    >>>> result = d[5]
    >>>> result
    computing...
    10
    >>>> type(d[10])
    computing...
    <type 'int'>
    >>>> d[10]
    45

It is interesting to note that this lazy-computing Python extension 
is solely implemented in a small `objspace/thunk.py`_ file consisting 
of around 200 lines of code. Since the 0.8.0 release you 
can `translate PyPy with the thunk object space`_.

Logic programming
+++++++++++++++++

People familiar with logic programming languages will be interested to
know that PyPy optionally supports logic variables and constraint-based
programming.  Among the many interesting features of logic programming
-- like unification -- this subsumes the thunk object space by
providing a more extensive way to deal with laziness.

Try it out::

    cd pypy
    python bin/py.py -o logic

    >>>> X = newvar()         # a logic variable
    >>>> bind(X, 42)          # give it a value
    >>>> assert X / 2 == 21   # then use it
    >>>> assert type(X) is int

    >>>> X, Y, Z = newvar(), newvar(), newvar()  # three logic vars
    >>>> unify({'hello': Y, 'world': Z}, X)      # a complex unification
    >>>> bind(Y, 5)                              # then give values to Y
    >>>> bind(Z, 7)                              # ... and Z
    >>>> X
    {'hello': 5, 'world': 7}

    >>>> bind(Z, 8)
    RuntimeError: Cannot bind twice

Read more about `Logic Object space features`_.

.. _`Logic Object space features`: howto-logicobjspace.html


Aspect Oriented Programming
+++++++++++++++++++++++++++

PyPy provides an experimental `Aspect Oriented Programming` facility
in the 'aop' module. Be aware that this module works on the abstract
syntax tree level, and will leave traces in the byte-compiled ('.pyc')
files. Be sure to remove these files after playing with the module if
you do not intend to really use the aspect oriented programming in
your code. Even better, use the '--no-objspace-usepycfiles' command
line option, so that the byte-compiled modules are not written to
disk. 

Here is an example::

 cd pypy
 python bin/py.py --no-objspace-usepycfiles

 >>>> from aop import Aspect, before, PointCut
 >>>> class DemoAspect:
 ....     __metaclass__ = Aspect
 ....     @before(PointCut(func="^open$").call())
 ....     def before_open(self, joinpoint):
 ....         print "opening", joinpoint.arguments()
 .... 
 >>>> aspect = DemoAspect()
 >>>> f = open("/tmp/toto.txt", 'w')

To read more about this, try the `aop module documentation`_.

.. _`Aspect Oriented Programming`: http://aosd.net
.. _`aop module documentation`: aspect_oriented_programming.html

Running the tests
+++++++++++++++++

The PyPy project uses test-driven-development.  Right now, there are
a couple of different categories of tests which you can run.
To run all the unit tests::

    cd pypy
    python test_all.py

(this is not recommended, since it takes hours and uses huge amounts of RAM).
Alternatively, you may run subtests by going to the correct subdirectory
and running them individually::

    python test_all.py interpreter/test/test_pyframe.py

``test_all.py`` is actually just a synonym for `py.test`_ which is 
our external testing tool. If you have installed that you 
can as well just issue ``py.test DIRECTORY_OR_FILE`` in order 
to perform test runs or simply start it without arguments to 
run all tests below the current directory. 

Finally, there are the CPython regression tests which you can 
run like this (this will take hours and hours and hours)::

    cd lib-python/2.4.1/test 
    python ../../../pypy/test_all.py

or if you have `installed py.test`_ then you simply say::

    py.test

from the lib-python/2.4.1/test directory. You need to have a checkout of the
testresult directory. Running one of the above commands tells you how to
proceed.

.. _`installed py.test`: https://codespeak.net/py/current/doc/download.html

Demos
+++++

The `demo/`_ directory contains examples of various aspects of PyPy,
ranging from running regular Python programs (that we used as compliance goals) 
over experimental distribution mechanisms to examples translating 
sufficiently static programs into low level code. 

.. _`try out the translator`:

Trying out the translator
------------------------- 

The translator is a tool based on the PyPy interpreter which can translate
sufficiently static Python programs into low-level code. To be able to use it
you need to:

  * Download and install Pygame_ and ctypes_ if you do not already have them.

  * Have an internet connection. The flowgraph viewer connects to
    codespeak.net and lets it convert the flowgraph by a patched version of
    `Dot Graphviz`_ that does not crash. This is only needed if you want to
    look at the flowgraphs.

  * Use Python-2.4 for using translation tools because python2.5 
    is (as of revision 39130) not fully supported. 

To start the interactive translator shell do::

    cd pypy
    python bin/translatorshell.py

Test snippets of translatable code are provided in the file
``pypy/translator/test/snippet.py``, which is imported under the name
``snippet``.  For example::

    >>> t = Translation(snippet.is_perfect_number)
    >>> t.view()
        
After that, the graph viewer pops up, that lets you interactively inspect the
flow graph. To move around, click on something that you want to inspect.
To get help about how to use it, press 'H'. To close it again, press 'Q'.

Trying out the type annotator
+++++++++++++++++++++++++++++

We have a type annotator that can completely infer types for functions like
``is_perfect_number`` (as well as for much larger examples)::

    >>> t.annotate([int])
    >>> t.view()

Move the mouse over variable names (in red) to see their inferred types.


Translating the flow graph to C code
++++++++++++++++++++++++++++++++++++

The graph can be turned into C code::

   >>> t.rtype()
   >>> f = t.compile_c()

The first command replaces the operations with other low level versions that
only use low level types that are available in C (e.g. int). To try out the
compiled version::

   >>> f(5)
   False
   >>> f(6)
   True

.. _LLVM:

Translating the flow graph to LLVM code
+++++++++++++++++++++++++++++++++++++++

The LLVM or `low level virtual machine`_ project has, among other things,
defined a statically typed portable assembly language and a set of tools that
optimize and compile this assembly for a variety of platforms.  As such, this
assembly is a natural target for PyPy's translator.

To translate to LLVM assembly you must first have `LLVM version 1.9 installed`_
- the `how to install LLVM`_ page provides some helpful hints.

The LLVM backend is not as flexible as the C backend, and for example only
supports one garbage collection strategy.  Calling compiled LLVM code from
CPython is more restrictive than the C backend - the return type and the
arguments of the entry function must be ints, floats or bools - as the emphasis
of the LLVM backend is to compile standalone executables.

Here is a simple example to try::

   >>> t = Translation(snippet.my_gcd)
   >>> a = t.annotate([int, int])
   >>> t.rtype()
   >>> f = t.compile_llvm()
   >>> f(15, 10)
   5


Translating the flow graph to JavaScript code
+++++++++++++++++++++++++++++++++++++++++++++

The JavaScript backend is still experimental but was heavily improved
during last
years `Google summer of code`_. It contains some rudimentary support
for the document object model and a good integration with PyPy's unittesting
framework. Code can be tested with the `Spidermonkey`_ commandline JavaScript
interpreter in addition to a multitude of JavaScript capable browsers. 
The emphasis of the JavaScript backend is to compile RPython code into
JavaScript snippets that can be used in a range of browsers. The goal is
to make it more and more capable to produce full featured web applications.
Please see the pypy/translator/js/test directory for example unittests.

Here is a simple example to try::

   >>> t = Translation(snippet.my_gcd)
   >>> a = t.annotate([int, int])
   >>> source = t.source_js()

If you want to know more about the JavaScript backend please refer to the
`JavaScript docs`_.

.. _`JavaScript docs`: js/whatis.html

Translating the flow graph to CLI code
++++++++++++++++++++++++++++++++++++++

Use the `CLI backend`_ to translate the flow graphs into .NET executables:
``gencli`` is quite mature now and can also compile the whole
interpreter. You can try out the `CLI backend`_ from the interactive
translator shell::

    >>> def myfunc(a, b): return a+b
    ... 
    >>> t = Translation(myfunc)
    >>> t.annotate([int, int])
    >>> f = t.compile_cli()
    >>> f(4, 5)
    9

The object returned by ``compile_cli`` is a wrapper around the real
executable: the parameters are passed as command line arguments, and
the returned value is read from the standard output.

Once you have compiled the snippet, you can also try to launch the
executable directly from the shell; you can find the executable in one
of the ``/tmp/usession-*`` directories::

    $ mono /tmp/usession-<username>/main.exe 4 5
    9

To translate and run for CLI you must have the SDK installed: Windows
users need the `.NET Framework SDK 2.0`_, while Linux and Mac users
can use Mono_.

A slightly larger example
+++++++++++++++++++++++++

There is a small-to-medium demo showing the translator and the annotator::

    cd demo
    python bpnn.py

This causes ``bpnn.py`` to display itself as a call graph and class
hierarchy.  Clicking on functions shows the flow graph of the particular
function.  Clicking on a class shows the attributes of its instances.  All
this information (call graph, local variables' types, attributes of
instances) is computed by the annotator.

As soon as you close the PyGame window, the function is turned into C code,
compiled and executed.

.. _`translate itself to lower level languages`: 

Translating the PyPy Python interpreter
---------------------------------------

(**Note**: for some hints on how to translate the Python interpreter under
Windows, see the `windows document`_)

.. _`windows document`: windows.html

Not for the faint of heart nor the owner of a very old machine: you can
translate the PyPy's whole of Python interpreter to low level C code. This is
the largest and
ultimate example of source that our translation toolchain can process::

    cd pypy/translator/goal
    python translate.py --run targetpypystandalone.py

By default the translation process will try to use the
`Boehm-Demers-Weiser garbage collector`_ for the translated interpreter (Use
``--gc=framework`` to use our own exact mark-n-sweep implementation which
at the moment is slower but doesn't have external dependencies). Otherwise, be sure
to install Boehm before starting the translation (e.g. by running
``apt-get install libgc-dev`` on Debian).

This whole process will take some time and quite a lot of memory (although it
might just work on a machine with 512MB of RAM). If the translation is
finished running and after you closed the graph you will be greeted
(because of ``--run`` option) by the friendly prompt of a PyPy
executable that is not running on top of CPython any more::

    [translation:info] created: ./pypy-c
    [translation:info] Running compiled c source...
    debug: entry point starting
    debug:  argv -> ./pypy-c
    debug: importing code
    debug: calling code.interact()
    Python 2.4.1 (pypy 0.7.1 build 18929) on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    (InteractiveConsole)
    >>>> 1 + 1
    2
    >>>> 

With the default options, you can find the produced executable under the
name ``pypy-c``.  Type ``pypy-c --help`` to see the options it supports --
mainly the same basic options as CPython.  In addition, ``pypy-c --info``
prints the translation options that where used to produce this particular
executable.  This executable contains a lot of things that are hard-coded
for your particular system (including paths), so it's not really meant to
be installed or redistributed at the moment.

If you exit the interpreter you get a pygame window with all the flow graphs
plus a pdb prompt. Moving around in the resulting flow graph is difficult
because of the sheer size of the result. For this reason, the debugger prompt
you get at the end has been enhanced with commands to facilitate locating
functions and classes. Type ``help graphs`` for a list of the new commands.
Help is also available on each of these new commands.

The ``translate.py`` script itself takes a number of options controlling
what to translate and how.  See ``translate.py -h``. Some of the more
interesting options are:

   * ``--text``: don't show the flow graph after the translation is
     done. This is useful if you don't have pygame installed.

   * ``--stackless``: this produces a pypy-c that includes features
     inspired by `Stackless Python <http://www.stackless.com>`__.

   * ``--gc=boehm|ref|framework|stacklessgc``: choose between using
     the `Boehm-Demers-Weiser garbage collector`_, our reference
     counting implementation or our own implementation of a mark and
     sweep collector, with two different approaches for finding roots
     (as we have seen Boehm's collector is the default).

Find a more detailed description of the various options in our `configuration
sections`_.

You can also use the translate.py script to try out several smaller
programs, e.g. a slightly changed version of Pystone::

    cd pypy/translator/goal
    python translate.py targetrpystonedalone

This will produce the executable "targetrpystonedalone-c".

.. _`configuration sections`: config/index.html

.. _`translate PyPy with the thunk object space`:

Translating with the thunk object space
++++++++++++++++++++++++++++++++++++++++


It is also possible to experimentally translate a PyPy version using
the "thunk" object space::

    cd pypy/translator/goal
    python translate.py targetpypystandalone.py --objspace=thunk

the examples in `lazily computed objects`_ should work in the translated
result.

Translating using the LLVM backend
++++++++++++++++++++++++++++++++++


To create a standalone executable using the experimental LLVM_ compiler
infrastructure::

    ./translate.py --text --batch --backend=llvm targetpypystandalone.py

Translating using the CLI backend
+++++++++++++++++++++++++++++++++

To create a standalone .NET executable using the `CLI backend`_::

    ./translate.py --text --batch --backend=cli targetpypystandalone.py

The executable and all its dependecies will be stored in the
./pypy-cli-data directory. To run pypy.NET, you can run
./pypy-cli-data/main.exe. If you are using Linux or Mac, you can use
the convenience ./pypy-cli script::

  $ ./pypy-cli
  debug: entry point starting
  debug:  argv ->
  debug: importing code
  debug: calling code.interact()
  Python 2.4.1 (pypy 0.9.0 build 38134) on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  (InteractiveConsole)
   >>>> 1 + 1
   2
   >>>>

Unfortunately the interactive interpreter will not work with Mono
versions older than 1.1.17.2 due to a Mono bug. Everything else will
work fine, but not the interactive shell.

Moreover, at the moment it's not possible to do the full translation
using only the tools provided by the Microsoft .NET SDK, since
``ilasm`` crashes when trying to assemble the pypy-cli code due to its
size.  Microsoft .NET SDK 2.0.50727.42 is affected by this bug; other
version could be affected as well: if you find a version of the SDK
that works, please tell us.

Windows users that want to compile their own pypy-cli can install
Mono_: if a Mono installation is detected the translation toolchain
will automatically use its ``ilasm2`` tool to assemble the
executables.

Trying the experimental .NET integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can also try the still very experimental clr_ module that
enables integration with the surrounding .NET environment.

You can dynamically load .NET classes using the ``clr.load_cli_class``
method. After a class has been loaded, you can instantiate and use it
as it were a normal Python class. Special methods such as indexers and
properties are supported using the usual Python syntax::

    >>>> import clr
    >>>> ArrayList = clr.load_cli_class('System.Collections', 'ArrayList')
    >>>> obj = ArrayList()
    >>>> obj.Add(1)
    0
    >>>> obj.Add(2)
    1
    >>>> obj.Add("foo")
    2
    >>>> print obj[0], obj[1], obj[2]
    1 2 foo
    >>>> print obj.Count
    3

At the moment the only way to load a .NET class is to explicitly use
``clr.load_cli_class``; in the future they will be automatically
loaded when accessing .NET namespaces as they were Python modules, as
IronPython does.

Translate using the Build Tool
++++++++++++++++++++++++++++++

If you don't have access to the required resources to build PyPy yourself,
or want to build it on a platform you don't have access to, you can use
the 'build tool', a set of components that allow translating and
compiling PyPy on remote servers. The build tool provides a meta server
that manages a set of connected build servers; clients can request a build
by specifying target options and Subversion revision and path, and the meta
server will do its best to get the request fulfilled.
Currently you need to have a codespeak account to be able to use the build tool.

The meta server is made available on codespeak.net and there should always be
a number of build servers running. If you want to request a compilation you need
to checkout the build tool sources from:

    https://codespeak.net/svn/pypy/build/buildtool

Then you can issue::

  $ ./bin/startcompile.py [options] <email>

in the root of the 'buildtool' package replacing '[options]' with
the options you desire and '<email>' with your email address. Type::

  $ ./bin/startcompile.py --help

to see a list of available options.

To donate build server time (we'd be eternally thankful! ;) just run::

  $ ./bin/buildserver.py

without arguments. Do mind that you need to be a registered user on
'codespeak.net', and  that a decent machine with enough RAM is required
(about 1-2 GB is recommended).

Note: currently Windows is not supported.

.. _`start reading sources`: 

Where to start reading the sources
---------------------------------- 

PyPy is made from parts that are relatively independent from each other.
You should start looking at the part that attracts you most (all paths are
relative to the PyPy top level directory).  You may look at our `directory reference`_ 
or start off at one of the following points:

*  `pypy/interpreter`_ contains the bytecode interpreter: bytecode dispatcher
   in pyopcode.py_, frame and code objects in eval.py_ and pyframe.py_,
   function objects and argument passing in function.py_ and argument.py_,
   the object space interface definition in baseobjspace.py_, modules in
   module.py_ and mixedmodule.py_.  Core types supporting the bytecode 
   interpreter are defined in typedef.py_.

*  `pypy/interpreter/pyparser`_ contains a recursive descent parser,
   and input data files that allow it to parse both Python 2.3 and 2.4
   syntax.  Once the input data has been processed, the parser can be
   translated by the above machinery into efficient code.
 
*  `pypy/interpreter/astcompiler`_ contains the compiler.  This
   contains a modified version of the compiler package from CPython
   that fixes some bugs and is translatable.  That the compiler and
   parser are translatable is new in 0.8.0 and it makes using the
   resulting binary interactively much more pleasant.

*  `pypy/objspace/std`_ contains the `Standard object space`_.  The main file
   is objspace.py_.  For each type, the files ``xxxtype.py`` and
   ``xxxobject.py`` contain respectively the definition of the type and its
   (default) implementation.

*  `pypy/objspace`_ contains a few other object spaces: the thunk_,
   trace_ and flow_ object spaces.  The latter is a relatively short piece
   of code that builds the control flow graphs when the bytecode interpreter
   runs in it.

*  `pypy/translator`_ contains the code analysis and generation stuff.
   Start reading from translator.py_, from which it should be easy to follow
   the pieces of code involved in the various translation phases.

*  `pypy/annotation`_ contains the data model for the type annotation that
   can be inferred about a graph.  The graph "walker" that uses this is in
   `pypy/annotation/annrpython.py`_.

*  `pypy/rpython`_ contains the code of the RPython typer. The typer transforms
   annotated flow graphs in a way that makes them very similar to C code so
   that they can be easy translated. The graph transformations are controlled
   by the stuff in `pypy/rpython/rtyper.py`_. The object model that is used can
   be found in `pypy/rpython/lltypesystem/lltype.py`_. For each RPython type
   there is a file rxxxx.py that contains the low level functions needed for
   this type.

.. _optionaltool: 

Additional Tools for running (and hacking) PyPy 
-----------------------------------------------

We use some optional tools for developing PyPy. They are not required to run 
the basic tests or to get an interactive PyPy prompt but they help to
understand  and debug PyPy especially for the ongoing translation work.  

graphviz & pygame for flow graph viewing (highly recommended)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

graphviz and pygame are both necessary if you
want to look at generated flow graphs: 

	graphviz: http://www.graphviz.org/Download.php 

	pygame: http://www.pygame.org/download.shtml

CTypes (highly recommended)
++++++++++++++++++++++++++++

`ctypes`_ (version 0.9.9.6 or later) is required if you want to translate PyPy
to C. See the `download page of ctypes`_.

.. _`download page of ctypes`: http://sourceforge.net/project/showfiles.php?group_id=71702
.. _`ctypes`: http://starship.python.net/crew/theller/ctypes/


CLISP
+++++++

The CLISP backend is optional and not quite up-to-date with the rest of
PyPy. Still there are a few examples you can try our backend out on. 
Here is a link to a LISP implementation that should basically work: 

    http://clisp.cons.org/ 


.. _`py.test`:  

py.test and the py lib 
+++++++++++++++++++++++

The `py library`_ is used for supporting PyPy development and 
running our tests against code and documentation as well as
compliance tests.  You don't need to install the py library because 
it ships with PyPy and `pypy/test_all.py`_ is an alias for ``py.test``
but if you want to have the ``py.test`` tool generally in your 
path, you might like to visit: 

    http://codespeak.net/py/dist/download.html

Getting involved 
================================== 

PyPy employs an open development process.  You are invited to join our
`pypy-dev mailing list`_ or look at the other `contact
possibilities`_.  We are also doing coding Sprints which are
separately announced and often happen around Python conferences such
as EuroPython or Pycon. Take a look at the list of upcoming events_ to
plan where to meet with us.

.. _events: http://codespeak.net/pypy/dist/pypy/doc/news.html
.. _`pypy-dev mailing list`: http://codespeak.net/mailman/listinfo/pypy-dev
.. _`contact possibilities`: contact.html

.. _`py library`: http://codespeak.net/py 
.. _`PyPy/LLVM backend`: translation.html#llvm 
.. _`low level virtual machine`: http://llvm.org/
.. _`how to install LLVM`: http://llvm.org/docs/GettingStarted.html
.. _`LLVM mailing list`: http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev
.. _`LLVM version 1.9 installed`: http://llvm.org/releases

.. _`Spidermonkey`: http://www.mozilla.org/js/spidermonkey/
.. _`Google summer of code`: http://code.google.com/soc

.. _`.NET Framework SDK 2.0`: http://msdn.microsoft.com/netframework/downloads/updates/default.aspx
.. _Mono: http://www.mono-project.com/Main_Page
.. _`CLI backend`: cli-backend.html
.. _clr: clr-module.html

.. _Dot Graphviz:           http://www.graphviz.org/
.. _Pygame:                 http://www.pygame.org/
.. _pyopcode.py:            http://codespeak.net/svn/pypy/dist/pypy/interpreter/pyopcode.py
.. _eval.py:                http://codespeak.net/svn/pypy/dist/pypy/interpreter/eval.py
.. _pyframe.py:             http://codespeak.net/svn/pypy/dist/pypy/interpreter/pyframe.py
.. _function.py:            http://codespeak.net/svn/pypy/dist/pypy/interpreter/function.py
.. _argument.py:            http://codespeak.net/svn/pypy/dist/pypy/interpreter/argument.py
.. _baseobjspace.py:        http://codespeak.net/svn/pypy/dist/pypy/interpreter/baseobjspace.py
.. _module.py:              http://codespeak.net/svn/pypy/dist/pypy/interpreter/module.py
.. _mixedmodule.py:          http://codespeak.net/svn/pypy/dist/pypy/interpreter/mixedmodule.py
.. _typedef.py:             http://codespeak.net/svn/pypy/dist/pypy/interpreter/typedef.py
.. _Standard object space:  objspace.html#the-standard-object-space
.. _objspace.py:            ../../pypy/objspace/std/objspace.py
.. _thunk:                  ../../pypy/objspace/thunk.py
.. _trace:                  ../../pypy/objspace/trace.py
.. _flow:                   ../../pypy/objspace/flow/
.. _translator.py:          ../../pypy/translator/translator.py
.. _mailing lists:          contact.html
.. _documentation:          index.html 
.. _unit tests:             coding-guide.html#test-design
.. _bug reports:            https://codespeak.net/issue/pypy-dev/

.. _`directory reference`: index.html#directory-reference
.. _`Boehm-Demers-Weiser garbage collector`: http://www.hpl.hp.com/personal/Hans_Boehm/gc/

.. include:: _ref.txt