pypy / pypy / doc / windows.rst

 David Malcolm 47ed8f9 2011-03-16 David Malcolm 1e46012 2011-03-14 Armin Rigo a5b60f7 2013-06-02 David Malcolm 1e46012 2011-03-14 Armin Rigo 422233e 2013-08-21 David Malcolm 1e46012 2011-03-14 mattip 7f02f3e 2012-12-05 David Malcolm 1e46012 2011-03-14 kris...@kristjan… 428cdec 2011-04-29 mattip 861ddd5 2012-03-21 kris...@kristjan… 428cdec 2011-04-29 mattip d270e84 2012-04-24 kris...@kristjan… 428cdec 2011-04-29 mattip 861ddd5 2012-03-21 kris...@kristjan… 428cdec 2011-04-29 David Malcolm 1e46012 2011-03-14 Maciej Fijalkows… 3512ed4 2011-07-22 Amaury Forgeot d… 3540a8f 2011-07-22 Maciej Fijalkows… 3512ed4 2011-07-22 Amaury Forgeot d… 3540a8f 2011-07-22 Maciej Fijalkows… 3512ed4 2011-07-22 David Malcolm 1e46012 2011-03-14 mattip 9446c0b 2013-01-05 David Malcolm 1e46012 2011-03-14 mattip 9446c0b 2013-01-05 Armin Rigo 422233e 2013-08-21 mattip 9446c0b 2013-01-05 David Malcolm 1e46012 2011-03-14 mattip 0bd57f2 2013-04-05 Armin Rigo 422233e 2013-08-21 mattip 0bd57f2 2013-04-05 Armin Rigo 422233e 2013-08-21 mattip 0bd57f2 2013-04-05 David Malcolm 1e46012 2011-03-14 mattip d270e84 2012-04-24 David Malcolm 1e46012 2011-03-14 mattip d270e84 2012-04-24 David Malcolm 1e46012 2011-03-14 mattip 861ddd5 2012-03-21 David Malcolm 1e46012 2011-03-14 mattip 54f773d 2012-08-19 mattip 9d969f5 2012-03-21 David Malcolm 1e46012 2011-03-14 mattip 9d969f5 2012-03-21 mattip 861ddd5 2012-03-21 mattip 57206e9 2012-03-21 mattip 9d969f5 2012-03-21 David Malcolm 1e46012 2011-03-14 mattip 57206e9 2012-03-21 mattip 861ddd5 2012-03-21 David Malcolm 1e46012 2011-03-14 mattip 861ddd5 2012-03-21 mattip 57206e9 2012-03-21 mattip 861ddd5 2012-03-21 mattip 57206e9 2012-03-21 mattip 861ddd5 2012-03-21 Armin Rigo 94def3a 2013-06-03 mattip 9d969f5 2012-03-21 Armin Rigo 699be64 2013-06-03 mattip 861ddd5 2012-03-21 mattip 54f773d 2012-08-19 mattip 861ddd5 2012-03-21 Armin Rigo 422233e 2013-08-21 mattip 861ddd5 2012-03-21 David Malcolm 1e46012 2011-03-14 lac 1250ec0 2011-04-29 Armin Rigo 422233e 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo 422233e 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo 422233e 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo 422233e 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo 422233e 2013-08-21 Armin Rigo c6dd660 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo 422233e 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo c6dd660 2013-08-21 Armin Rigo 422233e 2013-08-21 Armin Rigo 8c97e81 2013-08-23 Armin Rigo c6dd660 2013-08-21   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 =============== PyPy on Windows =============== PyPy is supported on Windows platforms, starting with Windows 2000. The following text gives some hints about how to translate the PyPy interpreter. PyPy supports only being translated as a 32bit program, even on 64bit Windows. See at the end of this page for what is missing for a full 64bit translation. To build pypy-c you need a C compiler. Microsoft Visual Studio is preferred, but can also use the mingw32 port of gcc. Translating PyPy with Visual Studio ----------------------------------- We routinely test the RPython translation toolchain_ using Visual Studio 2008, Express Edition. Other configurations may work as well. The translation scripts will set up the appropriate environment variables for the compiler, so you do not need to run vcvars before translation. They will attempt to locate the same compiler version that was used to build the Python interpreter doing the translation. Failing that, they will pick the most recent Visual Studio compiler they can find. In addition, the target architecture (32 bits, 64 bits) is automatically selected. A 32 bit build can only be built using a 32 bit Python and vice versa. By default pypy is built using the Multi-threaded DLL (/MD) runtime environment. **Note:** PyPy is currently not supported for 64 bit Windows, and translation will fail in this case. The compiler is all you need to build pypy-c, but it will miss some modules that relies on third-party libraries. See below how to get and build them. Preping Windows for the Large Build ----------------------------------- Normally 32bit programs are limited to 2GB of memory on Windows. It is possible to raise this limit, to 3GB on Windows 32bit, and almost 4GB on Windows 64bit. On Windows 32bit, it is necessary to modify the system: follow http://usa.autodesk.com/adsk/servlet/ps/dl/item?siteID=123112&id=9583842&linkID=9240617 to enable the "3GB" feature, and reboot. This step is not necessary on Windows 64bit. Then you need to execute:: editbin /largeaddressaware pypy.exe on the pypy.exe file you compiled. Installing external packages ---------------------------- On Windows, there is no standard place where to download, build and install third-party libraries. We recommend installing them in the parent directory of the pypy checkout. For example, if you installed pypy in d:\pypy\trunk\ (This directory contains a README file), the base directory is d:\pypy. You must then set the INCLUDE, LIB and PATH (for DLLs) environment variables appropriately. Abridged method (for -Ojit builds using Visual Studio 2008) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download the versions of all the external packages from https://bitbucket.org/pypy/pypy/downloads/local.zip Then expand it into the base directory (base_dir) and modify your environment to reflect this:: set PATH=\bin;%PATH% set INCLUDE=\include;%INCLUDE% set LIB=\lib;%LIB% Now you should be good to go. Read on for more information. The Boehm garbage collector ~~~~~~~~~~~~~~~~~~~~~~~~~~~ This library is needed if you plan to use the --gc=boehm translation option (this is the default at some optimization levels like -O1, but unneeded for high-performance translations like -O2). You may get it at http://www.hpl.hp.com/personal/Hans_Boehm/gc/gc_source/gc-7.1.tar.gz Versions 7.0 and 7.1 are known to work; the 6.x series won't work with pypy. Unpack this folder in the base directory. Then open a command prompt:: cd gc-7.1 nmake -f NT_THREADS_MAKEFILE copy Release\gc.dll The zlib compression library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download http://www.gzip.org/zlib/zlib-1.2.3.tar.gz and extract it in the base directory. Then compile:: cd zlib-1.2.3 nmake -f win32\Makefile.msc copy zlib1.dll \zlib.dll The bz2 compression library ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download http://bzip.org/1.0.5/bzip2-1.0.5.tar.gz and extract it in the base directory. Then compile:: cd bzip2-1.0.5 nmake -f makefile.msc The sqlite3 database library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Download http://www.sqlite.org/2013/sqlite-amalgamation-3071601.zip and extract it into a directory under the base directory. Also get http://www.sqlite.org/2013/sqlite-dll-win32-x86-3071601.zip and extract the dll into the bin directory, and the sqlite3.def into the sources directory. Now build the import library so cffi can use the header and dll:: lib /DEF:sqlite3.def" /OUT:sqlite3.lib" copy sqlite3.lib path\to\libs The expat XML parser ~~~~~~~~~~~~~~~~~~~~ Download the source code of expat on sourceforge: http://sourceforge.net/projects/expat/ and extract it in the base directory. Version 2.1.0 is known to pass tests. Then open the project file expat.dsw with Visual Studio; follow the instruction for converting the project files, switch to the "Release" configuration, reconfigure the runtime for Multi-threaded DLL (/MD) and build the solution (the expat project is actually enough for pypy). Then, copy the file win32\bin\release\libexpat.dll somewhere in your PATH. The OpenSSL library ~~~~~~~~~~~~~~~~~~~ OpenSSL needs a Perl interpreter to configure its makefile. You may use the one distributed by ActiveState, or the one from cygwin. In both case the perl interpreter must be found on the PATH. Get http://www.openssl.org/source/openssl-0.9.8k.tar.gz and extract it in the base directory. Then compile:: perl Configure VC-WIN32 ms\do_ms.bat nmake -f ms\nt.mak install Using the mingw compiler ------------------------ You can compile pypy with the mingw compiler, using the --cc=mingw32 option; gcc.exe must be on the PATH. If the -cc flag does not begin with "ming", it should be the name of a valid gcc-derivative compiler, i.e. x86_64-w64-mingw32-gcc for the 64 bit compiler creating a 64 bit target. You probably want to set the CPATH, LIBRARY_PATH, and PATH environment variable to the header files, lib or dlls, and dlls respectively of the locally installed packages if they are not in the mingw directory heirarchy. libffi for the mingw compiler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To enable the _rawffi (and ctypes) module, you need to compile a mingw version of libffi. Here is one way to do this, wich should allow you to try to build for win64 or win32: #. Download and unzip a mingw32 build_ or mingw64 build_, say into c:\mingw #. If you do not use cygwin, you will need msys to provide make, autoconf tools and other goodies. #. Download and unzip a msys for mingw_, say into c:\msys #. Edit the c:\msys\etc\fstab file to mount c:\mingw #. Download and unzip the libffi source files_, and extract them in the base directory. #. Run c:\msys\msys.bat or a cygwin shell which should make you feel better since it is a shell prompt with shell tools. #. From inside the shell, cd to the libffi directory and do:: sh ./configure make cp .libs/libffi-5.dll If you can't find the dll, and the libtool issued a warning about "undefined symbols not allowed", you will need to edit the libffi Makefile in the toplevel directory. Add the flag -no-undefined to the definition of libffi_la_LDFLAGS If you wish to experiment with win64, you must run configure with flags:: sh ./configure --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 or such, depending on your mingw64 download. hacking on PyPy with the mingw compiler ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since hacking on PyPy means running tests, you will need a way to specify the mingw compiler when hacking (as opposed to translating). As of March 2012, --cc is not a valid option for pytest.py. However if you set an environment variable CC to the compliter exe, testing will use it. .. _mingw32 build: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win32/Automated%20Builds .. _mingw64 build: http://sourceforge.net/projects/mingw-w64/files/Toolchains%20targetting%20Win64/Automated%20Builds .. _msys for mingw: http://sourceforge.net/projects/mingw-w64/files/External%20binary%20packages%20%28Win64%20hosted%29/MSYS%20%2832-bit%29 .. _libffi source files: http://sourceware.org/libffi/ .. _RPython translation toolchain: translation.html What is missing for a full 64-bit translation --------------------------------------------- The main blocker is that we assume that the integer type of RPython is large enough to (occasionally) contain a pointer value cast to an integer. The simplest fix is to make sure that it is so, but it will give the following incompatibility between CPython and PyPy on Win64: CPython: sys.maxint == 2**32-1, sys.maxsize == 2**64-1 PyPy: sys.maxint == sys.maxsize == 2**64-1 ...and, correspondingly, PyPy supports ints up to the larger value of sys.maxint before they are converted to long. The first decision that someone needs to make is if this incompatibility is reasonable. Assuming that it is, the first thing to do is probably to hack *CPython* until it fits this model: replace the field in PyIntObject with a long long field, and change the value of sys.maxint. This might just work, even if half-brokenly: I'm sure you can crash it because of the precision loss that undoubtedly occurs everywhere, but try not to. :-) Such a hacked CPython is what you'll use in the next steps. We'll call it CPython64/64. It is probably not too much work if the goal is only to get a translated PyPy executable, and to run all tests before transaction. But you need to start somewhere, and you should start with some tests in rpython/translator/c/test/, like test_standalone.py and test_newgc.py: try to have them pass on top of CPython64/64. Keep in mind that this runs small translations, and some details may go wrong. The most obvious one is to check that it produces C files that use the integer type Signed --- but what is Signed defined to? It should be equal to long on every other platforms, but on Win64 it should be something like long long. What is more generally needed is to review all the C files in rpython/translator/c/src for the word long, because this means a 32-bit integer even on Win64. Replace it with Signed most of the times. You can replace one with the other without breaking anything on any other platform, so feel free to. Then, these two C types have corresponding RPython types: rffi.LONG and lltype.Signed respectively. The first should really correspond to the C long. Add tests that check that integers casted to one type or the other really have 32 and 64 bits respectively, on Win64. Once these basic tests work, you need to review rpython/rlib/ for usages of rffi.LONG versus lltype.Signed. The goal would be to fix some more LONG-versus-Signed issues, by fixing the tests --- as always run on top of CPython64/64. Note that there was some early work done in rpython/rlib/rarithmetic with the goal of running all the tests on Win64 on the regular CPython, but I think by now that it's a bad idea. Look only at CPython64/64. The major intermediate goal is to get a translation of PyPy with -O2 with a minimal set of modules, starting with --no-allworkingmodules; you need to use CPython64/64 to run this translation too. Check carefully the warnings of the C compiler at the end. I think that MSVC is "nice" in the sense that by default a lot of mismatches of integer sizes are reported as warnings. Then you need to review pypy/module/*/ for LONG-versus-Signed issues. At some time during this review, we get a working translated PyPy on Windows 64 that includes all --translationmodules, i.e. everything needed to run translations. When we are there, the hacked CPython64/64 becomes much less important, because we can run future translations on top of this translated PyPy. As soon as we get there, please *distribute* the translated PyPy. It's an essential component for anyone else that wants to work on Win64! We end up with a strange kind of dependency --- we need a translated PyPy in order to translate a PyPy ---, but I believe it's ok here, as Windows executables are supposed to never be broken by newer versions of Windows. Happy hacking :-)