Commits

Vinay Sajip  committed dc2c843 Merge

Merged upstream changes.

  • Participants
  • Parent commits edda4b3, 0a0efb7

Comments (0)

Files changed (56)

 .gdb_history
 .purify
+.project
+.cproject
 .svn/
 Makefile$
 Makefile.pre$
 config.status
 config.status.lineno
 db_home
+description-pak
+doc-pak/
 platform$
 pyconfig.h$
 python$
 libpython*.so*
 *.swp
 *.o
+*.deb
 *.pyc
 *.pyo
 *.pyd
 *.cover
+*.tgz
 *~
 Lib/_sysconfigdata.py
 Lib/lib2to3/*.pickle

File Doc/library/development.rst

    unittest.rst
    2to3.rst
    test.rst
+   venv.rst

File Doc/library/venv.rst

+:mod:`venv` --- Creation of virtual environments
+================================================
+
+.. module:: venv
+   :synopsis: Creation of virtual environments.
+.. moduleauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
+.. sectionauthor:: Vinay Sajip <vinay_sajip@yahoo.co.uk>
+
+
+.. index:: pair: Environments; virtual
+
+.. versionadded:: 3.3
+
+**Source code:** :source:`Lib/venv.py`
+
+--------------
+
+The :mod:`venv` module provides support for creating lightweight
+"virtual environments" with their own site directories, optionally
+isolated from system site directories.  Each virtual environment has
+its own Python binary (allowing creation of environments with various
+Python versions) and can have its own independent set of installed
+Python packages in its site directories.
+
+Creating virtual environments
+-----------------------------
+
+Creation of virtual environments is simplest executing the ``pyvenv``
+script::
+
+    pyvenv /path/to/new/virtual/environment
+
+Running this command creates the target directory (creating any parent
+directories that don't exist already) and places a ``pyvenv.cfg`` file
+in it with a ``home`` key pointing to the Python installation the
+command was run from.  It also creates a ``bin`` (or ``Scripts`` on
+Windows) subdirectory containing a copy of the ``python`` binary (or
+binaries, in the case of Windows) and the ``pysetup3`` script (to
+facilitate easy installation of packages from PyPI into the new virtualenv).
+It also creates an (initially empty) ``lib/pythonX.Y/site-packages``
+subdirectory (on Windows, this is ``Lib\site-packages``).
+
+.. highlight:: none
+
+On Windows, you may have to invoke the ``pyvenv`` script as follows, if you
+don't have the relevant PATH and PATHEXT settings::
+
+    c:\Temp>c:\Python33\python c:\Python33\Tools\Scripts\pyvenv.py myenv
+
+or equivalently::
+
+    c:\Temp>c:\Python33\python -m venv myenv
+
+The command, if run with ``-h``, will show the available options::
+
+    usage: pyvenv [-h] [--no-distribute] [--no-site-packages] [--clear]
+                  ENV_DIR [ENV_DIR ...]
+
+    Creates virtual Python environments in one or more target directories.
+
+    positional arguments:
+      ENV_DIR             A directory to create the environment in.
+
+    optional arguments:
+      -h, --help             show this help message and exit
+      --no-distribute        Don't install Distribute in the virtual environment.
+      --system-site-packages Give access to the global site-packages dir to the
+                             virtual environment.
+      --clear                Delete the environment directory if it already exists.
+                             If not specified and the directory exists, an error is
+                             raised.
+
+.. note::
+
+   The current version allows Distribute to be optionally installed within
+   environments, so facilitate installation of existing packages and
+   experimentation with virtual environments. This functionality will be
+   removed from the standard library before the release of Python 3.3: it is
+   expected that third-party tools using this module will fill the gap.
+
+
+If the target directory already exists an error will be raised, unless
+the ``--clear`` option was provided, in which case the target
+directory will be deleted and virtual environment creation will
+proceed as usual.
+
+The created ``pyvenv.cfg`` file also includes the
+``include-system-site-packages`` key, set to ``true`` if ``venv`` is
+run with the ``--use-site-packages`` option, ``false`` otherwise.
+
+Multiple paths can be given to ``pyvenv``, in which case an identical
+virtualenv will be created, according to the given options, at each
+provided path.
+
+
+API
+---
+
+The high-level method described above makes use of a simple API which provides
+mechanisms for third-party virtual environment creators to customize
+environment creation according to their needs.
+
+The :class:`EnvBuilder` class accepts the following keyword arguments on
+instantiation:
+
+   * ``system_site_packages`` - A Boolean value indicating that the
+     system Python site-packages should be available to the
+     environment (defaults to ``False``).
+
+   * ``clear`` - A Boolean value which, if True, will delete any
+     existing target directory instead of raising an exception
+     (defaults to ``False``).
+
+   * ``use_symlinks`` - A Boolean value indicating whether to attempt
+     to symlink the Python binary (and any necessary DLLs or other
+     binaries, e.g. ``pythonw.exe``), rather than copying. Defaults to
+     ``True`` on Linux and Unix systems, but ``False`` on Windows and
+     Mac OS X.
+
+The returned env-builder is an object which has a method, ``create``,
+which takes as required argument the path (absolute or relative to the current
+directory) of the target directory which is to contain the virtual environment.
+The ``create`` method will either create the environment in the specified
+directory, or raise an appropriate exception.
+
+Creators of third-party virtual environment tools will be free to use
+the provided ``EnvBuilder`` class as a base class.
+
+.. highlight:: python
+
+The ``venv`` module will also provide a module-level function as a
+convenience::
+
+    def create(env_dir,
+               system_site_packages=False, clear=False, use_symlinks=None):
+        builder = EnvBuilder(
+            system_site_packages=system_site_packages, clear=clear)
+        builder.create(env_dir)
+
+The ``create`` method of the ``EnvBuilder`` class illustrates the
+hooks available for customization::
+
+    def create(self, env_dir):
+        """
+        Create a virtualized Python environment in a directory.
+
+        :param env_dir: The target directory to create an environment in.
+
+        """
+        env_dir = os.path.abspath(env_dir)
+        context = self.create_directories(env_dir)
+        self.create_configuration(context)
+        self.setup_python(context)
+        self.setup_packages(context)
+        self.setup_scripts(context)
+
+Each of the methods ``create_directories``, ``create_configuration``,
+``setup_python``, ``setup_packages`` and ``setup_scripts`` can be
+overridden.  The functions of these methods are:
+
+   * ``create_directories`` - creates the environment directory and
+     all necessary directories, and returns a context object. This is
+     just a holder for attributes (such as paths), for use by the
+     other methods.
+
+   * ``create_configuration`` - creates the ``pyvenv.cfg``
+     configuration file in the environment.
+
+   * ``setup_python`` - creates a copy of the Python executable (and,
+     under Windows, DLLs) in the environment.
+
+   * ``setup_packages`` - A placeholder method which can be overridden
+     in third party implementations to pre-install packages in the
+     virtual environment.
+
+   * ``setup_scripts`` - A placeholder methd which can be overridden
+     in third party implementations to pre-install scripts (such as
+     activation and deactivation scripts) in the virtual environment.
+
+The ``DistributeEnvBuilder`` subclass in the reference implementation
+illustrates how these last two methods can be used in practice. It's
+not envisaged that ``DistributeEnvBuilder`` will be actually added to
+Python core, but it makes the reference implementation more
+immediately useful for testing and exploratory purposes.
+
+   * The ``setup_packages`` method installs Distribute in the target
+     environment. This is needed at the moment in order to actually
+     install most packages in an environment, since most packages are
+     not yet packaging / setup.cfg based.
+
+   * The ``setup_scripts`` method installs activation and pysetup3
+     scripts in the environment. This is also done in a configurable
+     way: A ``scripts`` property on the builder is expected to provide
+     a buffer which is a base64-encoded zip file. The zip file
+     contains directories "common", "nt", and "posix", each
+     containing scripts destined for the bin directory in the
+     environment. The contents of "common" and the directory
+     corresponding to the platform are copied after doing some
+     text replacement of placeholders:
+
+        * ``__VIRTUAL_ENV__`` is replaced with absolute path of the
+          environment directory.
+
+        * ``__VIRTUAL_PROMPT__`` is replaced with the environment
+          prompt prefix.
+
+        * ``__BIN_NAME__`` is replaced with the name of the bin
+          directory.
+
+        * ``__ENV_PYTHON__`` is replaced with the absolute path of the
+          environment's executable.
+
+

File Lib/compileall.py

             try:
                 ok = py_compile.compile(fullname, cfile, dfile, True,
                                         optimize=optimize)
+                if not quiet:
+                    print('Mode of {} is {:o}'.format(cfile,
+                           os.stat(cfile).st_mode & 0o777))
             except py_compile.PyCompileError as err:
                 if quiet:
                     print('*** Error compiling {!r}...'.format(fullname))

File Lib/distutils/command/build_ext.py

         # for extensions under windows use different directories
         # for Release and Debug builds.
         # also Python's library directory must be appended to library_dirs
+        exec_prefix = sys.site_exec_prefix
         if os.name == 'nt':
             # the 'libs' directory is for binary installs - we assume that
             # must be the *native* platform.  But we don't really support
             # cross-compiling via a binary install anyway, so we let it go.
-            self.library_dirs.append(os.path.join(sys.exec_prefix, 'libs'))
+            self.library_dirs.append(os.path.join(exec_prefix, 'libs'))
             if self.debug:
                 self.build_temp = os.path.join(self.build_temp, "Debug")
             else:
 
             # Append the source distribution include and library directories,
             # this allows distutils on windows to work in the source tree
-            self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC'))
+            self.include_dirs.append(os.path.join(exec_prefix, 'PC'))
             if MSVC_VERSION == 9:
                 # Use the .lib files for the correct architecture
                 if self.plat_name == 'win32':
                 else:
                     # win-amd64 or win-ia64
                     suffix = self.plat_name[4:]
-                new_lib = os.path.join(sys.exec_prefix, 'PCbuild')
+                new_lib = os.path.join(exec_prefix, 'PCbuild')
                 if suffix:
                     new_lib = os.path.join(new_lib, suffix)
                 self.library_dirs.append(new_lib)
 
             elif MSVC_VERSION == 8:
-                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                self.library_dirs.append(os.path.join(exec_prefix,
                                          'PC', 'VS8.0'))
             elif MSVC_VERSION == 7:
-                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                self.library_dirs.append(os.path.join(exec_prefix,
                                          'PC', 'VS7.1'))
             else:
-                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                self.library_dirs.append(os.path.join(exec_prefix,
                                          'PC', 'VC6'))
 
         # OS/2 (EMX) doesn't support Debug vs Release builds, but has the
         # import libraries in its "Config" subdirectory
         if os.name == 'os2':
-            self.library_dirs.append(os.path.join(sys.exec_prefix, 'Config'))
+            self.library_dirs.append(os.path.join(exec_prefix, 'Config'))
 
         # for extensions under Cygwin and AtheOS Python's library directory must be
         # appended to library_dirs
         if sys.platform[:6] == 'cygwin' or sys.platform[:6] == 'atheos':
-            if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
+            if sys.executable.startswith(os.path.join(exec_prefix, "bin")):
                 # building third party extensions
-                self.library_dirs.append(os.path.join(sys.prefix, "lib",
+                prefix = sys.site_prefix
+                self.library_dirs.append(os.path.join(prefix, "lib",
                                                       "python" + get_python_version(),
                                                       "config"))
             else:
         sysconfig.get_config_var('Py_ENABLE_SHARED')
         if (sys.platform.startswith(('linux', 'gnu', 'sunos'))
             and sysconfig.get_config_var('Py_ENABLE_SHARED')):
-            if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
+            if sys.executable.startswith(os.path.join(exec_prefix, "bin")):
                 # building third party extensions
                 self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
             else:

File Lib/distutils/command/install.py

                     raise DistutilsOptionError(
                           "must not supply exec-prefix without prefix")
 
-                self.prefix = os.path.normpath(sys.prefix)
-                self.exec_prefix = os.path.normpath(sys.exec_prefix)
+                prefix = sys.site_prefix
+                exec_prefix = sys.site_exec_prefix
+                self.prefix = os.path.normpath(prefix)
+                self.exec_prefix = os.path.normpath(exec_prefix)
 
             else:
                 if self.exec_prefix is None:
             self.select_scheme("unix_home")
         else:
             if self.prefix is None:
-                self.prefix = os.path.normpath(sys.prefix)
+                prefix = sys.site_prefix
+                self.prefix = os.path.normpath(prefix)
 
             self.install_base = self.install_platbase = self.prefix
             try:

File Lib/distutils/filelist.py

File contents unchanged.

File Lib/distutils/sysconfig.py

 from .errors import DistutilsPlatformError
 
 # These are needed in a couple of spots, so just compute them once.
+# NOTE: On OS X, just getting sys.site_XXX fails during regrtest execution,
+# in test_importhooks.
+# It needs to be investigated as to why this failure only seems to
+# happen on OS X, but for now we've resorted to getattr(sys, 'site_xxx', sys.xxx)
+SITE_PREFIX = os.path.normpath(sys.site_prefix)
+SITE_EXEC_PREFIX = os.path.normpath(sys.site_exec_prefix)
 PREFIX = os.path.normpath(sys.prefix)
 EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
 
     sys.exec_prefix -- i.e., ignore 'plat_specific'.
     """
     if prefix is None:
-        prefix = plat_specific and EXEC_PREFIX or PREFIX
+        if standard_lib:
+            prefix = plat_specific and EXEC_PREFIX or PREFIX
+        else:
+            prefix = plat_specific and SITE_EXEC_PREFIX or SITE_PREFIX
 
     if os.name == "posix":
         libpython = os.path.join(prefix,
         # Normalized versions of prefix and exec_prefix are handy to have;
         # in fact, these are the standard versions used most places in the
         # Distutils.
-        _config_vars['prefix'] = PREFIX
-        _config_vars['exec_prefix'] = EXEC_PREFIX
+        _config_vars['prefix'] = SITE_PREFIX
+        _config_vars['exec_prefix'] = SITE_EXEC_PREFIX
 
         # Convert srcdir into an absolute path if it appears necessary.
         # Normally it is relative to the build directory.  However, during

File Lib/packaging/command/__init__.py

File contents unchanged.

File Lib/packaging/command/bdist_archive.py

+"""Create an executable installer for Windows."""
+
+import sys
+import os
+
+from shutil import rmtree
+from sysconfig import get_python_version
+from packaging.command.cmd import Command
+from packaging.errors import PackagingOptionError, PackagingPlatformError
+from packaging import logger
+from packaging.util import get_platform
+
+
+class bdist_archive(Command):
+
+    description = "create an executable installer for Windows"
+
+    user_options = [('bdist-dir=', None,
+                     "temporary directory for creating the distribution"),
+                    ('plat-name=', 'p',
+                     "platform name to embed in generated filenames "
+                     "(default: %s)" % get_platform()),
+                    ('keep-temp', 'k',
+                     "keep the pseudo-installation tree around after " +
+                     "creating the distribution archive"),
+                    ('target-version=', None,
+                     "require a specific python version" +
+                     " on the target system"),
+                    ('no-target-compile', 'c',
+                     "do not compile .py to .pyc on the target system"),
+                    ('no-target-optimize', 'o',
+                     "do not compile .py to .pyo (optimized)"
+                     "on the target system"),
+                    ('dist-dir=', 'd',
+                     "directory to put final built distributions in"),
+                    ('bitmap=', 'b',
+                     "bitmap to use for the installer instead of python-powered logo"),
+                    ('title=', 't',
+                     "title to display on the installer background instead of default"),
+                    ('skip-build', None,
+                     "skip rebuilding everything (for testing/debugging)"),
+                    ('install-script=', None,
+                     "basename of installation script to be run after"
+                     "installation or before deinstallation"),
+                    ('pre-install-script=', None,
+                     "Fully qualified filename of a script to be run before "
+                     "any files are installed.  This script need not be in the "
+                     "distribution"),
+                    ('user-access-control=', None,
+                     "specify Vista's UAC handling - 'none'/default=no "
+                     "handling, 'auto'=use UAC if target Python installed for "
+                     "all users, 'force'=always use UAC"),
+                   ]
+
+    boolean_options = ['keep-temp', 'no-target-compile', 'no-target-optimize',
+                       'skip-build']
+
+    def initialize_options(self):
+        self.bdist_dir = None
+        self.plat_name = None
+        self.keep_temp = False
+        self.no_target_compile = False
+        self.no_target_optimize = False
+        self.target_version = None
+        self.dist_dir = None
+        self.bitmap = None
+        self.title = None
+        self.skip_build = None
+        self.install_script = None
+        self.pre_install_script = None
+        self.user_access_control = None
+
+
+    def finalize_options(self):
+        self.set_undefined_options('bdist', 'skip_build')
+
+        if self.bdist_dir is None:
+            if self.skip_build and self.plat_name:
+                # If build is skipped and plat_name is overridden, bdist will
+                # not see the correct 'plat_name' - so set that up manually.
+                bdist = self.distribution.get_command_obj('bdist')
+                bdist.plat_name = self.plat_name
+                # next the command will be initialized using that name
+            bdist_base = self.get_finalized_command('bdist').bdist_base
+            self.bdist_dir = os.path.join(bdist_base, 'wininst')
+
+        if not self.target_version:
+            self.target_version = ""
+
+        if not self.skip_build and self.distribution.has_ext_modules():
+            short_version = get_python_version()
+            if self.target_version and self.target_version != short_version:
+                raise PackagingOptionError("target version can only be %s, or the '--skip-build'" \
+                      " option must be specified" % (short_version,))
+            self.target_version = short_version
+
+        self.set_undefined_options('bdist', 'dist_dir', 'plat_name')
+
+        if self.install_script:
+            for script in self.distribution.scripts:
+                if self.install_script == os.path.basename(script):
+                    break
+            else:
+                raise PackagingOptionError("install_script '%s' not found in scripts" % \
+                      self.install_script)
+
+    def run(self):
+        if (sys.platform != "win32" and
+            (self.distribution.has_ext_modules() or
+             self.distribution.has_c_libraries())):
+            raise PackagingPlatformError \
+                  ("distribution contains extensions and/or C libraries; "
+                   "must be compiled on a Windows 32 platform")
+
+        if not self.skip_build:
+            self.run_command('build')
+
+        install = self.get_reinitialized_command('install_dist',
+                                                 reinit_subcommands=True)
+        install.root = self.bdist_dir
+        install.skip_build = self.skip_build
+        install.warn_dir = False
+        install.plat_name = self.plat_name
+
+        install_lib = self.get_reinitialized_command('install_lib')
+        # we do not want to include pyc or pyo files
+        install_lib.compile = False
+        install_lib.optimize = 0
+
+        if self.distribution.has_ext_modules():
+            # If we are building an installer for a Python version other
+            # than the one we are currently running, then we need to ensure
+            # our build_lib reflects the other Python version rather than ours.
+            # Note that for target_version!=sys.version, we must have skipped the
+            # build step, so there is no issue with enforcing the build of this
+            # version.
+            target_version = self.target_version
+            if not target_version:
+                assert self.skip_build, "Should have already checked this"
+                target_version = sys.version[0:3]
+            plat_specifier = ".%s-%s" % (self.plat_name, target_version)
+            build = self.get_finalized_command('build')
+            build.build_lib = os.path.join(build.build_base,
+                                           'lib' + plat_specifier)
+
+        # Use a custom scheme for the zip-file, because we have to decide
+        # at installation time which scheme to use.
+        for key in ('purelib', 'platlib', 'headers', 'scripts', 'data'):
+            value = key.upper()
+            if key == 'headers':
+                value = value + '/Include/$dist_name'
+            setattr(install,
+                    'install_' + key,
+                    value)
+
+        logger.info("installing to %s", self.bdist_dir)
+        install.ensure_finalized()
+
+        # avoid warning of 'install_lib' about installing
+        # into a directory not in sys.path
+        sys.path.insert(0, os.path.join(self.bdist_dir, 'PURELIB'))
+
+        install.run()
+
+        del sys.path[0]
+
+        # And make an archive relative to the root of the
+        # pseudo-installation tree.
+        from tempfile import NamedTemporaryFile
+        archive_basename = NamedTemporaryFile().name
+        fullname = self.distribution.get_fullname()
+        arcname = self.make_archive(archive_basename, "zip",
+                                    root_dir=self.bdist_dir)
+        # create an exe containing the zip-file
+        self.create_exe(arcname, fullname, self.bitmap)
+        if self.distribution.has_ext_modules():
+            pyversion = get_python_version()
+        else:
+            pyversion = 'any'
+        self.distribution.dist_files.append(('bdist_wininst', pyversion,
+                                             self.get_installer_filename(fullname)))
+        # remove the zip-file again
+        logger.debug("removing temporary file '%s'", arcname)
+        os.remove(arcname)
+
+        if not self.keep_temp:
+            logger.info('removing %s', self.bdist_dir)
+            if not self.dry_run:
+                rmtree(self.bdist_dir)
+
+    def get_inidata(self):
+        # Return data describing the installation.
+
+        lines = []
+        metadata = self.distribution.metadata
+
+        # Write the [metadata] section.
+        lines.append("[metadata]")
+
+        # 'info' will be displayed in the installer's dialog box,
+        # describing the items to be installed.
+        if 'long_description' in metadata:
+            info = metadata['long_description']
+        else:
+            info = metadata.get('description', '')
+        info += '\n'
+
+        # Escape newline characters
+        def escape(s):
+            return s.replace("\n", "\\n")
+
+        for name in ["author", "author_email", "description", "maintainer",
+                     "maintainer_email", "name", "url", "version"]:
+            data = metadata.get(name, "")
+            if data:
+                info = info + ("\n    %s: %s" % \
+                               (name.capitalize(), escape(data)))
+                lines.append("%s=%s" % (name, escape(data)))
+
+        # The [setup] section contains entries controlling
+        # the installer runtime.
+        lines.append("\n[Setup]")
+        if self.install_script:
+            lines.append("install_script=%s" % self.install_script)
+        lines.append("info=%s" % escape(info))
+        lines.append("target_compile=%d" % (not self.no_target_compile))
+        lines.append("target_optimize=%d" % (not self.no_target_optimize))
+        if self.target_version:
+            lines.append("target_version=%s" % self.target_version)
+        if self.user_access_control:
+            lines.append("user_access_control=%s" % self.user_access_control)
+
+        title = self.title or self.distribution.get_fullname()
+        lines.append("title=%s" % escape(title))
+        import time
+        import packaging
+        build_info = "Built %s with packaging-%s" % \
+                     (time.ctime(time.time()), packaging.__version__)
+        lines.append("build_info=%s" % build_info)
+        return "\n".join(lines)
+
+    def create_exe(self, arcname, fullname, bitmap=None):
+        import struct
+
+        self.mkpath(self.dist_dir)
+
+        cfgdata = self.get_inidata()
+
+        installer_name = self.get_installer_filename(fullname)
+        logger.info("creating %s", installer_name)
+
+        if bitmap:
+            with open(bitmap, "rb") as fp:
+                bitmapdata = fp.read()
+            bitmaplen = len(bitmapdata)
+        else:
+            bitmaplen = 0
+
+        with open(installer_name, "wb") as file:
+            file.write(self.get_exe_bytes())
+            if bitmap:
+                file.write(bitmapdata)
+
+            # Convert cfgdata from unicode to ascii, mbcs encoded
+            if isinstance(cfgdata, str):
+                try:
+                    cfgdata = cfgdata.encode("mbcs")
+                except LookupError:
+                    # building on a system without MBCS. Everything
+                    # had better be latin-1 ...
+                    cfgdata = cfgdata.encode("latin-1")
+
+            # Append the pre-install script
+            cfgdata = cfgdata + b"\0"
+            if self.pre_install_script:
+                # We need to normalize newlines, so we open in text mode and
+                # convert back to bytes. "latin-1" simply avoids any possible
+                # failures.
+                with open(self.pre_install_script, encoding="latin-1") as fp:
+                    script_data = fp.read().encode("latin-1")
+                cfgdata = cfgdata + script_data + b"\n\0"
+            else:
+                # empty pre-install script
+                cfgdata = cfgdata + b"\0"
+            file.write(cfgdata)
+
+            # The 'magic number' 0x1234567B is used to make sure that the
+            # binary layout of 'cfgdata' is what the wininst.exe binary
+            # expects.  If the layout changes, increment that number, make
+            # the corresponding changes to the wininst.exe sources, and
+            # recompile them.
+            header = struct.pack("<iii",
+                                 0x1234567B,       # tag
+                                 len(cfgdata),     # length
+                                 bitmaplen,        # number of bytes in bitmap
+                                 )
+            file.write(header)
+            with open(arcname, "rb") as fp:
+                file.write(fp.read())
+
+    def get_installer_filename(self, fullname):
+        # Factored out to allow overriding in subclasses
+
+        # For pure-Python distributions, hardcode the 'win32' suffix,
+        # even when building on Linux.
+        dist = self.distribution
+        if not dist.has_ext_modules() and not dist.has_c_libraries():
+            suffix = 'win32'
+        else:
+            suffix = self.plat_name
+        basename = '%s.%s' % (fullname, suffix)
+        if self.target_version:
+            # if we create an installer for a specific python version,
+            # it's better to include this in the name
+            basename += '-py%s' % self.target_version
+        installer_name = os.path.join(self.dist_dir, "%s.exe" % basename)
+        return installer_name
+
+    def get_exe_bytes(self):
+        from packaging.compiler.msvccompiler import get_build_version
+        # If a target-version other than the current version has been
+        # specified, then using the MSVC version from *this* build is no good.
+        # Without actually finding and executing the target version and parsing
+        # its sys.version, we just hard-code our knowledge of old versions.
+        # NOTE: Possible alternative is to allow "--target-version" to
+        # specify a Python executable rather than a simple version string.
+        # We can then execute this program to obtain any info we need, such
+        # as the real sys.version string for the build.
+        cur_version = get_python_version()
+        if self.target_version and self.target_version != cur_version:
+            # If the target version is *later* than us, then we assume they
+            # use what we use
+            # string compares seem wrong, but are what sysconfig.py itself uses
+            if self.target_version > cur_version:
+                bv = get_build_version()
+            else:
+                if self.target_version < "2.4":
+                    bv = 6.0
+                else:
+                    bv = 7.1
+        else:
+            # for current version - use authoritative check.
+            bv = get_build_version()
+
+        # wininst-x.y.exe is in the same directory as this file
+        directory = os.path.dirname(__file__)
+        # we must use a wininst-x.y.exe built with the same C compiler
+        # used for python.  XXX What about mingw, borland, and so on?
+
+        # if plat_name starts with "win" but is not "win32"
+        # we want to strip "win" and leave the rest (e.g. -amd64)
+        # for all other cases, we don't want any suffix
+        if self.plat_name != 'win32' and self.plat_name[:3] == 'win':
+            sfix = self.plat_name[3:]
+        else:
+            sfix = ''
+
+        filename = os.path.join(directory, "wininst-%.1f%s.exe" % (bv, sfix))
+        with open(filename, "rb") as fp:
+            return fp.read()

File Lib/packaging/command/bdist_wininst.py

 
         # 'info' will be displayed in the installer's dialog box,
         # describing the items to be installed.
-        info = (metadata.long_description or '') + '\n'
+        info = metadata.get('description', '') + '\n'
 
         # Escape newline characters
         def escape(s):
 
             # Convert cfgdata from unicode to ascii, mbcs encoded
             if isinstance(cfgdata, str):
-                cfgdata = cfgdata.encode("mbcs")
+                try:
+                    cfgdata = cfgdata.encode("mbcs")
+                except LookupError:
+                    # building on a system without MBCS. Everything
+                    # had better be latin-1 ...
+                    cfgdata = cfgdata.encode("latin-1")
 
             # Append the pre-install script
             cfgdata = cfgdata + b"\0"
 
     def get_installer_filename(self, fullname):
         # Factored out to allow overriding in subclasses
+
+        # For pure-Python distributions, hardcode the 'win32' suffix,
+        # even when building on Linux.
+        dist = self.distribution
+        if not dist.has_ext_modules() and not dist.has_c_libraries():
+            suffix = 'win32'
+        else:
+            suffix = self.plat_name
+        basename = '%s.%s' % (fullname, suffix)
         if self.target_version:
             # if we create an installer for a specific python version,
             # it's better to include this in the name
-            installer_name = os.path.join(self.dist_dir,
-                                          "%s.%s-py%s.exe" %
-                                           (fullname, self.plat_name, self.target_version))
-        else:
-            installer_name = os.path.join(self.dist_dir,
-                                          "%s.%s.exe" % (fullname, self.plat_name))
+            basename += '-py%s' % self.target_version
+        installer_name = os.path.join(self.dist_dir, "%s.exe" % basename)
         return installer_name
 
     def get_exe_bytes(self):

File Lib/packaging/command/build_ext.py

         elif isinstance(self.rpath, str):
             self.rpath = self.rpath.split(os.pathsep)
 
+        prefix = sys.site_prefix
+        exec_prefix = sys.site_exec_prefix
+
         # for extensions under windows use different directories
         # for Release and Debug builds.
         # also Python's library directory must be appended to library_dirs
             # the 'libs' directory is for binary installs - we assume that
             # must be the *native* platform.  But we don't really support
             # cross-compiling via a binary install anyway, so we let it go.
+            # Note that we must use sys.exec_prefix here rather than
+            # exec_prefix, since the Python libs are not copied to a virtual
+            # environment.
             self.library_dirs.append(os.path.join(sys.exec_prefix, 'libs'))
             if self.debug:
                 self.build_temp = os.path.join(self.build_temp, "Debug")
 
             # Append the source distribution include and library directories,
             # this allows distutils on windows to work in the source tree
-            self.include_dirs.append(os.path.join(sys.exec_prefix, 'PC'))
+            self.include_dirs.append(os.path.join(exec_prefix, 'PC'))
             if MSVC_VERSION == 9:
                 # Use the .lib files for the correct architecture
                 if self.plat_name == 'win32':
                 else:
                     # win-amd64 or win-ia64
                     suffix = self.plat_name[4:]
-                new_lib = os.path.join(sys.exec_prefix, 'PCbuild')
+                new_lib = os.path.join(exec_prefix, 'PCbuild')
                 if suffix:
                     new_lib = os.path.join(new_lib, suffix)
                 self.library_dirs.append(new_lib)
 
             elif MSVC_VERSION == 8:
-                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                self.library_dirs.append(os.path.join(exec_prefix,
                                          'PC', 'VS8.0'))
             elif MSVC_VERSION == 7:
-                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                self.library_dirs.append(os.path.join(exec_prefix,
                                          'PC', 'VS7.1'))
             else:
-                self.library_dirs.append(os.path.join(sys.exec_prefix,
+                self.library_dirs.append(os.path.join(exec_prefix,
                                          'PC', 'VC6'))
 
         # OS/2 (EMX) doesn't support Debug vs Release builds, but has the
         # import libraries in its "Config" subdirectory
         if os.name == 'os2':
-            self.library_dirs.append(os.path.join(sys.exec_prefix, 'Config'))
+            self.library_dirs.append(os.path.join(exec_prefix, 'Config'))
 
         # for extensions under Cygwin and AtheOS Python's library directory must be
         # appended to library_dirs
         if sys.platform[:6] == 'cygwin' or sys.platform[:6] == 'atheos':
-            if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
+            if sys.executable.startswith(os.path.join(exec_prefix, "bin")):
                 # building third party extensions
-                self.library_dirs.append(os.path.join(sys.prefix, "lib",
+                self.library_dirs.append(os.path.join(prefix, "lib",
                                   "python" + sysconfig.get_python_version(),
                                                       "config"))
             else:
         sysconfig.get_config_var('Py_ENABLE_SHARED')
         if (sys.platform.startswith(('linux', 'gnu', 'sunos'))
             and sysconfig.get_config_var('Py_ENABLE_SHARED')):
-            if sys.executable.startswith(os.path.join(sys.exec_prefix, "bin")):
+            if sys.executable.startswith(os.path.join(exec_prefix, "bin")):
                 # building third party extensions
                 self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
             else:

File Lib/packaging/command/build_scripts.py

 
 import os
 import re
+import sys
 import sysconfig
 from tokenize import detect_encoding
 
 
 
 # check if Python is called on the first line with this expression
-first_line_re = re.compile(b'^#!.*python[0-9.]*([ \t].*)?$')
+FIRST_LINE_RE = re.compile(b'^#!.*pythonw?[0-9.]*([ \t].*)?$')
+DOTTED_CALLABLE_RE = re.compile(r'''(?P<name>(\w|-)+)
+                                    \s*=\s*(?P<callable>(\w+)(\.\w+)+)
+                                    (?P<flags>(\s+\w+)*)''', re.VERBOSE)
+SCRIPT_TEMPLATE = '''%(shebang)s
+if __name__ == '__main__':
+    rc = 1
+    try:
+        import sys, re
+        sys.argv[0] = re.sub('-script.pyw?$', '', sys.argv[0])
+        from %(module)s import %(func)s
+        rc = %(func)s() # None interpreted as 0
+    except Exception:
+        # use syntax which works with either 2.x or 3.x
+        sys.stderr.write('%%s\\n' %% sys.exc_info()[1])
+    sys.exit(rc)
+'''
+
+if os.name == 'nt':
+    # Executable launcher support
+    import zipfile
+    import struct
+    
+    def get_launcher(kind):
+        if struct.calcsize('P') == 8:   # 64-bit
+            bits = '64'
+        else:
+            bits = '32'
+        fname = '%s%s.exe' % (kind, bits)
+        zname = os.path.join(os.path.dirname(__file__), 'launchers.zip')
+        with zipfile.ZipFile(zname) as z:
+            result = z.read(fname)
+        return result
 
 class build_scripts(Command, Mixin2to3):
 
                                    'use_2to3', 'use_2to3_fixers',
                                    'convert_2to3_doctests', 'force',
                                    'executable')
+        self.force = True   # shebangs in scripts may need regeneration
         self.scripts = self.distribution.scripts
 
     def get_source_files(self):
         if self.use_2to3 and copied_files:
             self._run_2to3(copied_files, fixers=self.use_2to3_fixers)
 
+    def get_shebang(self, encoding, post_interp=b''):
+        if not sysconfig.is_python_build():
+            if sys.platform == 'darwin' and ('__PYTHONV_LAUNCHER__'
+                                             in os.environ):
+                executable =  os.environ['__PYTHONV_LAUNCHER__']
+            else:
+                executable = self.executable
+        elif sys.site_prefix != sys.prefix:
+            executable = os.path.join(
+                sysconfig.get_path('scripts'),
+               'python%s' % sysconfig.get_config_var('EXE'))
+        else:
+            executable = os.path.join(
+                sysconfig.get_config_var('BINDIR'),
+               'python%s%s' % (sysconfig.get_config_var('VERSION'),
+                               sysconfig.get_config_var('EXE')))
+        executable = os.fsencode(executable)
+        shebang = b'#!' + executable + post_interp + b'\n'
+        # Python parser starts to read a script using UTF-8 until
+        # it gets a #coding:xxx cookie. The shebang has to be the
+        # first line of a file, the #coding:xxx cookie cannot be
+        # written before. So the shebang has to be decodable from
+        # UTF-8.
+        try:
+            shebang.decode('utf-8')
+        except UnicodeDecodeError:
+            raise ValueError(
+                'The shebang ({!r}) is not decodable '
+                'from utf-8'.format(shebang))
+        # If the script is encoded to a custom encoding (use a
+        # #coding:xxx cookie), the shebang has to be decodable from
+        # the script encoding too.
+        if encoding != 'utf-8':
+            try:
+                shebang.decode(encoding)
+            except UnicodeDecodeError:
+                raise ValueError(
+                    'The shebang ({!r}) is not decodable '
+                    'from the script encoding ({})'
+                    .format(shebang, encoding))
+        return shebang
+
+    def make_script(self, name, path, flags, outfiles):
+        module, func = path.rsplit('.', 1)
+        flags = flags.split()
+        shebang = self.get_shebang('utf-8').decode('utf-8')
+        if 'gui' in flags and os.name == 'nt':
+            shebang = shebang.replace('python', 'pythonw')
+        script = SCRIPT_TEMPLATE % locals()
+        outfile = os.path.join(self.build_dir, name)
+        use_launcher = (os.name == 'nt')
+        if use_launcher:
+            exename = '%s.exe' % outfile
+            if 'gui' in flags:
+                ext = 'pyw'
+                launcher = get_launcher('w')
+            else:
+                ext = 'py'
+                launcher = get_launcher('t')
+            outfile = '%s-script.%s' % (outfile, ext)
+        with open(outfile, 'wb') as f:
+            f.write(script.encode('utf-8'))
+        outfiles.append(outfile)
+        if use_launcher:
+            with open(exename, 'wb') as f:
+                f.write(launcher)
+            outfiles.append(exename)
+
+    def copy_and_convert(self, script, outfiles):
+        adjust = False
+        script = convert_path(script)
+        outfile = os.path.join(self.build_dir, os.path.basename(script))
+        outfiles.append(outfile)
+
+        if not self.force and not newer(script, outfile):
+            logger.debug('not copying %s (up-to-date)', script)
+            return
+
+        # Always open the file, but ignore failures in dry-run mode --
+        # that way, we'll get accurate feedback if we can read the
+        # script.
+        try:
+            f = open(script, 'rb')
+        except IOError:
+            if not self.dry_run:
+                raise
+            f = None
+        else:
+            encoding, lines = detect_encoding(f.readline)
+            f.seek(0)
+            first_line = f.readline()
+            if not first_line:
+                logger.warning('%s: %s is an empty file (skipping)',
+                               self.get_command_name(),  script)
+                return
+
+            match = FIRST_LINE_RE.match(first_line)
+            if match:
+                adjust = True
+                post_interp = match.group(1) or b''
+
+        if not adjust:
+            if f:
+                f.close()
+            self.copy_file(script, outfile)
+        else:
+            logger.info('copying and adjusting %s -> %s', script,
+                     self.build_dir)
+            if not self.dry_run:
+                shebang = self.get_shebang(encoding, post_interp)
+                use_launcher = (os.name == 'nt')
+                if use_launcher:
+                    n, e = os.path.splitext(outfile)
+                    exename = n + '.exe'
+                    if b'pythonw' in first_line:
+                        launcher = get_launcher('w')
+                        suffix = '-script.pyw'
+                    else:
+                        launcher = get_launcher('t')
+                        suffix = '-script.py'
+                    outfile = n + suffix
+                    outfiles[-1] = outfile
+                with open(outfile, "wb") as outf:
+                    outf.write(shebang)
+                    outf.writelines(f.readlines())
+                if use_launcher:
+                    with open(exename, 'wb') as f:
+                        f.write(launcher)
+                    outfiles.append(exename)
+            if f:
+                f.close()
+
     def copy_scripts(self):
         """Copy each script listed in 'self.scripts'; if it's marked as a
-        Python script in the Unix way (first line matches 'first_line_re',
+        Python script in the Unix way (first line matches 'FIRST_LINE_RE',
         ie. starts with "\#!" and contains "python"), then adjust the first
         line to refer to the current Python interpreter as we copy.
         """
         self.mkpath(self.build_dir)
         outfiles = []
         for script in self.scripts:
-            adjust = False
-            script = convert_path(script)
-            outfile = os.path.join(self.build_dir, os.path.basename(script))
-            outfiles.append(outfile)
-
-            if not self.force and not newer(script, outfile):
-                logger.debug("not copying %s (up-to-date)", script)
-                continue
-
-            # Always open the file, but ignore failures in dry-run mode --
-            # that way, we'll get accurate feedback if we can read the
-            # script.
-            try:
-                f = open(script, "rb")
-            except IOError:
-                if not self.dry_run:
-                    raise
-                f = None
+            m = DOTTED_CALLABLE_RE.search(script)
+            if not m:
+                self.copy_and_convert(script, outfiles)
             else:
-                encoding, lines = detect_encoding(f.readline)
-                f.seek(0)
-                first_line = f.readline()
-                if not first_line:
-                    logger.warning('%s: %s is an empty file (skipping)',
-                                   self.get_command_name(),  script)
-                    continue
-
-                match = first_line_re.match(first_line)
-                if match:
-                    adjust = True
-                    post_interp = match.group(1) or b''
-
-            if adjust:
-                logger.info("copying and adjusting %s -> %s", script,
-                         self.build_dir)
-                if not self.dry_run:
-                    if not sysconfig.is_python_build():
-                        executable = self.executable
-                    else:
-                        executable = os.path.join(
-                            sysconfig.get_config_var("BINDIR"),
-                           "python%s%s" % (sysconfig.get_config_var("VERSION"),
-                                           sysconfig.get_config_var("EXE")))
-                    executable = os.fsencode(executable)
-                    shebang = b"#!" + executable + post_interp + b"\n"
-                    # Python parser starts to read a script using UTF-8 until
-                    # it gets a #coding:xxx cookie. The shebang has to be the
-                    # first line of a file, the #coding:xxx cookie cannot be
-                    # written before. So the shebang has to be decodable from
-                    # UTF-8.
-                    try:
-                        shebang.decode('utf-8')
-                    except UnicodeDecodeError:
-                        raise ValueError(
-                            "The shebang ({!r}) is not decodable "
-                            "from utf-8".format(shebang))
-                    # If the script is encoded to a custom encoding (use a
-                    # #coding:xxx cookie), the shebang has to be decodable from
-                    # the script encoding too.
-                    try:
-                        shebang.decode(encoding)
-                    except UnicodeDecodeError:
-                        raise ValueError(
-                            "The shebang ({!r}) is not decodable "
-                            "from the script encoding ({})"
-                            .format(shebang, encoding))
-                    with open(outfile, "wb") as outf:
-                        outf.write(shebang)
-                        outf.writelines(f.readlines())
-                if f:
-                    f.close()
-            else:
-                if f:
-                    f.close()
-                self.copy_file(script, outfile)
-
+                d = m.groupdict()
+                self.make_script(d['name'], d['callable'], d['flags'], outfiles)
         if os.name == 'posix':
             for file in outfiles:
                 if self.dry_run:
                                  file, oldmode, newmode)
                         os.chmod(file, newmode)
         return outfiles
+
+

File Lib/packaging/command/install_data.py

 # Contributed by Bastian Kleineidam
 
 import os
+import re
 from shutil import Error
 from sysconfig import get_paths, format_value
 from packaging import logger
-from packaging.util import convert_path
+from packaging.errors import PackagingOptionError
+from packaging.util import convert_path, split_multiline
 from packaging.command.cmd import Command
 
 
+CATEGORY_LINE = re.compile(r'^([A-Z]\w*)\s*=\s*(\S.*)$', re.IGNORECASE)
+
 class install_data(Command):
 
     description = "install platform-independent data files"
         self.force = False
         self.data_files = self.distribution.data_files
         self.warn_dir = True
+        self._categories = {}
 
+    @property
+    def categories(self):
+        return self._categories
+
+    @categories.setter
+    def categories(self, value):
+        if not isinstance(value, dict):
+            lines = split_multiline(value)
+            value = {}
+            for line in lines:
+                m = CATEGORY_LINE.match(line)
+                if m is None:
+                    raise PackagingOptionError('Invalid category '
+                                               'option: %r' % line)
+                key, val = m.group(1), m.group(2).strip()
+                value[key] = val
+        self._categories = value
+    
     def finalize_options(self):
         self.set_undefined_options('install_dist',
                                    ('install_data', 'install_dir'),
     def run(self):
         self.mkpath(self.install_dir)
         for _file in self.data_files.items():
-            destination = convert_path(self.expand_categories(_file[1]))
+            out = destination = convert_path(self.expand_categories(_file[1]))
             dir_dest = os.path.abspath(os.path.dirname(destination))
 
             self.mkpath(dir_dest)
             try:
-                out = self.copy_file(_file[0], dir_dest)[0]
+                if os.path.isdir(_file[0]):
+                    os.mkdir(destination)
+                else:
+                    out = self.copy_file(_file[0], dir_dest)[0]
             except Error as e:
                 logger.warning('%s: %s', self.get_command_name(), e)
-                out = destination
 
             self.outfiles.append(out)
             self.data_files_out.append((_file[0], destination))
 
     def expand_categories(self, path_with_categories):
         local_vars = get_paths()
+        local_vars.update(self.categories)
         local_vars['distribution.name'] = self.distribution.metadata['Name']
         expanded_path = format_value(path_with_categories, local_vars)
         expanded_path = format_value(expanded_path, local_vars)

File Lib/packaging/command/install_dist.py

                 if self.exec_prefix is not None:
                     raise PackagingOptionError(
                         "must not supply exec-prefix without prefix")
-
-                self.prefix = os.path.normpath(sys.prefix)
-                self.exec_prefix = os.path.normpath(sys.exec_prefix)
+                prefix = sys.site_prefix
+                exec_prefix = sys.site_exec_prefix
+                self.prefix = os.path.normpath(prefix)
+                self.exec_prefix = os.path.normpath(exec_prefix)
 
             else:
                 if self.exec_prefix is None:
             self.select_scheme("posix_home")
         else:
             if self.prefix is None:
-                self.prefix = os.path.normpath(sys.prefix)
+                prefix = sys.site_prefix
+                self.prefix = os.path.normpath(prefix)
 
             self.install_base = self.install_platbase = self.prefix
             try:

File Lib/packaging/command/install_distinfo.py

                                               'RESOURCES')
                 logger.info('creating %s', resources_path)
                 if not self.dry_run:
-                    with open(resources_path, 'w') as f:
+                    with open(resources_path, 'w', encoding='utf-8') as f:
                         writer = csv.writer(f, delimiter=',',
                                             lineterminator='\n',
                                             quotechar='"')
                     install = self.get_finalized_command('install_dist')
 
                     for fpath in install.get_outputs():
-                        if fpath.endswith('.pyc') or fpath.endswith('.pyo'):
+                        if (os.path.isdir(fpath) or fpath.endswith(('.pyc',
+                                                                    '.pyo'))):
                             # do not put size and md5 hash, as in PEP-376
                             writer.writerow((fpath, '', ''))
                         else:

File Lib/packaging/command/launchers.zip

Binary file added.

File Lib/packaging/create.py

                 path_tokens = sysconfig.get_paths(vars=vars).items()
                 # sort tokens to use the longest one first
                 path_tokens = sorted(path_tokens, key=lambda x: len(x[1]))
+                prefix = sys.site_prefix
                 for dest, srcs in (dist.data_files or []):
-                    dest = os.path.join(sys.prefix, dest)
+                    dest = os.path.join(prefix, dest)
                     dest = dest.replace(os.path.sep, '/')
                     for tok, path in path_tokens:
                         path = path.replace(os.path.sep, '/')

File Lib/packaging/database.py

         with self.get_distinfo_file('RECORD') as record:
             record_reader = csv.reader(record, delimiter=',',
                                        lineterminator='\n')
+
+            prefix = sys.site_prefix
+
             for row in record_reader:
                 missing = [None for i in range(len(row), 3)]
                 path, checksum, size = row + missing
                 if local:
                     path = path.replace('/', os.sep)
-                    path = os.path.join(sys.prefix, path)
+                    path = os.path.join(prefix, path)
                 results.append((path, checksum, size))
         return results
 
 
         :rtype: boolean
         """
+        prefix = sys.site_prefix
+
         for p, checksum, size in self._get_records():
-            local_absolute = os.path.join(sys.prefix, p)
+            local_absolute = os.path.join(prefix, p)
             if path == p or path == local_absolute:
                 return True
         return False

File Lib/packaging/install.py

     """
     path = os.path.abspath(path)
     if os.path.isdir(path):
-        logger.info('Installing from source directory: %r', path)
+        logger.info('Installing from source directory: %s', path)
         return _run_install_from_dir(path)
     elif _is_archive_file(path):
-        logger.info('Installing from archive: %r', path)
+        logger.info('Installing from archive: %s', path)
         _unpacked_dir = tempfile.mkdtemp()
         try:
             shutil.unpack_archive(path, _unpacked_dir)
+            if os.path.isfile(os.path.join(_unpacked_dir, 'setup.cfg')):
+                return _run_install_from_dir(_unpacked_dir)
             return _run_install_from_archive(_unpacked_dir)
         finally:
             shutil.rmtree(_unpacked_dir)
     else:
-        logger.warning('No project to install.')
+        logger.warning('No projects to install.')
         return False
 
 
 
             # reverting
             for installed_dist in installed_dists:
-                logger.info('Reverting %r', installed_dist)
+                logger.info('Reverting %s', installed_dist)
                 remove(installed_dist.name, paths)
             raise e
     return installed_dists
         if predicate.name.lower() != installed_project.name.lower():
             continue
         found = True
-        logger.info('Found %r %s', installed_project.name,
-                    installed_project.version)
+        logger.info('Found %s %s', installed_project.name,
+                    installed_project.metadata['version'])
 
         # if we already have something installed, check it matches the
         # requirements
-        if predicate.match(installed_project.version):
+        if predicate.match(installed_project.metadata['version']):
             return infos
         break
 
     try:
         release = index.get_release(requirements)
     except (ReleaseNotFound, ProjectNotFound):
-        raise InstallationException('Release not found: %r' % requirements)
+        raise InstallationException('Release not found: "%s"' % requirements)
 
     if release is None:
         logger.info('Could not find a matching project')
     dist = get_distribution(project_name, use_egg_info=True, paths=paths)
     if dist is None:
         raise PackagingError('Distribution %r not found' % project_name)
-    files = dist.list_installed_files(local=True)
+    # list_installed_files returns a generator, and we need the
+    # RECORD file itself closed so that we can move it - under Windows,
+    # you can't move a file which is still open.
+    files = list(dist.list_installed_files(local=True))
     rmdirs = []
     rmfiles = []
     tmp = tempfile.mkdtemp(prefix=project_name + '-uninstall')
                     error = _move_file(file_, tmpfile)
                     if error is not None:
                         success = False
+                        failed_on = file_
                         break
                 finally:
                     if not os.path.isfile(file_):
 
     if not success:
         logger.info('%r cannot be removed.', project_name)
-        logger.info('Error: %s', error)
+        logger.info('Error: %s: %r' % (error, failed_on))
         return False
 
     logger.info('Removing %r: ', project_name)
 
     except InstallationConflict as e:
         if logger.isEnabledFor(logging.INFO):
-            projects = ('%r %s' % (p.name, p.version) for p in e.args[0])
+            projects = ['%r %s' % (p.name, p.version) for p in e.args[0]]
             logger.info('%r conflicts with %s', project, ','.join(projects))
 
     return True

File Lib/packaging/manifest.py

File contents unchanged.

File Lib/packaging/tests/fake_dists/dory.zip

Binary file added.

File Lib/packaging/tests/test_command_build_scripts.py

         for name in expected:
             self.assertIn(name, built)
 
+    def test_full_installation(self):
+        def get_first_line(fn):
+            with open(fn, 'rb') as f:
+                s = f.readline()
+            return s.decode('utf-8')
+
+        import subprocess
+        import venv
+        target = self.mkdtemp()
+        venv.EnvBuilder(system_site_packages=False, clear=True).create(target)
+        source = os.path.dirname(__file__)
+        source = os.path.join(source, 'fake_dists', 'dory.zip')
+        if os.name == 'nt':
+            scriptdir = 'Scripts'
+            script = 'pysetup3.exe'
+        else:
+            scriptdir = 'bin'
+            script = 'pysetup3'
+        scriptdir = os.path.join(target, scriptdir)
+        exename = os.path.join(scriptdir, 'python')
+        script = os.path.join(scriptdir, script)
+        p = subprocess.Popen([script, 'install', source], shell=False,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        streams = p.communicate()
+        self.assertEqual(p.returncode, 0)
+        for fn in ('dory', 'demo1', 'demo2'):
+            if os.name == 'nt':
+                if fn == 'demo2':
+                    ext = 'pyw'
+                else:
+                    ext = 'py'
+                fn = '%s-script.%s' % (fn, ext)
+
+            pn = os.path.join(scriptdir, fn)
+            s = get_first_line(pn)
+            self.assertIn(exename, s)
+            if os.name == 'nt' and '%sdemo2' % os.sep in pn:
+                self.assertIn('pythonw', s)
+
 def test_suite():
     return unittest.makeSuite(BuildScriptsTestCase)
 

File Lib/packaging/tests/test_command_install_data.py

                 sysconfig._SCHEMES.set(scheme, option, value)
 
         self.addCleanup(restore)
+        packaging.database.clear_cache()
 
     def test_simple_run(self):
         pkg_dir, dist = self.create_dist()

File Lib/packaging/tests/test_config.py

 
 setup_hooks = %(setup-hooks)s
 
-
+[install_data]
+categories =
+    cat1 = /path/one
+# comment    
+    cat2 = /path/two
 
 [install_dist]
 sub_commands = foo
         self.assertIn('d', _COMPILERS)
         d = new_compiler(compiler='d')
         self.assertEqual(d.description, 'D Compiler')
+        cmd = dist.get_command_obj('install_data')
+        self.assertEqual(cmd.categories, {'cat1': '/path/one',
+                                          'cat2': '/path/two'})
 
         # check error reporting for invalid package_data value
         self.write_file('setup.cfg', SETUP_CFG_PKGDATA_BUGGY_1)

File Lib/packaging/tests/test_create.py

         os.chdir(self.wdir)
         # patch sysconfig
         self._old_get_paths = sysconfig.get_paths
+        prefix = sys.site_prefix
         sysconfig.get_paths = lambda *args, **kwargs: {
-            'man': sys.prefix + '/share/man',
-            'doc': sys.prefix + '/share/doc/pyxfoil', }
+            'man': prefix + '/share/man',
+            'doc': prefix + '/share/doc/pyxfoil', }
 
     def tearDown(self):
         sysconfig.get_paths = self._old_get_paths

File Lib/packaging/tests/test_database.py

 # TODO Add a test for absolute pathed RECORD items (e.g. /etc/myapp/config.ini)
 # TODO Add tests from the former pep376 project (zipped site-packages, etc.)
 
+prefix = sys.site_prefix
 
 def get_hexdigest(filename):
     with open(filename, 'rb') as file:
         found = dist.list_distinfo_files()
         self.assertEqual(sorted(found), sorted(distinfo_files))
         # Test for the iteration of local absolute paths
-        distinfo_files = [os.path.join(sys.prefix, distinfo_dir, path) for
+        distinfo_files = [os.path.join(prefix, distinfo_dir, path) for
                           path in distinfo_files]
         found = sorted(dist.list_distinfo_files(local=True))
         if os.sep != '/':

File Lib/packaging/tests/test_uninstall.py

 
         logs = [log for log in self.get_logs(logging.INFO)
                 if log.startswith('Error:')]
-        self.assertEqual(logs, ['Error: [Errno 42] impossible operation'])
+        self.assertEqual(len(logs), 1)
+        self.assertTrue(logs[0].startswith('Error: [Errno 42] impossible operation'))
 
         self.assertTrue(remove('Meh', paths=[site_packages]))
 
 resulting directories, if they exist, are appended to sys.path, and
 also inspected for path configuration files.
 
+If a file named "pyvenv.cfg" exists one directory above sys.executable,
+sys.site_prefix and sys.site_exec_prefix are set to that directory and
+it is also checked for site-packages and site-python (sys.prefix and
+sys.exec_prefix will always be the "real" prefixes of the Python
+installation). If "pyvenv.cfg" (a bootstrap configuration file) contains
+the key "include-system-site-packages" set to anything other than "false"
+(case-insensitive), the system-level prefixes will still also be
+searched for site-packages; otherwise they won't.
+
+All of the resulting site-specific directories, if they exist, are
+appended to sys.path, and also inspected for path configuration
+files.
+
 A path configuration file is a file whose name has the form
 <package>.pth; its contents are additional directories (one per line)
 to be added to sys.path.  Non-existing directories (or
 
 import sys
 import os
+import re
 import builtins
 
 # Prefixes for site-packages; add additional prefixes like /usr/local here
     sitedir, sitedircase = makepath(sitedir)
     if not sitedircase in known_paths:
         sys.path.append(sitedir)        # Add path component
+        known_paths.add(sitedircase)
     try:
         names = os.listdir(sitedir)
     except os.error:
         addsitedir(user_site, known_paths)
     return known_paths
 
-def getsitepackages():
+def getsitepackages(prefixes=None):
     """Returns a list containing all global site-packages directories
     (and possibly site-python).
 
-    For each directory present in the global ``PREFIXES``, this function
-    will find its `site-packages` subdirectory depending on the system
-    environment, and will return a list of full paths.
+    For each directory present in ``prefixes`` (or the global ``PREFIXES``),
+    this function will find its `site-packages` subdirectory depending on the
+    system environment, and will return a list of full paths.
     """
     sitepackages = []
     seen = set()
 
-    for prefix in PREFIXES:
+    if prefixes is None:
+        prefixes = PREFIXES
+
+    for prefix in prefixes:
         if not prefix or prefix in seen:
             continue
         seen.add(prefix)
                             sys.version[:3], "site-packages"))
     return sitepackages
 
-def addsitepackages(known_paths):
+def addsitepackages(known_paths, prefixes=None):
     """Add site-packages (and possibly site-python) to sys.path"""
-    for sitedir in getsitepackages():
+    for sitedir in getsitepackages(prefixes):
         if os.path.isdir(sitedir):
             addsitedir(sitedir, known_paths)
 
                 encodings.aliases.aliases[enc] = 'mbcs'
 
 
+CONFIG_LINE = re.compile(r'^(?P<key>(\w|[-_])+)\s*=\s*(?P<value>.*)$')
+
+def venv(known_paths):
+    global PREFIXES, ENABLE_USER_SITE
+
+    env = os.environ
+    if sys.platform == 'darwin' and '__PYTHONV_LAUNCHER__' in env:
+        executable = os.environ['__PYTHONV_LAUNCHER__']
+    else:
+        executable = sys.executable
+    executable_dir, executable_name = os.path.split(executable)
+    site_prefix = os.path.dirname(executable_dir)
+    if sys.platform == 'win32':
+        executable_name = os.path.splitext(executable_name)[0]
+    conf_basename = 'pyvenv.cfg'
+    candidate_confs = [
+        conffile for conffile in (
+            os.path.join(executable_dir, conf_basename),
+            os.path.join(site_prefix, conf_basename)
+            )
+        if os.path.isfile(conffile)
+        ]
+
+    if candidate_confs:
+        virtual_conf = candidate_confs[0]
+        system_site = "true"
+        with open(virtual_conf) as f:
+            for line in f:
+                line = line.strip()
+                m = CONFIG_LINE.match(line)
+                if m:
+                    d = m.groupdict()
+                    key, value = d['key'].lower(), d['value']
+                    if key == 'include-system-site-packages':
+                        system_site = value.lower()
+        sys.site_prefix = sys.site_exec_prefix = site_prefix
+
+        # Doing this here ensures venv takes precedence over user-site
+        addsitepackages(known_paths, [sys.site_prefix])
+
+        # addsitepackages will process site_prefix again if its in PREFIXES,
+        # but that's ok; known_paths will prevent anything being added twice
+        if system_site == "true":
+            PREFIXES.insert(0, sys.site_prefix)
+        else:
+            PREFIXES = [sys.site_prefix]
+            ENABLE_USER_SITE = False
+
+    return known_paths
+
+
 def execsitecustomize():
     """Run custom site specific code, if available."""
     try:
 
     abs_paths()
     known_paths = removeduppaths()
+    known_paths = venv(known_paths)
     if ENABLE_USER_SITE is None:
         ENABLE_USER_SITE = check_enableusersite()
     known_paths = addusersitepackages(known_paths)

File Lib/sysconfig.cfg

 # User resource directory
 local = ~/.local/{distribution.name}
 
-stdlib = {base}/lib/python{py_version_short}
+stdlib = {installed_base}/lib/python{py_version_short}
 platstdlib = {platbase}/lib/python{py_version_short}
 purelib = {base}/lib/python{py_version_short}/site-packages
 platlib = {platbase}/lib/python{py_version_short}/site-packages
-include = {base}/include/python{py_version_short}{abiflags}
-platinclude = {platbase}/include/python{py_version_short}{abiflags}
+include = {installed_base}/include/python{py_version_short}{abiflags}
+platinclude = {installed_platbase}/include/python{py_version_short}{abiflags}
 data = {base}
 
 [posix_home]
-stdlib = {base}/lib/python
+stdlib = {installed_base}/lib/python
 platstdlib = {base}/lib/python
 purelib = {base}/lib/python
 platlib = {base}/lib/python
-include = {base}/include/python
-platinclude = {base}/include/python
+include = {installed_base}/include/python
+platinclude = {installed_base}/include/python
 scripts = {base}/bin
 data = {base}
 
 [nt]
-stdlib = {base}/Lib
+stdlib = {installed_base}/Lib
 platstdlib = {base}/Lib
 purelib = {base}/Lib/site-packages
 platlib = {base}/Lib/site-packages
-include = {base}/Include
-platinclude = {base}/Include
+include = {installed_base}/Include
+platinclude = {installed_base}/Include
 scripts = {base}/Scripts
 data = {base}
 
 [os2]
-stdlib = {base}/Lib
+stdlib = {installed_base}/Lib
 platstdlib = {base}/Lib
 purelib = {base}/Lib/site-packages
 platlib = {base}/Lib/site-packages
-include = {base}/Include
-platinclude = {base}/Include
+include = {installed_base}/Include
+platinclude = {installed_base}/Include
 scripts = {base}/Scripts
 data = {base}
 

File Lib/sysconfig.py

 import os
 import re
 import sys
+import os
 from os.path import pardir, realpath
 from configparser import RawConfigParser
 
 _PY_VERSION = sys.version.split()[0]
 _PY_VERSION_SHORT = sys.version[:3]
 _PY_VERSION_SHORT_NO_DOT = _PY_VERSION[0] + _PY_VERSION[2]
+_SITE_PREFIX = os.path.normpath(sys.site_prefix)
 _PREFIX = os.path.normpath(sys.prefix)
+_SITE_EXEC_PREFIX = os.path.normpath(sys.site_exec_prefix)
 _EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
 _CONFIG_VARS = None
 _USER_BASE = None
                             done[name] = value
                         variables.remove(name)
 
-                        if name.startswith('PY_') \
-                        and name[3:] in renamed_variables:
+                        if (name.startswith('PY_') and
+                            name[3:] in renamed_variables):
 
                             name = name[3:]
                             if name not in done:
         # Normalized versions of prefix and exec_prefix are handy to have;
         # in fact, these are the standard versions used most places in the
         # packaging module.
-        _CONFIG_VARS['prefix'] = _PREFIX
-        _CONFIG_VARS['exec_prefix'] = _EXEC_PREFIX
+        _CONFIG_VARS['prefix'] = _SITE_PREFIX
+        _CONFIG_VARS['exec_prefix'] = _SITE_EXEC_PREFIX
         _CONFIG_VARS['py_version'] = _PY_VERSION
         _CONFIG_VARS['py_version_short'] = _PY_VERSION_SHORT
         _CONFIG_VARS['py_version_nodot'] = _PY_VERSION[0] + _PY_VERSION[2]
-        _CONFIG_VARS['base'] = _PREFIX
-        _CONFIG_VARS['platbase'] = _EXEC_PREFIX
+        _CONFIG_VARS['installed_base'] = _PREFIX
+        _CONFIG_VARS['base'] = _SITE_PREFIX
+        _CONFIG_VARS['installed_platbase'] = _EXEC_PREFIX
+        _CONFIG_VARS['platbase'] = _SITE_EXEC_PREFIX
         _CONFIG_VARS['projectbase'] = _PROJECT_BASE
         try:
             _CONFIG_VARS['abiflags'] = sys.abiflags

File Lib/test/test_capi.py

         (out, err) = p.communicate()
         self.assertEqual(out, b'')
         # This used to cause an infinite loop.
-        self.assertEqual(err.rstrip(),
+        self.assertTrue(err.rstrip().startswith(
                          b'Fatal Python error:'
-                         b' PyThreadState_Get: no current thread')
+                         b' PyThreadState_Get: no current thread'))
 
     def test_memoryview_from_NULL_pointer(self):
         self.assertRaises(ValueError, _testcapi.make_memoryview_from_NULL_pointer)

File Lib/test/test_faulthandler.py

 ^Fatal Python error: {name}
 
 {header}:
-  File "<string>", line {lineno} in <module>$
+  File "<string>", line {lineno} in <module>
 """.strip()
         regex = regex.format(
             lineno=line_number,

File Lib/test/test_gdb.py

 import re
 import subprocess
 import sys
+import sysconfig
 import unittest
 import locale
 
     raise unittest.SkipTest("gdb versions before 7.0 didn't support python embedding"
                             " Saw:\n" + gdb_version.decode('ascii', 'replace'))
 
+if not sysconfig.is_python_build():
+    raise unittest.SkipTest("test_gdb only works on source builds at the moment.")
+
 # Verify that "gdb" was built with the embedded python support enabled:
 cmd = "--eval-command=python import sys; print sys.version_info"
 p = subprocess.Popen(["gdb", "--batch", cmd],

File Lib/test/test_site.py

File contents unchanged.

File Lib/test/test_sysconfig.py

             if adapt:
                 global_path = global_path.replace(sys.exec_prefix, sys.prefix)
                 base = base.replace(sys.exec_prefix, sys.prefix)
+            elif sys.prefix != sys.site_prefix:
+                # virtual environment? Likewise, we have to adapt the paths
<