Commits

Éric Araujo  committed 567b8b2

Make the document ready for feedback and votes.

- Add a label to each proposal;
- Move the encoding from a comment into a proposal;
- Improve order;
- Fix minor things;
- Add a version number.

  • Participants
  • Parent commits b95024a

Comments (0)

Files changed (1)

File new-config-file.rst

 :Author: Éric Araujo <merwok@netwok.org>
 :Credits: Carl Meyer, folks at PyCon 2010, people in #distutils
 :License: PSF
+:Revision: 1.0
 
 .. warning::
 
   This document is not part of the documentation of Distutils2. It is a
   design/discussion document that serves to explain directions, collect
   feedback and votes, and will ultimately be rewritten as proper documentation
-  (without all the explanations about choices) and moved.
+  (without all the explanations about choices) and moved into the relevant doc
+  files.
 
 One goal of Distutils2 is to put all the information required to build and
 install a distribution into a static configuration file (:file:`setup.cfg`)
 `customization hooks`_.
 
 In the olden days, Distutils configuration files were used only to give
-options to commands. They were designed to be extensible: Third-party tools
-extending Distutils could tell their users to add a section in the
-distribution’s :file:`setup.cfg` file or in their user config file to give
-specific options to new commands or tools (e.g. testing tools). There is a
-simple API to get these options merged from all configuration files (which
-will be even simpler in Distutils2).
+options to commands. They were also designed to be extensible: Third-party
+tools relying on Distutils or providing new commands could tell their users
+to add a section in the distribution’s :file:`setup.cfg` file or in their
+user config file to set options. There is a simple API to get these options
+merged from all configuration files (which will be even simpler in
+Distutils2).
 
 .. i.e. d2.config.get_options('section') instead of
    distribution.metadata.get_option_dict('section')
   classifiers =
     Development Status :: 4 - Beta
     Environment :: Console (Text Based)
-    Environment :: X11 Applications :: GTK; sys.version < '3'
+    Environment :: X11 Applications :: GTK; python_version < '3'
     License :: OSI Approved :: MIT License
     Programming Language :: Python
     Programming Language :: Python :: 2
 follow.
 
 
+Encoding
+--------
+:codename: use-utf8
+
+The encoding of the config file is UTF-8. This encoding enables using Unicode
+characters in string fields, and is also a superset of ASCII.
+
+
 Avoid Metadata-Version
 ----------------------
+:codename: no-metadata-version
 
 This field does not have to be in the file, since the ``DistributionMetadata``
 class detects the right version from the fields that are present.
 
 Use CSV for Keywords and Requires-Python
 ----------------------------------------
+:codename: keywords-csv
 
 :PEP:`345` only says that the ``Keywords`` field is “a list”; the example
 uses space-delimited values, but distutils and distutils2 print out
 
   requires-python = >=2.4, <=3.0
 
+:codename: keywords-no-csv
 
 Alternatively, if it is deemed confusing to have two ways of giving
 multi-value fields, the field can be newline-separated like other fields
 
 Merge author and author-email
 -----------------------------
+:codename: merge-author-email
 
 Merge name and email in a single field for author (and maintainer)::
 
 
 Get description from a file
 ---------------------------
+:codename: desc-from-file
 
 Use the contents of a file as value for ``description``. Long descriptions
 typically contain blank lines, which are stripped by our `config file parser`_
 
 Listing modules and packages
 ----------------------------
+:codename: merge-mod-pkg
 
 For the new format, it is proposed that the three kinds of modules (Python
 modules, extension modules and packages) be merged into a single option.
             pirate
             ship
 
-(See below_ for the rationale to separate with newlines instead of arbitrary
-whitespace.)
+(See below__ for the rationale to separate with newlines instead of arbitrary
+whitespace. See appendix__ for implementation details explaining how this
+merged list will be easily parsed.)
 
-.. _below: `replacing conditionals`_
-
-Distutils2 code needs to sort this list into the three lists used by
-``Distribution``, following these simple rules:
-
-#. If the name is listed is the `extension modules`_ section, an instance of
-   ``distutils2.extension.Extension`` is created and added to
-   ``distribution.ext_modules``;
-#. If the name corresponds to an existing directory which contains an
-   ``__init__.py`` file, it is added to ``distribution.packages``;
-#. The name is added to ``distribution.py_modules``.
+.. __: `replacing conditionals`_
+.. __: `splitting the list of modules`_
 
 It is not possible to have a module and a package with the same name. In
 addition to being documented, this restriction could also be a runtime
 .. i.e. in d2.config.get_files('modules') + check command
 
 
-Recursion
-`````````
-Package addition is recursive; real-world use of
-:func:`setuptools.find_packages` shows that this is a useful feature. There is
-a way to control it::
-
-  modules = ship
-  exclude-modules = ship.hull
-
-Additionally, a boolean option could control all-or-nothing recursion::
-
-  modules = ship
-  recursive-modules = 0
-
-It is not clear that this would solve problems, e.g. for the mx project which
-is developped in one tree but packaged as separate PyPI projects; maybe it’s
-best not to define this option right now and try converting complex projects
-before revising this proposal.
-
-
 Naming
 ``````
 Calling packages “modules” may be confusing to some people, e.g. beginners,
 even if it’s technically correct. Other proposed names include “source” (too
 vague) and “importables” (unused in the documentation and ugly).
 
+:codename: no-merge-mod-pkg
+
 Alternatively, modules and packages could be listed separately. Since it
 appears that people tend to use either one or the other in their projects,
 there would be no cognitive overload in defining two fields instead of one::
 undestanding of file layout and possible detection (esp. recursion) issues.
 
 
+Recursion
+`````````
+:codename: pkg-recursion
+
+Package listing is recursive; real-world use of
+:func:`setuptools.find_packages` shows that this is a useful feature. There is
+a way to control it::
+
+  modules = ship
+  exclude-modules = ship.hull
+
+:codename: +pkg-recursion-boolean
+
+Additionally, a boolean option could control all-or-nothing recursion::
+
+  modules = ship
+  recursive-modules = 0
+
+It is not clear that this would solve problems, e.g. for the mx project which
+is developped in one tree but packaged as separate PyPI projects; maybe it’s
+best not to define this option right now, try to convert complex projects and
+then revise this proposal.
+
+
 Replacing package_dir
 `````````````````````
+:codename: pkg-dir-prefix
+
 Instead of replacing the ``package_dir`` argument with a field of the same
 name, it is proposed to merge it with the ``packages`` (or ``modules``)
 values::
 Note that these semantics are different from
 ``setup(packages=['thing'], package_dir={'thing': 'src2'}``: In the new
 proposal, ``src2:thing`` does not mean that the :file:`src2` directory is to
-be renamed :file:`thing` in the build directory, but that the directory
-:file:`src2` contains another directory named :file:`thing` (with its
-:file:`__init__.py` file and other submodules). The changed semantics are more
-intuitive.
+be renamed :file:`thing` in the build directory (reference__), but that the
+directory :file:`src2` contains another directory named :file:`thing` (with
+its :file:`__init__.py` file and other submodules). The changed semantics are
+more intuitive.
 
-Using ``:`` as a separator forbids using it in directory name, which would not
-be very sane anyway. It also allows putting packages in a deep subdirectory::
+.. __: http://docs.python.org/distutils/setupscript#listing-whole-packages
+
+Using ``:`` as a separator forbids using it in directory name, which would
+not be very sane anyway. Using it instead of a space or a slash also allows
+putting packages in a deep subdirectory::
 
   packages = client/bindings/python:parrotlib
 
 .. TODO kill use of “root package” in the docs and use a better term
         (say “top-level modules”)
 
+:codename: pkg-dir-no-prefix
+
 If a study of setup scripts in projects distributed on the Cheeseshop reveals
 that an overwhelming majority uses only one ``package_dir`` or none, this
 alternate, simpler proposal would be enough::
 
 Replacing conditionals
 ``````````````````````
+:codename: env-markers-for-files
+
 We can define the ``packages`` or ``modules`` field to be newline-separated
 and accept `PEP 345 environment markers`_ to support source distributions that
 contain e.g. code for both 2.x and 3.x, like httplib2_ does::
 
   packages =
-    python2:httplib2; sys.version < '3'
-    python3:httplib2; sys.version > '2'
+    python2:httplib2; python_version < '3'
+    python3:httplib2; python_version > '2'
+
+(Form using the alternate ``package_dir`` proposal::
+
+  packages = httplib2
+  package-dir = python2; python_version < '3'
+                python3; python_version > '2'
+
+Using a multi-line value for this field is kind of ugly, though.)
 
 Since there is no ``else``, each condition has to be written twice (once in
 normal form, once in reverse, which can be tricky and/or tedious), and the
 Example that can’t be translated:
 
 .. code-block:: python
-  
+
   if sys.platform == 'linux2' and os.uname()[2] > '2.6':
     # The inotify extension is only usable with Linux 2.6 kernels.
     ...
 
-For such cases, the solution seems to use a `pre-build hook`_ to edit the
+For such cases, the solution seems to use a `pre-build hook`__ to edit the
 lists of modules and packages. For trivial cases, environment markers provide
 a solution that does not require writing any code, so they’re still useful in
 the ``files`` section.
 
+.. __: `pre/post-command hooks`_
+
 
 Extension modules
 -----------------
+:codename: extensions-section
 
-A new section is proposed to describe extension modules, to replace
+A new section family is proposed to describe extension modules, to replace
 instantiation of :class:`Extension` objects with the right options in
-:file:`setup.py`.
+:file:`setup.py`. Each extension module has to be listed in the ``files``
+field and described in its own section::
 
-First proposal::
+  [files]
+  modules = ship.pirate
+
+  [extension: ship.pirate]
+  sources = ship/pirate.c
+  headers = Python.h pirate.h
+  include-dirs = include
+  optional = 1
+
+The section name is the string ``extension:`` followed by optional whitespace
+and the full name of the  module, field names are directly taken from
+:class:`Extension` arguments, values are simple adaptations (string arguments
+are single values, string lists are multi-value fields (whitespace-separated
+or newline-separated, to be decided), booleans are :mod:`ConfigParser`
+booleans).
+
+:codename: +vars-in-extmod
+
+If deemed useful, simple variables could be added to these sections; see
+definition__.
+
+.. __: variables_
+
+:codename: extensions-section-flat
+
+An alternate proposal that requires only one section but may prove more
+difficult to write and to parse is derived from the older ``Setup`` format,
+deprecated in Distutils and removed in Distutils2 (see
+:file:`{python3.2}/Lib/distutils/tests/Setup.sample`)::
 
   [extensions]
   pirate = pirate.c
   ship.hull = ship/hull.c
 
+.. This format effectively supersedes the old ``Setup`` format, supported by
+   :func:`distutils.extension.read_setup_file` (removed in Distutils2). Python
+   itself still uses a file in this format called :file:`Setup.dist` but uses
+   its own helper to consume it, so not supporting it in Distutils2 is okay.
+
 The format is ``module = source files [arguments to the compiler]``. More
-involved example from SDL (converted from older ``Setup`` format, deprecated
-in Distutils and removed in Distutils2, see
-:file:`{python3.2}/Lib/distutils/tests/Setup.sample`)::
+involved example from SDL::
 
   [extensions]
   _camera = src/_camera.c src/camera_v4l2.c src/camera_v4l.c $SDL $DEBUG
   font = src/font.c $SDL $FONT $DEBUG
   scrap = src/scrap.c $SDL $SCRAP $DEBUG
 
+.. _variables:
+
 This example introduces variables, which can be any string. A simple proposal
 for the assignment syntax::
 
 In other words, every key starting with a dollar sign is a variable that will
 be usable in the regular fields of the same section.
 
+:codename: +vars-in-vars-for-extmod
+
 If useful, expanding already defined variables in other variable definitions
 could be allowed.
 
-If useful, environment markers could be allowed in fields describing extension
-modules. Same thing for variables.
+:codename: +env-markers-for-extmod
 
-This format effectively supersedes the old ``Setup`` format, supported by
-:func:`distutils.extension.read_setup_file` (removed in Distutils2). Python
-itself still uses a file in this format called :file:`Setup.dist` but uses its
-own helper to consume it, so not supporting it in Distutils2 is okay.
-
-An alternate proposal is a direct conversion of the :class:`Extension` class
-rather than ``Setup`` files::
-
-  [files]
-  modules = pirate
-
-  [extension: pirate]
-  sources = pirate.c
-  headers = Python.h pirate.h
-  optional = 1
-
-Field names are the same as valid :class:`Extension` arguments, values are
-simple adaptations (string arguments are single values, string lists are
-multi-value fields (whitespace-separated or newline-separated, to be decided),
-booleans are :mod:`ConfigParser` booleans).
-
-Variables from the other proposal could be added; environment markers could
-too.
+If useful, environment markers could be allowed in fields of this section.
 
 
 Listing scripts
 ---------------
+:codename: scripts
 
 The Python list naturally translates to a multi-value field::
 
     bin/taunt
 
 Paths are relative, with ``..`` component forbidden. Using multi-line instead
-of whitespace or comma-separated allows directory names with spaces and also
-environment markers::
+of whitespace or comma-separated allows directory names with spaces and is
+consistent with other multi-value fields.
+
+:codename: +env-markers-for-scripts
+
+:PEP:`345` markers can be useful to filter the list of scripts according to
+the build environment (sdist would still ship all scripts)::
 
   scripts =
     unit2.py; sys.platform == 'win32'
     unit2; sys.platform != 'win32'
+    unit2-gui; sys.platform == 'linux2' and python_version < '3'
 
-For this precise use case, distutils could add the ``.py`` so-called file
-extension on win32 itself (see `Python #870479`_). Environment markers could
-still be useful for other uses, or we could not use them and let people define
-a `pre-build hook`_ for such cases.
+Full control over the scripts is possible thanks to `pre-build hooks`__ but
+allowing environment markers covers common needs without requiring user code.
 
-Proposals like fixing ``sys.path`` for scripts run from an uninstalled source
-or supporting :mod:`pkg_resources`-like ``console_scripts`` are regular feature
-requests outside of the scope of this document which can be implemented with
-new fields or extended syntax for existing fields.
+.. __: `pre/post-command hooks`_
 
-People have wanted a way to install into ``$exec_prefix/sbin`` instead of
+On the other hand, disallowing environment markers may be the best thing to do
+in the short term. There are a number of feature requests (regarding
+platform-dependent handling in particular) in the Python bug tracker that need
+discussion and testing; since hooks provide an easy way to experiment, the
+features could be easily implemented outside of Distutils2 and eventually
+incorporated into the core if met with positive feedback.
+
+A new field implementing a feature similar to :mod:`pkg_resources`\ ’
+``console_scripts`` may also render scripts as we know them obsolete.
+
+People have also wanted a way to install into ``$exec_prefix/sbin`` instead of
 ``$exec_prefix/bin``. There is no proposal about that now, although it would
 not be hard to define, since it is believed that the future resources_ tagging
 will supersede the scripts section and address this feature request in a
 generic way.
 
+Therefore, rejecting environment markers in ``scripts`` may be the best
+choice right now.
+
 
 Additional files
 ----------------
+:codename: data-fields
 
-While the PEP on resources_ is not implemented, :file:`setup.cfg` will grow
-new fields that merely translate the :file:`setup.py` form without any added
+Until the PEP on resources_ is implemented, :file:`setup.cfg` will grow new
+fields that merely translate the :file:`setup.py` form without any added
 semantics or features. Example::
 
   [files]
     config = cfg/data.cfg
     /etc/init.d = init-script
 
-Comma-separated values allow paths with spaces.
+Comma-separated values allow paths with spaces and avoid the need to parse
+multi-line values in a multi-line value.
 
 .. XXX check whether package_data={'': 'something'} works
 
 
 Replacing MANIFEST.in
 ---------------------
+:codename: remove-manifest.in
 
 The `design document`_ for resources_ states the goal to remove redundant
-listing of files in favor of :file:`setup.cfg`. Files listed as modules,
-packages, scripts, source files for extension modules (maybe tricky), package
-data, extra data files or description file will automatically be included in
-the manifest object used by the distribution. Before resources_ tagging is
+listing of files in favor of :file:`setup.cfg`. Presently, files listed as
+modules, packages, scripts, source files for extension modules, package data,
+extra data files or description file will automatically be included in the
+manifest object used by the distribution. Until resources tagging is
 implemented, we could either still support the :file:`MANIFEST.in` file or
 move its contents to :file:`setup.cfg`, retaining strict syntactic and
 semantic compatibility::
 .. TODO check if paths with spaces work/should work
 
 .. TODO check if MANIFEST is obsoleted by PEP 376 RECORD file
-   I think not, since it’s not included in sdists/bdists
+   I think not, since it’s not included in sdists/bdists.
+   MANIFEST and record file *should* be replaced by PEP 276 formats IMO.
 
 
 Psychic Mode
 ------------
+:codename: auto-detection-option
 
 This proposal takes advantage from convention over configuration, following
 the lead of DistUtilsExtra.auto_. The idea is to automatically detect specific
 :file:`POTFILES.in` depends on consensus that i18n-related tasks are in the
 scope of Distutils, which is not reached right now.
 
-Alternative: Require that people run ``distutils2.mkpkg`` to trigger detection
+:codename: auto-detection-script
+
+Alternative: Require that people run ``mkpkg`` to trigger detection
 and have the configuration file created or updated. Then they can check that
 the update is right thanks to ``$vcs diff setup.cfg`` or ``editor setup.cfg``.
+See appendix__.
+
+.. __: `appendix: making things simple for users`_
 
 
 Resources
 
 Customization hooks
 """""""""""""""""""
+:codename: distclass-cmdclass
 
 (This is not related to `pre/post-command hooks`_, which will probably be set
 in the relevant command sections or in a new one.)
 
   .. code-block:: python
 
-    setup(..., cmdclass={'build_py': build_py_2to3, 'test': TestCommand})
+    setup(..., cmdclass={'build_py': build_py_2to3, 'lint': LintCommand})
 
 :script_name:
   Usually ``sys.argv[0]``, used to generate error messages with the correct
   [global]
   distclass = shop.cheese.HamDistribution
   cmdclass =
-    build_py = distutils2.build_py.build_py_2to3; sys.version > '3'
+    build_py = distutils2.build_py.build_py_2to3; python_version >= '3'
     test = lib:_buildhelper.TestCommand
 
-As you can see, environment markers and `source directory`_ specifiers are
+As you can see, environment markers and `source directory`__ specifiers are
 allowed. The fields are located in the ``global`` section, alongside
 `command-packages`_.
 
+.. __: `replacing package_dir`_
+
+:codename: +rename-cmdclass
+
 Additional proposal: Give the field a clearer name. ``command-classes`` or
 ``commands`` (and change it in the Python code too). If there is a good reason
 to keep it short, it should at least be a plural form, i.e. ``cmdclasses``.
 
 Appendix: Making things simple for users
 """"""""""""""""""""""""""""""""""""""""
+Distutils2 will ship with a little program called ``mkpkg`` (which will soon
+get a better name) that generates a :file:`setup.cfg` file thanks to questions
+asked to the user (what is the project name, its version, etc.). As much as
+possible, the program will propose answers (e.g. using the
+:func:`find_package` function to get the list of packages, mocking
+``sys.modules['distutils']`` to run :file:`setup.py` scripts in a sandbox and
+get information from it, etc.) so that the user just has to press
+:kbd:`Enter` to validate, or write the correct value and validate.
 
-mkpkg (to be given a better name) will help people do the right thing, e.g.
-use a compliant version number, fill the ``license`` field only when there is
-no suitable Trove classifier for the chosen license, in other words give
+The program will also help people do the right thing, e.g. use a version
+number compliant with :PEP:`386`, fill the ``license`` field only when there
+is no suitable Trove classifier for the chosen license, in other words give
 useful hints for people that don’t read PEPs or documentation.
 
-mkpkg is an interactive mini-program that will generate a :file:`setup.cfg`
-file thanks to questions asked to the user (what is the project name, its
-version, etc.). As much as possible, the program will propose answers (e.g.
-using the :func:`find_package` function to get the list of packages) so that
-the user just has to press :kbd:`Enter` to validate, or write the correct
-value and validate.
+Some values could be specified in the user configuration file::
 
-Some values could be specified in the user configuration file, e.g. the author
-name or a template for the project homepage (e.g.
-``http://example.org/hacking/projects/{name}``). It’s tempting to reuse the
-``metadata`` section name but it may be confusing, since it would mean that
-the same section is used by two commands depending on the origin file, which
-is not an expectation in the Distutils model (which I will already change to
-introduce sections that are not allowed at all config file levels). It’s
-cleaner if the program uses its own section in the user config file.
+  [mkpkg]
+  author = John Smith <john@example.org>
+  project-url-template =
+    Code repository, http://example.org/hacking/projects/{name}
+    Documentation, http://packages.python.org/{name}
+
+..  It’s tempting to reuse the ``metadata`` section name but it may be
+    confusing, since it would mean that the same section is used by two
+    commands depending on the origin file, which is not an expectation in
+    the Distutils model (which I will already change to introduce sections
+    that are not allowed at all config file levels). It’s cleaner if the
+    program uses its own section in the user config file.
 
 For people wanting to upgrade progressively, Distutils2 includes a
 lib2to3-based converter to rewrite imports (:mod:`distutils2` provides a
 Appendix: Implementation details
 """"""""""""""""""""""""""""""""
 
+Multi-value fields
+------------------
 The `config file parser`_ strips leading and trailing whitespace for free, we
 just have to handle the case of the first line being empty (in the line
 ``spam =\nham``, the config file format considers there is an empty line after
 the equals). Handling that case is as simple as
 ``value.strip().splitlines())``.
 
-Helper functions to split the `source directory`_ specifier, resolve a dotted
+
+Support code
+------------
+Helper functions to split the `source directory`__ specifier, resolve a dotted
 name and split an environment marker will be provided in :mod:`util` and
 :mod:`config` for use by third-party tools. :mod:`config` will also provide
 higler-level functions to get a :class:`DistributionMetadata` instance from a
 filtered according to the environment, access a config section defined by a
 Distutils2 extension, and so on.
 
-The encoding of the config file is UTF-8.
+.. __: `replacing package_dir`_
 
-.. XXX use a special section name prefix for those extensions?
+.. _splitting the list of modules:
+
+If the proposal to merge the lists of modules and packages is accepted,
+Distutils2 code will have to sort this list into the three lists used by
+``Distribution``, following these simple rules:
+
+#. If the name has an ``extension`` section (or if it is listed in the
+   ``extensions`` section, depending on the proposal__ that gets accepted), an
+   instance of :class:`Extension` is created and added to
+   ``distribution.ext_modules``;
+#. If the name corresponds to an existing directory which contains an
+   ``__init__.py`` file, it is added to ``distribution.packages``;
+#. The name is added to ``distribution.py_modules``.
+
+.. __: `extension modules`_
+
+.. XXX use a special section name prefix for third parties extending d2?
 
 
 Appendix: Mapping from arguments to fields
 """"""""""""""""""""""""""""""""""""""""""
 
-============================ ==============================
+============================ ========================================
 Argument in :file:`setup.py` Field in :file:`setup.cfg`
-============================ ==============================
+============================ ========================================
 description                  summary
 long_description             description (changed meaning)
 author                       author
 
 packages                     packages or modules
 py_modules                   modules
-ext_modules                  modules (+ ``[extensions]``)
+ext_modules                  modules (+ extension(s) section)
 ext_package                  unsupported
 
 distclass                    distclass
 cmdclass                     cmdclasses
 script_name                  N/A
 script_args                  N/A
-options                      fields under ``[{command}]``
-============================ ==============================
+options                      fields in sections named after commands
+============================ ========================================
 
 
 Appendix: Rejected ideas
 
 .. Footnotes and References
    """"""""""""""""""""""""
-.. _source directory: `replacing package_dir`_
-.. _pre-build hook: `pre/post-command hooks`_
 .. _PEP 345 environment markers:
    http://www.python.org/dev/peps/pep-0345/#environment-markers
 .. _design document:
 .. _Python #8252: http://bugs.python.org/issue8252
 .. _Python #8253: http://bugs.python.org/issue8253
 .. _Python #870479: http://bugs.python.org/issue870479
-.. _pre/post-command hooks: http://bugs.python.org/issue8312
+.. _pre/post-command hooks:
+   http://distutils2.notmyidea.org/command_hooks.html
 .. _config file parser:
    http://docs.python.org/library/configparser#ConfigParser.RawConfigParser
 .. _command-packages:
 .. _httplib2: http://code.google.com/p/httplib2/source/browse/setup.py
 .. _DistUtilsExtra.auto:
    http://bazaar.launchpad.net/~python-distutils-extra-hackers/
-   python-distutils-extra/debian/annotate/head:/doc/README   
+   python-distutils-extra/debian/annotate/head:/doc/README