Source

python-peps / pep-0421.txt

The default branch has multiple heads

Full commit
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
PEP: 421
Title: Adding sys.implementation
Version: $Revision$
Last-Modified: $Date$
Author: Eric Snow <ericsnowcurrently@gmail.com>
BDFL-Delegate: Barry Warsaw
Status: Final
Type: Standards Track
Content-Type: text/x-rst
Created: 26-April-2012
Post-History: 26-April-2012
Resolution: http://mail.python.org/pipermail/python-dev/2012-May/119683.html


Abstract
========

This PEP introduces a new attribute for the ``sys`` module:
``sys.implementation``.  The attribute holds consolidated information
about the implementation of the running interpreter.  Thus
``sys.implementation`` is the source to which the standard library may
look for implementation-specific information.

The proposal in this PEP is in line with a broader emphasis on making
Python friendlier to alternate implementations.  It describes the new
variable and the constraints on what that variable contains.  The PEP
also explains some immediate use cases for ``sys.implementation``.


Motivation
==========

For a number of years now, the distinction between Python-the-language
and CPython (the reference implementation) has been growing.  Most of
this change is due to the emergence of Jython, IronPython, and PyPy as
viable alternate implementations of Python.

Consider, however, the nearly two decades of CPython-centric Python
(i.e. most of its existence).  That focus has understandably
contributed to quite a few CPython-specific artifacts both in the
standard library and exposed in the interpreter.  Though the core
developers have made an effort in recent years to address this, quite
a few of the artifacts remain.

Part of the solution is presented in this PEP: a single namespace in
which to consolidate implementation specifics.  This will help focus
efforts to differentiate the implementation specifics from the
language.  Additionally, it will foster a multiple-implementation
mindset.


Proposal
========

We will add a new attribute to the ``sys`` module, called
``sys.implementation``, as an object with attribute-access (as opposed
to a mapping).  It will contain implementation-specific information.

The attributes of this object will remain fixed during interpreter
execution and through the course of an implementation version.  This
ensures behaviors don't change between versions which depend on
attributes of ``sys.implementation``.

The object has each of the attributes described in the `Required
Attributes`_ section below.  Those attribute names will never start
with an underscore.  The standard library and the language definition
will rely only on those required attributes.

This proposal takes a conservative approach in requiring only a small
number of attributes.  As more become appropriate, they may be added
with discretion, as described in `Adding New Required Attributes`_.

While this PEP places no other constraints on ``sys.implementation``,
it also recommends that no one rely on capabilities outside those
described here.  The only exception to that recommendation is for
attributes starting with an underscore.  Implementers may use those
as appropriate to store per-implementation data.


Required Attributes
-------------------

These are attributes in ``sys.implementation`` on which the standard
library and language definition will rely, meaning implementers must
define them:

**name**
   A lower-case identifier representing the implementation.  Examples
   include 'pypy', 'jython', 'ironpython', and 'cpython'.

**version**
   The version of the implementation, as opposed to the version of the
   language it implements.  This value conforms to the format described
   in `Version Format`_.

**hexversion**
   The version of the implementation in the same hexadecimal format as
   ``sys.hexversion``.

**cache_tag**
   A string used for the PEP 3147 cache tag [#cachetag]_.  It would
   normally be a composite of the name and version (e.g. 'cpython-33'
   for CPython 3.3).  However, an implementation may explicitly use a
   different cache tag.  If ``cache_tag`` is set to None, it indicates
   that module caching should be disabled.


Adding New Required Attributes
------------------------------

In time more required attributes will be added to
``sys.implementation``.  However, each must have a meaningful use case
across all Python implementations in order to be considered.  This is
made most clear by a use case in the standard library or language
specification.

All proposals for new required attributes will go through the normal
PEP process.  Such a PEP need not be long, just long enough.  It will
need to sufficiently spell out the rationale for the new attribute,
its use cases, and the impact it will have on the various Python
implementations.


Version Format
--------------

A main point of ``sys.implementation`` is to contain information that
will be used internally in the standard library.  In order to
facilitate the usefulness of the version attribute, its value should
be in a consistent format across implementations.

As such, the format of ``sys.implementation.version`` will follow that
of ``sys.version_info``, which is effectively a named tuple.  It is a
familiar format and generally consistent with normal version format
conventions.


Rationale
=========

The status quo for implementation-specific information gives us that
information in a more fragile, harder to maintain way.  It is spread
out over different modules or inferred from other information, as we
see with `platform.python_implementation()`_.

This PEP is the main alternative to that approach.  It consolidates
the implementation-specific information into a single namespace and
makes explicit that which was implicit.


Type Considerations
-------------------

It's very easy to get bogged down in discussions about the type of
``sys.implementation``.  However, its purpose is to support the
standard library and language definition.  As such, there isn't much
that really matters regarding its type, as opposed to a feature that
would be more generally used.  Thus characteristics like immutability
and sequence-ness have been disregarded.

The only real choice has been between an object with attribute access
and a mapping with item access.  This PEP espouses dotted access to
reflect the relatively fixed nature of the namespace.


Non-Required Attributes
-----------------------

Earlier versions of this PEP included a required attribute called
``metadata`` that held any non-required, per-implementation data
[#Nick]_.  However, this proved to be an unnecessary addition
considering the purpose of ``sys.implementation``.

Ultimately, non-required attributes are virtually ignored in this PEP.
They have no impact other than that careless use may collide with
future required attributes.  That, however, is but a marginal concern
for ``sys.implementation``.


Why a Part of ``sys``?
----------------------

The ``sys`` module holds the new namespace because ``sys`` is the depot
for interpreter-centric variables and functions.  Many
implementation-specific attributes are already found in ``sys``.


Why Strict Constraints on Any of the Values?
--------------------------------------------

As already noted in `Version Format`_, values in
``sys.implementation`` are intended for use by the standard library.
Constraining those values, essentially specifying an API for them,
allows them to be used consistently, regardless of how they are
otherwise implemented.  However, care should be take to not
over-specify the constraints.


Discussion
==========

The topic of ``sys.implementation`` came up on the python-ideas list
in 2009, where the reception was broadly positive [#original]_.  I
revived the discussion recently while working on a pure-python
``imp.get_tag()`` [#revived]_.  Discussion has been ongoing
[#feedback]_.  The messages in `issue #14673`_ are also relevant.

A good part of the recent discussion centered on the type to use for
``sys.implementation``.


Use-cases
=========

platform.python_implementation()
--------------------------------

"explicit is better than implicit"

The ``platform`` module determines the python implementation by looking
for clues in a couple different ``sys`` variables [#guess]_.  However,
this approach is fragile, requiring changes to the standard library
each time an implementation changes.  Beyond that, support in
``platform`` is limited to those implementations that core developers
have blessed by special-casing them in the ``platform`` module.

With ``sys.implementation`` the various implementations would
*explicitly* set the values in their own version of the ``sys``
module.

Another concern is that the ``platform`` module is part of the stdlib,
which ideally should minimize implementation details such as would be
moved to ``sys.implementation``.

Any overlap between ``sys.implementation`` and the ``platform`` module
would simply defer to ``sys.implementation`` (with the same interface
in ``platform`` wrapping it).


Cache Tag Generation in Frozen Importlib
----------------------------------------

PEP 3147 defined the use of a module cache and cache tags for file
names.  The importlib bootstrap code, frozen into the Python binary as
of 3.3, uses the cache tags during the import process.  Part of the
project to bootstrap importlib has been to clean code out of
`Python/import.c`_ that did not need to be there any longer.

The cache tag defined in ``Python/import.c`` was hard-coded to
``"cpython" MAJOR MINOR`` [#cachetag]_.  For importlib the options are
either hard-coding it in the same way, or guessing the implementation
in the same way as does ``platform.python_implementation()``.

As long as the hard-coded tag is limited to CPython-specific code, it
is livable.  However, inasmuch as other Python implementations use the
importlib code to work with the module cache, a hard-coded tag would
become a problem.

Directly using the ``platform`` module in this case is a non-starter.
Any module used in the importlib bootstrap must be built-in or frozen,
neither of which apply to the ``platform`` module.  This is the point
that led to the recent interest in ``sys.implementation``.

Regardless of the outcome for the implementation name used, another
problem relates to the version used in the cache tag.  That version is
likely to be the implementation version rather than the language
version.  However, the implementation version is not readily
identified anywhere in the standard library.


Implementation-Specific Tests
-----------------------------

Currently there are a number of implementation-specific tests in the
test suite under ``Lib/test``.  The test support module
(`Lib/test/support.py`_) provides some functionality for dealing with
these tests.  However, like the ``platform`` module, ``test.support``
must do some guessing that ``sys.implementation`` would render
unnecessary.


Jython's ``os.name`` Hack
-------------------------

In Jython, ``os.name`` is set to 'java' to accommodate special
treatment of the java environment in the standard library [#os_name]_
[#javatest]_.  Unfortunately it masks the os name that would otherwise
go there.  ``sys.implementation`` would help obviate the need for this
special case.  Currently Jython sets ``os._name`` for the normal
``os.name`` value.


The Problem With ``sys.(version|version_info|hexversion)``
----------------------------------------------------------

Earlier versions of this PEP made the mistake of calling
``sys.version_info`` (and friends) the version of the Python language,
in contrast to the implementation.  However, this is not the case.
Instead, it is the version of the CPython implementation.  Incidentally,
the first two components of ``sys.version_info`` (major and minor) also
reflect the version of the language definition.

As Barry Warsaw noted, the "semantics of sys.version_info have been
sufficiently squishy in the past" [#Barry]_.  With
``sys.implementation`` we have the opportunity to improve this
situation by first establishing an explicit location for the version of
the implementation.

This PEP makes no other effort to directly clarify the semantics of
``sys.version_info``.  Regardless, having an explicit version for the
implementation will definitely help to clarify the distinction from the
language version.


Feedback From Other Python Implementers
=======================================

IronPython
----------

Jeff Hardy responded to a request for feedback [#ironpython]_.  He
said, "I'll probably add it the day after it's approved"
[#jeff_hardy_2012]_.  He also gave useful feedback on both the type of
``sys.implementation`` and on the ``metadata`` attribute (which has
since been removed from the PEP).

Jython
------

In 2009 Frank Wierzbicki said this (relative to Jython implementing the
required attributes) [#frank_wierzbicki_2009]_::

   Speaking for Jython, so far it looks like something we would adopt
   soonish after it was accepted (it looks pretty useful to me).

PyPy
----

Some of the PyPy developers have responded to a request for feedback
[#pypy]_.  Armin Rigo said the following [#armin_rigo_2012]_::

   For myself, I can only say that it looks like a good idea, which we
   will happily adhere to when we migrate to Python 3.3.

He also expressed support for keeping the required list small.  Both
Armin and Laura Creighton indicated that an effort to better catalog
Python's implementation would be welcome.  Such an effort, for which
this PEP is a small start, will be considered separately.


Past Efforts
============

``PEP 3139``
------------

PEP 3139, from 2008, recommended a clean-up of the ``sys`` module in
part by extracting implementation-specific variables and functions
into a separate module.  PEP 421 is less ambitious version of that
idea.  While PEP 3139 was rejected, its goals are reflected in PEP 421
to a large extent, though with a much lighter approach.


``PEP 399``
-----------

PEP 399 dictates policy regarding the standard library, helping to make
it friendlier to alternate implementations.  PEP 421 is proposed in
that same spirit.


The Bigger Picture
==================

It's worth noting again that this PEP is a small part of a larger
on-going effort to identify the implementation-specific parts of Python
and mitigate their impact on alternate implementations.

``sys.implementation`` is a focal point for implementation-specific
data, acting as a nexus for cooperation between the language, the
standard library, and the different implementations.  As time goes by
it is feasible that ``sys.implementation`` will assume current
attributes of ``sys`` and other builtin/stdlib modules, where
appropriate.  In this way, it is a PEP 3137-lite, but starting as
small as possible.

However, as already noted, many other efforts predate
``sys.implementation``.   Neither is it necessarily a major part of the
effort.  Rather, consider it as part of the infrastructure of the
effort to make Python friendlier to alternate implementations.


Alternatives
============

Since the single-namespace-under-sys approach is relatively
straightforward, no alternatives have been considered for this PEP.


Examples of Other Attributes
============================

These are examples only and not part of the proposal.  Most of them
were suggested during previous discussions, but did not fit into the
goals of this PEP.  (See `Adding New Required Attributes`_ if they get
you excited.)

**common_name**
   The case-sensitive name by which the implementation is known.

**vcs_url**
   A URL for the main VCS repository for the implementation project.

**vcs_revision_id**
   A value that identifies the VCS revision of the implementation.

**build_toolchain**
   The tools used to build the interpreter.

**build_date**
   The timestamp of when the interpreter was built.

**homepage**
   The URL of the implementation's website.

**site_prefix**
   The preferred site prefix for the implementation.

**runtime**
   The run-time environment in which the interpreter is running, as
   in "Common Language *Runtime*" (.NET CLR) or "Java *Runtime*
   Executable".

**gc_type**
   The type of garbage collection used, like "reference counting" or
   "mark and sweep".


Open Issues
===========

Currently none.


Implementation
==============

The implementation of this PEP is covered in `issue #14673`_.


References
==========

.. [#original] The 2009 sys.implementation discussion:
   http://mail.python.org/pipermail/python-dev/2009-October/092893.html

.. [#revived] The initial 2012 discussion:
   http://mail.python.org/pipermail/python-ideas/2012-March/014555.html
   (and http://mail.python.org/pipermail/python-ideas/2012-April/014878.html)

.. [#feedback] Feedback on the PEP:
   http://mail.python.org/pipermail/python-ideas/2012-April/014954.html

.. [#ironpython] Feedback from the IronPython developers:
   http://mail.python.org/pipermail/ironpython-users/2012-May/015980.html

.. [#dino_viehland_2009] (2009) Dino Viehland offers his opinion:
   http://mail.python.org/pipermail/python-dev/2009-October/092894.html

.. [#jeff_hardy_2012] (2012) Jeff Hardy offers his opinion:
   http://mail.python.org/pipermail/ironpython-users/2012-May/015981.html

.. [#jython] Feedback from the Jython developers:
   ???

.. [#frank_wierzbicki_2009] (2009) Frank Wierzbicki offers his opinion:
   http://mail.python.org/pipermail/python-dev/2009-October/092974.html

.. [#pypy] Feedback from the PyPy developers:
   http://mail.python.org/pipermail/pypy-dev/2012-May/009883.html

.. [#armin_rigo_2012] (2012) Armin Rigo offers his opinion:
   http://mail.python.org/pipermail/pypy-dev/2012-May/009884.html

.. [#guess] The ``platform`` code which divines the implementation name:
   http://hg.python.org/cpython/file/2f563908ebc5/Lib/platform.py#l1247

.. [#cachetag] The definition for cache tags in PEP 3147:
   http://www.python.org/dev/peps/pep-3147/#id53

.. [#tag_impl] The original implementation of the cache tag in CPython:
   http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c#l121

.. [#tests] Examples of implementation-specific handling in test.support:
   * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l509
   * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1246
   * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1252
   * http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l1275

.. [#os_name] The standard library entry for os.name:
   http://docs.python.org/3.3/library/os.html#os.name

.. [#javatest] The use of ``os.name`` as 'java' in the stdlib test suite.
   http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py#l512

.. [#Nick] Nick Coghlan's proposal for ``sys.implementation.metadata``:
   http://mail.python.org/pipermail/python-ideas/2012-May/014984.html

.. [#Barry] Feedback from Barry Warsaw:
   http://mail.python.org/pipermail/python-dev/2012-May/119374.html

.. _issue #14673: http://bugs.python.org/issue14673

.. _Lib/test/support.py: http://hg.python.org/cpython/file/2f563908ebc5/Lib/test/support.py

.. _Python/import.c:  http://hg.python.org/cpython/file/2f563908ebc5/Python/import.c


Copyright
=========

This document has been placed in the public domain.



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End: