Source

cpython_sandbox / Doc / library / codecs.rst

Full commit

:mod:`codecs` --- Codec registry and base classes

This module defines base classes for standard Python codecs (encoders and decoders) and provides access to the internal Python codec registry which manages the codec and error handling lookup process.

It defines the following functions:

To simplify access to the various codecs, the module provides these additional functions which use :func:`lookup` for the codec lookup:

To simplify working with encoded files or stream, the module also defines these utility functions:

The module also provides the following constants which are useful for reading and writing to platform dependent files:

Codec Base Classes

The :mod:`codecs` module defines a set of base classes which define the interface and can also be used to easily write your own codecs for use in Python.

Each codec has to define four interfaces to make it usable as codec in Python: stateless encoder, stateless decoder, stream reader and stream writer. The stream reader and writers typically reuse the stateless encoder/decoder to implement the file protocols.

The :class:`Codec` class defines the interface for stateless encoders/decoders.

To simplify and standardize error handling, the :meth:`encode` and :meth:`decode` methods may implement different error handling schemes by providing the errors string argument. The following string values are defined and implemented by all standard Python codecs:

Value Meaning
'strict' Raise :exc:`UnicodeError` (or a subclass); this is the default.
'ignore' Ignore the character and continue with the next.
'replace' Replace with a suitable replacement character; Python will use the official U+FFFD REPLACEMENT CHARACTER for the built-in Unicode codecs on decoding and '?' on encoding.
'xmlcharrefreplace' Replace with the appropriate XML character reference (only for encoding).
'backslashreplace' Replace with backslashed escape sequences (only for encoding).
'surrogateescape' Replace byte with surrogate U+DCxx, as defined in PEP 383.

In addition, the following error handlers are specific to a single codec:

Value Codec Meaning
'surrogatepass' utf-8 Allow encoding and decoding of surrogate codes in UTF-8.

The set of allowed values can be extended via :meth:`register_error`.

Codec Objects

The :class:`Codec` class defines these methods which also define the function interfaces of the stateless encoder and decoder:

The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide the basic interface for incremental encoding and decoding. Encoding/decoding the input isn't done with one call to the stateless encoder/decoder function, but with multiple calls to the :meth:`encode`/:meth:`decode` method of the incremental encoder/decoder. The incremental encoder/decoder keeps track of the encoding/decoding process during method calls.

The joined output of calls to the :meth:`encode`/:meth:`decode` method is the same as if all the single inputs were joined into one, and this input was encoded/decoded with the stateless encoder/decoder.

IncrementalEncoder Objects

The :class:`IncrementalEncoder` class is used for encoding an input in multiple steps. It defines the following methods which every incremental encoder must define in order to be compatible with the Python codec registry.

Constructor for an :class:`IncrementalEncoder` instance.

All incremental encoders must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry.

The :class:`IncrementalEncoder` may implement different error handling schemes by providing the errors keyword argument. These parameters are predefined:

  • 'strict' Raise :exc:`ValueError` (or a subclass); this is the default.
  • 'ignore' Ignore the character and continue with the next.
  • 'replace' Replace with a suitable replacement character
  • 'xmlcharrefreplace' Replace with the appropriate XML character reference
  • 'backslashreplace' Replace with backslashed escape sequences.

The errors argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`IncrementalEncoder` object.

The set of allowed values for the errors argument can be extended with :func:`register_error`.

IncrementalDecoder Objects

The :class:`IncrementalDecoder` class is used for decoding an input in multiple steps. It defines the following methods which every incremental decoder must define in order to be compatible with the Python codec registry.

Constructor for an :class:`IncrementalDecoder` instance.

All incremental decoders must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry.

The :class:`IncrementalDecoder` may implement different error handling schemes by providing the errors keyword argument. These parameters are predefined:

  • 'strict' Raise :exc:`ValueError` (or a subclass); this is the default.
  • 'ignore' Ignore the character and continue with the next.
  • 'replace' Replace with a suitable replacement character.

The errors argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`IncrementalDecoder` object.

The set of allowed values for the errors argument can be extended with :func:`register_error`.

The :class:`StreamWriter` and :class:`StreamReader` classes provide generic working interfaces which can be used to implement new encoding submodules very easily. See :mod:`encodings.utf_8` for an example of how this is done.

StreamWriter Objects

The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the following methods which every stream writer must define in order to be compatible with the Python codec registry.

Constructor for a :class:`StreamWriter` instance.

All stream writers must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry.

stream must be a file-like object open for writing binary data.

The :class:`StreamWriter` may implement different error handling schemes by providing the errors keyword argument. These parameters are predefined:

  • 'strict' Raise :exc:`ValueError` (or a subclass); this is the default.
  • 'ignore' Ignore the character and continue with the next.
  • 'replace' Replace with a suitable replacement character
  • 'xmlcharrefreplace' Replace with the appropriate XML character reference
  • 'backslashreplace' Replace with backslashed escape sequences.

The errors argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`StreamWriter` object.

The set of allowed values for the errors argument can be extended with :func:`register_error`.

In addition to the above methods, the :class:`StreamWriter` must also inherit all other methods and attributes from the underlying stream.

StreamReader Objects

The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the following methods which every stream reader must define in order to be compatible with the Python codec registry.

Constructor for a :class:`StreamReader` instance.

All stream readers must provide this constructor interface. They are free to add additional keyword arguments, but only the ones defined here are used by the Python codec registry.

stream must be a file-like object open for reading (binary) data.

The :class:`StreamReader` may implement different error handling schemes by providing the errors keyword argument. These parameters are defined:

  • 'strict' Raise :exc:`ValueError` (or a subclass); this is the default.
  • 'ignore' Ignore the character and continue with the next.
  • 'replace' Replace with a suitable replacement character.

The errors argument will be assigned to an attribute of the same name. Assigning to this attribute makes it possible to switch between different error handling strategies during the lifetime of the :class:`StreamReader` object.

The set of allowed values for the errors argument can be extended with :func:`register_error`.

In addition to the above methods, the :class:`StreamReader` must also inherit all other methods and attributes from the underlying stream.

The next two base classes are included for convenience. They are not needed by the codec registry, but may provide useful in practice.

StreamReaderWriter Objects

The :class:`StreamReaderWriter` allows wrapping streams which work in both read and write modes.

The design is such that one can use the factory functions returned by the :func:`lookup` function to construct the instance.

Creates a :class:`StreamReaderWriter` instance. stream must be a file-like object. Reader and Writer must be factory functions or classes providing the :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling is done in the same way as defined for the stream readers and writers.

:class:`StreamReaderWriter` instances define the combined interfaces of :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other methods and attributes from the underlying stream.

StreamRecoder Objects

The :class:`StreamRecoder` provide a frontend - backend view of encoding data which is sometimes useful when dealing with different encoding environments.

The design is such that one can use the factory functions returned by the :func:`lookup` function to construct the instance.

Creates a :class:`StreamRecoder` instance which implements a two-way conversion: encode and decode work on the frontend (the input to :meth:`read` and output of :meth:`write`) while Reader and Writer work on the backend (reading and writing to the stream).

You can use these objects to do transparent direct recodings from e.g. Latin-1 to UTF-8 and back.

stream must be a file-like object.

encode, decode must adhere to the :class:`Codec` interface. Reader, Writer must be factory functions or classes providing objects of the :class:`StreamReader` and :class:`StreamWriter` interface respectively.

encode and decode are needed for the frontend translation, Reader and Writer for the backend translation.

Error handling is done in the same way as defined for the stream readers and writers.

:class:`StreamRecoder` instances define the combined interfaces of :class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other methods and attributes from the underlying stream.

Encodings and Unicode

Strings are stored internally as sequences of codepoints in range 0 - 10FFFF (see PEP 393 for more details about the implementation). Once a string object is used outside of CPU and memory, CPU endianness and how these arrays are stored as bytes become an issue. Transforming a string object into a sequence of bytes is called encoding and recreating the string object from the sequence of bytes is known as decoding. There are many different methods for how this transformation can be done (these methods are also called encodings). The simplest method is to map the codepoints 0-255 to the bytes 0x0-0xff. This means that a string object that contains codepoints above U+00FF can't be encoded with this method (which is called 'latin-1' or 'iso-8859-1'). :func:`str.encode` will raise a :exc:`UnicodeEncodeError` that looks like this: UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in position 3: ordinal not in range(256).

There's another group of encodings (the so called charmap encodings) that choose a different subset of all Unicode code points and how these codepoints are mapped to the bytes 0x0-0xff. To see how this is done simply open e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on Windows). There's a string constant with 256 characters that shows you which character is mapped to which byte value.

All of these encodings can only encode 256 of the 1114112 codepoints defined in Unicode. A simple and straightforward way that can store each Unicode code point, is to store each codepoint as four consecutive bytes. There are two possibilities: store the bytes in big endian or in little endian order. These two encodings are called UTF-32-BE and UTF-32-LE respectively. Their disadvantage is that if e.g. you use UTF-32-BE on a little endian machine you will always have to swap bytes on encoding and decoding. UTF-32 avoids this problem: bytes will always be in natural endianness. When these bytes are read by a CPU with a different endianness, then bytes have to be swapped though. To be able to detect the endianness of a UTF-16 or UTF-32 byte sequence, there's the so called BOM ("Byte Order Mark"). This is the Unicode character U+FEFF. This character can be prepended to every UTF-16 or UTF-32 byte sequence. The byte swapped version of this character (0xFFFE) is an illegal character that may not appear in a Unicode text. So when the first character in an UTF-16 or UTF-32 byte sequence appears to be a U+FFFE the bytes have to be swapped on decoding. Unfortunately the character U+FEFF had a second purpose as a ZERO WIDTH NO-BREAK SPACE: a character that has no width and doesn't allow a word to be split. It can e.g. be used to give hints to a ligature algorithm. With Unicode 4.0 using U+FEFF as a ZERO WIDTH NO-BREAK SPACE has been deprecated (with U+2060 (WORD JOINER) assuming this role). Nevertheless Unicode software still must be able to handle U+FEFF in both roles: as a BOM it's a device to determine the storage layout of the encoded bytes, and vanishes once the byte sequence has been decoded into a string; as a ZERO WIDTH NO-BREAK SPACE it's a normal character that will be decoded like any other.

There's another encoding that is able to encoding the full range of Unicode characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two parts: marker bits (the most significant bits) and payload bits. The marker bits are a sequence of zero to four 1 bits followed by a 0 bit. Unicode characters are encoded like this (with x being payload bits, which when concatenated give the Unicode character):

Range Encoding
U-00000000 ... U-0000007F 0xxxxxxx
U-00000080 ... U-000007FF 110xxxxx 10xxxxxx
U-00000800 ... U-0000FFFF 1110xxxx 10xxxxxx 10xxxxxx
U-00010000 ... U-0010FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The least significant bit of the Unicode character is the rightmost x bit.

As UTF-8 is an 8-bit encoding no BOM is required and any U+FEFF character in the decoded string (even if it's the first character) is treated as a ZERO WIDTH NO-BREAK SPACE.

Without external information it's impossible to reliably determine which encoding was used for encoding a string. Each charmap encoding can decode any random byte sequence. However that's not possible with UTF-8, as UTF-8 byte sequences have a structure that doesn't allow arbitrary byte sequences. To increase the reliability with which a UTF-8 encoding can be detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls "utf-8-sig") for its Notepad program: Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: 0xef, 0xbb, 0xbf) is written. As it's rather improbable that any charmap encoded file starts with these byte values (which would e.g. map to

LATIN SMALL LETTER I WITH DIAERESIS
RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
INVERTED QUESTION MARK

in iso-8859-1), this increases the probability that a utf-8-sig encoding can be correctly guessed from the byte sequence. So here the BOM is not used to be able to determine the byte order used for generating the byte sequence, but as a signature that helps in guessing the encoding. On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file. On decoding utf-8-sig will skip those three bytes if they appear as the first three bytes in the file. In UTF-8, the use of the BOM is discouraged and should generally be avoided.

Standard Encodings

Python comes with a number of codecs built-in, either implemented as C functions or with dictionaries as mapping tables. The following table lists the codecs by name, together with a few common aliases, and the languages for which the encoding is likely used. Neither the list of aliases nor the list of languages is meant to be exhaustive. Notice that spelling alternatives that only differ in case or use a hyphen instead of an underscore are also valid aliases; therefore, e.g. 'utf-8' is a valid alias for the 'utf_8' codec.

Many of the character sets support the same languages. They vary in individual characters (e.g. whether the EURO SIGN is supported or not), and in the assignment of characters to code positions. For the European languages in particular, the following variants typically exist:

  • an ISO 8859 codeset
  • a Microsoft Windows code page, which is typically derived from a 8859 codeset, but replaces control characters with additional graphic characters
  • an IBM EBCDIC code page
  • an IBM PC code page, which is ASCII compatible
Codec Aliases Languages
ascii 646, us-ascii English
big5 big5-tw, csbig5 Traditional Chinese
big5hkscs big5-hkscs, hkscs Traditional Chinese
cp037 IBM037, IBM039 English
cp424 EBCDIC-CP-HE, IBM424 Hebrew
cp437 437, IBM437 English
cp500 EBCDIC-CP-BE, EBCDIC-CP-CH, IBM500 Western Europe
cp720   Arabic
cp737   Greek
cp775 IBM775 Baltic languages
cp850 850, IBM850 Western Europe
cp852 852, IBM852 Central and Eastern Europe
cp855 855, IBM855 Bulgarian, Byelorussian, Macedonian, Russian, Serbian
cp856   Hebrew
cp857 857, IBM857 Turkish
cp858 858, IBM858 Western Europe
cp860 860, IBM860 Portuguese
cp861 861, CP-IS, IBM861 Icelandic
cp862 862, IBM862 Hebrew
cp863 863, IBM863 Canadian
cp864 IBM864 Arabic
cp865 865, IBM865 Danish, Norwegian
cp866 866, IBM866 Russian
cp869 869, CP-GR, IBM869 Greek
cp874   Thai
cp875   Greek
cp932 932, ms932, mskanji, ms-kanji Japanese
cp949 949, ms949, uhc Korean
cp950 950, ms950 Traditional Chinese
cp1006   Urdu
cp1026 ibm1026 Turkish
cp1140 ibm1140 Western Europe
cp1250 windows-1250 Central and Eastern Europe
cp1251 windows-1251 Bulgarian, Byelorussian, Macedonian, Russian, Serbian
cp1252 windows-1252 Western Europe
cp1253 windows-1253 Greek
cp1254 windows-1254 Turkish
cp1255 windows-1255 Hebrew
cp1256 windows-1256 Arabic
cp1257 windows-1257 Baltic languages
cp1258 windows-1258 Vietnamese
cp65001   Windows only: Windows UTF-8 (CP_UTF8)
euc_jp eucjp, ujis, u-jis Japanese
euc_jis_2004 jisx0213, eucjis2004 Japanese
euc_jisx0213 eucjisx0213 Japanese
euc_kr euckr, korean, ksc5601, ks_c-5601, ks_c-5601-1987, ksx1001, ks_x-1001 Korean
gb2312 chinese, csiso58gb231280, euc- cn, euccn, eucgb2312-cn, gb2312-1980, gb2312-80, iso- ir-58 Simplified Chinese
gbk 936, cp936, ms936 Unified Chinese
gb18030 gb18030-2000 Unified Chinese
hz hzgb, hz-gb, hz-gb-2312 Simplified Chinese
iso2022_jp csiso2022jp, iso2022jp, iso-2022-jp Japanese
iso2022_jp_1 iso2022jp-1, iso-2022-jp-1 Japanese
iso2022_jp_2 iso2022jp-2, iso-2022-jp-2 Japanese, Korean, Simplified Chinese, Western Europe, Greek
iso2022_jp_2004 iso2022jp-2004, iso-2022-jp-2004 Japanese
iso2022_jp_3 iso2022jp-3, iso-2022-jp-3 Japanese
iso2022_jp_ext iso2022jp-ext, iso-2022-jp-ext Japanese
iso2022_kr csiso2022kr, iso2022kr, iso-2022-kr Korean
latin_1 iso-8859-1, iso8859-1, 8859, cp819, latin, latin1, L1 West Europe
iso8859_2 iso-8859-2, latin2, L2 Central and Eastern Europe
iso8859_3 iso-8859-3, latin3, L3 Esperanto, Maltese
iso8859_4 iso-8859-4, latin4, L4 Baltic languages
iso8859_5 iso-8859-5, cyrillic Bulgarian, Byelorussian, Macedonian, Russian, Serbian
iso8859_6 iso-8859-6, arabic Arabic
iso8859_7 iso-8859-7, greek, greek8 Greek
iso8859_8 iso-8859-8, hebrew Hebrew
iso8859_9 iso-8859-9, latin5, L5 Turkish
iso8859_10 iso-8859-10, latin6, L6 Nordic languages
iso8859_13 iso-8859-13, latin7, L7 Baltic languages
iso8859_14 iso-8859-14, latin8, L8 Celtic languages
iso8859_15 iso-8859-15, latin9, L9 Western Europe
iso8859_16 iso-8859-16, latin10, L10 South-Eastern Europe
johab cp1361, ms1361 Korean
koi8_r   Russian
koi8_u   Ukrainian
mac_cyrillic maccyrillic Bulgarian, Byelorussian, Macedonian, Russian, Serbian
mac_greek macgreek Greek
mac_iceland maciceland Icelandic
mac_latin2 maclatin2, maccentraleurope Central and Eastern Europe
mac_roman macroman, macintosh Western Europe
mac_turkish macturkish Turkish
ptcp154 csptcp154, pt154, cp154, cyrillic-asian Kazakh
shift_jis csshiftjis, shiftjis, sjis, s_jis Japanese
shift_jis_2004 shiftjis2004, sjis_2004, sjis2004 Japanese
shift_jisx0213 shiftjisx0213, sjisx0213, s_jisx0213 Japanese
utf_32 U32, utf32 all languages
utf_32_be UTF-32BE all languages
utf_32_le UTF-32LE all languages
utf_16 U16, utf16 all languages
utf_16_be UTF-16BE all languages
utf_16_le UTF-16LE all languages
utf_7 U7, unicode-1-1-utf-7 all languages
utf_8 U8, UTF, utf8 all languages
utf_8_sig   all languages

Python Specific Encodings

A number of predefined codecs are specific to Python, so their codec names have no meaning outside Python. These are listed in the tables below based on the expected input and output types (note that while text encodings are the most common use case for codecs, the underlying codec infrastructure supports arbitrary data transforms rather than just text encodings). For asymmetric codecs, the stated purpose describes the encoding direction.

The following codecs provide :class:`str` to :class:`bytes` encoding and :term:`bytes-like object` to :class:`str` decoding, similar to the Unicode text encodings.

Codec Aliases Purpose
idna   Implements RFC 3490, see also :mod:`encodings.idna`
mbcs dbcs Windows only: Encode operand according to the ANSI codepage (CP_ACP)
palmos   Encoding of PalmOS 3.5
punycode   Implements RFC 3492
raw_unicode_escape   Produce a string that is suitable as raw Unicode literal in Python source code
undefined   Raise an exception for all conversions. Can be used as the system encoding if no automatic coercion between byte and Unicode strings is desired.
unicode_escape   Produce a string that is suitable as Unicode literal in Python source code
unicode_internal   Return the internal representation of the operand

The following codecs provide :term:`bytes-like object` to :class:`bytes` mappings.

Codec Purpose Encoder/decoder
base64_codec [1] Convert operand to MIME base64 (the result always includes a trailing '\n') :meth:`base64.b64encode`, :meth:`base64.b64decode`
bz2_codec Compress the operand using bz2 :meth:`bz2.compress`, :meth:`bz2.decompress`
hex_codec Convert operand to hexadecimal representation, with two digits per byte :meth:`base64.b16encode`, :meth:`base64.b16decode`
quopri_codec Convert operand to MIME quoted printable :meth:`quopri.encodestring`, :meth:`quopri.decodestring`
uu_codec Convert the operand using uuencode :meth:`uu.encode`, :meth:`uu.decode`
zlib_codec Compress the operand using gzip :meth:`zlib.compress`, :meth:`zlib.decompress`
[1]Rather than accepting any :term:`bytes-like object`, 'base64_codec' accepts only :class:`bytes` and :class:`bytearray` for encoding and only :class:`bytes`, :class:`bytearray`, and ASCII-only instances of :class:`str` for decoding

The following codecs provide :class:`str` to :class:`str` mappings.

Codec Purpose
rot_13 Returns the Caesar-cypher encryption of the operand

:mod:`encodings.idna` --- Internationalized Domain Names in Applications

This module implements RFC 3490 (Internationalized Domain Names in Applications) and RFC 3492 (Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)). It builds upon the punycode encoding and :mod:`stringprep`.

These RFCs together define a protocol to support non-ASCII characters in domain names. A domain name containing non-ASCII characters (such as www.Alliancefrançaise.nu) is converted into an ASCII-compatible encoding (ACE, such as www.xn--alliancefranaise-npb.nu). The ACE form of the domain name is then used in all places where arbitrary characters are not allowed by the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so on. This conversion is carried out in the application; if possible invisible to the user: The application should transparently convert Unicode domain labels to IDNA on the wire, and convert back ACE labels to Unicode before presenting them to the user.

Python supports this conversion in several ways: the idna codec performs conversion between Unicode and ACE, separating an input string into labels based on the separator characters defined in section 3.1 (1) of RFC 3490 and converting each label to ACE as required, and conversely separating an input byte string into labels based on the . separator and converting any ACE labels found into unicode. Furthermore, the :mod:`socket` module transparently converts Unicode host names to ACE, so that applications need not be concerned about converting host names themselves when they pass them to the socket module. On top of that, modules that have host names as function parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host names (:mod:`http.client` then also transparently sends an IDNA hostname in the :mailheader:`Host` field if it sends that field at all).

When receiving host names from the wire (such as in reverse name lookup), no automatic conversion to Unicode is performed: Applications wishing to present such host names to the user should decode them to Unicode.

The module :mod:`encodings.idna` also implements the nameprep procedure, which performs certain normalizations on host names, to achieve case-insensitivity of international domain names, and to unify similar characters. The nameprep functions can be used directly if desired.

:mod:`encodings.mbcs` --- Windows ANSI codepage

Encode operand according to the ANSI codepage (CP_ACP).

Availability: Windows only.

:mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature

This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this is only done once (on the first write to the byte stream). For decoding an optional UTF-8 encoded BOM at the start of the data will be skipped.