Victor Stinner committed c61b891

pep cloexec: rewrite the rationale

  • Participants
  • Parent commits c73aee6

Comments (0)

Files changed (1)


+Inherited file descriptors issues
- * Third party modules may call ``fork()`` + ``exec()``
- * ``SocketServer``: "address already in use issue" error
- * ``fcntl()`` with ``FD_CLOEXEC`` flag is not atomic if another thread
-   calls ``fork()``
- * Leak of file descriptors: security vulnerability
- * ``os.O_CLOEXEC`` and ``os.pipe2()`` were added to Python 3.3
+On UNIX, subprocess closes file descriptors greater than 2 by default. All
+files opened by the parent process are automatically closed.
-New functions:
+There are other cases creating a subprocess or executing a new program where
+file descriptors are not closed: functiosn of the os.spawn*() family and third
+party modules calling ``exec()`` or ``fork()`` + ``exec()`` (using C or Python
+functions). In this case, file descriptors are shared between the parent and
+the child processes which is usually unexpected and causes various issues.
+Closing the file descriptor in the parent process doesn't close the related
+resource (file, socket, ...) because it is still open in the child process.
- * ``sys.getdefaultcloexec() -> bool``
- * ``sys.setdefaultcloexec(cloexec: bool)``
- * ``os.set_cloexec()``
+For example, the listening socket of TCPServer is not closed on ``exec()``: the
+child process is able to get connection from new clients; if the parent closes
+the socket and create a new listening socket on the same address, it would
+get an "address already is used" error.
-Add optional ``cloexec`` argument to:
+Leaking file descriptors is a major security vulnerability. An untrusted child
+process may read sensitive data like passwords and take control of the parent
+process though leaded file descriptors. It is for example a simple trick to
+escape from a chroot.
+Using ``fcntl()`` to set the close-on-exec flag is not safe in a multithreaded
+application: if a thread calls ``fork()`` and ``exec()`` between the creation
+of the file descriptor and the call to ``fcntl(fd, fcntl.F_SETFD, new_flags)``:
+the file descriptor will be inherited in the child process. More operating
+systems offer functions to set the flag during the creation of the file
+descriptor, which avoids the race condition.
+Python 3.2 added ``socket.SOCK_CLOEXEC``, Python 3.3 added ``os.O_CLOEXEC``
+flag and ``os.pipe2()`` function. So it is already possible to set atomically
+close-on-exec flag when opening a file  and creating a pipe or socket.
+The problem is that these flags and functions are not portable: only recent
+versions of operating systems support it. ``O_CLOEXEC`` flag was ignored by old
+Linux versions and so ``FD_CLOEXEC`` flag must be checked using ``fcntl()`` to
+ensure that ``O_CLOEXEC`` is supported by the Linux kernel version.
+This PEP proposes options to configure how the close-on-exec is set on file
+descriptors created by Python. The value can be changed globally, or explicitly
+when creating a new file descriptors.
+Add new functions:
+ * ``sys.getdefaultcloexec() -> bool``: get the current default value of the
+   close-on-exec flag
+ * ``sys.setdefaultcloexec(cloexec: bool)``: enable or disable close-on-exec
+   flag, the state of the flag can be overriden in each function creating a
+   file descriptor
+ * ``os.set_cloexec(fd: int, cloexec: bool)``: change the value of
+   close-on-exec flag
+Add a new optional ``cloexec`` argument to:
  * ``open()``: ``os.fdopen()`` is indirectly modified
  * ``os.dup()``, ``os.dup2()``
    * ``select.kqueue()``
    * ``socket.socket.recvmsg()``: use ``MSG_CMSG_CLOEXEC``, or ``os.set_cloexec()``
-The default value of ``cloexec`` is ``sys.getdefaultcloexec()``.
+The default value of the ``cloexec`` argument is ``sys.getdefaultcloexec()``.
+Only file descriptors created by the Python standard library will comply to
+The close-on-exec flag is unchanged for file descriptors created by third party
+modules calling directly C functions. Third party modules will have to be
+modified to read ``sys.getdefaultcloexec()`` to make them comply to this PEP.
+Applications still have to close explicitly file descriptors after a
+``fork()``.  The close-on-exec flag only closes file descriptors after
+``exec()``, and so after ``fork()`` + ``exec()``.
+Impacted functions and modules
+Impacted functions:
+ * ``os.forkpty()``
+ * ``http.server.CGIHTTPRequestHandler.run_cgi()``
 Impacted modules:
+ * ``multiprocessing``
  * ``socketserver``
- * ``xmlrpc.server``
  * ``subprocess``
  * ``tempfile``
+ * ``xmlrpc.server``
  * Maybe: ``signal``, ``threading``
-File descriptors created by third party modules do not respect
-``sys.setdefaultcloexec()`` if they are not created by functions of the Python
-standard library.
 XXX Should the ``cloexec`` argument be optional (default: True)? XXX
-Note: ``fcntl(fd, F_SETFD, flags)`` only supports one flag, so it would be
-possible to avoid ``fcntl(fd, F_GETFD)``. But it may drop other flags in
-the future, and so it is safer to keep the two functions calls.
+Note: ``fcntl(fd, F_SETFD, flags)`` only supports one flag (``FD_CLOEXEC``), so
+it would be possible to avoid ``fcntl(fd, F_GETFD)``. But it may drop other
+flags in the future, and so it is safer to keep the two functions calls.
- * ``dup3()`` with ``O_CLOEXEC`` flag [atomic]
  * ``fcntl(fd, F_DUPFD_CLOEXEC)`` [atomic]
  * ``dup()`` + ``os.set_cloexec(fd, True)`` [best-effort]
  * ``O_CLOEXEC``: available on Linux (2.6.23+), FreeBSD (8.3+).
    The flag is part of POSIX.1-2008.
  * ``socket()``: ``SOCK_CLOEXEC`` flag, available on Linux 2.6.27+
- * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux ???
- * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux ???
+ * ``fcntl()``: ``F_DUPFD_CLOEXEC`` flag, available on Linux 2.6.24+
+ * ``recvmsg()``: ``MSG_CMSG_CLOEXEC``, available on Linux 2.6.23+
 On Linux older than 2.6.23, ``O_CLOEXEC`` flag is simply ignored. So we have to
 check that the flag is supported by calling ``fcntl()``. If it doesn't work, we
 New functions:
- * ``dup3()``: available on Linux ???
- * ``pipe2()``: available on Linux ???
- * ``accept4()``: available on Linux 2.6.28
+ * ``dup3()``: available on Linux 2.6.27+ (and glibc 2.9)
+ * ``pipe2()``: available on Linux 2.6.27+ (and glibc 2.9)
+ * ``accept4()``: available on Linux 2.6.28+ (and glibc 2.10)
 If ``accept4()`` is called on Linux older than 2.6.28, ``accept4()`` returns
 ``-1`` (fail) and errno is set to ``ENOSYS``.
  * `Secure File Descriptor Handling
-   <>`_
+   <>`_ (Ulrich Drepper, 2008)
  * ` of the Tornado project
    emulate fcntl(fd, F_SETFD, FD_CLOEXEC) using