1. PyPA
  2. Python Packaging Authority Projects
  3. pypi
  4. Issues


Issue #148 wontfix

Support Markdown for readmes

created an issue

If you write your readme in Restructured Text, PyPI will render it on your project's page (eg. https://pypi.python.org/pypi/requests ). It should be possible to write your readme in Markdown too. Markdown is a pervasive format used on GitHub and StackOverflow. It shouldn't be necessary to learn a new markup language or rewrite your project's documentation to publish a Python package.

This is common frustration. The Python community suffers because it's a barrier to publishing packages and writing documentation . Please consider supporting Markdown!

Update 2014: Here's a pull request! https://bitbucket.org/pypa/pypi/pull-request/59/support-markdown-for-readmes-fixes-148/diff

Comments (46)

  1. Donald Stufft

    This is unlikely to happen in the near term. Warehouse (PyPI 2.0) is organized so that eventually we can support multiple markdown selections. The primary problem is we don't have any way to know if we should be rendering in markdown or rST. One way around that would be to render Markdown if rST fails, however some markdown is valid rST but it renders incorrectly in that case and looks really bad. It's also not a very particularly elegant solution especially if people want to do something else like asciidoc or some other formatting method. Likely the metadata 2.0 stuff will (eventually maybe?) support saying what markup your description is in and then we can render it using the appropriate renderer. It will also enable us to actually reject and upload if a package doesn't render correctly instead of just falling back to plaintext, because we'll know if it's supposed to be able to render or not.

  2. Hickford reporter

    Really? I understand PyPI's current algorithm to be

    If there's a readme.rst, readme.txt or readme (no extension) file, render that as restructured text. Otherwise, nothing.

    Here's a backwards compatible algorithm that would elate markdown authors:

    If there's a readme.rst, readme.txt or readme (no extension) file render that as restructured text. Else if there's a readme.md or readme.markdown file render that as Markdown. Otherwise, nothing.

    If you're concerned about projects with a readme.txt written in Markdown, they are already broken. This change wouldn't fix that, but it would help the majority of Markdown authors who use the .md or .markdown extensions. They do that because GitHub renders files with those extensions.

  3. Donald Stufft

    No, PyPI doesn't exactly do that. PyPI renders a long_description, if a long_description is not provided it will attempt to discover a README file and will insert that as the long_description.

  4. Will McKenzie

    A potential interim solution (until the metadata stuff that will let a package maintainer specify the format) could be to have a developer leave the long_description field blank and have a README.(md|markdown) file to be rendered in it's stead? I know with my packages I'm basically just reading the README.txt file and assigning it to the long description field (I didn't know how the algorithm worked).

  5. Marc Abramowitz

    I like Will's idea. Many, many Python projects have boilerplate in setup.py to set long_description to the contents of their README. That's because people want to have the best chance of their description getting picked up but almost everyone prefers editing a description in a separate file over something embedded in setup.py, plus the file extension tells you what format it's in.

    I think it would be great if PyPI/warehouse scanned for a README.rst, README.md, README.markdown and used that as the priority for the description to show. Perhaps setuptools could be modified to warn you in the unlikely event that your long_description has different data than your README (Jason R. Coombs: What do you think?).

    I think in the next version of the metadata standard I would drop long_description and instead offer something like long_description_filename.

    Maybe not a bad idea to add long_description_filename to setuptools now. Then package maintainers could start setting that and PyPI/Warehouse could use that in preference to long_description. That would let the server instantly know the format language from the extension and package maintainers could set it and remove lines from their setup.py to populate long_description from a file.

  6. Jason R. Coombs

    Marc Abramowitz I'm not sure your suggestion makes sense in the general case. What you and Will are suggesting is to create a new field in the metadata that references content outside the metadata. In particular, long_description_filename references a file that exists in one of the uploaded distribution packages, but which one? What if the distribution package is a wheel or egg or Windows installer? What if there are multiple distributions (Python version specific)? What if the metadata has been uploaded (e.g. through the register command) but no distribution has been uploaded? I see value in having the long description inline with the metadata in all cases, because that means that all metadata is represented in one entity and doesn't require steps to assemble it (and the complication that entails).

  7. Krisztian Fekete

    What about a new field: long_description_format defaulting to reST for backward compatibility, but enabling us to have markdown formatted descriptions?

    Lifting a README file into the manifest is a different feature, which can be solved with tooling as demonstrated by pbr.

  8. Jason R. Coombs

    I have no objection to supporting markdown or even moving toward preferring markdown. This poses a larger question of what should the metadata reflect to support this ability? Also, does the tool chain that renders documentation support markdown (namely Sphinx or Warehouse)? I'd recommend getting some feedback from pypa-dev on a change of this scope.

  9. Marc Abramowitz

    Hickford's patch would add Markdown to legacy PyPI. I'm thinking it could be tweaked a bit to look for a long_description_format property in the metadata and a similar change could be made to Warehouse (GitHub).

    Then we could make a setuptools extension to add that property to package metadata (use an extension so we don't have to modify setuptools unless it proves to be wildly popular).

  10. Jason R. Coombs

    I'm slighly -1 on adding another metadata field to specify the content type of an existing metadata field (i.e. long_description/long_description_format). If we did adopt that, I would recommend using MIME types (so something like "text/x-markdown; charset=utf-8" for markdown) to encourage alignment with other standards.

    Rather than storing the metadata for the long description in another field, perhaps the format could be inferred by a heuristic and specified for cases where the heuristic fails using out-of-band features of the format (i.e. comments). For example, the following two snippets could represent a way to specify the content type of the document:

    .. Content-Type: text/x-rst
    [//]: # Content-Type: text/x-markdown

    In this way, the long_description field becomes content-agnostic, but the tools now can infer the content type and users would have a means to force a content type.

    This approach would not require any changes to tooling except for those that render package metadata.

  11. Donald Stufft

    I'm of the opinion that a fallback on Markdown isn't very good.

    • Some Markdown is also valid ReST, but the symbols mean different things, this means that it's not true that a fallback enables proper Markdown support, it only enables markdown support if that markdown happens to not be renderable as ReST. I think depending on the fact that a particular chunk of text doesn't happen to also be valid for ReST is a pretty poor UX.
    • It doesn't scale well beyond Markdown. It might be said that why would anyone choose something else over Markdown and ReST but it seems to me that adopting a solution which makes it harder to add more things in the future is a bad long term solution.
    • It relies on the fact that we're shoving rendered HTML into the database, however Warehouse does not use that HTML and I want to get rid of it eventually. Relying on shoving html in the database also means that things that consume the data from PyPI cannot determine what syntax to render a description with, without downloading the project and trying to determine how PyPI got the long_description and if it's from a file that ends with .md or .rst.

    Overall I'd really like not to add more implementation defined stuff to the packaging toolchain. The processes for doing things should be documented in a PEP and someone should be able to replace each part of the toolchain by reading those PEPs and implementing them. It shouldn't require having to inspect the various implementations to figure things out.

  12. Marc Abramowitz

    For folks looking for Markdown support, have a look at https://pypi.python.org/pypi/setuptools-markdown which I uploaded to PyPI last night.

    This is very, very experimental. Make sure to test thoroughly before you put a package on PyPI that uses it. E.g.: I wouldn't want for folks who install your packages to have to install setuptools-markdown and pandoc; I think it should only be required for the package maintainer. But please check that. You can file GitHub issues, but PRs are much preferred!

  13. Hickford reporter

    That's unwelcoming :( I am not going to read any further or say any more, because quarrels between people who should be working together make me sad.

    Happily I've been fortunate in open source, everyone has been most helpful to me.

    Whether Markdown support is added to current PyPI or the new site (https://warehouse.python.org/ ), Markdown support is an important issue to many people. I'm sure we can work it out eventually.

  14. Hickford reporter

    Hi Donald. You are set that readme and format should be specified explicitly in setup.py ? That would work. I would also be happy with GitHub-style autodiscovery (use readme.rst, readme.md, or readme.txt).

  15. Donald Stufft

    There's no way to do auto discovery like Github does because we're not reading from a file (in some cases we are, but not generally), the description gets passed into the long_description field in the setup.py as a string and that gets sent as a POST argument when the file gets uploaded to PyPI. As a way of trying to find something PyPI will also, in the absence of that field, look for a README file in the archive but I personally feel that shouldn't be encouraged as it lives outside the metadata and instead we should focus on a reasonable way to represent it within the metadata.

  16. anatoly techtonik

    setup.py must die, also because it is impossible to parse correctly in all cases. The README.txt or README,md is already in the .zip archive. No need to reference it. The problem with PyPI is that it tries to provide an online editor for the metadata and mixes metadata with data. The README.txt content is pasted into PKG-INFO file, but it is a hack.

    The good way would be to render README file from archive if the Description field in PKG-INFO (long_description in setup.py) is absent. The context can be well detected based on extension.

  17. Nicholas Chammas

    For people looking for an interim solution, you can use pandoc to convert your long_description from Markdown to reStructuredText as part of setup.py.

    Here's an example:

    import setuptools
    import pypandoc
    long_description = pypandoc.convert('README.md', 'rst')

    It's straightforward and should work for most projects. You can add a fallback like in the Stack Overflow answer I linked to so that people running python setup.py without pandoc installed will get the plain Markdown README.

  18. Log in to comment