1. PyPA
  2. Python Packaging Authority Projects
  3. pypi
  4. Issues


Issue #161 resolved

ReST formatting fails, and there is no way to see why

Wichert Akkerman
created an issue

I just uploaded a new pyramid_sqlalchemy package to PyPI. Looking at the distribution page PyPI is not rendering ReST. I can’t figure out why. To debug this I all known (as far as I am aware) ways to debug this:

  • I ran python setup.py check, which showed no errors.
  • I ran python setup.py --long-description | rst2html-2.7.py > /dev/null. This also did not show any errors.
  • I copied the description manually from the PKG-INFO file in the sdist and ran rst2html on that. That also did not show any errors.

As an uploader to PyPI this is very frustrating. It would be very useful if PyPI can report any errors it finds in descriptions. I don't quite care how this is done. Possible options are:

  • when registering a distribution release with python setup.py register return all ReST errors so they are shown immediately when uploading.
  • email any ReST rendering errors after an upload to the uploader. Perhaps email it to all uploaders in case uploading happens using a role account for which email is never read.
  • show it in the web UI when updating the distribution information manually. This is slightly awkward since it requires an actual manual step to re-save the data without changing anything.

I don't think coming up with a way to run the syntax checks on a developer's machine, for example by creating a package which mimics PyPI's behaviour, as a workable approach: there is too much opportunity for environment changes (things like locale settings) and different package versions to lead to the wrong results.

Comments (30)

  1. Daniel Greenfeld

    I would love to see parsing errors displayed. Why?

    • Embarrassing for Python to have so many ugly pages on it's main package index.
    • Makes beginners feel like they are screwing up, when in reality it's just PyPI and it's super-strict RestructuredText settings.
    • Arguably obfuscates documentation!

    Perhaps rst2html could be used, then have it's output captured and displayed?

  2. Donald Stufft

    We can't return the RST errors directly because there's no way to prevent it from potentionally leaking data from the machines that run it. At least we'd need to seriously audit it to make sure it's not going to do that. In the works I have a thing that will run rst2html (basically) using the exact same settings as PyPI uses, I just haven't switched PyPI itself over to it yet.

  3. Donald Stufft

    Pelican and Sphinx and PyPI have two very different threat models.

    Pelican and Sphinx run on the developers local machine, so if you say... include a file, then at worst you're only going to include your own file.

    PyPI runs on the PyPI servers, so if you include a file, maybe you'll include config.ini which holds the database passwords for PyPI.

    Also Pelican and Sphinx tend to not be automatically ran on someone elses domain (RTD aside), so if you say inject a URL with a javacript: scheme the only site you're attacking is your own, vs on PyPI where you could use it steal PyPI cookies.

  4. Chris Jerdonek

    Since the Packaging User Guide recommends using the web form on PyPI (e.g. uploading a PKG-INFO file), it makes sense to prioritize that over the command-line user experience.

    If information from the parsing errors can't be provided to the user, can the web form at least say something like, "there were errors converting your long_description to reST, so PyPI is rendering it as plain-text. Go <here> for help on resolving the errors."

  5. Chris Jerdonek

    In the works I have a thing that will run rst2html (basically) using the exact same settings as PyPI uses, I just haven't switched PyPI itself over to it yet.

    Donald Stufft, is this in source control somewhere that people can look at?

  6. Marc Abramowitz

    See also https://bitbucket.org/pypa/pypi/pull-request/60/fix-bb-214-make-rst-rendering-not-fail-for/diff -- this proposes to increase the minimum level which causes docutils to halt. That prevents things like "title underline too short" an unused refs from halting rendering.

    It's not great that PyPI has a combination of being super strict and it doesn't tell you what the problems are. That's just not fair. I think it either has to loosen up or it needs to be able to report what the problems are (or at the very, very least that there was some problem as Chris Jerdonek suggests).

  7. Marc Abramowitz

    One more way to solve this is to offer a service that easily shows previews. Most markup thingies on the Web have previews -- e.g.: GitHub, BitBucket, JIRA/Confluence, Stack Overflow, etc. (plus they usually don't bail on rendering the entire document when they hit the first problem). So something perhaps like a hosted version of Marius Gedminas's restview with --pypi-strict.

  8. Marc Abramowitz

    sontek: You fixed https://pypi.python.org/pypi/pyramid_celery - what was it? (So that restview and other tools can catch it). Did you use restview --pypi-strict? That option makes it check some things that PyPI checks which are over and above what docutils checks.

    That just made me think of one tiny tweak to Chris Jerdonek's idea of showing a generic error. There should be at least two errors: one for docutils errors and one for PyPI errors. That at least gives a clue of what the problem is. Even better is to show these generic errors as the fallback but to show a specific carefully curated error for known common cases. My guess is a large percentage of problems are caused by a couple of issues with probably "title underline too short" being the #1 offender.

  9. Alexandre Conrad

    I've had the same issues with https://pypi.python.org/pypi/pycobertura which is a markdown document converted to rest using pypandoc. It's so frustrating when it breaks that I'm now extracting only a subset of the README to be included on Pypi because using the whole README is just too fragile and takes me forever to know which part of the document is the culprit.

    Here's the hack that extracts the most relevant part of the README to be submitted to pypi: https://github.com/SurveyMonkey/pycobertura/blob/master/setup.py#L9-L29

  10. Donald Stufft

    PyPI now uses the readme library for rendering. This doesn't yet have a CLI (but does have an API) and I'd like to add a CLI for it. This will render things exactly as how PyPI does it (because it's what PyPI uses).

  11. Chris Jerdonek

    Did you test how pypa/readme handles common errors such as title underline too short?

    Perhaps a follower to this issue could check the web UI and command-line (i.e. python setup.py register) experiences in this regard.

  12. Log in to comment