The package directory already hashes files by the first letter of
their name, meaning that they can reside on a filesystem with
directory size limits. However, the index files are not.
This adds an option to write the index files in a similar structure
but does not enable the option by default.
Note that this is not compatible with the way that pip expects to
find index pages (it requires a flat namespace) and so if this is
used, it must be used with a simple web application that can rewrite
the URLs that pip will use into directory-hashed URIs. A simple
two-line configuration for Apache's mod_rewrite is documented to
support this. Similarly, the main index page itself is written
to link to the non-hashed versions of the URLs.
With this change and a simple Apache configuration, bandersnatch
can now be used on a variety of filesystems, some of which are of
particular interest to mirror operators due to their transactional
nature and other features.