Issue #27 open

Stop using zipimport._zip_directory_cache

Brett Cannon
created an issue

setuptools uses a private API in zipimport called _zip_directory_cache. Obviously a no-no and has caused headaches in Python's standard library as removing the private API would break setuptools. So I would like to see its use go. =)

See http://bugs.python.org/issue2953 for what triggered this.

Comments (4)

  1. philip_thiem

    I found that filenames in the caches for an archive (the dictionary put into self.zipinfo) is not consistent across python implementations (for example, pypy uses the internal zip path but stock python replaces '/' with os.sep). Since it is a private interface, can't really tell deviating implementations to fix it. Seems like the primary issue that one would have to use _zip_directory_cache for is removal of a file from the cache. A grep on 0.6.31 didn't turn up anything of that sort, looks like it was mostly being used to store a manifest of the zipfile. I did workaround this to get a virtualenv test env to run for something else. Would I run into any problem if I were to just:

    diff -ru distribute-0.6.31-py2.7.orig.egg\pkg_resources.py distribute-0.6.31-py2.7.egg\pkg_resources.py
    --- distribute-0.6.31-py2.7.orig.egg\pkg_resources.py   Thu Jan 31 12:05:07 2013
    +++ distribute-0.6.31-py2.7.egg\pkg_resources.py    Thu Jan 31 12:18:40 2013
    @@ -13,7 +13,7 @@
     method.
     """
    
    -import sys, os, zipimport, time, re, imp, types
    +import sys, os, time, re, imp, types, zipfile, zipimport
     from urlparse import urlparse, urlunparse
    
     try:
    @@ -1363,7 +1363,33 @@
    
     empty_provider = EmptyProvider()
    
    +def build_zipmanifest(path):
    +    """
    +    This builds a similar dictionary to the zipimport directory
    +    caches.  However instead of tuples, ZipInfo objects are stored.
    
    +    The translation of the tuple is as follows:
    +      * [0] - zipinfo.filename on stock pythons this needs "/" --> os.sep
    +              on pypy it is the same (one reason why distribute did work
    +              in some cases on pypy and win32).
    +      * [1] - zipinfo.compress_type
    +      * [2] - zipinfo.compress_size
    +      * [3] - zipinfo.file_size
    +      * [4] - len(utf-8 encoding of filename) if zipinfo & 0x800
    +              len(ascii encoding of filename) otherwise
    +      * [5] - (zipinfo.date_time[0] - 1980) << 9 | 
    +               zipinfo.date_time[1] << 5 | zipinfo.date_time[2]
    +      * [6] - (zipinfo.date_time[3] - 1980) << 11 | 
    +               zipinfo.date_time[4] << 5 | (zipinfo.date_time[5] // 2)
    +      * [7] - zipinfo.CRC
    +    """
    +    zipinfo = dict()
    +    with zipfile.ZipFile(path) as zfile:
    +        for zitem in zfile.namelist():
    +            zpath = zitem.replace('/', os.sep)
    +            zipinfo[zpath] = zfile.getinfo(zitem)
    +            assert zipinfo[zpath] is not None
    +    return zipinfo
    
    
     class ZipProvider(EggProvider):
    @@ -1373,7 +1399,7 @@
    
         def __init__(self, module):
             EggProvider.__init__(self,module)
    -        self.zipinfo = zipimport._zip_directory_cache[self.loader.archive]
    +        self.zipinfo = build_zipmanifest(self.load.archive)
             self.zip_pre = self.loader.archive+os.sep
    
         def _zipinfo_name(self, fspath):
    @@ -1417,11 +1443,9 @@
                 return os.path.dirname(last)  # return the extracted directory name
    
             zip_stat = self.zipinfo[zip_path]
    -        t,d,size = zip_stat[5], zip_stat[6], zip_stat[3]
    -        date_time = (
    -            (d>>9)+1980, (d>>5)&0xF, d&0x1F,                      # ymd
    -            (t&0xFFFF)>>11, (t>>5)&0x3F, (t&0x1F) * 2, 0, 0, -1   # hms, etc.
    -        )
    +        size = zip_stat.file_size
    +        date_time = zip_stat.date_time + (0, 0, -1) #ymdhms+wday, yday, dst
    +        #1980 offset already done
             timestamp = time.mktime(date_time)
    
             try:
    @@ -1609,7 +1633,7 @@
         def __init__(self, importer):
             """Create a metadata provider from a zipimporter"""
    
    -        self.zipinfo = zipimport._zip_directory_cache[importer.archive]
    +        self.zipinfo = build_zipmanifest(importer.archive)
             self.zip_pre = importer.archive+os.sep
             self.loader = importer
             if importer.prefix:
    @@ -2848,4 +2872,5 @@
     # calling ``require()``) will get activated as well.
     add_activation_listener(lambda dist: dist.activate())
     working_set.entries=[]; map(working_set.add_entry,sys.path) # match order
    +
    
    Only in distribute-0.6.31-py2.7.egg: pkg_resources.py~
    
  2. Log in to comment