Issue #204 resolved

pkg_resources fails in buildout with two-level namespace packages

Aurélien Bompard
created an issue

I'm trying to run "buildout init" on a system where zc.buildout and distribute are already installed, and I get a traceback when pkg_resources tries to declare peak.utils as a namespace package : {{{ Getting distribution for 'distribute'. Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/setuptools/init.py", line 2, in <module> from setuptools.extension import Extension, Library File "/usr/lib/python2.7/site-packages/setuptools/extension.py", line 2, in <module> from setuptools.dist import _get_unpatched File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 6, in <module> from setuptools.command.install import install File "/usr/lib/python2.7/site-packages/setuptools/command/init.py", line 8, in <module> from setuptools.command import install_scripts File "/usr/lib/python2.7/site-packages/setuptools/command/install_scripts.py", line 3, in <module> from pkg_resources import Distribution, PathMetadata, ensure_directory File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2692, in <module> add_activation_listener(lambda dist: dist.activate()) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 668, in subscribe callback(dist) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2692, in <lambda> add_activation_listener(lambda dist: dist.activate()) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2196, in activate map(declare_namespace, self._get_metadata('namespace_packages.txt')) File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1777, in declare_namespace import(parent) ImportError: No module named peak An error occurred when trying to install distribute 0.6.16. Look above this message for any errors that were output by easy_install. }}}

peak and peak.utils are namespace packages, and I found a way to fix it : I check if the parent is not a namespace package itself before trying to import it. Here's the patch : {{{ --- /usr/lib/python2.7/site-packages/pkg_resources.py.orig 2011-05-13 06:44:46.165418517 +0200 +++ /usr/lib/python2.7/site-packages/pkg_resources.py 2011-05-13 07:50:14.827284238 +0200 @@ -1773,11 +1773,12 @@ if '.' in packageName: parent = '.'.join(packageName.split('.')[:-1]) declare_namespace(parent) - import(parent) - try: - path = sys.modules[parent].path - except AttributeError: - raise TypeError("Not a package:", parent) + if parent not in _namespace_packages: + import(parent) + try: + path = sys.modules[parent].path + except AttributeError: + raise TypeError("Not a package:", parent)

     # Track what packages are namespaces, so when new path items are added,
     # they can be updated

}}}

With this, buildout init works fine, and I did not see regressions anywhere. Does it look OK to you ?

Comments (8)

  1. Anonymous

    Your patch is, ironically enough, breaking my buildout.

    Specifically, it's causing the minitage.recipe.cmmi recipe to break. Log:

    While:
      Installing.
      Getting section protobuf.
      Initializing section protobuf.
      Loading zc.buildout recipe entry minitage.recipe.cmmi:default.
    
    An internal error occurred due to a bug in either zc.buildout or in a
    recipe being used:
    Traceback (most recent call last):
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 1805, in main
        getattr(buildout, command)(args)
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 468, in install
        [self[part]['recipe'] for part in install_parts]
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 1056, in __getitem__
        options._initialize()
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 1141, in _initialize
        recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/zc.buildout-1.5.2-py2.6.egg/zc/buildout/buildout.py", line 1101, in _install_and_load
        req.project_name, group, entry)
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/distribute-0.6.17-py2.6.egg/pkg_resources.py", line 337, in load_entry_point
        return get_distribution(dist).load_entry_point(group, name)
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/distribute-0.6.17-py2.6.egg/pkg_resources.py", line 2281, in load_entry_point
        return ep.load()
      File "/home/ehyde/dev/proj/org/codinghyde.enigma/eggs/distribute-0.6.17-py2.6.egg/pkg_resources.py", line 1991, in load
        entry = __import__(self.module_name, globals(),globals(), ['__name__'])
    ImportError: No module named cmmi.cmmi
    

    My buildout.cfg: https://github.com/codinghyde/codinghyde.enigma/blob/develop/buildout.cfg

    And an excerpt from the cmmi package's setup (it looks perfectly valid, I think)

    setup(
        name=name,
        version=version,
        description="zc.buildout recipes to compile and install software or python packages and generate scripts or configuration files  sponsored by Makina Corpus.",
        long_description= long_description,
        classifiers=[
            'Framework :: Buildout',
            'Intended Audience :: Developers',
            'License :: OSI Approved :: BSD License',
            'Topic :: Software Development :: Build Tools',
            'Topic :: Software Development :: Libraries :: Python Modules',
        ],
        keywords='development buildout recipe',
        author='Mathieu Pasquet',
        author_email='kiorky@cryptelium.net',
        url='http://cheeseshop.python.org/pypi/%s' % name,
        license='BSD',
        packages=find_packages('src'),
        package_dir = {'': 'src'},
        namespace_packages=['minitage', 'minitage.recipe', 'minitage.recipe.cmmi'],
        include_package_data=True,
        zip_safe=False,
        install_requires = [
            'zc.buildout',
            
            'minitage.core',
            'iniparse',
            'minitage.recipe.common',
        ],
        extras_require={'test': ['IPython', 'zope.testing', 'mocker']},
        #tests_require = ['zope.testing'],
        #test_suite = '%s.tests.test_suite' % name,
        # adding zdu, setuptools seems to order recipes executions
        # in akphabetical order for entry points
        # workaround when using the 2 recipes in the same buildout.
        entry_points = {
            'zc.buildout' : [
                'cmmi = %s:Recipe' % 'minitage.recipe.cmmi.cmmi',
                'default = %s:Recipe' % 'minitage.recipe.cmmi.cmmi',
            ]
        },
    )
    
  2. Aurélien Bompard reporter

    OK, I kind of understand where it comes from. The minitage.cmmi recipe's init.py files contain more than the simple declare_namespace that is recommanded here : http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages They added an ImportError handler :

    try:
        __import__('pkg_resources').declare_namespace(__name__)
    except ImportError:
        from pkgutil import extend_path
        __path__ = extend_path(__path__, __name__)
    

    As a result, the code worked without my patch. Now if I understand correctly, my patch bypasses the import call, but also bypasses the path assignment right under it. This may be why the import fails later on.

    I kind of see where it comes from, I've reproduced it, but I'm having problems finding a proper fix, since I don't know the details of the python import system. Here's a patch that seems to be fixing it :

    diff -r 9520b177d8a4 pkg_resources.py
    --- a/pkg_resources.py  Mon May 30 18:01:28 2011 +0200
    +++ b/pkg_resources.py  Tue May 31 17:02:08 2011 +0200
    @@ -1811,10 +1811,10 @@
                 declare_namespace(parent)
                 if parent not in _namespace_packages:
                     __import__(parent)
    -                try:
    -                    path = sys.modules[parent].__path__
    -                except AttributeError:
    -                    raise TypeError("Not a package:", parent)
    +            try:
    +                path = sys.modules[parent].__path__
    +            except AttributeError:
    +                raise TypeError("Not a package:", parent)
     
             # Track what packages are namespaces, so when new path items are added,
             # they can be updated
    

    Does it look OK ? Does it work for you ?

  3. Log in to comment