bdist_egg issues with python 3.3.0aX

behackett avatarbehackett created an issue

python3.3 setup.py bdist_egg fails with "ValueError: bad marshal data (unknown type code)":

Traceback (most recent call last):
  File "setup.py", line 294, in <module>
    **extra_opts
  File "/usr/local/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/lib/python3.3/distutils/dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.3/distutils/dist.py", line 936, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.3/site-packages/distribute-0.6.27dev-py3.3.egg/setuptools/command/bdist_egg.py", line 227, in run
    os.path.join(archive_root,'EGG-INFO'), self.zip_safe()
  File "/usr/local/lib/python3.3/site-packages/distribute-0.6.27dev-py3.3.egg/setuptools/command/bdist_egg.py", line 266, in zip_safe
    return analyze_egg(self.bdist_dir, self.stubs)
  File "/usr/local/lib/python3.3/site-packages/distribute-0.6.27dev-py3.3.egg/setuptools/command/bdist_egg.py", line 402, in analyze_egg
    safe = scan_module(egg_dir, base, name, stubs) and safe
  File "/usr/local/lib/python3.3/site-packages/distribute-0.6.27dev-py3.3.egg/setuptools/command/bdist_egg.py", line 429, in scan_module
    code = marshal.load(f);  f.close()
ValueError: bad marshal data (unknown type code)

This problem occurs using trunk cloned this morning as well as .25 and.26 (the only releases I've tried).

Comments (16)

  1. Marc Abramowitz
    • removed version
    • removed milestone

    I am also seeing this issue with Python 3.3.0a3 (v3.3.0a3:0b53b70a40a0, May 1 2012, 11:39:35).

    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ rm -rf build
    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ find . -name '*.pyc' -exec rm -rf {} \;
    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ find . -name '*.pyc'
    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ ../lexicon/.tox/py33/bin/python setup.py bdist_egg
    running bdist_egg
    running egg_info
    writing spec.egg-info/PKG-INFO
    writing top-level names to spec.egg-info/top_level.txt
    writing entry points to spec.egg-info/entry_points.txt
    writing requirements to spec.egg-info/requires.txt
    writing dependency_links to spec.egg-info/dependency_links.txt
    reading manifest file 'spec.egg-info/SOURCES.txt'
    writing manifest file 'spec.egg-info/SOURCES.txt'
    installing library code to build/bdist.macosx-10.6-intel/egg
    running install_lib
    running build_py
    creating build
    creating build/lib
    creating build/lib/spec
    copying spec/__init__.py -> build/lib/spec
    copying spec/cli.py -> build/lib/spec
    copying spec/plugin.py -> build/lib/spec
    copying spec/utils.py -> build/lib/spec
    Fixing build/lib/spec/__init__.py build/lib/spec/cli.py build/lib/spec/plugin.py build/lib/spec/utils.py
    Skipping implicit fixer: buffer
    Skipping implicit fixer: idioms
    Skipping implicit fixer: set_literal
    Skipping implicit fixer: ws_comma
    Fixing build/lib/spec/__init__.py build/lib/spec/cli.py build/lib/spec/plugin.py build/lib/spec/utils.py
    Skipping implicit fixer: buffer
    Skipping implicit fixer: idioms
    Skipping implicit fixer: set_literal
    Skipping implicit fixer: ws_comma
    creating build/bdist.macosx-10.6-intel
    creating build/bdist.macosx-10.6-intel/egg
    creating build/bdist.macosx-10.6-intel/egg/spec
    copying build/lib/spec/__init__.py -> build/bdist.macosx-10.6-intel/egg/spec
    copying build/lib/spec/cli.py -> build/bdist.macosx-10.6-intel/egg/spec
    copying build/lib/spec/plugin.py -> build/bdist.macosx-10.6-intel/egg/spec
    copying build/lib/spec/utils.py -> build/bdist.macosx-10.6-intel/egg/spec
    byte-compiling build/bdist.macosx-10.6-intel/egg/spec/__init__.py to __init__.cpython-33.pyc
    byte-compiling build/bdist.macosx-10.6-intel/egg/spec/cli.py to cli.cpython-33.pyc
    byte-compiling build/bdist.macosx-10.6-intel/egg/spec/plugin.py to plugin.cpython-33.pyc
    byte-compiling build/bdist.macosx-10.6-intel/egg/spec/utils.py to utils.cpython-33.pyc
    creating build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying spec.egg-info/PKG-INFO -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying spec.egg-info/SOURCES.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying spec.egg-info/dependency_links.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying spec.egg-info/entry_points.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying spec.egg-info/requires.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying spec.egg-info/top_level.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    zip_safe flag not set; analyzing archive contents...
    Traceback (most recent call last):
      File "setup.py", line 32, in <module>
        'Topic :: Software Development :: Testing'
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 917, in run_commands
        self.run_command(cmd)
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 936, in run_command
        cmd_obj.run()
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 227, in run
        os.path.join(archive_root,'EGG-INFO'), self.zip_safe()
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 266, in zip_safe
        return analyze_egg(self.bdist_dir, self.stubs)
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 402, in analyze_egg
        safe = scan_module(egg_dir, base, name, stubs) and safe
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 429, in scan_module
        code = marshal.load(f);  f.close()
    ValueError: bad marshal data (unknown type code)
    
  2. Marc Abramowitz

    I upgraded to Python 3.3.0a4 (v3.3.0a4:7c51388a3aa7, May 30 2012, 16:58:42) and now I get a different error:

    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ rm -rf build
    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ find . -name '*.pyc' -exec rm -rf {} \;
    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ find . -name '*.pyc'
    [last: 0] marca@scml-marca:~/dev/git-repos/spec$ ../lexicon/.tox/py33/bin/python setup.py bdist_egg
    Traceback (most recent call last):
      File "setup.py", line 1, in <module>
        from setuptools import setup, find_packages
      File "<frozen importlib._bootstrap>", line 1286, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1253, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
      File "<frozen importlib._bootstrap>", line 776, in load_module
      File "<frozen importlib._bootstrap>", line 757, in load_module
      File "<frozen importlib._bootstrap>", line 408, in module_for_loader_wrapper
      File "<frozen importlib._bootstrap>", line 645, in _load_module
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/__init__.py", line 2, in <module>
        from setuptools.extension import Extension, Library
      File "<frozen importlib._bootstrap>", line 1286, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1253, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
      File "<frozen importlib._bootstrap>", line 776, in load_module
      File "<frozen importlib._bootstrap>", line 757, in load_module
      File "<frozen importlib._bootstrap>", line 408, in module_for_loader_wrapper
      File "<frozen importlib._bootstrap>", line 645, in _load_module
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/extension.py", line 2, in <module>
        import distutils.core
      File "<frozen importlib._bootstrap>", line 1286, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1237, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 1286, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1253, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
      File "<frozen importlib._bootstrap>", line 776, in load_module
      File "<frozen importlib._bootstrap>", line 757, in load_module
      File "<frozen importlib._bootstrap>", line 408, in module_for_loader_wrapper
      File "<frozen importlib._bootstrap>", line 645, in _load_module
      File "/Users/marca/dev/git-repos/lexicon/.tox/py33/lib/python3.3/distutils/__init__.py", line 48, in <module>
        old_find_config_files = dist.Distribution.find_config_files
    AttributeError: 'module' object has no attribute 'Distribution'
    
  3. Marc Abramowitz

    It's still happening for me but I discovered that a workaround is to add zip_safe=False to setup.py and then bdist_egg works.

    Is this still bad though because a lot of packages may not have zip_safe=False and they could break with Python 3.3?

    Here's a little sample module to illustrate: https://bitbucket.org/msabramo/python3.3_bdist_egg_test/src

    ~/dev/git-repos$ git clone git@bitbucket.org:msabramo/python3.3_bdist_egg_test.git && cd python3.3_bdist_egg_test && python3.3 setup.py bdist_egg -q
    Cloning into 'python3.3_bdist_egg_test'...
    remote: Counting objects: 4, done.
    remote: Compressing objects: 100% (3/3), done.
    remote: Total 4 (delta 0), reused 0 (delta 0)
    Receiving objects: 100% (4/4), done.
    running bdist_egg
    running egg_info
    creating python3.3_bdist_egg_test.egg-info
    ...
    zip_safe flag not set; analyzing archive contents...
    Traceback (most recent call last):
      File "setup.py", line 11, in <module>
        url='https://bitbucket.org/msabramo/python3.3_bdist_egg_test',
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 917, in run_commands
        self.run_command(cmd)
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 936, in run_command
        cmd_obj.run()
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 227, in run
        os.path.join(archive_root,'EGG-INFO'), self.zip_safe()
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 266, in zip_safe
        return analyze_egg(self.bdist_dir, self.stubs)
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 402, in analyze_egg
        safe = scan_module(egg_dir, base, name, stubs) and safe
      File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.27-py3.3.egg/setuptools/command/bdist_egg.py", line 429, in scan_module
        code = marshal.load(f);  f.close()
    ValueError: bad marshal data (unknown type code)
    
    ~/dev/git-repos/python3.3_bdist_egg_test$ sed -i .bak 's/# zip_safe=False/zip_safe=False/' setup.py
    $ python3.3 setup.py bdist_eggrunning bdist_egg
    running egg_info
    writing python3.3_bdist_egg_test.egg-info/PKG-INFO
    writing top-level names to python3.3_bdist_egg_test.egg-info/top_level.txt
    writing dependency_links to python3.3_bdist_egg_test.egg-info/dependency_links.txt
    reading manifest file 'python3.3_bdist_egg_test.egg-info/SOURCES.txt'
    writing manifest file 'python3.3_bdist_egg_test.egg-info/SOURCES.txt'
    installing library code to build/bdist.macosx-10.6-intel/egg
    running install_lib
    running build_py
    copying python3.3_bdist_egg_test.egg-info/PKG-INFO -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying python3.3_bdist_egg_test.egg-info/SOURCES.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying python3.3_bdist_egg_test.egg-info/dependency_links.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying python3.3_bdist_egg_test.egg-info/not-zip-safe -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    copying python3.3_bdist_egg_test.egg-info/top_level.txt -> build/bdist.macosx-10.6-intel/egg/EGG-INFO
    creating dist
    creating 'dist/python3.3_bdist_egg_test-0.0.0-py3.3.egg' and adding 'build/bdist.macosx-10.6-intel/egg' to it
    removing 'build/bdist.macosx-10.6-intel/egg' (and everything under it)
    
  4. Marc Abramowitz

    Brett Cannon made a good point over at http://bugs.python.org/issue15031

    <blockquote>Why is distribute reading bytecode to begin with? What's the use case, especially considering that using bytecode screws over other Python VMs like Jython and IronPython.</blockquote>

    I wonder whether this code is still needed/desirable? I guess if we were to get rid of zipped distributions, then this wouldn't be necessary anymore?

  5. Antoine Pitrou

    Following patch works here:

    diff --git a/setuptools/command/bdist_egg.py b/setuptools/command/bdist_egg.py
    --- a/setuptools/command/bdist_egg.py
    +++ b/setuptools/command/bdist_egg.py
    @@ -425,7 +425,11 @@ def scan_module(egg_dir, base, name, stu
             return True     # Extension module
         pkg = base[len(egg_dir)+1:].replace(os.sep,'.')
         module = pkg+(pkg and '.' or '')+os.path.splitext(name)[0]
    -    f = open(filename,'rb'); f.read(8)   # skip magic & date
    +    f = open(filename,'rb')
    +    if sys.version_info < (3, 3):
    +        f.read(8)   # skip magic & date
    +    else:
    +        f.read(12)  # skip magic, date & size
         code = marshal.load(f);  f.close()
         safe = True
         symbols = dict.fromkeys(iter_symbols(code))
    
    
  6. Stefan H. Holek

    Thank you Marc for figuring out the file size issue. The patch however is not yet sufficient. Marshal.load() sometimes just happens to not raise but returns an integer!

    To reproduce, run easy_install docutils in Python 3.3.

    To fix the issue, I have simplified Marc's patch in that I made the number of bytes-to-skip solely depend on the Python version. This allows to install docutils in Python 3.3.

    Patch 'bdist_egg': https://bitbucket.org/stefanholek/distribute/changeset/1837e1e58ca8

  7. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.