Need a release 1.0.8 with Python 3 fixes

Create issue
Issue #69 resolved
Victor Stinner created an issue

Hi,

Could you please release a new version with my fixes for Python 3? I still need it to port the OpenStack Swift project to Python 3.

https://review.openstack.org/#/c/199034/

I'm my pending patch for Swift, I'm using sed on pyeclib, which is really ugly!

https://review.openstack.org/#/c/199034/9/tox.ini

There is an ongoing discussing on the new embedded liberasurecode tarball:

https://bitbucket.org/kmgreen2/pyeclib/commits/7be5f2fa7ec403a8f98d7ab43b145eb7#comment-2132577

I don't know if it's a new feature or a required fix to get a release. It looks like the current code works, but liberasurecode 1.0.8 may be reinstalled whereas it's already installed. The .so doesn't end with .so.1.0.8 on my Fedora 22, there is only a "liberasurecode.so" file.

If there are open issues which should be fixed before a release, can you please try to list them there?

Thank you!

Comments (24)

  1. Tushar Gohad

    @haypo .so symlinks are only installed as part of development (-dev/-devel) packages. Check out the output of "ldd" on a liberasurecode based program and you will see liberasurecode.so.1, which is the SONAME, and liberasurecode.so.1.0.8, which it symlinks to, is the real name. Here are some basics: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

    Now about the liberasurecode tarball issue, we have explained this previously: if we don't include it in the pyeclib package, given Openstack CI/Jenkins slave doesn't have liberasurecode installed, pyeclib build and thus the check job is going to fail. Hope you see it.

  2. Victor Stinner reporter

    I did some more tests.

    When liberasurecode is installed by PyEClib, it is installed in /usr/local. The problem is that the .so file is installed in /usr/local/lib, whereas .so files should go to /usr/lib64 and /usr/local/lib64. Because of that, setup.py is unable to find liberasurecode, and so calling "python setup.py install" again will reinstall liberasurecode.

    I installed manually liberasurecode in /usr using ./configure --prefix=/usr. With that, setup.py PyEClib finds the .so(.1.0.8) file and everything goes fine. I suggest this small change in setup.py:

    diff --git a/setup.py b/setup.py
    index c62400f..fd86e66 100644
    --- a/setup.py
    +++ b/setup.py
    @@ -120,7 +120,7 @@ class build(_build):
                 if (os.path.isdir(locallibsrcdir)):
                     curdir = os.getcwd()
                     os.chdir(locallibsrcdir)
    -                configure_cmd = ("./configure --prefix=/usr/local")
    +                configure_cmd = ("./configure --prefix=/usr")
                     print(configure_cmd)
                     retval = os.system(configure_cmd)
                     if retval != 0:
    

    Note: /usr/lib64/liberasurecode.so.1.0.8 file exists. Now I'm no more sure that liberasurecode.so.1.0.8 was not installed before.

    Note: there is a small typo in README:

    -This library makes use of Jesasure for Reed-Solomon as implemented by the
    +This library makes use of Jerasure for Reed-Solomon as implemented by the
    
  3. Tushar Gohad

    @haypo typically the "libdir" (lib or lib64) is determined automagically by autoconf. The only thing we have control over is the prefix. Changing it to "/usr" won't help much. I am looking into fixing it for the general case (by determining libdir correctly). Thanks.

  4. Victor Stinner reporter

    Victor Stinner typically the "libdir" (lib or lib64) is determined automagically by autoconf.

    autoconf uses /usr/local/lib when the prefix is /usr/local, but autoconf uses /usr/lib64 when the prefix is /usr.

    The only thing we have control over is the prefix. Changing it to "/usr" won't help much.

    setup.py only check /lib64 on 64-bit systems, and only /lib on 32-bit systems. For these reason, installing liberasurecode with --prefix=/usr helps. .so files are installed into /usr/lib64 and so setup.py is able to detect liberasurecode and it doesn't reinstall it.

    When using --prefix=/usr/local (current default, hardcoded in setup.py), setup.py is unable to find liberasurecode in /usr/local/lib.

    For me, there are two options:

    • (my favorite option) modify setup.py to install liberasurecode with --prefix=/usr: since PyEClib is also installed in /usr, it looks consistent to me. No?
    • modify setup.py to lookup for liberasurecode in /lib64 and /lib directories

    Example of patch for the second option:

    diff --git a/setup.py b/setup.py
    index c62400f..268b8c2 100644
    --- a/setup.py
    +++ b/setup.py
    @@ -51,24 +51,13 @@ default_python_libdir = get_python_lib()
    
     default_library_paths = [default_python_libdir]
    
    -if platform_arch[0].startswith('64') and os.path.exists('/lib64'):
    -    default_library_paths.append('/lib64')
    -else:
    -    default_library_paths.append('/lib')
    -if platform_arch[0].startswith('64') and os.path.exists('/usr/lib64'):
    -    default_library_paths.append('/usr/lib64')
    -else:
    -    default_library_paths.append('/usr/lib')
    -if platform_arch[0].startswith('64') and os.path.exists('/usr/local/lib64'):
    -    default_library_paths.append('/usr/local/lib64')
    -else:
    -    default_library_paths.append('/usr/local/lib')
    -if platform_arch[0].startswith('64') and os.path.exists('%s/lib64'
    -                                                        % _exec_prefix):
    -    default_library_paths.append('%s/lib64' % _exec_prefix)
    -else:
    -    default_library_paths.append('%s/lib' % _exec_prefix)
    -
    +arch64 = platform_arch[0].startswith('64')
    +for prefix in ('/', '/usr', '/usr/local', _exec_prefix):
    +    libdir = os.path.join(prefix, '/lib')
    +    libdir64 = os.path.join(prefix, '/lib64')
    +    if arch64 and os.path.exists(lib64):
    +        default_library_paths.append(libdir64)
    +    default_library_paths.append(libdir)
    
     # utility routine
     def _read_file_as_str(name):
    
  5. Victor Stinner reporter

    It became difficult to me to follow multiple discussions splitted in different issues, so I created the issue #72: "setup.py install" reinstalls liberasurecode 1.0.8 on Fedora 22.

    I also opened other issues for other problems that I saw.

  6. Victor Stinner reporter

    I uninstalled liberasurecode and I ran some tests on the master branch (rev 5347db2171d55cc4587529268c5e5c8d16a4f79d): everything is fine. "sudo python setup.py install" installs liberasurecode in /usr, setup.py is able to locate installed C headers, and PyEClib is correctly installed.

    For my usage of PyEClib: the master branch is ready for a release. Now it's up to you :-)

  7. Victor Stinner reporter

    "If there are open issues which should be fixed before a release, can you please try to list them there?"

    ping?

  8. Kevin Greenan repo owner

    @haypo We are testing the master branch on a few platforms (CentOS, Fedora, Mac OS X, etc.) before tagging. I talked with @tsg- last night and he mentioned having library path issues on CentOS, but I think he had a fix. We believe that the core functionality of the library is stable (i.e. erasure coding), but dealing with platform-specific stuff is always tricky and fickle.

  9. Kota Tsuyuzaki

    @haypo @tsg- will push latest 1.0.9 to PyPI tomorrow morning in US. The reason we didn't push 1.0.8 to PyPI is that it would break Swift gate on openstack gerrit because Swift has a bunch of tests using jerasure even though 1.0.8 doesn't bandle the jerasure binary.

    Recently the restriction of PyECLib was merged on Swift master so that Swift has now a constraint PyECLib==1.0.7 so it's time to push latest version into PyPI. I'm sure he was working a lot to make the bumping smoothly. Sorry for inconvenience.

  10. Tushar Gohad

    Thanks @bloodeagle40234. @haypo Kota described the situation accurately. We have 1.0.9 release ready - would you be able to give the latest master a test (with "python setup.py install --install-liberasurecode [ yes | no ]")?

  11. sdague

    This broke OpenStack because only the master case was addressed. OpenStack includes upgrade testing, and this broke the stable/kilo branches, so all OpenStack patches are blocked.

  12. Kevin Greenan repo owner

    @tsg- @bloodeagle40234 I am missing context on what @sdague is saying above. Does this mean you need to push the requirements.txt change to other branches?

  13. Kevin Greenan repo owner

    @sdague Got it! The requirements change allowed us to break much of the common C code into a separate library. This is a stop-gap that will give time for package maintainers to post the separated C library (liberasurecode).

    @tsg- Should the default behavior install the internal liberasurecode?

  14. sdague

    For a change that doesn't retain compatibility like this, it would be better to not have it be a Z bump, but be an X or Y bump in the version number. Z bumps are typically assumed to be safe and backwards compatible.

  15. Kevin Greenan repo owner

    @sdague Hrm... This may be a bug, as it should be installing the included library by default.

    @tsg- I wonder if this is an installation issue (e.g. change of install directory).

  16. Kevin Greenan repo owner

    This installs liberasurecode by default (after removing it) on both Linux 2.6.32-573.el6.x86_64 and Mac OS 11.4.2... Not sure why this is not working for the OpenStack Gate.

  17. Tushar Gohad

    @kmgreen2, yes both Kota and I missed the stable/kilo branch. This is being addressed here - https://review.openstack.org/#/c/222237 which @sdague has already approved.

    Still need to look why 1.0.9 is not installing liberasurecode by default and blowing up at gate .. we did test "pip install pyeclib" after the package was uploaded and it did work as expected.

  18. Kevin Greenan repo owner

    @tsg- Cool! Yeah, the install works fine from both pip and manually installing using setup.py. Not sure what is going on here...

  19. Log in to comment