setup.py misbehaves with respect to the location of shared objects

[Oliver]

Using SQLalchemy 0.8.2 I was trying to build and install it into a custom Python build (i.e. not the system one) on Scientific Linux 6.1.

I'm setting $PYTHONHOME before invoking setup.py install using the Python into which I want to install the SQLAlchemy module.

Since I am trying to build an RPM and the idea is to have a self-contained Python environment, everything should go into a particular folder (/opt/app) in this case.

I am using fakechroot to mimic the target environment on the build machine and it works for Python as well as the extensions I am trying to install. With the sole exception of SQLAlchemy and the .so files built for it. These end up inside:

/home/oliver/.python-eggs/SQLAlchemy-0.8.2-py2.7-linux-x86_64.egg-tmp/sqlalchemy/cutils.so
/home/oliver/.python-eggs/SQLAlchemy-0.8.2-py2.7-linux-x86_64.egg-tmp/sqlalchemy/cprocessors.so
/home/oliver/.python-eggs/SQLAlchemy-0.8.2-py2.7-linux-x86_64.egg-tmp/sqlalchemy/cresultproxy.so

within the faked chroot environment which is not the intended behavior. Instead they should all go into the site-package directory of the Python installation.

Am I doing something wrong or is this a defect?

Also note that from the output of setup.py install when run inside the faked chroot environment, I see:

creating stub loader for sqlalchemy/cprocessors.so
creating stub loader for sqlalchemy/cresultproxy.so
creating stub loader for sqlalchemy/cutils.so
byte-compiling build/bdist.linux-x86_64/egg/sqlalchemy/cprocessors.py to cprocessors.pyc
byte-compiling build/bdist.linux-x86_64/egg/sqlalchemy/cresultproxy.py to cresultproxy.pyc
byte-compiling build/bdist.linux-x86_64/egg/sqlalchemy/cutils.py to cutils.pyc

Thanks and best regards

[Mike Bayer]

SQLAlchemy's setup.py uses a system called "setuptools" to build its software, which itself builds upon Python's standard build system "distutils". "setuptools" itself is a de-facto standard in Python as well and is used by a large portion of Python projects.

All of the behaviors you're observing here are not specific to SQLAlchemy and are instead part of the behaviors of distutils and setuptools, there's nothing in SQLAlchemy itself that impacts the behavior of where .so files go or anything like that. SQLAlchemy does have some C extensions, which might not be the case for other packages you are comparing to, but for comparison, try installing a package like psycopg2 or lxml which also generates .so files and compare the behavior there.

This is a pretty fundamental Python install issue so I'd suggest asking on a general group such as comp.lang.python, or if you want to talk to distutils developers you can ask on http://www.python.org/community/sigs/current/distutils-sig/, they might be able to shed light on the mechanics of your issue here.

[Oliver]

Hi hi. Thanks for reading over it. I'm aware that you use setuptools. I am using the same for an extension (albeit proprietary) which I authored and I can tell you that this behavior cannot solely explained by setuptools.

My extension - by the way with the exact same environment - installs just fine. I'll see whether I can devise a change to setup.py to fix this. Would you be willing to take this in or not at all? You closed this pretty quickly, so I'm asking.

Thanks for SQLAlchemy, by the way.

[Mike Bayer]

The only function we call setup.py is setup(). There are no options to control where .so files go or how to respond to $PYTHONHOME, chroot environments or anything else like that (See http://docs.python.org/2/distutils/apiref.html#distutils.core.setup for core setup() and http://pythonhosted.org/distribute/setuptools.html#new-and-changed-setup-keywords for those added by setuptools). We are calling it in the identical form as other more widely used packages such as psycopg2, giving it a list of ext_modules, that's all it accepts.

If setup() is doing the wrong thing based on your environment, I'd assume the issue is with setuptools. I have no knowledge of what $PYTHONHOME even does in terms of setuptools, and based on my experience I'd guess it does not what you expect - $PYTHONHOME is not a variable that is commonly used these days, standard practice is to use virtualenv for local Python installations. So from all indications this certainly seems like a usage issue with setuptools overall and the folks on disutils-sig should have definitive information on how setuptools works - I'm only a user of it. If people with knowledge of setuptools can point to an actual bug in SQLAlchemy's code, reopen this ticket and we will fix it.

[Donald Stufft]

You probably want the --root option. It would not surprise me if setuptools gets confused by PYTHONHOME.

[Oliver]

Replying to dstufft:

You probably want the --root option. It would not surprise me if setuptools gets confused by PYTHONHOME.

Thanks, that is exactly it. Feels a bit awkward to say --root /, but it does the job. Thanks again for the help.