Django tests failing under 1.6.1

Issue #50 resolved
Anonymous created an issue

(Imported from Google Code)

della5@iinet.com.au wrote:

What steps will reproduce the problem?
1. Run the testsuite like so
2. nosetests -w build/lib
3.

What is the expected output? What do you see instead?
Pass, fails.

What version of the product are you using? On what operating system?
1.6.1; gentoo linux

Please provide any additional information below.

From tox.ini we have;

[testenv:py27]
deps =

nose
coverage
unittest2
py-bcrypt
bcryptor
django
M2Crypto

Installed are;

nose-1.3.0, coverage-3.6, unittest2 is there,
py-bcrypt-0.4, Bcryptor-1.2.2, django latest == 1.5.5,
m2crypto-0.21.1.

django 1,5,4 appeared to fair as poorly.

to get a clean tun it requires;

nosetests -w build/lib \

-e test_unsalted_sha1 -e test_77_fuzz_input \
-e test_config -e test_unsalted_md5

I can't help but notice the bugs point at the unreleased 1.6.2. Do you really have virt. no issues in 1.6.1? euscan tells me there's no later release than 1.6.1.

Comments (13)

  1. Eli Collins repo owner

    (Imported from Google Code)

    I can confirm those test failures.

    It looks like Django has altered some border cases in their hasher code, which is causing a number of passlib.ext.django-related tests to fail. I'm not that surprised, passlib's django integration tests were designed to be extra sensitive to changes in Django, to keep a tight leash on passlib.ext.django's behavior. (If you want, you can edit tox.ini to remove "django" from [testenv:py27], and all the tests should pass, confirming it's just the django integration that's failing).

    It doesn't look like any of the changes impact security, but they definitely need to be addressed to shore up Passlib's Django support. I think this by itself is enough to warrant a 1.6.2 release later this week.

    And yeah, while there are a few small enhancements slated for 1.6.2, the 1.6.1 release hasn't had any reported bugs until now :)

    Btw, thank you very much for the detailed report!

    • Eli
  2. Eli Collins repo owner

    (Imported from Google Code)

    1.6.2 has been released, and all test are passing for Django 1.2 - 1.6 now (at least on my systems). I'm closing this issue, but please let me know if any of your test runs are still failing... more environments == better testing :)

    - Eli

  3. Anonymous

    (Imported from Google Code)

    della5@iinet.com.au wrote:

    ~/cvsPortage/gentoo-x86/dev-python/passlib $ PYTHON_TARGETS=pypy2_0 ebuild passlib-1.6.2.ebuild clean test

    yields

    Ran 1926 tests in 131.211s

    FAILED (SKIP=613, errors=49)

    with a plethora of

    ERROR: django_bcrypt (pybcrypt backend): test encrypt() / genconfig() PLUS various individual test errors

    gees it's pypy. what's your take on pypy? Are you inclined to requesting a full log of error output for running under pypy?

    ~/cvsPortage/gentoo-x86/dev-python/passlib $ PYTHON_TARGETS=python2_7 ebuild passlib-1.6.2.ebuild clean test

    yields

    * python2_7: running distutils-r1_run_phase python_test
    ..................................................................................................................................S......SSS....S.SSS........S......SSS....S.SSS........S......SSS....S.SS.S........SSSSSSSSSS....S.SS.S................S......SS.S.......SSSSSSSSSS....S.SSSS..........SS.SSS.SSS....S.SS.S.........S......SSS....S.SSSS.......S........S....S.SSSS..............SSS......SS.S..........S........S....S.SSSS.......S........S....S.SS..........S........S....S.SSSS.......SSSSSSSSSS....S.SSSS.......SSSSSSSSSS....S.SSSS.......SSSSSSSSSS....S.SSSS.......SSSSSSSSSS....S.SSSS.......SSSSSSSSSS....S.SSSS.......SSSSSSSSSS....S.SS.S...S.............SSS......SSS......S....SSSSSSSSSS....S.SSSS........SSSSSSSSSS....S.SS.S.......S......SSS....S.SSS........S......SSS....S.SSS........SSSSSSSSSS....S.SSSS.......SSSSSSSSSS....S.SS.S...............SSS......SSS...........SSSSSSSSSS....S.SSSS..........SSSSSSSSSS....S..SSS..........S......SSS....S............S......SSS....S............SSSSSSSSSS....S.SS.S........SSSSSSSSSS....S.SS.S.......SSSSSSSSSS....S.SS.S.......SSSSSSSSSS....S.SS.S..........S......SSS....S.SSSS.......S........S....S.SSS........S........S....S.SSSS.......S........S....S.SSSS.......S.............S.SSS........SSSSSSSSSS....S.SSSSS......SSSSSSSSSS....S.SS.S..........S........S....S.SSS.........................S......SSS.................S......S.S....................S......S.S...........S........S....S.S.S........SSSSSSSSSS....S.SS.S..S.....SSSSSSSSSS....S.SSSSS.S......................S......S.............S.....S.S.S.................S.....S.SSSS...SS.............S......SSSS.........S......SSS....S..S...........SSSSSSSSSS....S..SSS..S....S........S....S.SSSS.........S........S....S.SSSS.........S......SSS....S.SS...........S......SSS....S.SS.................................................................................................S......SSS....S.SSSS...................SSSSSSSSSS....S.SSSS......
    ----------------------------------------------------------------------
    Ran 1942 tests in 116.199s

    OK (SKIP=591)

    with

    nosetests -w "${BUILD_DIR}"/lib \

    || die "Tests fail with ${EPYTHON}"

    which amounts to about PYTHONPATH=. nosetests .

  4. Eli Collins repo owner
    • changed status to open

    (Imported from Google Code)

    I am trying to support pypy if possible, so definitely interested in the full error log if it's not too much trouble (if you want to put it up on gist.github.com that'd probably be better than pasting it here).

    There's so many library combinations, looks like one slipped through the cracks. I'm guessing you're running the test under pypy with django installed, but without bcrypt? I'm curious, if you install "bcrypt" (not "py-bcrypt") as well, do the tests pass then? Also, along with the log if you could tell me what pypy & django versions you're testing under.

    Thanks!
    - Eli

  5. Anonymous

    (Imported from Google Code)

    della5@iinet.com.au wrote:

    Oh dang it!!!!!!

    [I] dev-python/Bcryptor

    Available versions: (~)1.2.2 (~)1.2.2[1] {test PYTHON_TARGETS="python2_6 python2_7"}
    Installed versions: 1.2.2[1](15:44:22 15/12/13)(-test PYTHON_TARGETS="python2_6 python2_7")
    Homepage: http://www.bitbucket.org/ares/bcryptor/
    Description: Python wrapper for bcrypt

    [D] dev-python/django

    Available versions: 1.4.8 (~)1.5.4 **9999 {doc mysql postgres sqlite test vhosts PYTHON_TARGETS="python2_6 python2_7 python3_2 python3_3"}
    Installed versions: 1.5.5(17:03:23 15/12/13)(-doc -mysql -postgres -sqlite -test PYTHON_TARGETS="python2_6 python2_7 python3_2 python3_3")

    [I] dev-python/pypy

    Available versions:
    (2.0) (~)2.0.2

    testuser@archtester ~/cvsPortage/gentoo-x86/dev-python/passlib $ qlist Bcryptor
    /usr/lib64/python2.7/site-packages/bcryptor/bcrypt.py
    plus the rest.

    so, I have the package Bcryptor installs a file bcryptor/bcrypt.py.

    Now I shall refrain from rushing to pypi and doing a search on pypi.org for all packages with the string and just ask you here. Are you referring to the package Bcryptor? I think yes. Your tox.ini has;

    [testenv:py-bcrypt-py3]
    basepython = python3
    deps =
    # bcryptor -- has py3 installation issues

    py-bcrypt
    {[testenv:py32]deps}
    commands =
    nosetests {posargs:passlib.tests.test_handlers_bcrypt}

    [testenv:pypy]
    # pypy (as of v1.6 - v2.2) targets Python 2.7
    basepython = pypy
    deps =

    bcrypt
    {[testenv]deps}

    Now recently I have had not one but 2 packages cause me confusion come grief by way of the package name. I specially added Bcryptor to portage because I found it in the tox.ini of the prior version. Now you ask do I have bcrypt installed. At this stage I simply don't know for sure.

    Now, I added Bcryptor with only py2.6 2.7 support because of this.

    ~/cvsPortage/gentoo-x86/dev-python/Bcryptor $ ebuild Bcryptor-1.2.2.ebuild clean install

    yielded

    warning: build_py: byte-compiling is disabled, skipping.

    running build_ext
    building 'bcryptor.engine' extension
    error: unknown file type '.pyx' (from 'lib/bcryptor/wrap/engine.pyx')

    so the source of the bcryptor or Bcryptor fooled the installed pypy on its attempt to build the source code. This is likely a cython thingy.

    That py3 has installation issues is consistent with my findings to have added the package Bcryptor without py3 support. However it seems you have SOMETHING installed that props up pypy for doing the testsuite pulling on a bcrypt.py.

    I think the answer lies in getting these deps clearly understood and the log of the output may ultimately not be required and is only an issue because it's quite long and too long to put here.

    I do like your support of pypy, it's coming on nicely over time and really needs devs like you to embrace it. I await your response.

  6. Eli Collins repo owner

    (Imported from Google Code)

    Yeah, the package naming for the bcrypt libraries in particular is rather poor. My tox files probably aren't the best place to find answers, either, they're rather a mess at this point :)

    There's actually three main bcrypt libraries for python. I'm not sure what they're named under other packaging systems like portage, but on pypi they're named 'bcryptor' (https://pypi.python.org/pypi/Bcryptor), 'py-bcrypt' (https://pypi.python.org/pypi/py-bcrypt/), and 'bcrypt' (https://pypi.python.org/pypi/bcrypt/1.0.2).

    Personally, I'd avoid 'bcryptor'... I support it in passlib, but it's cython bindings always cause me grief, and also make it incompatible with pypy (as far as I can tell). It certainly sounds like it's managed to half-install itself and foul things up on your system.

    'py-bcrypt' is the oldest of the three, and it's pretty solid. But it's also a plain old C extension, and won't work with pypy.

    'bcrypt' is the one you want. It's a cffi-based extension, so it's the only one of the three which will work under pypy as well as normal C python. It's also the "something" that was being used in my tox config to test passlib with pypy. I'd recommend eliminate all traces of the others, and trying to get that one installed.

    Let me know if that helps straighten things out!

    - Eli

  7. Anonymous

    (Imported from Google Code)

    della5@iinet.com.au wrote:

    right it does straighten things out. I suspected it to be a separate package, so I shall get it and make an ebuild and install it to the system to give it a full whirl. Your interest and co-operation warrants it, but whether it finds its way to portage proper is another matter. I'll prob. do that tomorrow.

  8. Anonymous

    (Imported from Google Code)

    della5@iinet.com.au wrote:

    Eli; righty.

    What I have after all this further testing is a headache. I have in fact made an ebuild and installed it into my system by gentoo dev know how, and yep pypy2_0 does in fact pass the whole testsuite. I uninstalled the other 2, and then I got 3 Errors 1 Fail under py3, and 3 Errors under py2. This makes for a headache. On re-installing py-bcrypt they disappeared, however your test setup has finally defeated portage's capacity to cope by a 50m. pole vault.

    The tox.ini and the whole testsuite has a separate list of req'd deps which once installed see it pass under all python 'impls'. In gentoo to cater to this we need install any and all deps, and have it covered for any possible scenario or python impl. Here is the problem.

    ~/improvise/dev-python/bcrypt $ sudo ebuild bcrypt-1.0.2.ebuild clean merge

    ===========================================================

    * dev-python/py-bcrypt-0.4:0::gentoo
    * /usr/lib64/pypy2.0/site-packages/bcrypt/__init__.py
    * /usr/lib64/python2.6/site-packages/bcrypt/__init__.py
    * /usr/lib64/python2.7/site-packages/bcrypt/__init__.py
    * /usr/lib64/python3.2/site-packages/bcrypt/__init__.py
    * /usr/lib64/python3.3/site-packages/bcrypt/__init__.py
    *
    * Package 'dev-python/bcrypt-1.0.2' NOT merged due to file collisions.
    * If necessary, refer to your elog messages for the whole content of the
    * above message.

    These similarly named libs have gone and assigned identically named system installed files. tilt. This means that gentoo has to do some serious tampering so as to get the 2 separate libs installed into the one common gentoo system.

    tilt

    Without this tampering, gentoo is obliged to do some compromising. We can have this or that installed under this or that condition or needed python implementation.

    I think it's not the first time I have seen this, however the file collisions phenomenon raises the degree of difficulty or incompatibility to another level or 3.
    The nice news is that the testsuite will pass all once setup to the individual req's stipulated by the source code. At this point there ia a most minimal compromise option thought spells itself out for a final ebuild, however a compromise is a compromise is a compromise.

  9. Eli Collins repo owner

    (Imported from Google Code)

    Oh yeah... I forgot to mention in my bcrypt library description that the 'bcrypt' package is meant to be a drop-in replacement for 'py-bcrypt', so they use a number of the exact same python files. Which is why none of my test environments within tox.ini use both at the same time.

    I'm not a gentoo user, so I'm not quite clear why you're trying to install both of them. It shouldn't be needed for testing: if you're running tox, it should be handling its own installation of the libraries inside an isolated python virtualenv, without affecting what's installed on your system. Whereas in an actual deployment, applications that use passlib are welcome to require either (or neither) of them... all the python bcrypt libraries are optional dependencies for passlib, it will use them if it finds them.

    I hope the tox.ini suite isn't too much of a beast. For my own release testing purposes, I tried to include a large variety of implementation-and-optional-dependency combinations... but that does make it require a bit of setup and cpu cycles for the full thing to run :)

    - Eli

  10. Anonymous

    (Imported from Google Code)

    della5@iinet.com.au wrote:

    I'm not a gentoo user, so I'm not quite clear why you're trying to install both of them.

    For what it's worth I can explain in a line or 2. The explanation once given changes nothing though. Here;

    The use of tox, while not banned, is strongly discouraged and in practise rarely ever done in a python package ebuild. Gentoo's portage has a rule that bans the importing of, or connecting to the network, in any way beyond the fetch phase of an emerge which is fundamentally a system install, == apt-get or rpm -i.

    "if you're running tox, it should be handling its own installation of the libraries inside an isolated python virtualenv, without affecting what's installed on your system"

    This is what's called the perfect mis-match. Basically gentoo's policies makes for "it's got to work by utilising and drawing from (importing and so forth) the system installed packages' files". The use of tox 'conveniently' skirts this imposition. As mis-matches go, that makes for perfection.

    ok, how about a line or 20.

    I hope the tox.ini suite isn't too much of a beast.

    It is, without a doubt, a beast. However all is not lost. We @ gentoo have dealt before with such beasts, and are sure to have to do so again. The thing about your testsuite, tox aside, is that it is very comprehensive. and that's a good thingy. I've established in my own gentoo system that they do pass once given a lib combo to suite a given python 'impl'. That it draws upon optional libraries for optional testing provides us with some room to move. With this room, 'we' can make and invoke a compromise that will give a gentoo ebuild a worable and working test phase.

    Thx for your solid explanations and support of pypy.

  11. Log in to comment