Binary type returned as type str instead of type bytes in 0.6beta1 on Python 3.1.1+ (Ubuntu Karmic)

Issue #1639 resolved
Anonymous created an issue

Could not find a bug report with this error.

I am using pickle.dumps() to create a binary string to insert into a database (sqlite). b'\x80\x03c_PM\ntvcapex\nq\x00.'

The string appears to be stored as a BLOB in the database. When querying the object back during a session the returned value is of type string(notice the quotation marks):

"b'\x80\x03c_PM\ntvcapex\nq\x00.'"

and results in an error when trying to use pickle.loads():

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.1/pickle.py", line 1369, in loads raise TypeError("Can't load pickle from unicode string") TypeError: Can't load pickle from unicode string

The same problem occurs when the tables are mapped with the PickleType instead of Binary.

Comments (12)

  1. Michael Bayer repo owner
    • changed component to engine
    • changed milestone to py3k

    Need to know what backend you're using since this is usually a highly backend-specific behavior.

    Note that Py3K tests are not currently passing 100% as 0.6's priority is Python 2.xx for the initial beta releases.

  2. Anonymous

    Not too sure what you mean by backend but the database is sqlite otherwise I don't know what you are referring to.

    I have tested the data directly with the sqlite3 dbapi and the data is returned as expected (bytes). I would be happy to look for the piece of code that is causing the problem but would need some direction as I have only just started to use sqlalchemy.

    I see that you have changed it to engine for the component so I will have a poke around there for the rest of my day.

    Kind Regards

  3. Michael Bayer repo owner

    "sqlite" is then your backend. the test for binary behavior is test/sql/test_types.py, using the BinaryTest class. Read the doc on README.unittests to see how to run the tests.

  4. Anonymous

    I ran the test_types.py and the resulting error is detailed below:

    I had to change line 26 of properties.py (/sqlalchemy/orm/) to NoneType = type(None) to get the script to run. When it runs it fails with the error traceback below. I am not familiar with decorators and how they work. Unfortunately for now I have found my limit. I will read over this failure again in the next few days to see if it makes any more sense.

    XXXXX@XXXXXX:~/sqlalchemy/test/sql$ nosetests3 test_types.py -v Failure: AttributeError ('NoneType' object has no attribute 'unbounded_varchar') ... ERROR

    ====================================================================== ERROR: Failure: AttributeError ('NoneType' object has no attribute 'unbounded_varchar')


    Traceback (most recent call last): File "/usr/local/lib/python3.1/dist-packages/nose/failure.py", line 27, in runTest reraise(self.exc_class, self.exc_val, self.tb) File "/usr/local/lib/python3.1/dist-packages/nose/_3.py", line 7, in reraise raise exc_class(exc_val).with_traceback(tb) File "/usr/local/lib/python3.1/dist-packages/nose/loader.py", line 372, in loadTestsFromName addr.filename, addr.module) File "/usr/local/lib/python3.1/dist-packages/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/lib/python3.1/dist-packages/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/XXXXX/sqlalchemy/test/sql/test_types.py", line 752, in <module> class StringTest(TestBase, AssertsExecutionResults): File "/home/XXXXX/sqlalchemy/test/sql/test_types.py", line 754, in StringTest @testing.requires.unbounded_varchar AttributeError: 'NoneType' object has no attribute 'unbounded_varchar'


    Ran 1 test in 0.004s

    FAILED (errors=1)

  5. Anonymous

    (original author: ged) Replying to zzzeek:

    Need to know what backend you're using since this is usually a highly backend-specific behavior.

    Note that Py3K tests are not currently passing 100% as 0.6's priority is Python 2.xx for the initial beta releases.

    No need to know the backend, as returning strings is the normal behavior for Binary. This is a bit of a strange contract IMO, but it is what you get across the board. It probably feels even weirder on Py3k but that's to be expected too because we actively convert to string in all cases and don't have specific code for Py3k there.

  6. Anonymous

    (original author: ged) Replying to guest:

    XXXXX@XXXXXX:~/sqlalchemy/test/sql$ nosetests3 test_types.py -v Failure: AttributeError ('NoneType' object has no attribute 'unbounded_varchar') ... ERROR

    I suspect you did not install SQLAlchemy's nose plugin, which is required to run the test suite; see README.unittests for details. Note that running the test suite for this particular ticket is pointless IMO since the behavior you get is the expected one. The behavior might not be appropriate, but it's expected.

  7. Anonymous

    Thanks for the feedback ged. This may be a silly question but if the returned object is of type str how would I go about converting the returned string:

    "b'\x80\x03c_PM\ntvcapex\nq\x00.'"

    to a bytes class:

    b'\x80\x03c_PM\ntvcapex\nq\x00.'

    ? As it is the binary object that I am insterested not the string representation of it.

  8. Michael Bayer repo owner

    the return result should definitely not be a repr() of a 'b' string. whether or not it round trips, that's a bug. as far as if it should be str() or bytes, I haven't though about that yet. This is py3k we're talking about. On Py2k binary is str().

  9. Anonymous

    (original author: ged) Replying to zzzeek:

    as far as if it should be str() or bytes, I haven't though about that yet. This is py3k we're talking about.

    On Py3K, we should definitely return bytes and not strings. This is indeed a non issue on Py2K.

  10. Log in to comment