Binary type returned as type str instead of type bytes in 0.6beta1 on Python 3.1.1+ (Ubuntu Karmic)

[Former user]

Could not find a bug report with this error.

I am using pickle.dumps() to create a binary string to insert into a database (sqlite). b'\x80\x03c_PM\ntvcapex\nq\x00.'

The string appears to be stored as a BLOB in the database. When querying the object back during a session the returned value is of type string(notice the quotation marks):

"b'\x80\x03c_PM\ntvcapex\nq\x00.'"

and results in an error when trying to use pickle.loads():

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.1/pickle.py", line 1369, in loads raise TypeError("Can't load pickle from unicode string") TypeError: Can't load pickle from unicode string

The same problem occurs when the tables are mapped with the PickleType instead of Binary.

[Mike Bayer]

Need to know what backend you're using since this is usually a highly backend-specific behavior.

Note that Py3K tests are not currently passing 100% as 0.6's priority is Python 2.xx for the initial beta releases.

[Former user]

Not too sure what you mean by backend but the database is sqlite otherwise I don't know what you are referring to.

I have tested the data directly with the sqlite3 dbapi and the data is returned as expected (bytes). I would be happy to look for the piece of code that is causing the problem but would need some direction as I have only just started to use sqlalchemy.

I see that you have changed it to engine for the component so I will have a poke around there for the rest of my day.

Kind Regards

[Mike Bayer]

"sqlite" is then your backend. the test for binary behavior is test/sql/test_types.py, using the BinaryTest class. Read the doc on README.unittests to see how to run the tests.

[Former user]

I ran the test_types.py and the resulting error is detailed below:

I had to change line 26 of properties.py (/sqlalchemy/orm/) to NoneType = type(None) to get the script to run. When it runs it fails with the error traceback below. I am not familiar with decorators and how they work. Unfortunately for now I have found my limit. I will read over this failure again in the next few days to see if it makes any more sense.

XXXXX@XXXXXX:~/sqlalchemy/test/sql$ nosetests3 test_types.py -v Failure: AttributeError ('NoneType' object has no attribute 'unbounded_varchar') ... ERROR

====================================================================== ERROR: Failure: AttributeError ('NoneType' object has no attribute 'unbounded_varchar')

---

Traceback (most recent call last): File "/usr/local/lib/python3.1/dist-packages/nose/failure.py", line 27, in runTest reraise(self.exc_class, self.exc_val, self.tb) File "/usr/local/lib/python3.1/dist-packages/nose/_3.py", line 7, in reraise raise exc_class(exc_val).with_traceback(tb) File "/usr/local/lib/python3.1/dist-packages/nose/loader.py", line 372, in loadTestsFromName addr.filename, addr.module) File "/usr/local/lib/python3.1/dist-packages/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/usr/local/lib/python3.1/dist-packages/nose/importer.py", line 84, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/XXXXX/sqlalchemy/test/sql/test_types.py", line 752, in <module> class StringTest(TestBase, AssertsExecutionResults): File "/home/XXXXX/sqlalchemy/test/sql/test_types.py", line 754, in StringTest @testing.requires.unbounded_varchar AttributeError: 'NoneType' object has no attribute 'unbounded_varchar'

---

Ran 1 test in 0.004s

FAILED (errors=1)

[Former user]

(original author: ged) Replying to zzzeek:

Need to know what backend you're using since this is usually a highly backend-specific behavior.

Note that Py3K tests are not currently passing 100% as 0.6's priority is Python 2.xx for the initial beta releases.

No need to know the backend, as returning strings is the normal behavior for Binary. This is a bit of a strange contract IMO, but it is what you get across the board. It probably feels even weirder on Py3k but that's to be expected too because we actively convert to string in all cases and don't have specific code for Py3k there.

[Former user]

(original author: ged) Replying to guest:

XXXXX@XXXXXX:~/sqlalchemy/test/sql$ nosetests3 test_types.py -v Failure: AttributeError ('NoneType' object has no attribute 'unbounded_varchar') ... ERROR

I suspect you did not install SQLAlchemy's nose plugin, which is required to run the test suite; see README.unittests for details. Note that running the test suite for this particular ticket is pointless IMO since the behavior you get is the expected one. The behavior might not be appropriate, but it's expected.

[Former user]

Thanks for the feedback ged. This may be a silly question but if the returned object is of type str how would I go about converting the returned string:

"b'\x80\x03c_PM\ntvcapex\nq\x00.'"

to a bytes class:

b'\x80\x03c_PM\ntvcapex\nq\x00.'

? As it is the binary object that I am insterested not the string representation of it.

[Mike Bayer]

the return result should definitely not be a repr() of a 'b' string. whether or not it round trips, that's a bug. as far as if it should be str() or bytes, I haven't though about that yet. This is py3k we're talking about. On Py2k binary is str().

[Former user]

(original author: ged) A patch which should fix the issue on py3k (for dialects which return bytes)

[Former user]

(original author: ged) Replying to zzzeek:

as far as if it should be str() or bytes, I haven't though about that yet. This is py3k we're talking about.

On Py3K, we should definitely return bytes and not strings. This is indeed a non issue on Py2K.

[Mike Bayer]

9ee458a619b443a24b195d4374759cff2e54e4bd

[Mike Bayer]

Removing milestone: py3k (automated comment)