Unicode-related error when using sqlite with str containing non-ascii characters

[Former user]

See attached testcase (testcase.py) and output (output.txt). 'In the wild,' this error occurs when trying to INSERT the content of a binary file into a Text column. In the testcase, this is simulated using a str object that is populated with a unicode escape.

Original code looked somewhat like this:

strval = "" with open("somepdf.pdf") as f: strval = f.read()

In other words, strval is filled with binary data from a file.

When executed: - using sqlite url: first commit succeeds, second fails - using mysql url: both commits succeed as expected

[Former user]

Minmial testcase

[Former user]

full output (stderr and stdout) of python testcase.py

[Former user]

Description pseudocode with formatting:

strval = ""
with open("somepdf.pdf") as f:
    strval = f.read()

[Mike Bayer]

Sorry, there's no bug here. A Python bytestring has no known encoding so SQLAlchemy cannot affect it in any way, and modern versions of pysqlite make the same decision. I would recommend using UnicodeText instead of Text(convert_unicode=True), the former emits a warning when a non-unicode object is passed.

[Former user]

Thanks for the quick response -- I may try Binary instead when reading file data.

[Mike Bayer]

Removing milestone: 0.6.xx (automated comment)