- attached testcase.py
Unicode-related error when using sqlite with str containing non-ascii characters
See attached testcase (testcase.py) and output (output.txt). 'In the wild,' this error occurs when trying to INSERT the content of a binary file into a Text column. In the testcase, this is simulated using a str object that is populated with a unicode escape.
Original code looked somewhat like this:
strval = "" with open("somepdf.pdf") as f: strval = f.read()
In other words, strval is filled with binary data from a file.
When executed: - using sqlite url: first commit succeeds, second fails - using mysql url: both commits succeed as expected
Comments (6)
-
Account Deleted -
Account Deleted - attached output.txt
full output (stderr and stdout) of
python testcase.py
-
Account Deleted Description pseudocode with formatting:
strval = "" with open("somepdf.pdf") as f: strval = f.read()
-
repo owner - changed status to invalid
Sorry, there's no bug here. A Python bytestring has no known encoding so SQLAlchemy cannot affect it in any way, and modern versions of pysqlite make the same decision. I would recommend using UnicodeText instead of Text(convert_unicode=True), the former emits a warning when a non-unicode object is passed.
-
Account Deleted Thanks for the quick response -- I may try Binary instead when reading file data.
-
repo owner - removed milestone
Removing milestone: 0.6.xx (automated comment)
- Log in to comment
Minmial testcase