Unicode-related error when using sqlite with str containing non-ascii characters

Issue #2076 resolved
Former user created an issue

See attached testcase (testcase.py) and output (output.txt). 'In the wild,' this error occurs when trying to INSERT the content of a binary file into a Text column. In the testcase, this is simulated using a str object that is populated with a unicode escape.

Original code looked somewhat like this:

strval = "" with open("somepdf.pdf") as f: strval = f.read()

In other words, strval is filled with binary data from a file.

When executed: - using sqlite url: first commit succeeds, second fails - using mysql url: both commits succeed as expected

Comments (6)

  1. Former user Account Deleted

    Description pseudocode with formatting:

    strval = ""
    with open("somepdf.pdf") as f:
        strval = f.read()
    
  2. Mike Bayer repo owner

    Sorry, there's no bug here. A Python bytestring has no known encoding so SQLAlchemy cannot affect it in any way, and modern versions of pysqlite make the same decision. I would recommend using UnicodeText instead of Text(convert_unicode=True), the former emits a warning when a non-unicode object is passed.

  3. Former user Account Deleted

    Thanks for the quick response -- I may try Binary instead when reading file data.

  4. Log in to comment