Unicode decoding errors not configurable

The Python codec encode/decode methods support the option of specifying how to handle encoding/decoding errors.

For example:

>>> value = '\x92'
>>> value.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 0: unexpected code byte

>>> value.decode('utf-8','replace')
(u'\ufffd', 1)

This is a problem for me because we have what is supposed to be unicode data in our database, but they were able to coerce it to store non-unicode data. I cannot retrieve anything right now.

I propose changing 3 lines:

sqlalchemy/types.py

class String(Concatenable, TypeEngine):

    def __init__(self, length=None, convert_unicode=False, assert_unicode=None, unicode_errors='strict'):
        self.length = length
        self.convert_unicode = convert_unicode
        self.assert_unicode = assert_unicode
        self.unicode_errors = unicode_errors

    def result_processor(self, dialect):
        if self.convert_unicode or dialect.convert_unicode:
            def process(value):
                if value is not None and not isinstance(value, unicode):
                    return value.decode(dialect.encoding,self.unicode_errors)
                else:
                    return value
            return process
        else:
            return None

Comments (8)