Updating character or memo fields with bytes always attempts to decode using the ascii codec

Lucas Taylor avatarLucas Taylor created an issue

The update field functions for character and memo fields always attempt to decode bytes to unicode using the ascii codec. This occurs regardless of the specified codepage. If you update a field with a unicode value, the decoding step is skipped, and the update function (correctly) uses the defined codepage for encoding.

In this example, the codepage is latin1, but the decoding is attempted using ascii

>>> table = dbf.Table('test_latin')
>>> table.open()
dbf.Table('test_latin.dbf', status='read-write')
>>> table.codepage
CodePage('latin1', 'latin1', '\x00')
>>> table.append({'memo': chr(148)})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "dbf.py", line 4670, in append
    gather(newrecord, dictdata, drop=drop)
  File "dbf.py", line 7394, in gather
    record._commit_flux()
  File "dbf.py", line 2235, in _commit_flux
    raise DbfError("unable to write updates to disk, original data restored: %r" % (exc,))
dbf.DbfError: unable to write updates to disk, original data restored: UnicodeDecodeError('ascii', '\x94', 0, 1, 'ordinal not in range(128)')
>>> table.append({'memo': unichr(148)})
>>> print table[-1]
  0 - memo      : u'\x94'

The issue appears to be a bug in the choice of default decoders in Record._update_field_value and RecordTemplate._update_field_value

It looks like those functions should use self._meta.decoder instead of self._meta.input_decoder

Comments (1)

  1. stoneleaf

    Ideally, all strings going into dbf are already in unicode format. When a string is passed in that is not unicode, and the field type is not binary (not possible for db3), then the input_decoding setting is used -- which defaults to ascii, and is what is used to set the self._meta.input_decoder.

  2. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.