Unicode in warning message results in unicode error being reported

Issue #165 resolved
Jonathan Pyle
created an issue

The warning message about duplicate keys includes some of the YAML source in the error message. If the YAML source contains Unicode, however, the printing of the warning message itself produces a unicode error.

One could get around this by disabling the warning. However, when this happened to me, it was hard to figure out that "duplicate keys" was even the problem, since the only error I was seeing was "ordinal not in range."

Maybe when warnings and errors are printed along with the source context, there could be a try/except on printing the source context, and if there is an exception, the warning/error could be printed in a way that does not include the source context.

>>> import ruamel.yaml
>>> ruamel.yaml.safe_load('type: Domestic\ntypetwo: International')
{'typetwo': 'International', 'type': 'Domestic'}
>>> ruamel.yaml.safe_load('type: Domestic\ntype: International')
ructor.py:262: DuplicateKeyFutureWarning: while constructing a mapping
  in "<byte string>", line 1, column 1:
    type: Domestic
    ^ (line: 1)
found duplicate key "type" with value "International" (original value: "Domestic")
  in "<byte string>", line 2, column 1:
    type: International
    ^ (line: 2)

To suppress this check see:
    http://yaml.readthedocs.io/en/latest/api.html#duplicate-keys

Duplicate keys will become and error in future releases, and are errors
by default when using the new API.

  warnings.warn(DuplicateKeyFutureWarning(*args))
>>> ruamel.yaml.safe_load('type: Doméstica\ntypetwo: International')
{'typetwo': 'International', 'type': u'Dom\xe9stica'}
>>> ruamel.yaml.safe_load('type: Doméstica\ntype: International')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 664, in safe_load
    return load(stream, SafeLoader, version)
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 635, in load
    return loader._constructor.get_single_data()  # type: ignore
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 104, in get_single_data
    return self.construct_document(node)
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 114, in construct_document
    for dummy in generator:
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 611, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 356, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 238, in construct_mapping
    self.check_mapping_key(node, key_node, mapping, key, value)
  File "/usr/share/docassemble/local/local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 251, in check_mapping_key
    key, value, mapping.get(key)), key_node.start_mark,
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 3: ordinal not in range(128)

Comments (1)

  1. Ruamel/Anthon van der Neut repo owner

    fix issue #165 Python2 raising error when Unicode text involved in duplicate key warning.

    I could only reproduce this in Python2, finally fixed it. Thanks for reporting

    When this change indeed resolves your problem, please Close this issue. (You can do so using the WorkFlow pull-down (close to the top right of this page))

    → <<cset 11f34d8d5dba>>

  2. Log in to comment