Commits

Jason R. Coombs committed fee9e1b

Protect from UnicodeDecodeError when decoding data on the wire when data is not properly encoded in ASCII or UTF-8.

  • Participants
  • Parent commits 18e799c
  • Tags 3.2.2

Comments (0)

Files changed (2)

+3.2.2
+=====
+
+* Protect from UnicodeDecodeError when decoding data on the wire when data is
+  not properly encoded in ASCII or UTF-8.
+
 3.2.1
 =====
 
     [u'barbar', u'baz']
     >>> len(b)
     0
+
+    Some clients will feed latin-1 or other encodings. This client assumes
+    UTF-8 and any other encodings will produce the "replacement character",
+    �.
+    >>> b.feed(b'Ol\xe9\n')
+    >>> list(b.lines())
+    [u'Ol\ufffd']
     """
     line_sep_exp = re.compile(b'\r?\n')
     encoding = 'utf-8'
         lines = self.line_sep_exp.split(self.buffer)
         # save the last, unfinished, possibly empty line
         self.buffer = lines.pop()
-        return (line.decode(self.encoding) for line in lines)
+        return (
+            line.decode(self.encoding, errors='replace')
+            for line in lines
+        )
 
     def __iter__(self):
         return self.lines()