Lennart Regebro committed 226bd89

We need to make sure that the result is always a str, even if the result is an error response. Otherwise
you get an error when trying to pattern match in line 206.

Comments (0)

Files changed (1)


         base = f.url     # handle redirects
         page =
-        if sys.version_info >= (3,) and not isinstance(f, urllib2.HTTPError):
-            charset = f.headers.get_param('charset') or 'latin-1'
+        if not isinstance(page, str): # We are in Python 3 and got bytes. We want str.
+            if isinstance(f, urllib2.HTTPError):
+                # Errors have no charset, assume latin1:
+                charset = 'latin-1'
+            else:
+                charset = f.headers.get_param('charset') or 'latin-1'
             page = page.decode(charset, "ignore")
         for match in HREF.finditer(page):