TypeError("Unicode-objects must be encoded before hashing")

Issue #316 open
Wise Kaa created an issue

What is it ?

2018-05-07 18:34:08.877 INFO  [kallithea.lib.base] IP: 46.18.200.242 User: <AuthUser('id:1[default] auth:True')> accessed /_admin/register
Error - <type 'exceptions.TypeError'>: Unicode-objects must be encoded before hashing
URL: http://git.kngk.org/_admin/register
File '/usr/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
  app_iter = self.application(environ, sr_checker)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/middleware/sessionmiddleware.py', line 62 in __call__
  return self.wrap_app(environ, session_start_response)
File '/usr/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
  response = self.app(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 103 in __call__
  response = self.dispatch(controller, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 313 in dispatch
  return controller(environ, start_response)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/base.py', line 446 in __call__
  return WSGIController.__call__(self, environ, start_response)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 214 in __call__
  response = self._dispatch_call()
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 164 in _dispatch_call
  response = self._inspect_call(func)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
  result = self._perform_call(func, args)
File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 57 in _perform_call
  return func(**args)
File '<decorator-gen-2>', line 2 in register
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 857 in __wrapper
  return func(*fargs, **fkwargs)
File '/usr/local/lib/python2.7/site-packages/kallithea/controllers/login.py', line 151 in register
  UserModel().create_registration(form_result)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 186 in create_registration
  new_user = self.create(form_data)
File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 92 in create
  v = get_crypt_password(v)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 136 in get_crypt_password
  return KallitheaCrypto.hash_string(password)
File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 110 in hash_string
  return bcrypt.hashpw(str_, bcrypt.gensalt(10))
File '/usr/local/lib/python2.7/site-packages/bcrypt/__init__.py', line 62 in hashpw
  raise TypeError("Unicode-objects must be encoded before hashing")
TypeError: Unicode-objects must be encoded before hashing

Kallithea 0.3.4

Comments (21)

  1. Wise Kaa reporter

    Yes. It is on register page. I don't Unicode strings, just ascii string.
    Username: stas
    Password: QazWsx321

  2. Thomas De Schampheleire

    Can you try following patch?

    diff --git a/kallithea/lib/auth.py b/kallithea/lib/auth.py
    --- a/kallithea/lib/auth.py
    +++ b/kallithea/lib/auth.py
    @@ -103,6 +103,12 @@ class KallitheaCrypto(object):
    
             :param password: password to hash
             """
    +        try:
    +            password = str(password)
    +        except UnicodeEncodeError:
    +            log.warning('rejecting non-ascii password')
    +            return False
    +        log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
             if is_windows:
                 return hashlib.sha256(str_).hexdigest()
             elif is_unix:
    @@ -127,6 +133,7 @@ class KallitheaCrypto(object):
             except UnicodeEncodeError:
                 log.warning('rejecting non-ascii password')
                 return False
    +        log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
             if is_windows:
                 return hashlib.sha256(password).hexdigest() == hashed
             elif is_unix:
    @@ -138,6 +145,7 @@ class KallitheaCrypto(object):
    
    
     def get_crypt_password(password):
    +    log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
         return KallitheaCrypto.hash_string(password)
    

    It seems that the method used for calculating the password hash on registration is different than the one fixed in the commit Mads referenced, used to verify a password on login.

    A simple test with the credentials you mentioned: 'stas' / 'QazWsx321' does not cause this problem on my side, though.

    Please send the output of the extra '>>>>>>>>' log statements from your test. We should be able to see exactly what Kallithea sees as password.

    What version of bcrypt is in use? (Use 'pip freeze | grep bcrypt').
    On my side, I have py-bcrypt 0.3, but I tested with 0.4 as well.

    How did you install Kallithea?

  3. Wise Kaa reporter

    Bcrypt:
    bcrypt==3.1.4
    py-bcrypt==0.4

    My machine is FreeBSD 10.4-STABLE.

    2018-05-08 08:54:49.112 INFO  [kallithea.lib.base] IP: 46.18.200.242 User: <AuthUser('id:1[default] auth:True')> accessed /_admin/register
    2018-05-08 08:54:49.205 INFO  [kallithea.lib.auth] >>>>>>>>>>>>>>>>>> get_crypt_password(u'QweAsd321')
    Error - <type 'exceptions.UnboundLocalError'>: local variable 'password' referenced before assignment
    URL: http://git.kngk.org/_admin/register
    File '/usr/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
      app_iter = self.application(environ, sr_checker)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/middleware/sessionmiddleware.py', line 62 in __call__
      return self.wrap_app(environ, session_start_response)
    File '/usr/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
      response = self.app(environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 103 in __call__
      response = self.dispatch(controller, environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 313 in dispatch
      return controller(environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/base.py', line 446 in __call__
      return WSGIController.__call__(self, environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 214 in __call__
      response = self._dispatch_call()
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 164 in _dispatch_call
      response = self._inspect_call(func)
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
      result = self._perform_call(func, args)
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 57 in _perform_call
      return func(**args)
    File '<decorator-gen-2>', line 2 in register
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 864 in __wrapper
      return func(*fargs, **fkwargs)
    File '/usr/local/lib/python2.7/site-packages/kallithea/controllers/login.py', line 151 in register
      UserModel().create_registration(form_result)
    File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 186 in create_registration
      new_user = self.create(form_data)
    File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 92 in create
      v = get_crypt_password(v)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 143 in get_crypt_password
      return KallitheaCrypto.hash_string(password)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 107 in hash_string
      password = str(password)
    UnboundLocalError: local variable 'password' referenced before assignment
    
  4. Wise Kaa reporter

    I changed password to str_

    2018-05-08 09:03:16.038 INFO  [kallithea.lib.base] IP: 46.18.200.242 User: <AuthUser('id:1[default] auth:True')> accessed /_admin/register
    2018-05-08 09:03:16.141 INFO  [kallithea.lib.auth] >>>>>>>>>>>>>>>>>> get_crypt_password(u'QweAsd321')
    2018-05-08 09:03:16.141 INFO  [kallithea.lib.auth] >>>>>>>>>>>>>>>>>> hash_string(u'QweAsd321')
    Error - <type 'exceptions.TypeError'>: Unicode-objects must be encoded before hashing
    URL: http://git.kngk.org/_admin/register
    File '/usr/local/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
      app_iter = self.application(environ, sr_checker)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/middleware/sessionmiddleware.py', line 62 in __call__
      return self.wrap_app(environ, session_start_response)
    File '/usr/local/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
      response = self.app(environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 103 in __call__
      response = self.dispatch(controller, environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/pylons/wsgiapp.py', line 313 in dispatch
      return controller(environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/base.py', line 446 in __call__
      return WSGIController.__call__(self, environ, start_response)
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 214 in __call__
      response = self._dispatch_call()
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 164 in _dispatch_call
      response = self._inspect_call(func)
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
      result = self._perform_call(func, args)
    File '/usr/local/lib/python2.7/site-packages/pylons/controllers/core.py', line 57 in _perform_call
      return func(**args)
    File '<decorator-gen-2>', line 2 in register
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 864 in __wrapper
      return func(*fargs, **fkwargs)
    File '/usr/local/lib/python2.7/site-packages/kallithea/controllers/login.py', line 151 in register
      UserModel().create_registration(form_result)
    File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 186 in create_registration
      new_user = self.create(form_data)
    File '/usr/local/lib/python2.7/site-packages/kallithea/model/user.py', line 92 in create
      v = get_crypt_password(v)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 143 in get_crypt_password
      return KallitheaCrypto.hash_string(password)
    File '/usr/local/lib/python2.7/site-packages/kallithea/lib/auth.py', line 116 in hash_string
      return bcrypt.hashpw(str_, bcrypt.gensalt(10))
    File '/usr/local/lib/python2.7/site-packages/bcrypt/__init__.py', line 62 in hashpw
      raise TypeError("Unicode-objects must be encoded before hashing")
    TypeError: Unicode-objects must be encoded before hashing
    
  5. Wise Kaa reporter

    I was correct method:

     diff auth.py auth.py.orig
    106,111d105
    <         try:
    <             password = str(str_)
    <         except UnicodeEncodeError:
    <             log.warning('rejecting non-ascii password')
    <             return False
    <         log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
    113c107
    <             return hashlib.sha256(password).hexdigest()
    ---
    >             return hashlib.sha256(str_).hexdigest()
    116c110
    <             return bcrypt.hashpw(password, bcrypt.gensalt(10))
    ---
    >             return bcrypt.hashpw(str_, bcrypt.gensalt(10))
    136d129
    <         log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
    148d140
    <     log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
    

    Now it is OK.

  6. Wise Kaa reporter
     diff auth.py auth.py.orig
    106,111d105
    <         try:
    <             password = str(str_)
    <         except UnicodeEncodeError:
    <             log.warning('rejecting non-ascii password')
    <             return False
    <         log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
    113c107
    <             return hashlib.sha256(password).hexdigest()
    ---
    >             return hashlib.sha256(str_).hexdigest()
    116c110
    <             return bcrypt.hashpw(password, bcrypt.gensalt(10))
    ---
    >             return bcrypt.hashpw(str_, bcrypt.gensalt(10))
    136d129
    <         log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
    148d140
    <     log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
    
  7. Wise Kaa reporter

    Corrected:

    diff auth.py auth.py.orig
    106,111d1054-STABLE (WEB) #3 r328435: Fri Jan 26 16:30:17 MSK 2018
    <         try:]# mc
    <             password = str(str_)
    <         except UnicodeEncodeError:
    <             log.warning('rejecting non-ascii password')
    <             return False
    <         log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
    113c107
    <             return hashlib.sha256(password).hexdigest()
    ---
    >             return hashlib.sha256(str_).hexdigest()
    116c110
    <             return bcrypt.hashpw(password, bcrypt.gensalt(10))
    ---
    >             return bcrypt.hashpw(str_, bcrypt.gensalt(10))
    136,141d129
    <         try:
    <             hashed = str(hashed)
    <         except UnicodeEncodeError:
    <             log.warning('rejecting non-ascii hashed password')
    <             return False
    <         log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
    153d140
    <     log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
    
  8. Thomas De Schampheleire

    In unified diff, the total change you did seems to be:

    diff --git a/kallithea/lib/auth.py b/kallithea/lib/auth.py
    --- a/kallithea/lib/auth.py
    +++ b/kallithea/lib/auth.py
    @@ -103,11 +103,17 @@ class KallitheaCrypto(object):
    
             :param password: password to hash
             """
    +        try:
    +            password = str(str_)
    +        except UnicodeEncodeError:
    +            log.warning('rejecting non-ascii password')
    +            return False
    +        log.info(">>>>>>>>>>>>>>>>>> hash_string(%s)", repr(str_))
             if is_windows:
    -            return hashlib.sha256(str_).hexdigest()
    +            return hashlib.sha256(password).hexdigest()
             elif is_unix:
                 import bcrypt
    -            return bcrypt.hashpw(str_, bcrypt.gensalt(10))
    +            return bcrypt.hashpw(password, bcrypt.gensalt(10))
             else:
                 raise Exception('Unknown or unsupported platform %s' \
                                 % __platform__)
    @@ -127,6 +133,12 @@ class KallitheaCrypto(object):
             except UnicodeEncodeError:
                 log.warning('rejecting non-ascii password')
                 return False
    +        try:
    +            hashed = str(hashed)
    +        except UnicodeEncodeError:
    +            log.warning('rejecting non-ascii hashed password')
    +            return False
    +        log.info(">>>>>>>>>>>>>>>>>> hash_check(%s)", repr(password))
             if is_windows:
                 return hashlib.sha256(password).hexdigest() == hashed
             elif is_unix:
    @@ -138,6 +150,7 @@ class KallitheaCrypto(object):
    
    
     def get_crypt_password(password):
    +    log.info(">>>>>>>>>>>>>>>>>> get_crypt_password(%s)", repr(password))
         return KallitheaCrypto.hash_string(password)
    

    Where the real change is to also verify the second parameter of hash_check.

    It puzzles me a bit of what was really failing in the original situation. For registration, only hash_check seems to be used, at least that is in your traceback, and you showed the same problem with my first (corrected) patch.

    Without changes (back to the original situation), do you see the same problem with any username password combination, e.g. 'foobar':'barfoo' ?

  9. Mads Kiilerich

    Why was the issue closed? It hasn't been fixed upstream?

    Also, can you confirm that you are running in an environment where LANG=C?

  10. Thomas De Schampheleire

    @Wise Kaa Could you check the locale settings in the terminal in which you start Kallithea? E.g. by running:

    env | grep LANG
    env | grep LC_
    
  11. Richard H.

    Hi there!

    I just stumbled upon this problem. And I seem to have found the culprit - by incident.

    Dec 16 01:43:06 pkg: py27-py-bcrypt-0.3 deinstalled

    Dec 16 01:43:16 pkg: py27-bcrypt-3.1.4_1 installed

    I just updated pkg and it changed my versions! So it seems there is a mismatch between ports version and pkg version of py27-bcrypt. I restored the package from ports and all is well again. When I installed kallithea, it chose this version after all.

    Using FreeBSD 10.4-RELEASE too by the way.

  12. Mads Kiilerich

    Oh. Right. These two projects both provide a 'bcrypt' module. But they should also conflict. So weird that despite setup.py requiring bcrypt >= 3.1.0, at runtime it ended up using py-bcrypt 0.3 .

    Since the "right" bcrypt has __version__ and the other doesn't, we could perhaps add an

    assert getattr(bcrypt, "__version__", '').split(".", 2)[:2] == ["3", "1"]

    or

    assert getattr(bcrypt, "__version__", '').startswith("3.1.")

  13. Richard H.

    It seems the error just happens, when the update happens without restarting the service. When trying to restart it, the dependency check fails. Well, FreeBSD is quite behind as well, just wanted to share my insight so if anybody else stumbles, the information will be there :)

  14. Thomas De Schampheleire

    @Richard H. So it seems you are not using a virtualenv, but rather are relying on system packages, is that correct?

    @Mads Kiilerich An assert like you proposed could be done. But we'd need to reference it from the setup.py dependency list to keep them in sync.
    More generally though, this problem could happen with every package: someone could update a package to a version outside of the supported range.

  15. Richard H.

    @Thomas De Schampheleire Yes, that's right. The port did not install using a virtualenv and since it worked, I never gave it much thought for now. The dependencies weren't heavy either. I might switch to a virtualenv later, but as of right now I am quite new to Python.

  16. Log in to comment