0.4.0: dispatch.wsgi terminates due to LC_ALL or LANG encoding.

Issue #340 resolved
Edmund Wong created an issue

I upgraded 0.3.x to 0.4.0 and came across a LC_ALL/LANG issue.

[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.caching] Caching enabled: True -> {'data_dir': '/var/www/kali/srv/data/cache/data', 'cache_regions': {'long_term': {'expire': 36000, 'data_dir': '/var/www/kali/srv/data/cache/data', 'type': 'memory', 'enabled': True, 'lock_dir': '/var/www/kali/srv/data/cache/lock', 'key_length': 256}, 'sql_cache_short': {'expire': 10, 'data_dir': '/var/www/kali/srv/data/cache/data', 'type': 'memory', 'enabled': True, 'lock_dir': '/var/www/kali/srv/data/cache/lock', 'key_length': 256}, 'short_term': {'expire': 60, 'data_dir': '/var/www/kali/srv/data/cache/data', 'type': 'memory', 'enabled': True, 'lock_dir': '/var/www/kali/srv/data/cache/lock', 'key_length': 256}}, 'expire': None, 'log_file': None, 'type': 'memory', 'lock_dir': '/var/www/kali/srv/data/cache/lock'}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.mingflush] MingSessionFlush enabled: False -> {'autoflush': False}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.transaction_manager] TransactionManager enabled: False -> {'attempts': 1, 'enabled': False, 'commit_veto': None}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.696 DEBUG [tg.appwrappers.errorpage] ErrorPageApplicationWrapper enabled: True -> {'content_types': ['text/html', None], 'handle_exceptions': True, 'path': '/error/document', 'status_codes': [400, 401, 403, 404], 'enabled': True}
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.697 ERROR [kallithea.config.app_cfg] Cannot encode Unicode paths to file system encoding 'ANSI_X3.4-1968'
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.697 ERROR [kallithea.config.app_cfg] Note: Environment variable LANG is 'en_US.UTF-8' - perhaps change it to some other value from 'locale -a', like 'C.UTF-8' or 'en_US.UTF-8'
[Mon Apr 01 09:58:06 2019] [error] 2019-04-01 09:58:06.697 ERROR [kallithea.config.app_cfg] Terminating ...

Looking at the code in app_cfg.py, I noticed it's testing out u\xe9'.encode(sys.getfilesystemencoding())

But it ends up failing and Terminating.

I go into python and get the following results:

import sys

print(sys.getfilesystemencoding())

prints out 'UTF-8'.

So I copy part of the code form setup-configuration(app) as follows:

import os
import sys
 try:
     u'\xe9'.encode(sys.getfilesystemencoding())
 except UnicodeEncodeError:
     print("Cannot encode Unicode paths to file system encoding %r", sys.getfilesystemencoding())
     for var in ['LC_CTYPE', 'LC_ALL', 'LANG']:
         if var in os.environ:
             val = os.environ[var]
             print("%s is %r" % (var, val))
             break
         else:
             print("No locale setting found...")
     print("Terminating...")

prints : '\xc3\xa9'

so I add the following before the try: block:

print(sys.getfilesystemencoding())

and it prints :

sys : 'ANSI_X3.4-1968'

So I add:

os.environ["LC_ALL"] = "en_US.UTF-8" os.environ["LANG"] = "en_US.UTF-8"

to the dispatch.wsgi.

But I still get the same ANSI_X3.4-1968 result.

These are the settings I have in my.ini:

i18n.enabled = false lang =

I think I might have screwed up the upgrade... though I'm not sure where as I followed the upgrade instructions.. (though I hazard a guess I missed some points somewhere...)

Comments (9)

  1. Thomas De Schampheleire

    What is the output of 'locale -a' on the server? It should contain en_US.UTF-8, the value you specify. Alternatively, use another UTF-8 value which is present in the output of locale -a.

  2. Mads Kiilerich

    Also: Notice that Python does evil things with setting system encoding based on environment at startup. Setting it inside Python in os.environ will probably not work as expected. It should probably be set in the environment before launching Python.

  3. Edmund Wong reporter

    @Thomas De Schampheleire locale -a apparently lists en_US.utf8 instead of what I had thought would be (en_US.UTF-8).

    @Mads Killerich bingo. With the results from 'locale -a' and your suggestion, I added "lang=en_US.utf8" to the WSGIDaemonProcess entry in the apache configuration. Now it's working

    Should this be written as a note in the documentation? I'm running Kallithea in a CentOS 6.9 system.

  4. Mads Kiilerich

    Are you sure en_US.UTF-8 doesn't work? I thought it aliased one spelling to the other. But perhaps not in all places. That would be nice to get verified an extra time, now when you get it working and reliably can test while changing one parameter at a time.

    Indeed - we should update documentation to cover it better. We added this big early failure, just so we can get reliable feedback early, instead of random and hard-to-reproduce failures later on at runtime.

  5. Thomas De Schampheleire

    If I set an invalid 'LANG' like 'foo', I get an error from TurboGears2:

    2019-04-01 20:49:24.344 ERROR [gearbox] Failed to load application
    2019-04-01 20:49:24.345 ERROR [gearbox] unsupported locale setting
    Traceback (most recent call last):
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 172, in _run_subcommand
        return cmd.run(parsed_args)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/command.py", line 31, in run
        self.take_action(parsed_args)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commands/serve.py", line 280, in take_action
        relative_to=base, global_conf=parsed_vars)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commands/serve.py", line 311, in loadapp
        return loadapp(app_spec, name=name, relative_to=relative_to, **kw)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp
        return loadobj(APP, uri, name=name, **kw)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 277, in loadobj
        global_conf=global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 302, in loadcontext
        global_conf=global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 326, in _loadconfig
        return loader.get_context(object_type, name, global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 459, in get_context
        section)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 481, in _context_from_use
        object_type, name=use, global_conf=global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 412, in get_context
        global_conf=global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 302, in loadcontext
        global_conf=global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 334, in _loadegg
        return loader.get_context(object_type, name, global_conf)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 625, in get_context
        object_type, name=name)
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 651, in find_egg_entry_point
        possible.append((entry.load(), protocol, entry.name))
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2411, in load
        return self.resolve()
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
        module = __import__(self.module_name, fromlist=['__name__'], level=0)
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/config/middleware.py", line 17, in <module>
        from kallithea.config.app_cfg import base_config
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/config/app_cfg.py", line 34, in <module>
        from kallithea.lib.middleware.simplegit import SimpleGit
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/middleware/simplegit.py", line 42, in <module>
        from kallithea.lib.base import BaseVCSController, check_locking_state
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/base.py", line 53, in <module>
        from kallithea.lib import auth_modules
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/auth_modules/__init__.py", line 24, in <module>
        from kallithea.lib.auth import PasswordGenerator, AuthUser
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/auth.py", line 54, in <module>
        from kallithea.lib.utils import get_repo_slug, get_repo_group_slug, \
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 39, in <module>
        from kallithea.lib.vcs.utils.hgcompat import ui, config
      File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/vcs/utils/hgcompat.py", line 11, in <module>
        from mercurial import localrepo
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/localrepo.py", line 26, in <module>
        from . import (
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/context.py", line 26, in <module>
        from . import (
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/subrepo.py", line 23, in <module>
        from . import (
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/cmdutil.py", line 23, in <module>
        from . import (
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/mercurial/crecord.py", line 33, in <module>
        locale.setlocale(locale.LC_ALL, u'')
      File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib64/python2.7/locale.py", line 581, in setlocale
        return _setlocale(category, locale)
    Error: unsupported locale setting
    

    This is also true for LANG settings that are in itself valid but are not present on the system (i.e. not listed in locale -a), for example: nl_BE.utf8.

    When the locale exists but is not unicode capable, like LANG=C, then the error is:

    2019-04-01 20:52:48.329 ERROR [kallithea.config.app_cfg] Cannot encode Unicode paths to file system encoding 'ANSI_X3.4-1968'
    2019-04-01 20:52:48.329 ERROR [kallithea.config.app_cfg] Note: Environment variable LANG is 'C' - perhaps change it to some other value from 'locale -a', like 'C.UTF-8' or 'en_US.UTF-8'
    2019-04-01 20:52:48.329 ERROR [kallithea.config.app_cfg] Terminating ...
    

    If I set LANG to a valid locale, present on the system, Kallithea starts correctly, regardless of the exact spelling of the utf-8 suffix:

    en_US.utf8
    en_US.utf-8
    en_US.UTF8
    en_US.UTF-8
    

    So, I think the main clue that fixed the problem for @Edmund Wong was adding a valid locale (whatever the spelling) in the apache configuration under 'WSGIDaemonProcess'.

    I think that the "Apache with mod_wsgi" section docs/setup.rst should indeed be adapted to give a better hint about this. @Mads Kiilerich, if you have experience with Apache, please propose a modification, I didn't use it yet.

  6. Mads Kiilerich

    I have only used Kallithea in production in an international environment where all filenames were pure ASCII.

    I guess docs/overview.rst has to mention the locale challenges and say something about UTF-8 being highly recommend (and nothing else feasible), and that a valid utf8 locale has to be set in the environment before starting the Python interpreter. I don't know what we can say about setting the right valid locale. But the "Apache with mod_wsgi" section of docs/setup.rst should probably use the lang and locale options in the examples.

    http://blog.dscpl.com.au/2014/09/setting-lang-and-lcall-when-using.html and https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html says everything pretty well. I'm not sure we can say it better. So perhaps just apply that advice in our examples, and link to them?

    (In general, searching for "Graham Dumpleton mod_wsgi kallithea" gives some very qualified hits.)

    For other ways of hosting the WSGI process (and for other Kallithea invocations), we should probably also mention the need for setting environment variables. (If Kallihea were to take care of that, we would probably have to use wrapper scripts. That would probably not be a good idea.)

  7. Log in to comment