Option Item UnicodeEncodeError

Issue #1320 resolved
Martin Tiršel created an issue

If I add in Product > Option item the "Display value" some non ASCII characters (e.g "černý lak s prosklenými dvířky"), I get unicode errors at logging:

{{{

!python

Environment:

Request Method: GET Request URL: http://192.168.1.6:8000/product/kamna-jotul-f-600/ Django Version: 1.2.5 Python Version: 2.6.5 Installed Applications: ['django.contrib.sites', 'debug_toolbar', 'satchmo_store.shop', 'django.contrib.admin', 'django.contrib.admindocs', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.comments', 'django.contrib.sessions', 'django.contrib.sitemaps', 'south', 'registration', 'sorl.thumbnail', 'keyedcache', 'livesettings', 'l10n', 'satchmo_utils.thumbnail', 'satchmo_store.contact', 'tax', 'tax.modules.percent', 'shipping', 'shipping.modules.tiered', 'satchmo_ext.recentlist', 'product', 'product.modules.configurable', 'product.modules.custom', 'satchmo_ext.brand', 'payment', 'payment.modules.purchaseorder', 'satchmo_ext.satchmo_toolbar', 'satchmo_utils', 'app_plugins', 'tkshop.localsite', 'tkshop.customization', 'tkshop.satchmo_extensions', 'pagination'] Installed Middleware: ('django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'debug_toolbar.middleware.DebugToolbarMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.middleware.doc.XViewMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'threaded_multihost.middleware.ThreadLocalMiddleware', 'satchmo_store.shop.SSLMiddleware.SSLRedirect', 'pagination.middleware.PaginationMiddleware')

Traceback: File "/home/bruce/.virtualenv/dj3-satchmo/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response 100. response = callback(request, callback_args, callback_kwargs) File "/home/bruce/.virtualenv/dj3-satchmo/src/satchmo/satchmo/apps/product/views/init.py" in get_product 157. default_view_tax=default_view_tax) File "/home/bruce/.virtualenv/dj3-satchmo/src/satchmo/satchmo/apps/product/models.py" in add_template_context 1176. context = subtype.add_template_context(context, *args, kwargs) File "/home/bruce/.virtualenv/dj3-satchmo/src/satchmo/satchmo/apps/product/modules/configurable/models.py" in add_template_context 217. options = serialize_options(self, selected_options) File "/home/bruce/.virtualenv/dj3-satchmo/src/satchmo/satchmo/apps/product/utils.py" in serialize_options 216. log.debug('serialized: %s', values) File "/usr/lib/python2.6/logging/init.py" in debug 1036. self._log(DEBUG, msg, args, *kwargs) File "/usr/lib/python2.6/logging/init.py" in _log 1165. self.handle(record) File "/usr/lib/python2.6/logging/init.py" in handle 1175. self.callHandlers(record) File "/usr/lib/python2.6/logging/init.py" in callHandlers 1212. hdlr.handle(record) File "/usr/lib/python2.6/logging/init.py" in handle 673. self.emit(record) File "/home/bruce/.virtualenv/dj3-satchmo/lib/python2.6/site-packages/debug_toolbar/panels/logger.py" in emit 52. 'message': record.getMessage(), File "/usr/lib/python2.6/logging/init.py" in getMessage 306. msg = msg % self.args

Exception Type: UnicodeEncodeError at /product/kamna-jotul-f-600/ Exception Value: 'ascii' codec can't encode character u'\u010d' in position 9: ordinal not in range(128)

}}}

This exception is printed in console, not in browser unitl I enable debug toolbar, then I get exception in browser too.

Comments (12)

  1. Chris Moffitt repo owner

    I can confirm this is an issue where the log statement is having problems unpacking the message passed to it. I haven't figured out a good way to fix yet. Will continue to look at it.

  2. Alexander Clausen

    As this breaks both the logging panel of Django Debug Toolbar and Sentry logging, I'm really interested in how to fix this. A possible fix I thought up was to JSON-Serialize "values" before logging, but I guess thats overkill.

    Related python bug? http://bugs.python.org/issue5876

  3. Hynek Cernoch

    @Alexander: serialize before logging?
    No. An important feature of logging is that objects are formated only after verifiing all conditions that it should be logged.


    Exporting of lazy objects which are intended only to be logged on debugging level can be costly as can cause db queries etc.

    My idea how to workaround python bug in logging:

    # -- coding: utf-8 --
    import logging
    
    class U(object):
        """
        Object helper for international character support in logging
        with mixed unicode and UTF-8.
        """
        # can be later extended
        # 1) for more complicated cases if it is important
        # 2) look which of methods __str__, __unicode__ and __repr__ the object has
        #    in order to optimize efficiency for very "expensive" objects
    
        def __init__(self, obj):
            self.obj = obj
    
        def __unicode__(self):
            try:
                return unicode(self.obj)
            except UnicodeDecodeError:
                return self.obj.decode('utf-8')
    
        def __str__(self):
            try:
                return str(self.obj)
            except UnicodeEncodeError:
                return self.obj.encode('utf-8')
    
    logging.basicConfig(level=logging.DEBUG)
    log = logging.getLogger()
    # examples can be complicated objects
    supernova = 'Sěščřžý'
    galaxy = u'Gěščřžý'
    log.debug('New supernova %s in %s.', U(supernova), U(galaxy))
    log.debug(u'New supernova %s in %s.', U(supernova), U(galaxy))
    log.debug('%s', galaxy)
    log.debug(u'%s', galaxy)
    log.debug('%s', supernova)
    # faling original examples:
    # Nothing useful but the same short traceback inside logging logged
    # however the program continues.
    log.warning('failing:')
    log.debug(u'%s', supernova)
    log.warning('failing:')
    log.debug('%s %s', supernova, galaxy)
    log.warning('failing:')
    log.debug('%s %s', galaxy, supernova)
    log.debug('END')
    

    It works. If you can improve what is noted to be improved I will be glad.


    @Alexander: Related python bug? http://bugs.python.org/issue5876//

    I think No. IMO the most complicated silly bug was in some earlier python version but all should be fixed by my solution..

  4. Alexander de Jong

    Please don't do @Alexander bitbucket sends me e-mails for that. The username you are actually looking for is: alexc.

  5. Chris Moffitt repo owner

    I am not sure if this approach will work. The problem is that the result is a complex object that includes a dictionary and tuple so I'm not sure that this approach works. I played around trying to find a robust solution and haven't found anything yet.

  6. Hynek Cernoch

    OK. I have forgotten type conversions in the "except" parts. It works now with complex expressions like [{'ěš': u'čř', u'žý': 'ú'}, ('ů', 'ň')] .

    It must work with every object now because every object must have str or unicode methods.

    The method str of basic python types does escaping of international characters which is not nice (e.g. '\xc4\x8dern\xc3\xbd lak s prosklen\xc3\xbdmi dv\xc3\xad\xc5\x99ky' is uglier than '"černý lak s prosklenými dvířky') but it is often in the logs.

    # -- coding: utf-8 --
    import logging
    logging.basicConfig(level=logging.DEBUG)
    log = logging.getLogger()
    
    class U(object):
        def __init__(self, obj):
            self.obj = obj
        def __unicode__(self):
            try:
                return unicode(self.obj)
            except UnicodeDecodeError:
                return str(self.obj).decode('utf-8')
        def __str__(self):
            try:
                return str(self.obj)
            except UnicodeEncodeError:
                return unicode(self.obj).encode('utf-8')
    
    log.debug('something %s', U([{'ěš': u'čř', u'žý': 'ú'}, ('ů', 'ň')]))
    log.debug(u'something %s', U([{'ěš': u'čř', u'žý': 'ú'}, ('ů', 'ň')]))
    
  7. Hynek Cernoch

    Hmm. Yes, class U expects that every object has either str or unicode methods functional but this was not the case of 'product.models.Option' model for non ASCII characters.

    Problem was in its repr method which should return ASCII characters as it is usual for repr function. Then was not important double consecutive similar logging and I removed it. That's all.

    Fix 6817b1a74b96

  8. Log in to comment