Incorrect display of unicode symbols in file view

Issue #451 new
VilenT created an issue

See attached screenshot

If i click it, it shows:
{{{

There is no file nor directory at the given path: 'Eis.Preloader.Data/Data/\xd0\xbf\xd1\x97\xd0\x85\xd0\xbf\xd1\x97\xd0\x85\xd0\xbf\xd1\x97\xd0\x85' at revision 'b750436539ad'

}}}

Comments (8)

  1. VilenT reporter

    What? This is how my files are represented in the repository, I do not control that at all. This is windows setup. Btw, zip files are created with the same wrong encoded symbols

  2. Marcin Kuzminski repo owner

    Hmm maybe try plaing with encoding settings in the .ini files ? In general try to not use non-ascii filenames it's always troubles

  3. VilenT reporter

    Hmm maybe try plaing with encoding settings in the .ini files ? In general try to not use non-ascii filenames it's always troubles

    Are you serious? Why dont you use unicode throughout the application and configured system encoding(or ini-file set encoding) for all IO ?

  4. Marcin Kuzminski repo owner

    Yes I'm serious.

    First of all creating non-ascii file/directory names is a bad habit. Try not to do this.

    I use unicode on whole app with few exceptions on when it's not possible. eg. in mercurial filectx nodes when they simply require string paths to be able to get them from the hg index. I know when you use utf8(the default encoding in rhodecode used for str<->unicode translations) it works fine see here for example: http://demo.rhodecode.org/japanese-test/files/tip/

    utf8(default) encoding is controlled by the rhodeocde .ini file on running server instance and can be changed to what other encoding you use on your windows machine. I'm pretty sure that it is not utf8 if filenames are messed up.

  5. VilenT reporter

    First of all creating non-ascii file/directory names is a bad habit. Try not to do this.

    It is a bad habit to ignore non-latin writing users, moreover it is a bad habit to ignore windows users

    So, now I've changed the encoding to cp1251 (it is my win locale encoding) and now the file is displayed correctly I suppose rhodecode should try to detect the correct locale/encoding.

  6. VilenT reporter

    But now, after clicking it I got error 500 https://sdkdevtest2/Eis/files/2099a3d8cbf46d5f59f208db9b0a017465ca76b3/Eis.Preloader.Data/Data/%D0%95%D0%98%D0%A1

    Error - <type 'exceptions.UnicodeDecodeError'>: 'utf8' codec can't decode byte 0xc5 in position 24: invalid continuation byte
    URL: https://127.0.0.1:5000/Eis/files/2099a3d8cbf46d5f59f208db9b0a017465ca76b3/Eis.Preloader.Data/Data/%D0%95%D0%98%D0%A1
    File 'c:\\Python27\\lib\\site-packages\\weberror-0.10.3-py2.7.egg\\weberror\\errormiddleware.py', line 162 in __call__
      app_iter = self.application(environ, sr_checker)
    File 'c:\\Python27\\lib\\site-packages\\beaker-1.6.3-py2.7.egg\\beaker\\middleware.py', line 155 in __call__
      return self.wrap_app(environ, session_start_response)
    File 'c:\\Python27\\lib\\site-packages\\routes-1.13-py2.7.egg\\routes\\middleware.py', line 131 in __call__
      response = self.app(environ, start_response)
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\wsgiapp.py', line 107 in __call__
      response = self.dispatch(controller, environ, start_response)
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\wsgiapp.py', line 312 in dispatch
      return controller(environ, start_response)
    File 'c:\\Python27\\lib\\site-packages\\rhodecode-1.4.0beta-py2.7.egg\\rhodecode\\lib\\base.py', line 180 in __call__
      return WSGIController.__call__(self, environ, start_response)
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\controllers\\core.py', line 211 in __call__
      response = self._dispatch_call()
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\controllers\\core.py', line 162 in _dispatch_call
      response = self._inspect_call(func)
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\controllers\\core.py', line 105 in _inspect_call
      result = self._perform_call(func, args)
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\controllers\\core.py', line 57 in _perform_call
      return func(**args)
    File '<string>', line 2 in index
    File 'c:\\Python27\\lib\\site-packages\\rhodecode-1.4.0beta-py2.7.egg\\rhodecode\\lib\\auth.py', line 528 in __wrapper
      return func(*fargs, **fkwargs)
    File 'c:\\Python27\\lib\\site-packages\\rhodecode-1.4.0beta-py2.7.egg\\rhodecode\\controllers\\files.py', line 164 in index
      return render('files/files.html')
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\templating.py', line 243 in render_mako
      cache_type=cache_type, cache_expire=cache_expire)
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\templating.py', line 218 in cached_template
      return render_func()
    File 'c:\\Python27\\lib\\site-packages\\pylons-1.0-py2.7.egg\\pylons\\templating.py', line 240 in render_template
      return literal(template.render_unicode(**globs))
    File 'c:\\Python27\\lib\\site-packages\\mako-0.7.0-py2.7.egg\\mako\\template.py', line 406 in render_unicode
      as_unicode=True)
    File 'c:\\Python27\\lib\\site-packages\\mako-0.7.0-py2.7.egg\\mako\\runtime.py', line 764 in _render
      **_kwargs_for_callable(callable_, data))
    File 'c:\\Python27\\lib\\site-packages\\mako-0.7.0-py2.7.egg\\mako\\runtime.py', line 796 in _render_context
      _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
    File 'c:\\Python27\\lib\\site-packages\\mako-0.7.0-py2.7.egg\\mako\\runtime.py', line 817 in _exec_template
      _render_error(template, context, e)
    File 'c:\\Python27\\lib\\site-packages\\mako-0.7.0-py2.7.egg\\mako\\runtime.py', line 826 in _render_error
      result = template.error_handler(context, error)
    File 'c:\\Python27\\lib\\site-packages\\mako-0.7.0-py2.7.egg\\mako\\runtime.py', line 815 in _exec_template
      callable_(context, *args, **kwargs)
    File 'e:\\Team\\rhodecode\\data\\templates\\base\\root.html.py', line 64 in render_body
      __M_writer(escape(next.body()))
    File 'e:\\Team\\rhodecode\\data\\templates\\base\\base.html.py', line 75 in render_body
      __M_writer(escape(next.main()))
    File 'e:\\Team\\rhodecode\\data\\templates\\files\\files.html.py', line 82 in render_main
      __M_writer(escape(h.url("files_nodelist_home",repo_name=c.repo_name,revision=c.changeset.raw_id,f_path=c.file.path)))
    File 'c:\\Python27\\lib\\site-packages\\paste-1.7.5.1-py2.7.egg\\paste\\registry.py', line 155 in __call__
      return self._current_obj()(*args, **kw)
    File 'c:\\Python27\\lib\\site-packages\\routes-1.13-py2.7.egg\\routes\\util.py', line 393 in __call__
      url = self.mapper.generate(*route_args, **newargs)
    File 'c:\\Python27\\lib\\site-packages\\routes-1.13-py2.7.egg\\routes\\mapper.py', line 826 in generate
      path = route.generate(**kargs)
    File 'c:\\Python27\\lib\\site-packages\\routes-1.13-py2.7.egg\\routes\\route.py', line 697 in generate
      if val and not self.req_regs[key].match(self.make_unicode(val)):
    File 'c:\\Python27\\lib\\site-packages\\routes-1.13-py2.7.egg\\routes\\route.py', line 139 in make_unicode
      return s.decode(self.encoding)
    File 'c:\\Python27\\lib\\encodings\\utf_8.py', line 16 in decode
      return codecs.utf_8_decode(input, errors, True)
    UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 24: invalid continuation byte
    
    
    CGI Variables
    -------------
      CONTENT_LENGTH: '0'
      CONTENT_TYPE: '; charset="utf-8"'
      HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
      HTTP_ACCEPT_LANGUAGE: 'ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3'
      HTTP_CONNECTION: 'Keep-Alive'
      HTTP_COOKIE: 'rhodecode=b87a9bb36627485eacbd840d5dfdf04d'
      HTTP_DNT: '1'
      HTTP_HOST: '127.0.0.1:5000'
      HTTP_MAX_FORWARDS: '10'
      HTTP_REFERER: 'https://sdkdevtest2/Eis/files/2099a3d8cbf46d5f59f208db9b0a017465ca76b3/Eis.Preloader.Data/Data'
      HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0'
      HTTP_X_ARR_LOG_ID: '7d60b8ea-a995-4a81-a521-b5f5315ad03b'
      HTTP_X_ARR_SSL: '2048|128|CN=Sdkdevtest2.SDK_GARANT|CN=Sdkdevtest2.SDK_GARANT'
      HTTP_X_FORWARDED_FOR: '10.121.8.40:10522'
      HTTP_X_ORIGINAL_URL: '/Eis/files/2099a3d8cbf46d5f59f208db9b0a017465ca76b3/Eis.Preloader.Data/Data/%D0%95%D0%98%D0%A1'
      PATH_INFO: '/Eis/files/2099a3d8cbf46d5f59f208db9b0a017465ca76b3/Eis.Preloader.Data/Data/\xd0\x95\xd0\x98\xd0\xa1'
      REMOTE_ADDR: '127.0.0.1'
      REQUEST_METHOD: 'GET'
      SERVER_NAME: '0.0.0.0'
      SERVER_PORT: '5000'
      SERVER_PROTOCOL: 'HTTP/1.1'
    
    
    WSGI Variables
    --------------
      application: <beaker.middleware.SessionMiddleware object at 0x03105370>
      beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x03105370>>
      beaker.session: {'_authentication_token': '5c0ddffb4b8e100646bde32185e114f2ae756a79', 'rhodecode_user': {'username': u'tambovtsev', 'is_authenticated': True, 'user_id': 3}, '_accessed_time': 1338272516.174, '_creation_time': 1338272295.298}
      paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at 0x01C562F0>
      paste.registry: <paste.registry.Registry object at 0x04B2E8D0>
      paste.throw_errors: True
      pylons.action_method: <bound method FilesController.index of <rhodecode.controllers.files.FilesController object at 0x042CBC90>>
      pylons.controller: <rhodecode.controllers.files.FilesController object at 0x042CBC90>
      pylons.environ_config: {'session': 'beaker.session', 'cache': 'beaker.cache'}
      pylons.pylons: <pylons.util.PylonsContext object at 0x042CB730>
      pylons.routes_dict: {'action': u'index', 'controller': u'files', 'revision': u'2099a3d8cbf46d5f59f208db9b0a017465ca76b3', 'f_path': u'Eis.Preloader.Data/Data/\u0415\u0418\u0421', 'repo_name': u'Eis'}
      routes.route: <routes.route.Route object at 0x02F712D0>
      routes.url: <routes.util.URLGenerator object at 0x0337D4B0>
      webob._parsed_query_vars: (GET([]), '')
      webob.adhoc_attrs: {'user': <AuthUser('id:3:tambovtsev|True')>, 'language': 'en-us'}
      wsgi process: 'Multithreaded'
      wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x0337D4B0>, {'action': u'index', 'controller': u'files', 'revision': u'2099a3d8cbf46d5f59f208db9b0a017465ca76b3', 'f_path': u'Eis.Preloader.Data/Data/\u0415\u0418\u0421', 'repo_name': u'Eis'})
    ------------------------------------------------------------
    
    
    
  7. Log in to comment