1. Travis Shirk
  2. eyeD3
  3. Issues
Issue #81 resolved

eyed3 on Windows, filename containing special characters - 'charmap' codec can't encode character error

Torsten Enderling
created an issue

Hi,

I'm using eyed3 on Windows 8.1 64 bit; I have Python 2.7.8.10 installed (ActivePython, 64 bit version).

I have some Hawaiian music files whose names contain a special polynesian character called the Okina (http://en.wikipedia.org/wiki/%CA%BBOkina).

The character looks like this: ʻ

When eyed3 encounters one of the files with that charater in its name (Windows displays the character just fine), it will abort and output the following message:

←[31mUncaught exception: 'charmap' codec can't encode character u'\u02bb' in position 21: character maps to <undefined>
←[39m
eyed3:ERROR: 'charmap' codec can't encode character u'\u02bb' in position 21: character maps to <undefined>
Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\eyed3\main.py", line 284, in <module>
    retval = mainFunc(args, config)
  File "C:\Python27\lib\site-packages\eyed3\main.py", line 52, in main
    fs_encoding=args.fs_encoding)
  File "C:\Python27\lib\site-packages\eyed3\utils\__init__.py", line 135, in walk
    handler.handleFile(f)
  File "C:\Python27\lib\site-packages\eyed3\plugins\classic.py", line 427, in handleFile
    self.printHeader(f)
  File "C:\Python27\lib\site-packages\eyed3\plugins\classic.py", line 490, in printHeader
    HEADER_COLOR(), size_str, Fore.RESET))
  File "C:\Python27\lib\site-packages\eyed3\utils\console.py", line 499, in printMsg
    s = _encode(s)
  File "C:\Python27\lib\site-packages\eyed3\utils\console.py", line 487, in _encode
    return s.encode(LOCAL_ENCODING)
  File "C:\Python27\lib\encodings\cp1252.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u02bb' in position 21: character maps to <undefined>

I tried experimenting with various codepage settings for the commandline instance, like chcp 65001 and similar codepages. I cannot get this working on my own. Most webpages talking about python and codepages provide code examples for python, I could not find a way for example to enforce a certain encoding for the python interpreter.

Do you have any ideas?

Comments (12)

  1. Travis Shirk repo owner

    Do you know the vaue of LOCAL_ENCODING, which is typically the 'locale' on Unix. Often this problem is not with parsing, but like the above, it occurs when displaying. If LOCAL_ENCODING were ASCII, for example, the display would fail.

  2. Torsten Enderling reporter

    I'm not sure - is there any way to find out, for example using a small Python script to determine this? I've done some Google searches, for example see this article:

    https://github.com/pytest-dev/pytest-bdd/issues/45

    The way I understand it (not having any clue about Python), is that Python will use codepage 1252 on Windows; that is a standard ASCII charset. Is LOCAL_ENCODING an environment variable that I could configure in the DOS window before executing the script, or is that an internal thing? I had done some searches before, but could not find any input regarding how to configure the codepage for Python on Windows. Windows will display the character just fine, and as mentioned before I did attempt to change the codepage in a DOS box before executing eyed3.

  3. Unknown Name

    I'm having the same issue with the album Концерт by Billy Joel.

    Windows 8.1

    Python 2.7.9

    eyeD3 0.7.5

    E:\Media\Music\Billy Joel\???????>python -m eyed3.main --no-color "01. Odoya (feat. Zhournalist).mp3"
    01. Odoya (feat. Zhournalist).mp3       [ 1.20 MB ]
    -------------------------------------------------------------------------------
    Time: 01:18     MPEG1, Layer III        [ 128 kb/s @ 44100 Hz - Stereo ]
    -------------------------------------------------------------------------------
    ID3 v2.3:
    title: Odoya (feat. Zhournalist)
    artist: Billy Joel
    Uncaught exception: 'charmap' codec can't encode characters in position 7-13: character maps to <undefined>
    
    eyed3:ERROR: 'charmap' codec can't encode characters in position 7-13: character maps to <undefined>
    Traceback (most recent call last):
      File "C:\Python27\lib\site-packages\eyed3\main.py", line 284, in <module>
        retval = mainFunc(args, config)
      File "C:\Python27\lib\site-packages\eyed3\main.py", line 52, in main
        fs_encoding=args.fs_encoding)
      File "C:\Python27\lib\site-packages\eyed3\utils\__init__.py", line 125, in walk
        handler.handleFile(os.path.abspath(path))
      File "C:\Python27\lib\site-packages\eyed3\plugins\classic.py", line 447, in handleFile
        self.printTag(self.audio_file.tag)
      File "C:\Python27\lib\site-packages\eyed3\plugins\classic.py", line 527, in printTag
        printMsg("%s: %s" % (boldText("album"), album))
      File "C:\Python27\lib\site-packages\eyed3\utils\console.py", line 499, in printMsg
        s = _encode(s)
      File "C:\Python27\lib\site-packages\eyed3\utils\console.py", line 487, in _encode
        return s.encode(LOCAL_ENCODING)
      File "C:\Python27\lib\encodings\cp1252.py", line 12, in encode
        return codecs.charmap_encode(input,errors,encoding_table)
    UnicodeEncodeError: 'charmap' codec can't encode characters in position 7-13: character maps to <undefined>
    

    Mutagen doesn't throw an exception, but displays '???????' like the file path.

    E:\Media\Music\Billy Joel\???????>python c:\Python27\Scripts\mutagen-inspect "01. Odoya (feat. Zhournalist).mp3"
    -- 01. Odoya (feat. Zhournalist).mp3
    - MPEG 1 layer 3, 128000 bps, 44100 Hz, 78.35 seconds (audio/mp3)
    TALB=???????
    

    Hopefully it isn't a big change to gracefully handle non-ASCII characters.

    -Bearded Gnome

  4. Unknown Name

    Here's the values of LOCAL_ENCODING and LOCAL_FS_ENCODING:

    >>> import locale
    >>> locale.getpreferredencoding(do_setlocale=True)
    'cp1252'
    
    >>> import sys
    >>> sys.getfilesystemencoding()
    'mbcs'
    

    Edit: Possible solution is to change _encode in console.py:

    def _encode(s):
        if PY2:
            if isinstance(s, unicode):
                return s.encode(LOCAL_ENCODING, 'replace')
            elif isinstance(s, str):
                return s
            else:
                raise TypeError("Argument must be str or unicode")
        else:
            assert(isinstance(s, str))
            return s
    

    Specifically, adding 'replace' to encode().

    Still working on filenames.

    -Bearded Gnome

  5. Torsten Enderling reporter

    Sorry to bother you again; I have installed the latest dev branch, and I now receive a different error message:

    -------------------------------------------------------------------------------
    Uncaught exception: global name 'compat' is not defined
    
    eyed3:ERROR: global name 'compat' is not defined
    Traceback (most recent call last):
      File "c:\python27\lib\site-packages\eyed3\main.py", line 281, in <module>
        retval = mainFunc(args, config)
      File "c:\python27\lib\site-packages\eyed3\main.py", line 49, in main
        fs_encoding=args.fs_encoding)
      File "c:\python27\lib\site-packages\eyed3\utils\__init__.py", line 85, in walk
        handler.handleFile(f)
      File "c:\python27\lib\site-packages\eyed3\plugins\classic.py", line 436, in handleFile
        save_tag = (self.handleEdits(self.audio_file.tag) or
      File "c:\python27\lib\site-packages\eyed3\plugins\classic.py", line 955, in handleEdits
        fid = compat.BytesType(fid, "ascii")
    NameError: global name 'compat' is not defined
    

    Any idea? I receive this error message regardless of which file I use it on. Should I wait for the stable release?

  6. Log in to comment