PostgreSQL non-native Enum fails with unicode

Issue #3354 resolved
Thomas Tanner created an issue

sqlalchemy 0.9.9, python 2.7.9, osx, PostgreSQL database with encoding unicode.

# -*- coding: utf8 -*-

from sqlalchemy.types import String, Integer, Enum
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, create_engine
from sqlalchemy.orm import Session
import sys

Base = declarative_base()

class Thing(Base):
    __tablename__ = 'thing'
    id = Column(Integer, primary_key=True)
    name = Column(String(1))
    value = Column(Enum('B',u'Ü',name='myvalue', native_enum=False, convert_unicode=None))

engine = create_engine(sys.argv[1], echo=True,
client_encoding='utf8')
Base.metadata.create_all(engine)
sess = Session(engine)
sess.add(Thing(name='A', value=u'Ü'))
sess.commit()

sess.query(Thing).filter_by(name='A').first()
Traceback (most recent call last):
  File "./utfbug.py", line 23, in <module>
    sess.query(Thing).filter_by(name='A').first()
  File "<path>/sqlalchemy/orm/query.py", line 2370, in first
    ret = list(self[0:1])
  File "<path>/sqlalchemy/orm/query.py", line 2231, in __getitem__
    return list(res)
  File "<path>/sqlalchemy/orm/loading.py", line 103, in instances
    util.raise_from_cause(err)
  File "<path>/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "<path>/sqlalchemy/orm/loading.py", line 74, in instances
    rows = [process[0](row, None) for row in fetch]
  File "<path>/sqlalchemy/orm/loading.py", line 459, in _instance
    populate_state(state, dict_, row, isnew, only_load_props)
  File "<path>/sqlalchemy/orm/loading.py", line 309, in populate_state
    populator(state, dict_, row)
  File "<path>/sqlalchemy/orm/strategies.py", line 169, in fetch_col
    dict_[key] = row[col]
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 0: ordinal not in range(128)

if I set convert_unicode=False, it works fine. mysql ist not affected.

Comments (3)

  1. Mike Bayer repo owner

    this one is sort of funny because each time I look at it the cause gets more and more intricate. First I thought it was just the "convert_unicode" param being passed to String incorrectly. Then I realized it's this: https://bitbucket.org/zzzeek/sqlalchemy/src/ee40c7afaabe33e2080607de0045a9fd0a4748d8/lib/sqlalchemy/dialects/postgresql/psycopg2.py?at=master#cl-357 and I figured, that must be some old legacy thing, but then I realized, oh, we actually need that for a "native" enum still, and the reason it's not checking for "native_enum" is because that code precedes when we had the generic "Enum" type; it used to be a standalone object.

  2. Mike Bayer repo owner
    • Fixed a long-standing bug where the :class:.Enum type as used with the psycopg2 dialect in conjunction with non-ascii values and native_enum=False would fail to decode return results properly. This stemmed from when the PG :class:.postgresql.ENUM type used to be a standalone type without a "non native" option. fixes #3354
    • corrected the assertsql comparison rule to expect a non-ascii SQL string

    → <<cset ecd7b31d5eae>>

  3. Mike Bayer repo owner
    • Fixed a long-standing bug where the :class:.Enum type as used with the psycopg2 dialect in conjunction with non-ascii values and native_enum=False would fail to decode return results properly. This stemmed from when the PG :class:.postgresql.ENUM type used to be a standalone type without a "non native" option. fixes #3354 cherry pick from ecd7b31d5eaed138e699293719f70260da3c978d

    → <<cset 0775c90ce090>>

  4. Log in to comment