Unicode Comparison Warning

Issue #3971 resolved
Lukas Siemon created an issue

Getting a warning

.../env/local/lib/python2.7/site-packages/sqlalchemy/sql/type_api.py:359: 
UnicodeWarning: Unicode equal comparison failed to convert 
both arguments to Unicode - interpreting them as being unequal
  return x == y

Unfortunately we treat warnings as exceptions, so I'm kind of stuck here. This seems like a problem with SQLAlchemy? Any advice what best to do here?

Test Case:

# coding=utf-8
import unittest
from sqlalchemy import Column, String, text, inspect
from sqlalchemy.dialects.postgresql import UUID
from sqlalchemy.ext.declarative import declarative_base
from flask import Flask
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = (
    'postgres://postgres:password@localhost:5432/tmp')
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True
db = SQLAlchemy(app)

Base = declarative_base()


class Test(db.Model):
    __tablename__ = 'test'
    id = Column(UUID, primary_key=True, nullable=False,
                server_default=text('uuid_generate_v4()'))
    name = Column(String, nullable=False)

db.create_all()


class TestClass(unittest.TestCase):

    def test_function(self):
        test = Test(name="♥ This")
        db.session.add(test)
        db.session.commit()
        db.session.expire_all()
        test = Test.query.filter_by(name="♥ This").first()
        test.name = "Still ♥ This!"
        inspected = inspect(test)
        assert getattr(inspected.attrs, 'name').history.has_changes()

Comments (2)

  1. Mike Bayer repo owner

    The test here is incorrect in that you aren't using Python unicode objects:

       test = Test.query.filter_by(name="♥ This").first()
       test.name = "Still ♥ This!"
    

    since this is Python 2.7 and I see no unicode_literals future import, this should be:

       test = Test.query.filter_by(name=u"♥ This").first()
       test.name = u"Still ♥ This!"
    

    Otherwise you are comparing the bytestring in Python (unknown encoding) to the Unicode object you get back from the database (Postgresql database likely defaulting to utf-8 so the non-Unicode object is persisted correctly, psycopg2 gives you back Unicode).

    changing even just the one line:

    test.name = u"Still ♥ This!"

    allows it to succeed.

    use echo='debug' to see things like this happening:

    db.engine.echo='debug'
    
  2. Lukas Siemon reporter

    So the test case didn't actually reflect what I'm seeing in the code base.

    I don't fully understand what is going on, but even though the pg encoding is set to utf8, the fields are returned as str (and not unicode). I wasn't able to figure out where this happens. So now I just used the dirty reload hack and set the python encoding to utf8. Time to move on to python 3 I guess...

  3. Log in to comment