zzzeek / sqlalchemy_old / issues / #1991 - pymysql dialect — Bitbucket

Issue #1991 resolved

Mike Bayer repo owner created an issue 2010-12-02

patch is attached. remote ticket is http://code.google.com/p/pymysql/issues/detail?id=41#c0

Comments (5)

Mike Bayer reporter

changed milestone to 0.7.xx

The tests don't pass, in particular the unicode round trip tests fail here. It appears that its required for "use_unicode=1" and "charset=utf8" to be specified. However, setting this flag then causes binary data to fail.

I've attached both of these bugs to the PyMySQL tracker: http://code.google.com/p/pymysql/issues/detail?id=45 http://code.google.com/p/pymysql/issues/detail?id=46 .

An updated patch against 0.7 is attached.

Unicode Failures:

======================================================================
FAIL: test.sql.test_types.UnicodeTest.test_ignoring_unicode_error
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "<string>", line 1, in <lambda>
  File "/Users/classic/dev/sqlalchemy/./test/lib/testing.py", line 53, in decorate
    return fn(*args, **kw)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 670, in test_ignoring_unicode_error
    eq_(x, unicodedata)
  File "/Users/classic/dev/sqlalchemy/./test/lib/testing.py", line 501, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xc3\xb4le de petite voix m\xe2\x80\x99a r\xc3\xa9veill\xc3\xa9. Elle disait: \xc2\xab S\xe2\x80\x99il vous pla\xc3\xaet\xe2\x80\xa6 dessine-moi un mouton! \xc2\xbb' != u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xf4le de petite voix m\u2019a r\xe9veill\xe9. Elle disait: \xab S\u2019il vous pla\xeet\u2026 dessine-moi un mouton! \xbb'

======================================================================
FAIL: test.sql.test_types.UnicodeTest.test_native_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 473, in test_native_unicode
    testing.db.dialect.returns_unicode_strings)
AssertionError: name: mysql driver pymysql returns_unicode_strings=True

======================================================================
FAIL: test.sql.test_types.UnicodeTest.test_round_trip
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 485, in test_round_trip
    eq_(x['unicode_varchar']('unicode_varchar'), unicodedata)
  File "/Users/classic/dev/sqlalchemy/./test/lib/testing.py", line 501, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xc3\xb4le de petite voix m\xe2\x80\x99a r\xc3\xa9veill\xc3\xa9. Elle disait: \xc2\xab S\xe2\x80\x99il vous pla\xc3\xaet\xe2\x80\xa6 dessine-moi un mouton! \xc2\xbb' != u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xf4le de petite voix m\u2019a r\xe9veill\xe9. Elle disait: \xab S\u2019il vous pla\xeet\u2026 dessine-moi un mouton! \xbb'

======================================================================
FAIL: test.sql.test_types.UnicodeTest.test_round_trip_executemany
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 503, in test_round_trip_executemany
    eq_(x['unicode_varchar']('unicode_varchar'), unicodedata)
  File "/Users/classic/dev/sqlalchemy/./test/lib/testing.py", line 501, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xc3\xb4le de petite voix m\xe2\x80\x99a r\xc3\xa9veill\xc3\xa9. Elle disait: \xc2\xab S\xe2\x80\x99il vous pla\xc3\xaet\xe2\x80\xa6 dessine-moi un mouton! \xc2\xbb' != u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xf4le de petite voix m\u2019a r\xe9veill\xe9. Elle disait: \xab S\u2019il vous pla\xeet\u2026 dessine-moi un mouton! \xbb'

======================================================================
FAIL: test.sql.test_types.UnicodeTest.test_union
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 522, in test_union
    eq_(x['unicode_varchar']('unicode_varchar'), unicodedata)
  File "/Users/classic/dev/sqlalchemy/./test/lib/testing.py", line 501, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xc3\xb4le de petite voix m\xe2\x80\x99a r\xc3\xa9veill\xc3\xa9. Elle disait: \xc2\xab S\xe2\x80\x99il vous pla\xc3\xaet\xe2\x80\xa6 dessine-moi un mouton! \xc2\xbb' != u'Alors vous imaginez ma surprise, au lever du jour, quand une dr\xf4le de petite voix m\u2019a r\xe9veill\xe9. Elle disait: \xab S\u2019il vous pla\xeet\u2026 dessine-moi un mouton! \xbb'

----------------------------------------------------------------------
Ran 60 tests in 0.668s

binary failures:

======================================================================
ERROR: test.sql.test_types.BinaryTest.test_comparison
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "<string>", line 1, in <lambda>
  File "/Users/classic/dev/sqlalchemy/./test/lib/testing.py", line 112, in decorate
    return fn(*args, **kw)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 864, in test_comparison
    binary_table.insert().execute(data=data)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/sql/expression.py", line 2623, in execute
    return e._execute_clauseelement(self, multiparams, params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1879, in _execute_clauseelement
    return connection._execute_clauseelement(elem, multiparams, params)
    execute(clauseelement, *multiparams, **params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1176, in execute
    params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1283, in _execute_clauseelement
    compiled_sql, params
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1365, in _execute_context
    context)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/default.py", line 318, in do_execute
    cursor.execute(statement, parameters)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py", line 96, in execute
    query = query % conn.escape(args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py", line 575, in escape
    return escape_item(obj, self.charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py", line 22, in escape_item
    return escape_sequence(val, charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py", line 45, in escape_sequence
    quoted = escape_item(item, charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py", line 27, in escape_item
    val = val.decode(charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb4 in position 0: invalid start byte

======================================================================
ERROR: test.sql.test_types.BinaryTest.test_round_trip
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
    self.test(*self.arg)
  File "/Users/classic/dev/sqlalchemy/./test/sql/test_types.py", line 823, in test_round_trip
    mypickle=testobj3)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/sql/expression.py", line 2623, in execute
    return e._execute_clauseelement(self, multiparams, params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1879, in _execute_clauseelement
    return connection._execute_clauseelement(elem, multiparams, params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1986, in _execute_clauseelement
    **(params or {}))
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1975, in execute
    execute(clauseelement, *multiparams, **params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1176, in execute
    params)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1283, in _execute_clauseelement
    compiled_sql, params
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/base.py", line 1365, in _execute_context
    context)
  File "/Users/classic/dev/sqlalchemy/./lib/sqlalchemy/engine/default.py", line 318, in do_execute
    cursor.execute(statement, parameters)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/cursors.py", line 96, in execute
    query = query % conn.escape(args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/connections.py", line 575, in escape
    return escape_item(obj, self.charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py", line 22, in escape_item
    return escape_sequence(val, charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py", line 45, in escape_sequence
    quoted = escape_item(item, charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymysql/converters.py", line 27, in escape_item
    val = val.decode(charset)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf2 in position 1: invalid continuation byte

----------------------------------------------------------------------
Ran 2 tests in 0.040s

2010-12-27T16:57:00+00:00

Mike Bayer reporter
- changed watchers to google@evax.fr
- 2010-12-27T16:57:55+00:00
Former user Account Deleted
Thanks for the work on this issue and sorry for the lack of coordination.

We posted two patches fixing the issues here and there at the end of December.

As the maintainer didn't answer yet we've setup a fork with the patches applied on bitbucket.

Evax.
- 2011-01-10T04:20:47+00:00
Mike Bayer reporter
- changed status to resolved
- changed milestone to 0.7.0
The latest SVN of pymysql passes all tests 100%, very nice !

1515073b960b2319bd77ed6e9f6e04e458636a1e
- 2011-01-26T11:18:35+00:00
Mike Bayer reporter
- removed milestone
Removing milestone: 0.7.0 (automated comment)
- 2014-02-23T18:14:45+00:00
Log in to comment

Assignee: –

Type: enhancement

Priority: major

Status: resolved

Component: mysql

Milestone: –

Votes: 0

Watchers: 0

Jira: the preferred issue tracker for Bitbucket. Join the team!