MySQL Connector, Python 3.4, unable to get unicode strings [Was: MySQL Connector >= 2.0.0, Python 3.4 always returns bytearrays instead of (unicode) strings]

[Andrea Gronchi]

I'm using SQLAlchemy 9.8.0 and MySQL Connector 2.0.3.

I've found myself unable to get unicode data (intending the PY3 type "str"), every text field was always obtained as byte array data.

Passing explicitly convert_unicode=True had no effect; declaring columns as UnicodeText had no effect; coercing connection options "use_unicode=1&charset=…" had no effect.

I have gave out and moved on to PyMySql; everything worked instantly.

Then I found this release note http://dev.mysql.com/doc/relnotes/connector-python/en/news-2-0-0.html , which basically notifies about the braking change of always returning bytearray data while using raw cursors. I don't know if this might be the case but it's probably worth looking into.

[Andrea Gronchi]

Updated.

It turns out, by testing the issue using mysql-connector-python-1.2.3, that the connector simply returns 'bytes', not 'bytearray'. Different type, the originating issue still stands: unable to get unicode data out of varchar and text fields.

This may very well be an issue on my side though.

[Mike Bayer]

mysqlconnector supports unicode w/o issue, our tests pass, need a test case here. please reopen with a short script, thanks

[Mike Bayer]

here's a case which shows it:

#!

 python3 -m pytest test/dialect/mysql/test_dialect.py --db mysqlconnector -k test_special_encodings
================================================ test session starts ================================================
platform darwin -- Python 3.4.2 -- py-1.4.26 -- pytest-2.5.2 -- /usr/local/bin/python3
collected 11 items 

test/dialect/mysql/test_dialect.py:89: DialectTest_mysql_mysqlconnector.test_special_encodings FAILED

===================================================== FAILURES ======================================================
______________________________ DialectTest_mysql_mysqlconnector.test_special_encodings ______________________________
Traceback (most recent call last):
  File "<string>", line 2, in test_special_encodings
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/testing/exclusions.py", line 95, in decorate
    return self._do(config._current, fn, *args, **kw)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/testing/exclusions.py", line 124, in _do
    self._expect_failure(config, ex, name=fn.__name__)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/testing/exclusions.py", line 136, in _expect_failure
    util.raise_from_cause(ex)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/testing/exclusions.py", line 122, in _do
    return_value = fn(*args, **kw)
  File "/Users/classic/dev/sqlalchemy/test/dialect/mysql/test_dialect.py", line 96, in test_special_encodings
    conn = eng.connect()
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 2005, in connect
    return self._connection_cls(self, **kwargs)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 72, in __init__
    if connection is not None else engine.raw_connection()
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 2091, in raw_connection
    self.pool.unique_connection, _connection)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 2061, in _wrap_pool_connect
    return fn()
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 279, in unique_connection
    return _ConnectionFairy._checkout(self)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 644, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 439, in checkout
    rec = pool._do_get()
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 964, in _do_get
    self._dec_overflow()
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 961, in _do_get
    return self._create_connection()
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 284, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/pool.py", line 415, in __init__
    exec_once(self.connection, self)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/event/attr.py", line 248, in exec_once
    self(*args, **kw)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/event/attr.py", line 258, in __call__
    fn(*args, **kw)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/util/langhelpers.py", line 1319, in go
    return once_fn(*arg, **kw)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/strategies.py", line 157, in first_connect
    dialect.initialize(c)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/dialects/mysql/base.py", line 2578, in initialize
    self._detect_ansiquotes(connection)
  File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/dialects/mysql/base.py", line 2857, in _detect_ansiquotes
    self._server_ansiquotes = 'ANSI_QUOTES' in mode
TypeError: Type str doesn't support the buffer API
============================================== short test summary info ==============================================
FAIL test/dialect/mysql/test_dialect.py::DialectTest_mysql_mysqlconnector::()::test_special_encodings
================================= 10 tests deselected by '-ktest_special_encodings' =================================
====================================== 1 failed, 10 deselected in 0.04

[Mike Bayer]

seems like a different issue since that's use_unicode=0 and it's only within those special operations, #3336 is created.