py3k support for MySQLdb via pymysql

[Thomas Tanner]

sqlalchemy HEAD, mysql 5.5.38, pymysql 0.6.6, mysql=mysql+pysql://user:password@localhost/test_schema running py.test --db mysql --dropfirst --backend-only --log-info=sqlalchemy.orm.mapper --log-debug=sqlalchemy.pool --log-info=sqlalchemy.engine

platform darwin -- Python 3.4.3 -- py-1.4.26 -- pytest-2.7.0 --

DEBUG:sqlalchemy.pool.QueuePool:Created new connection <pymysql.connections.Connection object at 0x10800cf60>
INFO:sqlalchemy.engine.base.Engine:SHOW VARIABLES LIKE 'sql_mode'
INFO:sqlalchemy.engine.base.Engine:()
INFO:sqlalchemy.engine.base.Engine:SELECT DATABASE()
INFO:sqlalchemy.engine.base.Engine:()
INFO:sqlalchemy.engine.base.Engine:show collation where `Charset` = 'utf8' and `Collation` = 'utf8_bin'
INFO:sqlalchemy.engine.base.Engine:()
INFO:sqlalchemy.engine.base.Engine:SELECT CAST('test plain returns' AS CHAR(60)) AS anon_1
INFO:sqlalchemy.engine.base.Engine:()
INFO:sqlalchemy.engine.base.Engine:SELECT CAST('test unicode returns' AS CHAR(60)) AS anon_1
INFO:sqlalchemy.engine.base.Engine:()
INFO:sqlalchemy.engine.base.Engine:SELECT CAST('test collated returns' AS CHAR CHARACTER SET utf8) COLLATE utf8_bin AS anon_1
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW FULL TABLES FROM `test_schema`
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW FULL TABLES FROM `test_schema`
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
DEBUG:sqlalchemy.pool.QueuePool:Created new connection <pymysql.connections.Connection object at 0x108031c18>
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW FULL TABLES FROM `test_schema`
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> checked out from pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW FULL TABLES FROM `test_schema`
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> rollback-on-return
============================= test session starts ==============================
platform darwin -- Python 3.4.3 -- py-1.4.26 -- pytest-2.7.0 -- 

collecting ... DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SELECT CURRENT_TIMESTAMP AS current_timestamp_1
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW VARIABLES LIKE 'lower_case_table_names'
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW VARIABLES LIKE 'lower_case_table_names'
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW VARIABLES LIKE 'lower_case_table_names'
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> checked out from pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x108031c18> rollback-on-return
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> checked out from pool
INFO:sqlalchemy.engine.base.Engine:SHOW VARIABLES LIKE 'lower_case_table_names'
INFO:sqlalchemy.engine.base.Engine:()
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> being returned to pool
DEBUG:sqlalchemy.pool.QueuePool:Connection <pymysql.connections.Connection object at 0x10800cf60> rollback-on-return
collected 1846 items

test/aaa_profiling/test_compiler.py::CompileTest_mysql_pymysql::test_insert PASSED
test/aaa_profiling/test_compiler.py::CompileTest_mysql_pymysql::test_select PASSED
test/aaa_profiling/test_compiler.py::CompileTest_mysql_pymysql::test_select_labels PASSED
test/aaa_profiling/test_compiler.py::CompileTest_mysql_pymysql::test_update PASSED
test/aaa_profiling/test_compiler.py::CompileTest_mysql_pymysql::test_update_whereclause PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_DecimalResultProcessor_init PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_DecimalResultProcessor_process PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_UnicodeResultProcessor_init PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_ad_hoc_types PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_alias_pathing PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_fixture PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_join_cache PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_key_fallback_result PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_many_discarded_relationships PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_many_updates PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_mapper_reset PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_orm_many_engines PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_path_registry PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_session PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_sessionmaker PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_type_compile PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_unicode_warnings PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_warnings_util PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_with_inheritance PASSED
test/aaa_profiling/test_memusage.py::MemUsageTest_mysql_pymysql::test_with_manytomany PASSED
test/aaa_profiling/test_resultset.py::ExecutionTest_mysql_pymysql::test_minimal_connection_execute FAILEDINFO:sqlalchemy.pool.QueuePool:Pool disposed. Pool size: 5  Connections in pool: 0 Current Overflow: -5 Current Checked out connections: 0
INFO:sqlalchemy.pool.QueuePool:Pool recreating
Encountered a stray connection in test cleanup: {<sqlalchemy.pool._ConnectionRecord object at 0x10ad34f28>}
Traceback (most recent call last):
  File "bin/py.test", line 9, in <module>
    load_entry_point('pytest==2.7.0', 'console_scripts', 'py.test')()
  File "$site/_pytest/config.py", line 41, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "$site/_pytest/core.py", line 521, in __call__
    return self._docall(self.methods, kwargs)
  File "$site/_pytest/core.py", line 528, in _docall
    firstresult=self.firstresult).execute()
  File "$site/_pytest/core.py", line 394, in execute
    res = method(*args)
  File "$site/_pytest/main.py", line 116, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "$site/_pytest/main.py", line 109, in wrap_session
    exitstatus=session.exitstatus)
  File "$site/_pytest/core.py", line 521, in __call__
    return self._docall(self.methods, kwargs)
  File "$site/_pytest/core.py", line 528, in _docall
    firstresult=self.firstresult).execute()
  File "$site/_pytest/core.py", line 393, in execute
    return wrapped_call(method(*args), self.execute)
  File "$site/_pytest/core.py", line 109, in wrapped_call
    wrap_controller.send(call_outcome)
  File "$site/_pytest/terminal.py", line 356, in pytest_sessionfinish
    outcome.get_result()
  File "$site/_pytest/core.py", line 137, in get_result
    raise ex[1].with_traceback(ex[2])
  File "$site/_pytest/core.py", line 123, in __init__
    self.result = func()
  File "$site/_pytest/core.py", line 394, in execute
    res = method(*args)
  File "$site/_pytest/runner.py", line 55, in pytest_sessionfinish
    session._setupstate.teardown_all()
  File "$site/_pytest/runner.py", line 375, in teardown_all
    self._pop_and_teardown()
  File "$site/_pytest/runner.py", line 348, in _pop_and_teardown
    self._teardown_with_finalization(colitem)
  File "$site/_pytest/runner.py", line 366, in _teardown_with_finalization
    self._callfinalizers(colitem)
  File "$site/_pytest/runner.py", line 363, in _callfinalizers
    py.builtin._reraise(*exc)
  File "$site/py/_builtin.py", line 227, in _reraise
    raise cls.with_traceback(val, tb)
  File "$site/_pytest/runner.py", line 356, in _callfinalizers
    fin()
  File "$path/test/../lib/sqlalchemy/testing/plugin/pytestplugin.py", line 149, in finalize
    class_teardown(item.parent.parent)
  File "$path/test/../lib/sqlalchemy/testing/plugin/pytestplugin.py", line 178, in class_teardown
    plugin_base.stop_test_class(item.cls)
  File "$path/test/../lib/sqlalchemy/testing/plugin/plugin_base.py", line 426, in stop_test_class
    assertions.global_cleanup_assertions()
  File "$path/test/../lib/sqlalchemy/testing/assertions.py", line 165, in global_cleanup_assertions
    _assert_no_stray_pool_connections()
  File "$path/test/../lib/sqlalchemy/testing/assertions.py", line 203, in _assert_no_stray_pool_connections
    "after gc.collect(): %s" % err
AssertionError: Stray connection refused to leave after gc.collect(): {<sqlalchemy.pool._ConnectionRecord object at 0x10ad34f28>}

without logging and with duplicate errors removed (table users exists):

==================================== ERRORS ====================================
 ERROR at setup of ComponentReflectionTest_mysql_pymysql.test_autoincrement_col 
Traceback (most recent call last):
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 768, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "$site/pymysql/connections.py", line 929, in _read_query_result
    result.read()
  File "$site/pymysql/connections.py", line 1125, in read
    first_packet = self.connection._read_packet()
  File "$site/pymysql/connections.py", line 893, in _read_packet
    packet.check_error()
  File "$site/pymysql/connections.py", line 369, in check_error
    err.raise_mysql_exception(self._data)
  File "$site/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "$site/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1050, "Table 'users' already exists")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "$path/test/../lib/sqlalchemy/testing/fixtures.py", line 83, in setup_class
    cls._setup_once_tables()
  File "$path/test/../lib/sqlalchemy/testing/fixtures.py", line 112, in _setup_once_tables
    cls.metadata.create_all(cls.bind)
  File "$path/test/../lib/sqlalchemy/sql/schema.py", line 3611, in create_all
    tables=tables)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1843, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1471, in _run_visitor
    **kwargs).traverse_single(element)
  File "$path/test/../lib/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 727, in visit_metadata
    _is_metadata_operation=True)
  File "$path/test/../lib/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 761, in visit_table
    include_foreign_key_constraints=include_foreign_key_constraints
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 968, in _execute_ddl
    compiled
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1332, in _handle_dbapi_exception
    exc_info
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 768, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "$site/pymysql/connections.py", line 929, in _read_query_result
    result.read()
  File "$site/pymysql/connections.py", line 1125, in read
    first_packet = self.connection._read_packet()
  File "$site/pymysql/connections.py", line 893, in _read_packet
    packet.check_error()
  File "$site/pymysql/connections.py", line 369, in check_error
    err.raise_mysql_exception(self._data)
  File "$site/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "$site/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1050, "Table 'users' already exists") [SQL: '\nCREATE TABLE test_schema.users (\n\tuser_id INTEGER NOT NULL AUTO_INCREMENT, \n\ttest1 CHAR(5) NOT NULL, \n\ttest2 FLOAT(5) NOT NULL, \n\tparent_user_id INTEGER, \n\tPRIMARY KEY (user_id), \n\tFOREIGN KEY(parent_user_id) REFERENCES test_schema.users (user_id)\n)ENGINE=InnoDB\n\n']

...
=================================== FAILURES ===================================
_________________ EnumSetTest_mysql_pymysql.test_unicode_enum __________________
Traceback (most recent call last):
  File "<string>", line 2, in test_unicode_enum
  File "$path/test/../lib/sqlalchemy/testing/util.py", line 195, in provide_metadata
    return fn(*args, **kw)
  File "$path/test/dialect/mysql/test_types.py", line 914, in test_unicode_enum
    metadata.create_all()
  File "$path/test/../lib/sqlalchemy/sql/schema.py", line 3611, in create_all
    tables=tables)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1843, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1471, in _run_visitor
    **kwargs).traverse_single(element)
  File "$path/test/../lib/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 727, in visit_metadata
    _is_metadata_operation=True)
  File "$path/test/../lib/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 761, in visit_table
    include_foreign_key_constraints=include_foreign_key_constraints
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 968, in _execute_ddl
    compiled
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1335, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 766, in query
    sql = sql.encode(self.encoding, 'surrogateescape')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 95: ordinal not in range(256)
_______________ EnumSetTest_mysql_pymysql.test_unicode_roundtrip _______________
Traceback (most recent call last):
  File "<string>", line 2, in test_unicode_roundtrip
  File "$path/test/../lib/sqlalchemy/testing/util.py", line 195, in provide_metadata
    return fn(*args, **kw)
  File "$path/test/dialect/mysql/test_types.py", line 830, in test_unicode_roundtrip
    set_table.create()
  File "$path/test/../lib/sqlalchemy/sql/schema.py", line 725, in create
    checkfirst=checkfirst)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1843, in _run_visitor
    conn._run_visitor(visitorcallable, element, **kwargs)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1471, in _run_visitor
    **kwargs).traverse_single(element)
  File "$path/test/../lib/sqlalchemy/sql/visitors.py", line 121, in traverse_single
    return meth(obj, **kw)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 761, in visit_table
    include_foreign_key_constraints=include_foreign_key_constraints
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 968, in _execute_ddl
    compiled
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1335, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 766, in query
    sql = sql.encode(self.encoding, 'surrogateescape')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 87: ordinal not in range(256)
=========================== short test summary info ============================
FAIL test/dialect/mysql/test_types.py::EnumSetTest_mysql_pymysql::()::test_unicode_enum
FAIL test/dialect/mysql/test_types.py::EnumSetTest_mysql_pymysql::()::test_unicode_roundtrip
!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 25 failures !!!!!!!!!!!!!!!!!!!!
========= 2 failed, 95 passed, 334 skipped, 23 error in 62.13 seconds ==========

[Mike Bayer]

you need to run with this DBURI:

--dburi "mysql+pymysql://scott:tiger@127.0.0.1:3306/test?charset=utf8"

the latin-1 error occurs without the utf8.

As far as the warnings and the "users" table, that doesn't reproduce for me even if I run the tests with the failing latin-1 assertion. Whether it's something in the test suite or something local to your environment we'll get it with #3358. Please let's expand that single issue for any additional Py3K + MySQL test failures you're looking into for now, thanks.

[Thomas Tanner]

sorry for continuing this but I think it is still relevant:

re-running the tests with sqlalchemy 0.9.9 and the ?charset=utf8 option I get (table exists errors removed)

=================================== FAILURES ===================================
________________ TypesTest_mysql_pymysql.test_bit_50_roundtrip _________________
Traceback (most recent call last):
  File "<string>", line 2, in test_bit_50_roundtrip
  File "$path/test/../lib/sqlalchemy/testing/exclusions.py", line 84, in decorate
    return self._do(config._current, fn, *args, **kw)
  File "$path/test/../lib/sqlalchemy/testing/exclusions.py", line 114, in _do
    self._expect_failure(config, ex, name=fn.__name__)
  File "$path/test/../lib/sqlalchemy/testing/exclusions.py", line 126, in _expect_failure
    util.raise_from_cause(ex)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 182, in reraise
    raise value
  File "$path/test/../lib/sqlalchemy/testing/exclusions.py", line 112, in _do
    return_value = fn(*args, **kw)
  File "<string>", line 2, in test_bit_50_roundtrip
  File "$path/test/../lib/sqlalchemy/testing/util.py", line 191, in provide_metadata
    return fn(*args, **kw)
  File "$path/test/dialect/mysql/test_types.py", line 331, in test_bit_50_roundtrip
    roundtrip([0] * 8)
  File "$path/test/dialect/mysql/test_types.py", line 327, in roundtrip
    print("Found %s" % list(row))
  File "$path/test/../lib/sqlalchemy/dialects/mysql/base.py", line 876, in process
    for i in map(ord, value):
TypeError: ord() expected string of length 1, but int found
----------------------------- Captured stdout call -----------------------------
Storing [0, 0, 0, 0, 0, 0, 0, 0]
Expected [0, 0, 0, 0, 0, 0, 0, 0]
__________________ CompiledCacheTest_mysql_pymysql.test_cache __________________
Traceback (most recent call last):
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 768, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "$site/pymysql/connections.py", line 929, in _read_query_result
    result.read()
  File "$site/pymysql/connections.py", line 1125, in read
    first_packet = self.connection._read_packet()
  File "$site/pymysql/connections.py", line 893, in _read_packet
    packet.check_error()
  File "$site/pymysql/connections.py", line 369, in check_error
    err.raise_mysql_exception(self._data)
  File "$site/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "$site/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1054, "Unknown column 'user_name' in 'field list'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "$path/test/engine/test_execute.py", line 730, in test_cache
    cached_conn.execute(ins, {'user_name': 'u1'})
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 841, in execute
    return meth(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 938, in _execute_clauseelement
    compiled_sql, distilled_params
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1070, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1271, in _handle_dbapi_exception
    exc_info
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 768, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "$site/pymysql/connections.py", line 929, in _read_query_result
    result.read()
  File "$site/pymysql/connections.py", line 1125, in read
    first_packet = self.connection._read_packet()
  File "$site/pymysql/connections.py", line 893, in _read_packet
    packet.check_error()
  File "$site/pymysql/connections.py", line 369, in check_error
    err.raise_mysql_exception(self._data)
  File "$site/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "$site/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (InternalError) (1054, "Unknown column 'user_name' in 'field list'") 'INSERT INTO users (user_name) VALUES (%s)' ('u1',)
______ CompiledCacheTest_mysql_pymysql.test_keys_independent_of_ordering _______
Traceback (most recent call last):
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 768, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "$site/pymysql/connections.py", line 929, in _read_query_result
    result.read()
  File "$site/pymysql/connections.py", line 1125, in read
    first_packet = self.connection._read_packet()
  File "$site/pymysql/connections.py", line 893, in _read_packet
    packet.check_error()
  File "$site/pymysql/connections.py", line 369, in check_error
    err.raise_mysql_exception(self._data)
  File "$site/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "$site/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
pymysql.err.InternalError: (1054, "Unknown column 'user_name' in 'field list'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "$path/test/engine/test_execute.py", line 741, in test_keys_independent_of_ordering
    {"user_id": 1, "user_name": "u1", "extra_data": "e1"})
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 841, in execute
    return meth(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 938, in _execute_clauseelement
    compiled_sql, distilled_params
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1070, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1271, in _handle_dbapi_exception
    exc_info
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "$path/test/../lib/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "$path/test/../lib/sqlalchemy/engine/base.py", line 1063, in _execute_context
    context)
  File "$path/test/../lib/sqlalchemy/engine/default.py", line 442, in do_execute
    cursor.execute(statement, parameters)
  File "$site/pymysql/cursors.py", line 134, in execute
    result = self._query(query)
  File "$site/pymysql/cursors.py", line 282, in _query
    conn.query(q)
  File "$site/pymysql/connections.py", line 768, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "$site/pymysql/connections.py", line 929, in _read_query_result
    result.read()
  File "$site/pymysql/connections.py", line 1125, in read
    first_packet = self.connection._read_packet()
  File "$site/pymysql/connections.py", line 893, in _read_packet
    packet.check_error()
  File "$site/pymysql/connections.py", line 369, in check_error
    err.raise_mysql_exception(self._data)
  File "$site/pymysql/err.py", line 120, in raise_mysql_exception
    _check_mysql_exception(errinfo)
  File "$site/pymysql/err.py", line 115, in _check_mysql_exception
    raise InternalError(errno, errorvalue)
sqlalchemy.exc.InternalError: (InternalError) (1054, "Unknown column 'user_name' in 'field list'") 'INSERT INTO users (user_id, user_name, extra_data) VALUES (%s, %s, %s)' (1, 'u1', 'e1')
________________ UnicodeTest_mysql_pymysql.test_native_unicode _________________
Traceback (most recent call last):
  File "$path/test/sql/test_types.py", line 943, in test_native_unicode
    expected
  File "$path/test/../lib/sqlalchemy/testing/assertions.py", line 187, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: True != False
=========================== short test summary info ============================
FAIL test/dialect/mysql/test_types.py::TypesTest_mysql_pymysql::()::test_bit_50_roundtrip
FAIL test/engine/test_execute.py::CompiledCacheTest_mysql_pymysql::()::test_cache
FAIL test/engine/test_execute.py::CompiledCacheTest_mysql_pymysql::()::test_keys_independent_of_ordering
FAIL test/sql/test_types.py::UnicodeTest_mysql_pymysql::()::test_native_unicode
======== 4 failed, 1263 passed, 423 skipped, 30 error in 149.96 seconds ========

[Thomas Tanner]

is the AssertionError: Stray connection refused to leave after gc.collect(): {<sqlalchemy.pool._ConnectionRecord object at 0x10ad34f28>} above expected?

[Mike Bayer]

the warning is sometimes a side effect of fixtures going bad, which seems to be happening with your "users" table issue. It's not a big deal.

The "unknown column "user_name" " error is again a fixture not working.

As for the BIT issue:

#!

File "$path/test/dialect/mysql/test_types.py", line 327, in roundtrip
    print("Found %s" % list(row))
  File "$path/test/../lib/sqlalchemy/dialects/mysql/base.py", line 876, in process
    for i in map(ord, value):
TypeError: ord() expected string of length 1, but int found

that's not SQLAlchemy master, nor is it 0.9. Line 876 of mysql/base.py is in the middle of the MEDIUMINT type and we do not use map(ord, value). This is our result processor:

    def result_processor(self, dialect, coltype):
        """Convert a MySQL's 64 bit, variable length binary string to a long.

        TODO: this is MySQL-db, pyodbc specific.  OurSQL and mysqlconnector
        already do this, so this logic should be moved to those dialects.

        """

        def process(value):
            if value is not None:
                v = 0
                for i in value:
                    if not isinstance(i, int):
                        i = ord(i)  # convert byte to int on Python 2
                    v = v << 8 | i
                return v
            return value
        return process

The code you are looking at, involving map(), was fixed in #3333.

So this looks like a dupe of #3333.

[Mike Bayer]

oh, you said running against 0.9.9. Please run against master and/or rel_0_9 from git only, thanks.