pysqlite 2.8.2 does not return None for cursor.description, breaks downstream

[Edd Barrett]

Hi,

Trying to update OpenBSD's py-sqlalchemy package to 1.0.14. Three tests currently fail. py.test output below.

Thanks

> /usr/ports/pobj/py-sqlalchemy-1.0.14/SQLAlchemy-1.0.14/lib/sqlalchemy/testing/suite/test_update_delete.py(54)

  48         def test_delete(self):
  49             t = self.tables.plain_pk
  50             r = config.db.execute(
  51                 t.delete().where(t.c.id == 2)
  52             )
  53             assert not r.is_insert
  54  ->         assert not r.returns_rows
  55             eq_(
  56                 config.db.execute(t.select().order_by(t.c.id)).fetchall(),
  57                 [
  58                     (1, "d1"),
  59                     (3, "d3")
  60                 ]
  61             )

  30         def test_update(self):
  31             t = self.tables.plain_pk
  32             r = config.db.execute(
  33                 t.update().where(t.c.id == 2),
  34                 data="d2_new"
  35             )
  36             assert not r.is_insert
  37  ->         assert not r.returns_rows
  38
  39             eq_(
  40                 config.db.execute(t.select().order_by(t.c.id)).fetchall(),
  41                 [
  42                     (1, "d1"),
  43                     (2, "d2_new"),
  44                     (3, "d3")
  45                 ]
  46             )

980         def test_no_rowcount_on_selects_inserts(self):
 981             """assert that rowcount is only called on deletes and updates.
 982
 983             This because cursor.rowcount may can be expensive on some dialects
 984             such as Firebird, however many dialects require it be called
 985             before the cursor is closed.
 986
 987             """
 988
 989             metadata = self.metadata
 990
 991             engine = engines.testing_engine()
 992
 993             t = Table('t1', metadata,
 994                       Column('data', String(10))
 995                       )
 996             metadata.create_all(engine)
 997
 998             with patch.object(
 999                     engine.dialect.execution_ctx_cls, "rowcount") as mock_rowcount:
1000                 mock_rowcount.__get__ = Mock()
1001                 engine.execute(t.insert(),
1002                                {'data': 'd1'},
1003                                {'data': 'd2'},
1004                                {'data': 'd3'})
1005
1006                 eq_(len(mock_rowcount.__get__.mock_calls), 0)
1007
1008                 eq_(
1009                     engine.execute(t.select()).fetchall(),
1010                     [('d1', ), ('d2', ), ('d3', )]
1011                 )
1012                 eq_(len(mock_rowcount.__get__.mock_calls), 0)
1013
1014                 engine.execute(t.update(), {'data': 'd4'})
1015
1016  >>             eq_(len(mock_rowcount.__get__.mock_calls), 1)
1017
1018                 engine.execute(t.delete())
1019  ->             eq_(len(mock_rowcount.__get__.mock_calls), 2)
(Pdb++) len(mock_rowcount.__get__.mock_calls)
0

[Mike Bayer]

Hi -

Unfortunately, I can't fix an issue without any details, such as:

1. version of Python?

2. version of SQLite?

3. how are tests being run ?

4. Full output of the test run, including all stack traces etc?

5. reproducible every time?

6. any downstream context? openbsd issue tracker, continuous integration servers, mailing list, etc?

[Edd Barrett]

OK, let's start working through these.

1) Python 2.7.12

2) SQLite 3.9.2

3) I just used py.test in the source directory (but with 1.0.14 installed into the system Python tree via the OS package manager).

4) See below

5) Yes, but I notice it only fires when the Python sqlite2 package is installed.

6) No, I'm an OpenBSD developer discovering these issues for the first time.

Thanks

============================= test session starts ==============================
platform openbsd6 -- Python 2.7.12, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /usr/local/bin/python2.7
cachedir: .cache
rootdir: /usr/ports/pobj/SQLAlchemy-1.0.14/SQLAlchemy-1.0.14, inifile: setup.cfg
plugins: capturelog-0.7
collecting ... collected 7374 items

test/aaa_profiling/test_compiler.py::CompileTest_sqlite_pysqlite::test_insert <- lib/sqlalchemy/testing/profiling.py SKIPPED
test/aaa_profiling/test_compiler.py::CompileTest_sqlite_pysqlite::test_select SKIPPED
test/aaa_profiling/test_compiler.py::CompileTest_sqlite_pysqlite::test_select_labels SKIPPED
...
test/dialect/test_suite.py::SimpleUpdateDeleteTest_sqlite_pysqlite::test_delete <- lib/sqlalchemy/testing/suite/test_update_delete.py FAILED
test/dialect/test_suite.py::SimpleUpdateDeleteTest_sqlite_pysqlite::test_update <- lib/sqlalchemy/testing/suite/test_update_delete.py FAILED
...
test/sql/test_resultset.py::ResultProxyTest_sqlite_pysqlite::test_no_rowcount_on_selects_inserts FAILED
...
=========================== short test summary info ============================
FAIL test/dialect/test_suite.py::SimpleUpdateDeleteTest_sqlite_pysqlite::()::test_delete
FAIL test/dialect/test_suite.py::SimpleUpdateDeleteTest_sqlite_pysqlite::()::test_update
FAIL test/sql/test_resultset.py::ResultProxyTest_sqlite_pysqlite::()::test_no_rowcount_on_selects_inserts

=================================== FAILURES ===================================
______________ SimpleUpdateDeleteTest_sqlite_pysqlite.test_delete ______________
Traceback (most recent call last):
  File "/usr/ports/pobj/SQLAlchemy-1.0.14/SQLAlchemy-1.0.14/test/../lib/sqlalchemy/testing/suite/test_update_delete.py", line 54, in test_delete
    assert not r.returns_rows
AssertionError: assert not True
 +  where True = <sqlalchemy.engine.result.ResultProxy object at 0x330562daa90>.returns_rows
______________ SimpleUpdateDeleteTest_sqlite_pysqlite.test_update ______________
Traceback (most recent call last):
  File "/usr/ports/pobj/SQLAlchemy-1.0.14/SQLAlchemy-1.0.14/test/../lib/sqlalchemy/testing/suite/test_update_delete.py", line 37, in test_update
    assert not r.returns_rows
AssertionError: assert not True
 +  where True = <sqlalchemy.engine.result.ResultProxy object at 0x3305ba38f10>.returns_rows
_____ ResultProxyTest_sqlite_pysqlite.test_no_rowcount_on_selects_inserts ______
Traceback (most recent call last):
  File "/usr/ports/pobj/SQLAlchemy-1.0.14/SQLAlchemy-1.0.14/test/sql/test_resultset.py", line 1016, in test_no_rowcount_on_selects_inserts
    eq_(len(mock_rowcount.__get__.mock_calls), 1)
  File "/usr/ports/pobj/SQLAlchemy-1.0.14/SQLAlchemy-1.0.14/test/../lib/sqlalchemy/testing/assertions.py", line 209, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
AssertionError: 0 != 1
=== 3 failed, 6581 passed, 790 skipped, 1 pytest-warnings in 288.64 seconds ====

[Mike Bayer]

Yes, but I notice it only fires when the Python sqlite2 package is installed.

OK lets dig into this, since there is no Python sqlite2 package. do you mean:

https://docs.python.org/2/library/sqlite3.html

or

https://pypi.python.org/pypi/pysqlite/2.8.2

?

next, note that SQLite itself "vendors" itself statically into a consuming application. So the version of SQLite you have there may not be what's being used. Go into Python to get the right versions of things (send me this output):

#!

$ python
Python 2.7.12 (default, Jul 18 2016, 10:55:51) 
[GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.13.0'
>>> from pysqlite2 import dbapi2
>>> dbapi2.sqlite_version
'3.13.0'

the issue we're seeing above is basically related to sqlite and/or the DBAPI itself.

[Mike Bayer]

hey good news, pysqlite locally does the same thing. this is a pysqlite vs. sqlite3 difference.

[Mike Bayer]

upstream pysqlite issue: https://github.com/ghaering/pysqlite/issues/104

[Mike Bayer]

fixed upstream in pysqlite 2.8.3 https://pypi.python.org/pypi/pysqlite/2.8.3

[Edd Barrett]

Many thanks :)