cursor.executemany should accept iterators

Issue #13 closed
Anonymous created an issue

The current implementation of cursor.executemany does not accept iterators. If one invokes the method as follows:

# ....
records = iter([1, 2, 3, 4, 5])
cursor.executemany(my_insert_statement, records)

The following TypeError is returned

TypeError: must be list, not list_iterator

When working with large data, it becomes impractical store all records in memory. Having the ability to supply an iterator would alleviate this constraint. This would also make cx_Oracle compatible with other libraries such as petl.

Acceptance Criteria

cursor.executemany should accept iterators as well as in memory data structures.

Comments (4)

  1. Shai Berger

    The request to support iterators may be valid, but both the reason given for it and the example are wrong.

    executemany() stores all the records in memory whether it is given them as a list or as an iterator; that's just the way it works, at the C level. For iterators to have any effect on memory here you need to implement batching of the records -- and that can easily be done in Python (there is a recipe for it in itertools' documentation) and then you don't need the cx_Oracle feature anyway.

    The example uses records = iter([1, 2, 3, 4, 5]), but executemany() expects a sequence of sequences, so it should be either records = iter([[1, 2, 3, 4, 5]]) or iter([[1], [2], [3], [4], [5]])

  2. Krishna Mohan Itikarlapalli

    Iterators may want to supply the data coming from a source on the fly (from another query result or from a stream). And it is possible to supply the values to OCI/C on the fly too (using dynamic binds). But would like to know the actual requirement from the poster.

  3. Log in to comment