Overview

Arrayterator creates a buffered iterator for reading big arrays in small contiguous blocks. The class is useful for objects stored in the file system. It allows iteration over the object without reading everything in memory; instead, small blocks are read and iterated over.

Arrayterator can be used with any object that supports multidimensional slices. This includes NumPy arrays, but also variables from Scientific.IO.NetCDF or pynetcdf for example.

Examples

>>> import numpy as np
>>> a = np.arange(3 * 4 * 5 * 6).reshape(3, 4, 5, 6)
>>> a_itor = np.lib.arrayterator.Arrayterator(a, 2)
>>> a_itor.shape
(3, 4, 5, 6)

Now we can iterate over a_itor, and it will return arrays of size two. Since buf_size was smaller than any dimension, the first dimension will be iterated over first:

>>> for subarr in a_itor:
...     if not subarr.all():
...         print subarr, subarr.shape
...
[[[[0 1]]]] (1, 1, 1, 2)