Issue #276 resolved

Put too large exception on indexing

Anonymous created an issue

Hi, i'm using Whoosh on top of GAE with large amount of data and I am getting "The request to API call datastore_v3.Put() was too large." exception after sometime in the indexing process.The process is divided into separate GAE's tasks. I'm also getting "Values may not be more than 1000000 bytes in length; received 1038568 bytes" sometimes.

What do you think is happening? Am i'm doing something wrong or it's a GAE limitation?

Thanks

Comments (2)

  1. Matt Chaput repo owner

    The current GAE storage implementation is more-or-less a toy contribution from a user long ago that "fakes" files using Datastore and MemCache and has a 1MB limit. There are currently efforts to replace it to at least use the Blobstore API to remove the 1MB limit, but I need to find someone with knowledge of GAE/Blobstore to help. There is currently a thread on the mailing list if you're interested in helping.

  2. Log in to comment