[s3boto] use multipart uploads to stream writes

Issue #110 resolved
Gordon NA created an issue

With the current implementation, a write requires an entire file to fit in memory because S3BotoFile uses StringIO. Multipart uploads allow for writing large files to s3 without loading the entire file locally.

I have a basic implementation here (up to revision 04710871bf02): https://bitbucket.org/pendletongp/django-storages-s3boto/

It can be improved to retry failed uploads. Another improvement would be to parallelize the part uploads.

Comments (1)

  1. Log in to comment