[s3boto] S3ResponseError: RequestTimeout - saving files ec2 to s3

Issue #85 new
Luke Opperman
created an issue

S3ResponseError: 400 Bad Request <?xml version="1.0" encoding="UTF-8"?> <Error><Code>RequestTimeout</Code><Message>Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.</Message><RequestId>...</RequestId><HostId>...</HostId></Error>

Truncated Traceback:

{{{ File ".../lib/python2.6/site-packages/django/db/models/fields/files.py" in pre_save 255. file.save(file.name, file, save=False) File ".../lib/python2.6/site-packages/django/db/models/fields/files.py" in save 92. self.name = self.storage.save(name, content) File ".../lib/python2.6/site-packages/django/core/files/storage.py" in save 48. name = self._save(name, content) File ".../lib/python2.6/site-packages/storages/backends/s3boto.py" in _save 197. reduced_redundancy=self.reduced_redundancy) File ".../src/boto/boto/s3/key.py" in set_contents_from_file 625. self.send_file(fp, headers, cb, num_cb, query_args) File ".../src/boto/boto/s3/key.py" in send_file 517. query_args=query_args) File ".../src/boto/boto/s3/connection.py" in make_request 400. override_num_retries=override_num_retries) File ".../src/boto/boto/connection.py" in make_request 534. override_num_retries) File ".../src/boto/boto/connection.py" in _send_http_request 525. http_request.host, sender, override_num_retries) File ".../src/boto/boto/connection.py" in _mexe 436. response = sender(connection, method, path, data, headers) File ".../src/boto/boto/s3/key.py" in sender 490. response.status, response.reason, body) }}}

Versions: Django: 1.2.5 django-storages: https://bitbucket.org/david/django-storages/changeset/47673439a27f * boto: https://github.com/hmarr/boto/commits/ses

This is on EC2 talking to S3, so maybe related to <<issue 20>>, perhaps a boto issue not django-storages.

This happens regularly but not consistently for us, both on initial upload (browser->ec2->s3) and also using django-imagekit to generate cached thumbnails (so just ec2->s3). Files are typically 50k-500k. Typically will go through after a few manual retries by the user.

Comments (5)

  1. Steve Jalim

    I had something similar to this issue a few years ago when first using django-storages. My solution was to set a longer querystring expiry parameter (eg 600, ie, 10 mins) rather than the default 60).


    I'm not sure this is the best way to do it, but it worked for me

  2. Ian Lewis

    Are you using querystring auth? Does the suggestion that Steve Jalim mentioned work for you?

    I'm afraid something like this would best be solved by sub-classing the S3BotoStorage class and supporting retries to a level sufficient for your application. Or supporting retries somehow within your application code.

  3. Log in to comment