storage.exists() + S3Boto, Storages is making a ton of requests and downloading large amounts of XML
When using storage.exists() + S3Boto, I ran into a big issue with easy-thumbnails. When saving several different thumbnails for just one file, the memory for my python process would grow to 2.75 GB. I tracked this down to storage.exists() making tons of requests and downloading large amounts of data. This is because I'm using a bucket on S3 with over 300K objects.
I submitted a patch to easy-thumbnails here, https://github.com/epicserve/easy-thumbnails/commit/e389235ca53462e87d335ace0cca8558f52a8a03.
Is there a way to optimize Storages, so storage.exists() would work faster and not use large amounts of data?