[s3boto] Seperate buckets for Static and Media Content

Issue #93 new
Former user created an issue

First, I do not see any option to set these to two separate buckets. The problem lies in the fact that I am unable to set "MEDIA_URL" and "STATIC_URL" to the same location. Because of this, when I use a third party application that handles user uploaded media (for example, User Avatars in django-userena) then my Media files are pointed towards S3 directly as opposed to using CloudFront like we do for the static files.

I'm sure in a custom application I can easily get around this problem but I'm not sure how to go about doing it with third party applications aside from modifying their code directly.

Comments (16)

  1. Former user Account Deleted

    I'm interested in the same thing. I don't necessarily need them to go to separate buckets, but being able to have my static files go into "my_bucket/static/*" and media files into "my_bucket/media/*" would be fine. Then I can have one bucket (and an optional CloudFront cdn linked to that bucket) and my MEDIA_URL and STATIC_URL would look like "http://cdn.mywebsite.com/media" and "http://cdn.mywebsite.com/static" (or just http://my_bucket.s3.amazonaws.com/[media | static]). Maybe if I subclass the django-storages s3boto backend and make specific settings for static and media subfolders inside a bucket (or separate buckets even), then that would work just fine. What do the developers think is the best solution? Others I'm sure would benefit from this advice or an elegant solution.

  2. Ian Lewis

    This is kind of a limitation in Django and staticfiles. I had created a fork of django-staticfiles previously that addresses this issue but it was rejected. I'm not sure we should be trying to solve a problem in Django (media) and staticfiles in django-storages.

  3. Phillip Marshall

    I'm in the middle of moving a semibig project to heroku/s3, so I'm also interested in this (either having 2 buckets or one bucket with 2 folders is fine). It's a frustrating problem because django demands that we seperate static from media and then only has the limitation of only allowing one storage. I'm just thinking out loud here, but what if we break that first part and put media inside static? I know it's not clean, but it could work until django core decides we can have more than one storage without forking anything.

    + css/
    - favicon.ico
    + images/
    + javascript/
    + media/ #not in local static folder 
      + avatars/
      + movies/
      - etc.txt

    Then `manage.py collectstatic` could dump the contents of our local STATICFILES_DIRS = ("/wherever-dev/static",) into the storage root like it so demands, and we can just specify the media folder when saving user created media from within our views since they are more flexible. As long as they stay out of each-other's way, it should be fine, yes? Correct me if i'm being extremely naive. This probably would not solve the issue for someone demanding they be in separate buckets for third party apps of course.

  4. christianverkerk+bitbucket

    I'm interested in this same thing, I was kind of assuming that your MEDIA_URL could have a /media/ appended to it and your STATIC_URL /static/. Does django-storages look at the MEDIA_URL and STATIC_URL at all? Or is it purely based on the keys? I'm assuming the normal way to circumvent this shortcoming is to create two buckets mysite-static and mysite-media?

  5. Chee-Hyung Yoon

    Although the suggestion by wizpig64 can work, it would be nicer to have support for separate static and media folders. What's the timeline for the implementation? Should I go ahead to create a patch myself?

  6. Ian Lewis

    No, we probably wont and never will have support for separate static and media folders. The django storage API is made such that you should subclass the S3 backend to supply the settings you need for media and use that class.

    Creating support for separate static and media folders requires ugly hacks and have thus far not been accepted for that the reasons I've mentioned in many many issues and pull requests. Django's DEFAULT_STORAGE just doesn't support this in any meaningful way that would allow us to support it without ugly hacks. Please don't write a patch. It will likely not get accepted.

    I talked with jezdez at PyCon about this and other issues and I think the solution is not to support it directly but to re-engineer the s3boto API and docs so that it's easier and clearer how to do this. Until that gets done I don't think we will be able to support this on a level that satisfies everyone.

  7. Chris Jones

    I ran into this problem as well. I don't like the idea of static media and uploaded media being smashed into the same bucket. My solution to this was quite simple. In my "generic" app I created a copy of the S3BotoStorage backend and added a single setting to my Django settings. `AWS_STATIC_STORAGE_BUCKET_NAME` holds the name of my static files bucket name.

    DEFAULT_FILE_STORAGE is set to the standard S3BotoStorage backend with all standard settings. STATICFILES_STORAGE is set to my copy of the S3 backend with one change, the STORAGE_BUCKET_NAME gets the AWS_STATIC_STORAGE_BUCKET_NAME setting.

    This isn't ideal but was quick and simple to implement. This should honestly be relatively simple to implement. Basically subclass the S3BotoStorage backend, call it S3BotoStaticStorage and the only override is another setting for the name of the static bucket.

    If I can get some time I'll try to see if I can submit a patch. I've been using this in production but where it would fall apart on me is when we update django-storages and I forget to update my copy of the backend if anything critical changes.

  8. Ian Lewis

    The solution is to subclass the boto backend and override the parts you need and use that backend for static storage. Can't say that enough here.

  9. Richard Leland

    This is a good candidate for adding to the docs. I've had to do the subclassing a few times and can throw a good example in. Will modify S3 docs.

  10. Benjamin Stookey

    I thought it was also a good candidate for a package. FWIW I put https://github.com/jamstooks/django-s3-folder-storage together so I could use a similar setup in a number of projects. It's pretty easy. It's just a few changes to my settings:

    DEFAULT_FILE_STORAGE = 's3_folder_storage.s3.DefaultStorage'
    DEFAULT_S3_PATH = "media"
    STATICFILES_STORAGE = 's3_folder_storage.s3.StaticStorage'
    STATIC_S3_PATH = "static"

    I hope others find it useful.

  11. Log in to comment