[s3boto] admin prefix issue with Django 1.4

Issue #121 new
Albert O'Connor created an issue

In the admin the admin_media_prefix ends up with the following value:

{{{ <script type="text/javascript"> window.admin_media_prefix = "https://BUCKETNAME.s3.amazonaws.com/admin"; </script> }}}

But it should be "https://BUCKETNAME.s3.amazonaws.com/admin/" for javascript which loads resources to function.

The code in the Django template for generating the prefix is

{{{ <script type="text/javascript">window.admin_media_prefix = "{% filter escapejs %}{% static "admin/" %}{% endfilter %}";</script> }}}

In Django 1.4 the static block is provided by the current storage backend, in my case it calls url on the S3BotoStorage class, which in turns _clean_name(name) with "admin/" as the value of name.

{{{ name = self._normalize_name(self._clean_name(name)) }}}

Which takes us to http://code.larlet.fr/django-storages/src/83fa2f0ba20c/storages/backends/s3boto.py#cl-225 where os.path.normpath is called name. Name has the value "admin/" but normpath returns "admin".

The javascript to render images for the calendar picker relies on the trailing slash on the prefix, without you get:

{{{ <img src="https://BUCKETNAME.s3.amazonaws.com/adminimg/icon_calendar.gif" alt="Calendar"> }}}

Which results in a broken image.

One solution would be to not use normpath, but that would result in other types of strange paths not being normalized. Some types of issues would resolved by other code in the backend, so the risks of doing this might be minor.

I am going to likely fork it and apply the change or removing normpath, but I am not sure if that is really the best answer.

Comments (31)

  1. Former user Account Deleted

    Noticing this as well, but the absolute strangest thing is that locally it works just fine (even when using S3 for static files), but it fails when deployed (to Heroku in this case). Madness...

  2. Wilerson de Oliveira

    I have noticed this issue as well, and the behavior is even worse if AWS_QUERYSTRING_AUTH is True (which is the default), as not only the trailing slash is removed, the querystring AWS parameters are added just between "admin" and "img/icon_calendar.gif", e.g.:

    <img src="https://BUCKETNAME.s3.amazonaws.com/admin?Signature=SIGNATURE&Expires=1340738491&AWSAccessKeyId=ACCESSKEYimg/icon_calendar.gif" alt="calendar"/>

    I'm not sure if this is really a bug with django-storages or if django.contrib.admin is misusing the static block by storing the prefix to use in the js files. Either way, this is a nasty behavior for people that rely heavily on the admin.

  3. codysoyland

    Here is my solution as a workaround until this gets fixed.

    I created a subclass of S3BotoStorage and added this method:

        def url(self, name):
            url = super(CustomS3BotoStorage, self).url(name)
            if name.endswith('/') and not url.endswith('/'):
                url += '/'
            return url

    After changing my storage engine setting to use my subclass, it works normally.

  4. Bob Spryn

    Is the maintainer maintaining this anymore? There are a bunch of bugs and pulls that aren't receiving any attention.

  5. unbracketed

    I've tried the workaround proposed by @codysoyland but it doesn't work for me. I've replaced every instance of S3BotoStorage in my application with my custom class. For some reason, the icon_calendar is never processed by the storage class.

    It appears that DateTimeShortcuts.js writes the image tag out uses the value of DateTimeShortcuts.admin_media_prefix (also the same as window.admin_media_prefix) which is set to:


    I'm not sure django-storages is the culprit.

  6. ThomasPurchas

    "window.admin_media_prefix" is set in the "base.html" file. If you look inside you will find the line

        <script type="text/javascript">window.__admin_media_prefix__ = "{% filter escapejs %}{% static "admin/" %}{% endfilter %}";</script>

    which sets "window.admin_media_prefix" to whatever boto returns as the url resolution to "admin/" which should return a url with a trailing slash when "AWS_QUERYSTRING_AUTH" is false, but currently doesn't.

    Additionally if you bucket is named so that you can CNAME your own domain address to it (meaning that your bucket is named "sub.domain.co.uk"), then trying to access the subdomain version of it over "https://" results in a certificate error in most browsers as the S3 ssl certificate only covers a single subdomain below "s3.amazonaws.com".

  7. Arun Ravindran


    I am facing the exact same issue and the workaround doesn't work for me as well. I can confirm that the bug appears on Chrome and Firefox. This appears to be a problem if the static assets are deployed to Amazon S3.

    Any ideas on how to fix this?

  8. Jonatan Heyman

    I had the same issue with the S3BotoStorage generating URLs in the format "http(s):bucket.name.s3.amazonaws.com/.../" which caused SSL certificate problems when buckets had dots in their name.

    I've subclassed the S3BotoStorage and re-implemented the url function to solve it:

    class FixedS3BotoStorage(S3BotoStorage):
        def url(self, name):
            Change URL's generated in the following format: http(s)://bucket.name.s3.amazonaws.com/.../
            Into the format: http(s)://s3.amazonaws.com/bucket.name/.../
            url = super(FixedS3BotoStorage, self).url(name)
            bucket_name = self.bucket.name
            if "%s.s3.amazonaws.com" % bucket_name in url:
                url = url.replace("%s.s3.amazonaws.com/" % bucket_name, "s3.amazonaws.com/%s/" % bucket_name)
            return url
  9. Richard Leland

    OK so I see a few issues in this thread. I'm going to try to get them all down here.

    1. Certain assets are missing a trailing slash on admin in the URL when viewing the Django admin
    2. If AWS_QUERYSTRING_AUTH is True (default), it adds query params, which improperly get sandwiched between admin and img/icon_calendar.gif for example
    3. Subdomains are not being properly generated

    If you see anything else, let me know. I'm going to attempt to tackle the above.

  10. Richard Leland

    In regards to the 3rd item - it's not that the subdomains aren't being properly generated, it's that there are complexities around S3 SSL, bucket naming, etc. May be able to solve this with documentation.

  11. Richard Leland

    In regards to the 2nd item, I need to find out if that's an underlying boto issue or just happening because of the trailing slash issue. Setting up test...

  12. Richard Leland

    OK I did some research on this today and from what I can tell every one of these issues can be resolved by understanding boto's calling formats and the limitations of S3 buckets with regards to SSL.

    I'll spend some time improving the docs, outlining different scenarios and appropriate settings for each. I'll also add docs on using a subclass for separate buckets for static and media.

  13. John Brewer

    Can you be specific about how to solve #2 in your list? It's a pretty basic need for the Django admin as it keeps that icon_calendar.gif from being referenced properly.

  14. ├ćndrew Rininsland

    Experiencing both issues 1 and 2 in this thread, using Django 1.6.1 and django-storages 1.1.8. I can get by without querystring auth, but the broken images in Django admin are a bit annoying. I'll PR my changes if I fix it myself.

  15. Zhenxi Yang

    @aendrew , in case you are still working on this. I think I got a fix to work around the issue 2.

    You need to set AWS_QUERYSTRING_AUTH = False in your setting.py.

    After reading the whole thread, I think the problem is in django admin app, https://github.com/django/django/blob/master/django/contrib/admin/templates/admin/base.html#L9 It generate admin prefix url by {% static 'admin/' %}, which will be handled by django-storages storages.backends.s3boto.S3BotoStorage.url() function.

    If you have AWS_QUERYSTRING_AUTH=true, the url() function will add additional parameter in the end, which accounts for those auth parameters. That breaks the url after combines the prefix with the actual image location.

  16. Jonathan Wiklund

    I've got a variant of Querystring as well. Although it is not squished between admin and asset, but in the asset filepath itself..

  17. Damon Timm

    Was having the same problem (at the OP) in that a trailing slash was missing from my Admin media URLs. Used the recommended fix from the comments along with this gem to separate static files and media uploads. Here is what I ended up with finally:

    from storages.backends.s3boto import S3BotoStorage
    class FixedS3BotoStorage(S3BotoStorage):
        def url(self, name):
            url = super(FixedS3BotoStorage, self).url(name)
            if name.endswith('/') and not url.endswith('/'):
                url += '/'
            return url
    StaticRootS3BotoStorage = lambda: FixedS3BotoStorage(location='static')
    MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media')
  18. Radek Wojcik

    Same problem on django 1.6.8 and django-storages 1.1.18. This is my fix:

    from urlparse import urljoin
    from django.conf import settings
    from storages.backends.s3boto import S3BotoStorage
    # Overridden location property allows us to keep media and static files
    # in separate subfolders on S3
    class StaticS3Storage(S3BotoStorage):
        location = 'static'
        def modified_time(self, path):
            # XXX
            # Force collectstatic to not check times
            # We were having issues with files not collecting due to timestamps
            # being weird (the backend was thinking the local files were out of date
            # compared to the stale ones on S3)
            raise NotImplementedError
        def url(self, name):
            # Fix for django error abusing {% static %}
            url = super(StaticS3Storage, self).url(name)
            if name.endswith('/') and not url.endswith('/'):
                url += '/'
            return url
  19. Log in to comment