Commits

David Larlet committed af5d6b4

Add support for MogileFS storage, thanks Andrew McClain.

See http://blog.fluther.com/blog/2008/09/04/mogilefs-for-django/ for details.

Comments (0)

Files changed (2)

MogileFSStorage.py

+import urlparse
+from StringIO import StringIO
+from mimetypes import guess_type
+
+from django.core.files.storage import Storage
+from django.core.exceptions import ImproperlyConfigured
+from django.conf import settings
+from django.utils.text import force_unicode
+from django.http import HttpResponse, HttpResponseNotFound
+from django.core.cache import cache
+
+import mogilefs
+
+
+class MogileFSStorage(Storage):
+    """MogileFS filesystem storage"""
+    def __init__(self, base_url=settings.MEDIA_URL):
+        
+        # the MOGILEFS_MEDIA_URL overrides MEDIA_URL
+        if hasattr(settings, 'MOGILEFS_MEDIA_URL'):
+            self.base_url = settings.MOGILEFS_MEDIA_URL
+        else:
+            self.base_url = base_url
+                
+        for var in ('MOGILEFS_TRACKERS', 'MOGILEFS_DOMAIN',):
+            if not hasattr(settings, var):
+                raise ImproperlyConfigured, "You must define %s to use the MogileFS backend." % var
+            
+        self.trackers = settings.MOGILEFS_TRACKERS
+        self.domain = settings.MOGILEFS_DOMAIN
+        self.client = mogilefs.Client(self.domain, self.trackers)
+    
+    def get_mogile_paths(self, filename):
+        return self.client.get_paths(filename)  
+    
+    # The following methods define the Backend API
+
+    def filesize(self, filename):
+        raise NotImplemented
+        #return os.path.getsize(self._get_absolute_path(filename))
+    
+    def path(self, filename):
+        paths = self.get_mogile_paths(filename)
+        if paths:
+            return self.get_mogile_paths(filename)[0]
+        else:
+            return None
+    
+    def url(self, filename):
+        return urlparse.urljoin(self.base_url, filename).replace('\\', '/')
+
+    def open(self, filename, mode='rb'):
+        raise NotImplemented
+        #return open(self._get_absolute_path(filename), mode)
+
+    def exists(self, filename):
+        return filename in self.client
+
+    def save(self, filename, raw_contents):
+        filename = self.get_available_filename(filename)
+        
+        if not hasattr(self, 'mogile_class'):
+            self.mogile_class = None
+
+        # Write the file to mogile
+        success = self.client.send_file(filename, StringIO(raw_contents), self.mogile_class)
+        if success:
+            print "Wrote file to key %s, %s@%s" % (filename, self.domain, self.trackers[0])
+        else:
+            print "FAILURE writing file %s" % (filename)
+
+        return force_unicode(filename.replace('\\', '/'))
+
+    def delete(self, filename):
+        
+        self.client.delete(filename)
+            
+        
+def serve_mogilefs_file(request, key=None):
+    """
+    Called when a user requests an image.
+    Either reproxy the path to perlbal, or serve the image outright
+    """
+    # not the best way to do this, since we create a client each time
+    mimetype = guess_type(key)[0] or "application/x-octet-stream"
+    client = mogilefs.Client(settings.MOGILEFS_DOMAIN, settings.MOGILEFS_TRACKERS)
+    if hasattr(settings, "SERVE_WITH_PERLBAL") and settings.SERVE_WITH_PERLBAL:
+        # we're reproxying with perlbal
+        
+        # check the path cache
+        
+        path = cache.get(key)
+
+        if not path:
+            path = client.get_paths(key)
+            cache.set(key, path, 60)
+    
+        if path:
+            response = HttpResponse(content_type=mimetype)
+            response['X-REPROXY-URL'] = path[0]
+        else:
+            response = HttpResponseNotFound()
+    
+    else:
+        # we don't have perlbal, let's just serve the image via django
+        file_data = client[key]
+        if file_data:
+            response = HttpResponse(file_data, mimetype=mimetype)
+        else:
+            response = HttpResponseNotFound()
+    
+    return response

django-mogilefs-storage.txt

+================
+MogileFS Storage
+================
+
+The MogileFS storage backend is fairly simple: it uses URLs (or, rather, 
+parts of URLs) as keys into the mogile database. When the user requests a file 
+stored by mogile (say, an avatar), the URL gets passed to a view which, using 
+a client to the mogile tracker, retrieves the "correct" path (the path that 
+points to the actual file data). The view will then either return the path(s) 
+to perlbal to reproxy, or, if you’re not using perlbal to reproxy 
+(which you should), it serves the data of the file directly from django.
+
+In order for the backend to work, we need to add a few settings variables:
+
+    * ``MOGILEFS_DOMAIN``: The mogile domain that files should read 
+      from/written to, e.g “production”
+    * ``MOGILEFS_TRACKERS``: A list of trackers to connect to, 
+      e.g. ["foo.sample.com:7001", "bar.sample.com:7001"]
+    * ``MOGILEFS_MEDIA_URL`` (optional): The prefix for URLs that point to 
+      mogile files. This is used in a similar way to ``MEDIA_URL``, 
+      e.g. "/mogilefs/"
+    * ``SERVE_WITH_PERLBAL``: Boolean that, when True, will pass the paths 
+      back in the response in the ``X-REPROXY-URL`` header. If False, django 
+      will serve all mogile media files itself (bad idea for production, 
+      but useful if you’re testing on a setup that doesn’t have perlbal 
+      running)
+    * ``DEFAULT_FILE_STORAGE``: This is the class that’s used for the backend.
+      You’ll want to set this to ``project.app.storages.MogileFSStorage``
+      (or wherever you’ve installed the backend) 
+
+ 
+
+Getting files into mogile
+-------------------------
+
+The great thing about file backends is that we just need to specify the 
+backend in the model file and everything is taken care for us — all the 
+default save() methods work correctly.
+
+For Fluther, we have two main media types we use mogile for: avatars and 
+thumbnails. Mogile defines “classes” that dictate how each type of file is 
+replicated — so you can make sure you have 3 copies of the original avatar 
+but only 1 of the thumbnail.
+
+In order for classes to behave nicely with the backend framework, we’ve had to 
+do a little tomfoolery. (This is something that may change in future versions 
+of the filestorage framework).
+
+Here’s what the models.py file looks like for the avatars::
+
+    from django.core.filestorage import storage
+    
+    # TODO: Find a better way to deal with classes. Maybe a generator?
+    class AvatarStorage(storage.__class__):
+        mogile_class = ‘avatar’ 
+    
+    class ThumbnailStorage(storage.__class__):
+        mogile_class = ‘thumb’
+    
+    class Avatar(models.Model):
+        user = models.ForeignKey(User, null=True, blank=True)
+        image = models.ImageField(storage=AvatarStorage())
+        thumb = models.ImageField(storage=ThumbnailStorage())
+
+Each of the custom storage classes defines a ``class`` attribute which gets 
+passed to the mogile backend behind the scenes.  If you don’t want to worry 
+about mogile classes, don’t need to define a custom storage engine or specify 
+it in the field — the default should work just fine.
+
+Serving files from mogile
+-------------------------
+
+Now, all we need to do is plug in the view that serves up mogile data. 
+
+Here’s what we use::
+
+  urlpatterns += patterns(”,
+      (r’^%s(?P<key>.*)’ % settings.MOGILEFS_MEDIA_URL[1:], 
+          'MogileFSStorage.serve_mogilefs_file')
+  )
+
+Any url beginning with the value of ``MOGILEFS_MEDIA_URL`` will get passed to 
+our view. Since ``MOGILEFS_MEDIA_URL`` requires a leading slash (like 
+``MEDIA_URL``), we strip that off and pass the rest of the url over to the 
+view.
+
+That’s it! Happy mogiling!
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.