1. Brendan McAdams
  2. lighttpd-gridfs

Source

lighttpd-gridfs /

Filename Size Date modified Message
6.3 KB
- Mostly "finished".
5.4 KB
Initial checkin, not the complete backend code yet but want to get a repository created.
264 B
- Tracking down final elusive bug (crashy type)
3.5 KB
- Seems that the contents of the gridfile were acting weird because the string went out of scope. Now strdup() wrapping the creation of it's c_str() in the gridfile_t return struct. Which fixes the issue. Needs to be freed of course when done.
1.1 KB
- Mostly "finished".
26.6 KB
- Tracking down final elusive bug (crashy type)
5.2 KB
- Seems that the contents of the gridfile were acting weird because the string went out of scope. Now strdup() wrapping the creation of it's c_str() in the gridfile_t return struct. Which fixes the issue. Needs to be freed of course when done.
-*- markdown -*-

# lighttpd-gridfs
*Authors*:

* Brendan W. McAdams <bwmcadams@evilmonkeylabs.com> (Lighttpd port)  
* Mike Dirolf <mike@dirolf.com> (Original author of the [original nginx-gridfs module](http://github.com/mdirolf/nginx-gridfs/), gridfs logic/helpers)  

## About

**lighttpd-gridfs** is a [Lighttpd](http://lighttpd.net) module to serve content
directly from [MongoDB](http://www.mongodb.org)'s [GridFS](http://www.mongodb.org/display/DOCS/GridFS).  
This module is based upon [code Mike Dirolf wrote](http://github.com/mdirolf/nginx-gridfs) to provide this functionality
for the Nginx webserver.  In porting to Lighttpd I followed the codebases of Lighttpd's mod_fastcgi/mod_scgi/mod_proxy modules, allowing
for multiple configured paths & extensions. It should also make for an easy addition of load balanced multiple gridfs backends in the near future.


## Dependencies

**lighttpd-gridfs** requires the MongoDB C++ client library (which is installed by default when MongoDB is installed).  It also needs to link against the [Boost libraries](http://www.boost.org) (specifically libboost_thread and libboost_filesystem, and for older boost versions libboost_system as well), as the MongoDB Client depends upon them.  

Due to the fact that I'm subverting Lighttpd's optional scons based build system rather than Make/Autoconf, you'll also need the [Scons Package](http://www.scons.org) installed.

Finally, (It will eventually be optional) this module now requires the development package for libmagic, to determine mime types.  On Debian/Ubuntu it is "libmagic-dev" (e.g. `sudo apt-get install libmagic-dev`) ... on RHEL/CentOS/Fedora it's likely "libmagic-devel".  Libmagic itself is typically installed on unixy systems.

## Installation
Installing Lighttpd modules requires rebuilding the entire Lighttpd tree from source.  I have done my development against Lighttpd 1.4.19.  Rather than messing with autoconf and make files, I currently only utilize Lighttpd's [Scons](http://www.scons.org) build system.  This code should work on newer versions of Lighttpd as well, you just may need to manually adjust the build file patches.

* Grab the [Lighttpd source](http://lighttpd.net).  On Debian/Ubuntu you can run `sudo apt-get source lighttpd` for the version appropriate to your sytem. I  have not dealt with patching the dpkg build scripts yet to produce a new package. 
* Clone the [lighttpd-gridfs repository](http://bitbucket.org/bwmcadams/lighttpd-gridfs/) somewhere on your system, or download a source package from bitbucket.
* Change to the directory containing the lighttpd source.
* Patch the build system to support the new module:

        % patch -p1 < $PATH_TO_LIGHTTPD_GRIDFS/build-lighttpd-1_4_19.patch 
        % cp $PATH_TO_LIGHTTPD_GRIDFS/build_gridfs_config.py .

* You may wish to adjust the directories in build_gridfs_config.py, which sets the search path for MongoDB's libraries and includes, for your system. 
* Copy the mod_gridfs files into $LIGHTTPD_SRC/src:
    
        % cp $LIGHTTPD_GRIDFS_SRC/mod_gridfs.c $LIGHTTPD_GRIDFS_SRC/gridfs_c_helpers.cpp $LIGHTTPD_GRIDFS_SRC/gridfs_c_helpers.h $LIGHTTPD_SRC/src 

* Build Lighttpd, enabling the gridfs module:
    
        % scons with_gridfs=yes

* If all goes well, you now have a built lighttpd with the module.  I believe, as long as versions match up, you should be able to simply deploy build/mod_gridfs.so in  your existing Lighttpd deployment and it'll work.  YMMV... 

## Configuration

The entries in lighttpd.conf (or, as I prefer, a separate included conf / debian & ubuntu conf-enabled/available system) for gridfs should look very similar to a fastcgi/scgi/proxy config.  You can map on an extension or path.  Extension maps aren't yet supported.  The basic testing setup I use is:

    server.modules +=  ( 
      "mod_gridfs"
    )
         
    gridfs.server = (
      "/gridfs/" => ((
        "mongod_host" => "127.0.0.1:27017",
        "gridfs_db"   => "http_gridfs",
        "gridfs_root_collection" => "fs",
      ))
    )


This will pass all requests to `/gridfs/` through to the gridfs plugin, using the mongo server at `127.0.0.1:27017`, looking in the Mongo database `http_gridfs` and using the collection `fs` as the gridfs root (e.g. `fs.chunks` and `fs.files`).  **NOTE: Config settings are in flight, and I am a fickle coder.  They are still subject to change (names of variables, but not strategy [e.g. matching fcgi/scgi config])**  

It may be buggy at the moment but the system SHOULD set defaults on all 3 internal variables  requiring you to just define `"/path/" => (())` ... The example above represents the actual default values.  If you have issues with the defaults please file a bug.

### Known issues / TODO / Things You Should Hack On

(*This list is taken originally from Mike Dirolf's Nginx plugin, I've marked the items from his list I've fixed/think I fixed and added some of my own*)

* Some issues with large files (_bwm_:not confirmed this is issue on Lighttpd like it is on Nginx as they're diff architectures, but assume it's still broken and submit a useful patch to correct it...)
* Need to use Mimetypes in GridFS File (or maybe fall back to guessing by extension?) (_bwm_: Linux has mime-magic, and there should be a c call to determine mimetype based on magic number)
* HTTP Range support for partial downloads (There are some existing lighttpd modules that do this that should be usable as examples
* URL Decode filenames (_bwm_: Lighttpd provides calls to url decode and simplify (remove /../'s) requested paths which I invoke at request-path-parse-time, and *SHOULD* solve this issue.)
* Better error handling / loggign (_bwm_: This is still mostly an issue on lighttpd as well, although the more complicated Lighttpd plugin structure also lends itself to more automatic error handling/recovery.  Need logging, including support for the gridfs.debug boolean I coded into the config parser but am not using yet )
* Better config / build process (_bwm_: I think I sort of fixed this in the Lighttpd port, but it still requires patching the existing Sconstruct files and doesnt' provide make support.  Also need to add support to spit out rpms/deb packages...)
* Support for getting files by _id (in case there are duplicate filenames) 

### Credits

* Sho Fukamachi (sho) - towards compatibility with newer boost versions

### License

**lighttpd-gridfs**, is licensed under the Apache License, Version 2.0.  See *LICENSE* for details.