1. LAIKA Open Source
  2. thingfish

Commits

Michael Granger  committed ce17220

Add initial implementation of processors

  • Participants
  • Parent commits 733677a
  • Branches default

Comments (0)

Files changed (16)

File .hgignore

View file
 doc/
 ^ChangeLog$
 integration/
+logs/

File Processors.rdoc

View file
+= ThingFish Processors
+
+== Processor Objects
+
+Processors are programs that modify or enhance uploaded assets in one or more of three ways:
+
+1. Modify uploaded assets synchronously before they're saved
+2. Modify assets synchronously on the fly when they're downloaded
+3. Modify uploaded assets asynchronously after they're saved
+
+
+The basic Processor interface is:
+
+  class Thingfish::Processor
+  
+    def process_request( request )
+      # No-op by default
+    end
+  
+    def process_response( response )
+      # No-op by default
+    end
+  
+    def process( io )
+      # No-op by default
+    end
+  
+  end
+
+
+The first two cases run inside the Thingfish daemon, and the third runs inside an asynchronous
+processor daemon. Thingfish writes the UUID of every newly-uploaded asset to a PUB zeromq endpoint,
+so asynchronous processing systems need only SUBscribe to that endpoint, and dispatch processing
+jobs to one or more daemons when a new uuid is read.
+
+
+=== Request (Synchronous) Processor
+
+An example of a synchronous upload processor adds metadata extracted from the ID3 tags of uploaded
+MP3 files:
+
+  class Thingfish::ID3Processor < Thingfish::Processor
+  
+    def handle_request( request )
+      return unless request.content_type == 'audio/mp3'
+  
+      mp3 = request.body.read
+      metadata = extract_some_id3_shit( mp3 )
+      request.add_metadata( metadata )
+    end
+  
+  end
+
+
+=== Request/Response (Synchronous) Processor
+
+An example of a processor that adds two watermarks to images:
+
+* when an image is uploaded, it adds an invisible watermark to the image data, which then becomes a
+  permanent part of the asset.
+* when an image is downloaded, it adds a visible watermark to the image if it's being downloaded
+  from an external site.
+
+  class Thingfish::WaterMarker < Thingfish::Processor
+  
+    def handle_request( request )
+      return unless request.content_type =~ /^image/
+  
+      image = request.body.read
+      watermarked_image = add_invisible_watermark( image )
+      request.body.rewind
+      request.body.write( watermarked_image )
+    end
+  
+    def handle_response( response )
+      return unless response.content_type =~ /^image/ &&
+        !from_internal_network( response )
+  
+      image = response.body.read
+      watermarked_image = add_visible_watermark( image )
+      response.body.rewind
+      response.body.write( watermarked_image )
+    end
+  
+  end
+
+
+=== Asynchronous-only Processor
+
+This is an example of a processor that adds a thumbnail to videos that have been uploaded
+based on the video's keyframes. It doesn't touch uploads or downloads, but since it's 
+time-consuming, it can be run asynchronously at any point after the upload.
+
+  class Thingfish::VideoProcessor < Thingfish::Processor
+  
+    def process_async( asset )
+      asset.metadata[:thumbnail] = extract_keyframe( asset.data )
+    end
+  
+  end
+
+
+

File bin/tfprocessord

View file
+#!/usr/bin/env ruby
+
+require 'thingfish/processordaemon'
+
+Thingfish::ProcessorDaemon.run( ARGV )
+

File bin/thingfishd

View file
 #!/usr/bin/env ruby
 #encoding: utf-8
 
-require 'thingfish'
+require 'strelka'
+require 'thingfish/handler'
 
 configpath = ARGV.shift || 'etc/thingfish.conf'
 
 Strelka.load_config( configpath )
-Thingfish.run
+Thingfish::Handler.run

File example/m2-config.rb

View file
 	name         'Thingfish Examples'
 	default_host 'localhost'
 
-	access_log   '/logs/access.log'
-	error_log    '/logs/error.log'
-	chroot       '.'
-	pid_file     '/run/mongrel2.pid'
+	access_log   'logs/access.log'
+	error_log    'logs/error.log'
+	chroot       ''
+	pid_file     'run/mongrel2.pid'
 
 	bind_addr    '0.0.0.0'
 	port         3474
 setting "zeromq.threads", 1
 
 setting 'limits.content_length', 100 * 1024 * 1024
+setting 'server.daemonize', false
 setting 'upload.temp_store', 'var/uploads/mongrel2.upload.XXXXXX'
 
 mkdir_p 'var/uploads'

File example/mongrel2.sqlite

Binary file modified.

File example/thingfish.conf

View file
 ---
 logging:
-  __default__: warn STDERR
+  __default__: debug STDERR
 mongrel2:
-  configdb: mongrel2.sqlite
+  configdb: example/mongrel2.sqlite
 app:
   devmode: false
   app_glob_pattern: '{apps,handlers}/**/*'

File experiments/event_grabber.rb

View file
+#!/usr/bin/env ruby
+#
+
+require 'zmq'
+
+ctx = ZMQ::Context.new
+sock = ctx.socket( ZMQ::SUB )
+sock.setsockopt( ZMQ::SUBSCRIBE, 'create' )
+sock.connect( 'tcp://127.0.0.1:3475' )
+
+while msg = sock.recv
+	puts msg
+end
+

File lib/thingfish.rb

View file
 #!/usr/bin/env ruby
 
-require 'strelka'
-require 'strelka/app'
+require 'loggability'
 
 #
 # Network-accessable datastore service
 # * Michael Granger <ged@FaerieMUD.org>
 # * Mahlon E. Smith <mahlon@martini.nu>
 #
-class Thingfish < Strelka::App
+module Thingfish
+	extend Loggability
+
 
 	# Loggability API -- log all Thingfish-related stuff to a separate logger
 	log_as :thingfish
 	# Version control revision
 	REVISION = %q$Revision$
 
-	# Configurability API -- set config defaults
-	CONFIG_DEFAULTS = {
-		datastore: 'memory',
-		metastore: 'memory'
-	}
-
-	# Metadata keys which aren't directly modifiable via the REST API
-	OPERATIONAL_METADATA_KEYS = %w[
-		format
-		extent
-		created
-		uploadaddress
-	]
-
-
-	require 'thingfish/mixins'
-	require 'thingfish/datastore'
-	require 'thingfish/metastore'
-	extend MethodUtilities
-
 
 	### Get the library version. If +include_buildnum+ is true, the version string will
 	### include the VCS rev ID.
 	end
 
 
-	##
-	# Configurability API
-	config_key :thingfish
-
-	# The configured datastore type
-	singleton_attr_accessor :datastore
-
-	# The configured metastore type
-	singleton_attr_accessor :metastore
-
-
-	### Configurability API -- install the configuration
-	def self::configure( config=nil )
-		config ||= self.defaults
-		self.datastore = config[:datastore] || 'memory'
-		self.metastore = config[:metastore] || 'memory'
-	end
-
-
-	### Set up blerghh...
-	def initialize( * )
-		super
-
-		@datastore = Thingfish::Datastore.create( self.class.datastore )
-		@metastore = Thingfish::Metastore.create( self.class.metastore )
-	end
-
-
-	######
-	public
-	######
-
-	# The datastore
-	attr_reader :datastore
-
-	# The metastore
-	attr_reader :metastore
-
-
-	########################################################################
-	### P L U G I N S
-	########################################################################
-
-	#
-	# Global parmas
-	#
-	plugin :parameters
-	param :uuid
-	param :key, :word
-	param :limit, :integer, "The maximum number of records to return."
-	param :offset, :integer, "The offset into the result set to use as the first result."
-	param :order, /^(?<fields>[[:word:]]+(?:,[[:word:]]+)*)/,
-		"The name(s) of the fields to order results by."
-	param :direction, /^(asc|desc)$/i, "The order direction (ascending or descending)"
-	param :casefold, :boolean, "Whether or not to convert to lowercase before matching"
-
-
-	#
-	# Content negotiation
-	#
-	plugin :negotiation
-
-
-	#
-	# Filters
-	#
-	plugin :filters
-
-	### Modify outgoing headers on all responses to include version info.
-	###
-	filter :response do |res|
-		res.headers.x_thingfish = Thingfish.version_string( true )
-	end
-
-
-	#
-	# Routing
-	#
-	plugin :routing
-	router :exclusive
-
-	# GET /serverinfo
-	# Return various information about the handler configuration.
-	get '/serverinfo' do |req|
-		res  = req.response
-		info = {
-			:version   => Thingfish.version_string( true ),
-			:metastore => self.metastore.class.name,
-			:datastore => self.datastore.class.name
-		}
-
-		res.for( :text, :json, :yaml ) { info }
-		return res
-	end
-
-
-	#
-	# Datastore routes
-	#
-
-	# GET /
-	# Fetch a list of all objects
-	get do |req|
-		finish_with HTTP::BAD_REQUEST, req.params.error_messages.join(', ') unless req.params.okay?
-
-		uuids = self.metastore.search( req.params.valid )
-		self.log.debug "UUIDs are: %p" % [ uuids ]
-
-		base_uri = req.base_uri
-		list = uuids.collect do |uuid|
-			uri = base_uri.dup
-			uri.path += uuid
-
-			metadata = self.metastore.fetch( uuid )
-			metadata['uri'] = uri.to_s
-			metadata['uuid'] = uuid
-
-			metadata
-		end
-
-		res = req.response
-		res.for( :json, :yaml ) { list }
-		res.for( :text ) do
-			list.collect {|entry| "%s [%s, %0.2fB]" % entry.values_at(:url, :format, :extent) }
-		end
-
-		return res
-	end
-
-
-	# GET /«uuid»
-	# Fetch an object by ID
-	get ':uuid' do |req|
-		uuid = req.params[:uuid]
-		object = self.datastore.fetch( uuid ) or
-			finish_with HTTP::NOT_FOUND, "No such object."
-		metadata = self.metastore.fetch( uuid )
-
-		res = req.response
-		res.body = object
-		res.content_type = metadata['format']
-
-		return res
-	end
-
-
-	# POST /
-	# Upload a new object.
-	post do |req|
-		metadata = self.extract_header_metadata( req )
-		metadata.merge!( self.extract_default_metadata(req) )
-
-		uuid = self.datastore.save( req.body )
-		self.metastore.save( uuid, metadata )
-
-		url = req.base_uri.dup
-		url.path += uuid
-
-		res = req.response
-		res.headers.location = url
-		res.status = HTTP::CREATED
-
-		res.for( :text, :json, :yaml ) { metadata }
-
-		return res
-	end
-
-
-	# PUT /«uuid»
-	# Replace the data associated with +uuid+.
-	put ':uuid' do |req|
-		metadata = self.extract_default_metadata( req )
-
-		uuid = req.params[:uuid]
-		object = self.datastore.fetch( uuid ) or
-			finish_with HTTP::NOT_FOUND, "No such object."
-
-		self.datastore.replace( uuid, req.body )
-		self.metastore.merge( uuid, metadata )
-
-		res = req.response
-		res.status = HTTP::NO_CONTENT
-
-		return res
-	end
-
-
-	# DELETE /«uuid»
-	# Remove the object associated with +uuid+.
-	delete ':uuid' do |req|
-		uuid = req.params[:uuid]
-
-		self.datastore.remove( uuid ) or finish_with( HTTP::NOT_FOUND, "No such object." )
-		metadata = self.metastore.remove( uuid )
-
-		res = req.response
-		res.status = HTTP::OK
-
-		# TODO: Remove in favor of default metadata when the metastore
-		# knows what that is.
-		res.for( :text ) do
-			"%d bytes for %s deleted." % [ metadata['extent'], uuid ]
-		end
-		res.for( :json, :yaml ) {{ uuid: uuid, extent: metadata['extent'] }}
-
-		return res
-	end
-
-
-	#
-	# Metastore routes
-	#
-
-	# GET /«uuid»/metadata
-	# Fetch all metadata for «uuid».
-	get ':uuid/metadata' do |req|
-		uuid = req.params[:uuid]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-
-		res = req.response
-		res.status = HTTP::OK
-		res.for( :json, :yaml ) { self.metastore.fetch( uuid ) }
-
-		return res
-	end
-
-
-	# GET /«uuid»/metadata/«key»
-	# Fetch metadata value associated with «key» for «uuid».
-	get ':uuid/metadata/:key' do |req|
-		uuid = req.params[:uuid]
-		key  = req.params[:key]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-
-		res = req.response
-		res.status = HTTP::OK
-		res.for( :json, :yaml ) { self.metastore.fetch_value( uuid, key ) }
-
-		return res
-	end
-
-
-	# POST /«uuid»/metadata/«key»
-	# Create a metadata value associated with «key» for «uuid».
-	post ':uuid/metadata/:key' do |req|
-		uuid = req.params[:uuid]
-		key  = req.params[:key]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-		finish_with( HTTP::CONFLICT, "Key already exists." ) unless
-			self.metastore.fetch_value( uuid, key ).nil?
-
-		self.metastore.merge( uuid, key => req.body.read )
-
-		res = req.response
-		res.headers.location = req.uri.to_s
-		res.body = nil
-		res.status = HTTP::CREATED
-
-		return res
-	end
-
-
-	# PUT /«uuid»/metadata/«key»
-	# Replace or create a metadata value associated with «key» for «uuid».
-	put ':uuid/metadata/:key' do |req|
-		uuid = req.params[:uuid]
-		key  = req.params[:key]
-
-		finish_with( HTTP::FORBIDDEN, "Protected metadata." ) if
-			OPERATIONAL_METADATA_KEYS.include?( key )
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-		previous_value = self.metastore.fetch( uuid, key )
-
-		self.metastore.merge( uuid, key => req.body.read )
-
-		res = req.response
-		res.body = nil
-
-		if previous_value
-			res.status = HTTP::NO_CONTENT
-		else
-			res.headers.location = req.uri.to_s
-			res.status = HTTP::CREATED
-		end
-
-		return res
-	end
-
-
-	# PUT /«uuid»/metadata
-	# Replace user metadata for «uuid».
-	put ':uuid/metadata' do |req|
-		uuid = req.params[:uuid]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-
-		op_metadata = self.metastore.fetch( uuid, *OPERATIONAL_METADATA_KEYS )
-		new_metadata = self.extract_metadata( req )
-		self.metastore.save( uuid, new_metadata.merge(op_metadata) )
-
-		res = req.response
-		res.status = HTTP::NO_CONTENT
-
-		return res
-	end
-
-
-	# POST /«uuid»/metadata
-	# Merge new metadata into the existing metadata for «uuid».
-	post ':uuid/metadata' do |req|
-		uuid = req.params[:uuid]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-
-		new_metadata = self.extract_metadata( req )
-		self.metastore.merge( uuid, new_metadata )
-
-		res = req.response
-		res.status = HTTP::NO_CONTENT
-
-		return res
-	end
-
-
-	# DELETE /«uuid»/metadata
-	# Remove all (but operational) metadata associated with «uuid».
-	delete ':uuid/metadata' do |req|
-		uuid = req.params[:uuid]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-
-		self.metastore.remove_except( uuid, *OPERATIONAL_METADATA_KEYS )
-
-		res = req.response
-		res.status = HTTP::NO_CONTENT
-
-		return res
-	end
-
-
-	# DELETE /«uuid»/metadata
-	# Remove the metadata associated with «key» for the given «uuid».
-	delete ':uuid/metadata/:key' do |req|
-		uuid = req.params[:uuid]
-		key = req.params[:key]
-
-		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
-		finish_with( HTTP::FORBIDDEN, "Protected metadata." ) if
-			OPERATIONAL_METADATA_KEYS.include?( key )
-
-		self.metastore.remove( uuid, key )
-
-		res = req.response
-		res.status = HTTP::NO_CONTENT
-
-		return res
-	end
-
-
-	#########
-	protected
-	#########
-
-
-	### Overridden from the base handler class to allow spooled uploads.
-	def handle_async_upload_start( request )
-		self.log.info "Starting asynchronous upload: %s" %
-			[ request.headers.x_mongrel2_upload_start ]
-	end
-
-
-	### Return a Hash of default metadata extracted from the given +request+.
-	def extract_default_metadata( request )
-		return {
-			'useragent'     => request.headers.user_agent,
-			'extent'        => request.headers.content_length,
-			'uploadaddress' => request.remote_ip,
-			'format'        => request.content_type,
-			'created'       => Time.now.getgm,
-		}
-	end
-
-
-	### Extract and validate supplied metadata from the +request+.
-	def extract_metadata( req )
-		new_metadata = req.params.fields.dup
-		new_metadata.delete( :uuid )
-
-		protected_keys = OPERATIONAL_METADATA_KEYS & new_metadata.keys
-
-		unless protected_keys.empty?
-			finish_with HTTP::FORBIDDEN,
-				"Unable to alter protected metadata. (%s)" % [ protected_keys.join(', ') ]
-		end
-
-		return new_metadata
-	end
-
-
-	### Extract metadata from X-ThingFish-* headers from the given +request+ and return
-	### them as a Hash.
-	def extract_header_metadata( request )
-		self.log.debug "Extracting metadata from headers: %p" % [ request.headers ]
-		metadata = {}
-		request.headers.each do |header, value|
-			name = header.downcase[ /^x_thingfish_(?<name>[[:alnum:]\-]+)$/i, :name ] or next
-			self.log.debug "Found metadata header %p" % [ header ]
-			metadata[ name ] = value
-		end
-
-		return metadata
-	end
-
-end # class Thingfish
+end # module Thingfish
 
 # vim: set nosta noet ts=4 sw=4:

File lib/thingfish/handler.rb

View file
+# -*- ruby -*-
+#encoding: utf-8
+
+require 'strelka'
+require 'strelka/app'
+
+require 'configurability'
+require 'loggability'
+
+require 'thingfish' unless defined?( Thingfish )
+require 'thingfish/processor'
+
+#
+# Network-accessable datastore service
+#
+class Thingfish::Handler < Strelka::App
+	extend Loggability,
+	       Configurability
+
+
+	# Strelka App ID
+	ID = 'thingfish'
+
+
+	# Loggability API -- log to the :thingfish logger
+	log_to :thingfish
+
+
+	# Configurability API -- set config defaults
+	CONFIG_DEFAULTS = {
+		datastore: 'memory',
+		metastore: 'memory',
+		processors: [],
+		event_socket_uri: 'tcp://127.0.0.1:3475',
+	}
+
+	# Metadata keys which aren't directly modifiable via the REST API
+	OPERATIONAL_METADATA_KEYS = %w[
+		format
+		extent
+		created
+		uploadaddress
+	]
+
+
+	require 'thingfish/mixins'
+	require 'thingfish/datastore'
+	require 'thingfish/metastore'
+	extend Thingfish::MethodUtilities
+
+
+	##
+	# Configurability API
+	config_key :thingfish
+
+	# The configured datastore type
+	singleton_attr_accessor :datastore
+
+	# The configured metastore type
+	singleton_attr_accessor :metastore
+
+	# The ZMQ socket for publishing various resource events.
+	singleton_attr_accessor :event_socket_uri
+
+	# The list of configured processors.
+	singleton_attr_accessor :processors
+
+
+	### Configurability API -- install the configuration
+	def self::configure( config=nil )
+		config ||= self.defaults
+		self.datastore = config[:datastore] || self.defaults[:datastore]
+		self.metastore = config[:metastore] || self.defaults[:metastore]
+		self.event_socket_uri = config[:event_socket_uri] || self.defaults[:event_socket_uri]
+
+		self.processors = self.load_processors( config[:processors] )
+		self.processors.each do |processor|
+			self.filter( :request, &processor.method(:process_request) )
+			self.filter( :response, &processor.method(:process_response) )
+		end
+	end
+
+
+	### Load the Thingfish::Processors in the given +processor_list+ and return an instance
+	### of each one.
+	def self::load_processors( processor_list )
+		processors = []
+
+		processor_list.each do |processor_type|
+			begin
+				processors << Thingfish::Processor.create( processor_type )
+			rescue LoadError => err
+				self.log.error "%p: %s while loading the %s processor" %
+					[ err.class, err.message, processor_type ]
+			end
+		end
+
+		return processors
+	end
+
+
+	### Set up the metastore, datastore, and event socket when the handler is
+	### created.
+	def initialize( * ) # :notnew:
+		super
+
+		@datastore = Thingfish::Datastore.create( self.class.datastore )
+		@metastore = Thingfish::Metastore.create( self.class.metastore )
+		@event_socket = Mongrel2.zmq_context.socket( ZMQ::PUB )
+		@event_socket.setsockopt( ZMQ::LINGER, 0 )
+		@event_socket.bind( self.class.event_socket_uri )
+	end
+
+
+	######
+	public
+	######
+
+	# The datastore
+	attr_reader :datastore
+
+	# The metastore
+	attr_reader :metastore
+
+	# The PUB socket on which resource events are published
+	attr_reader :event_socket
+
+
+	### Shutdown handler hook.
+	def shutdown
+		self.event_socket.close if self.event_socket
+		super
+	end
+
+
+	### Restart handler hook.
+	def restart
+		if self.event_socket
+			oldsock = @event_socket
+			@event_socket = @event_socket.dup
+			oldsock.close
+		end
+
+		super
+	end
+
+
+	########################################################################
+	### P L U G I N S
+	########################################################################
+
+	#
+	# Global parmas
+	#
+	plugin :parameters
+	param :uuid
+	param :key, :word
+	param :limit, :integer, "The maximum number of records to return."
+	param :offset, :integer, "The offset into the result set to use as the first result."
+	param :order, /^(?<fields>[[:word:]]+(?:,[[:word:]]+)*)/,
+		"The name(s) of the fields to order results by."
+	param :direction, /^(asc|desc)$/i, "The order direction (ascending or descending)"
+	param :casefold, :boolean, "Whether or not to convert to lowercase before matching"
+
+
+	#
+	# Content negotiation
+	#
+	plugin :negotiation
+
+
+	#
+	# Filters
+	#
+	plugin :filters
+
+	### Modify outgoing headers on all responses to include version info.
+	###
+	filter :response do |res|
+		res.headers.x_thingfish = Thingfish.version_string( true )
+	end
+
+
+	#
+	# Routing
+	#
+	plugin :routing
+	router :exclusive
+
+	# GET /serverinfo
+	# Return various information about the handler configuration.
+	get '/serverinfo' do |req|
+		res  = req.response
+		info = {
+			:version   => Thingfish.version_string( true ),
+			:metastore => self.metastore.class.name,
+			:datastore => self.datastore.class.name
+		}
+
+		res.for( :text, :json, :yaml ) { info }
+		return res
+	end
+
+
+	#
+	# Datastore routes
+	#
+
+	# GET /
+	# Fetch a list of all objects
+	get do |req|
+		finish_with HTTP::BAD_REQUEST, req.params.error_messages.join(', ') unless req.params.okay?
+
+		uuids = self.metastore.search( req.params.valid )
+		self.log.debug "UUIDs are: %p" % [ uuids ]
+
+		base_uri = req.base_uri
+		list = uuids.collect do |uuid|
+			uri = base_uri.dup
+			uri.path += uuid
+
+			metadata = self.metastore.fetch( uuid )
+			metadata['uri'] = uri.to_s
+			metadata['uuid'] = uuid
+
+			metadata
+		end
+
+		res = req.response
+		res.for( :json, :yaml ) { list }
+		res.for( :text ) do
+			list.collect {|entry| "%s [%s, %0.2fB]" % entry.values_at(:url, :format, :extent) }
+		end
+
+		return res
+	end
+
+
+	# GET /«uuid»
+	# Fetch an object by ID
+	get ':uuid' do |req|
+		uuid = req.params[:uuid]
+		object = self.datastore.fetch( uuid ) or
+			finish_with HTTP::NOT_FOUND, "No such object."
+		metadata = self.metastore.fetch( uuid )
+
+		res = req.response
+		res.body = object
+		res.content_type = metadata['format']
+
+		return res
+	end
+
+
+	# POST /
+	# Upload a new object.
+	post do |req|
+		metadata = self.extract_header_metadata( req )
+		metadata.merge!( self.extract_default_metadata(req) )
+
+		uuid = self.datastore.save( req.body )
+		self.metastore.save( uuid, metadata )
+		self.send_event( :created, :uuid => uuid )
+
+		url = req.base_uri.dup
+		url.path += uuid
+
+		res = req.response
+		res.headers.location = url
+		res.status = HTTP::CREATED
+
+		res.for( :text, :json, :yaml ) { metadata }
+
+		return res
+	end
+
+
+	# PUT /«uuid»
+	# Replace the data associated with +uuid+.
+	put ':uuid' do |req|
+		metadata = self.extract_default_metadata( req )
+
+		uuid = req.params[:uuid]
+		object = self.datastore.fetch( uuid ) or
+			finish_with HTTP::NOT_FOUND, "No such object."
+
+		self.datastore.replace( uuid, req.body )
+		self.metastore.merge( uuid, metadata )
+
+		res = req.response
+		res.status = HTTP::NO_CONTENT
+
+		return res
+	end
+
+
+	# DELETE /«uuid»
+	# Remove the object associated with +uuid+.
+	delete ':uuid' do |req|
+		uuid = req.params[:uuid]
+
+		self.datastore.remove( uuid ) or finish_with( HTTP::NOT_FOUND, "No such object." )
+		metadata = self.metastore.remove( uuid )
+
+		res = req.response
+		res.status = HTTP::OK
+
+		# TODO: Remove in favor of default metadata when the metastore
+		# knows what that is.
+		res.for( :text ) do
+			"%d bytes for %s deleted." % [ metadata['extent'], uuid ]
+		end
+		res.for( :json, :yaml ) {{ uuid: uuid, extent: metadata['extent'] }}
+
+		return res
+	end
+
+
+	#
+	# Metastore routes
+	#
+
+	# GET /«uuid»/metadata
+	# Fetch all metadata for «uuid».
+	get ':uuid/metadata' do |req|
+		uuid = req.params[:uuid]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+
+		res = req.response
+		res.status = HTTP::OK
+		res.for( :json, :yaml ) { self.metastore.fetch( uuid ) }
+
+		return res
+	end
+
+
+	# GET /«uuid»/metadata/«key»
+	# Fetch metadata value associated with «key» for «uuid».
+	get ':uuid/metadata/:key' do |req|
+		uuid = req.params[:uuid]
+		key  = req.params[:key]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+
+		res = req.response
+		res.status = HTTP::OK
+		res.for( :json, :yaml ) { self.metastore.fetch_value( uuid, key ) }
+
+		return res
+	end
+
+
+	# POST /«uuid»/metadata/«key»
+	# Create a metadata value associated with «key» for «uuid».
+	post ':uuid/metadata/:key' do |req|
+		uuid = req.params[:uuid]
+		key  = req.params[:key]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+		finish_with( HTTP::CONFLICT, "Key already exists." ) unless
+			self.metastore.fetch_value( uuid, key ).nil?
+
+		self.metastore.merge( uuid, key => req.body.read )
+
+		res = req.response
+		res.headers.location = req.uri.to_s
+		res.body = nil
+		res.status = HTTP::CREATED
+
+		return res
+	end
+
+
+	# PUT /«uuid»/metadata/«key»
+	# Replace or create a metadata value associated with «key» for «uuid».
+	put ':uuid/metadata/:key' do |req|
+		uuid = req.params[:uuid]
+		key  = req.params[:key]
+
+		finish_with( HTTP::FORBIDDEN, "Protected metadata." ) if
+			OPERATIONAL_METADATA_KEYS.include?( key )
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+		previous_value = self.metastore.fetch( uuid, key )
+
+		self.metastore.merge( uuid, key => req.body.read )
+
+		res = req.response
+		res.body = nil
+
+		if previous_value
+			res.status = HTTP::NO_CONTENT
+		else
+			res.headers.location = req.uri.to_s
+			res.status = HTTP::CREATED
+		end
+
+		return res
+	end
+
+
+	# PUT /«uuid»/metadata
+	# Replace user metadata for «uuid».
+	put ':uuid/metadata' do |req|
+		uuid = req.params[:uuid]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+
+		op_metadata = self.metastore.fetch( uuid, *OPERATIONAL_METADATA_KEYS )
+		new_metadata = self.extract_metadata( req )
+		self.metastore.save( uuid, new_metadata.merge(op_metadata) )
+
+		res = req.response
+		res.status = HTTP::NO_CONTENT
+
+		return res
+	end
+
+
+	# POST /«uuid»/metadata
+	# Merge new metadata into the existing metadata for «uuid».
+	post ':uuid/metadata' do |req|
+		uuid = req.params[:uuid]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+
+		new_metadata = self.extract_metadata( req )
+		self.metastore.merge( uuid, new_metadata )
+
+		res = req.response
+		res.status = HTTP::NO_CONTENT
+
+		return res
+	end
+
+
+	# DELETE /«uuid»/metadata
+	# Remove all (but operational) metadata associated with «uuid».
+	delete ':uuid/metadata' do |req|
+		uuid = req.params[:uuid]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+
+		self.metastore.remove_except( uuid, *OPERATIONAL_METADATA_KEYS )
+
+		res = req.response
+		res.status = HTTP::NO_CONTENT
+
+		return res
+	end
+
+
+	# DELETE /«uuid»/metadata
+	# Remove the metadata associated with «key» for the given «uuid».
+	delete ':uuid/metadata/:key' do |req|
+		uuid = req.params[:uuid]
+		key = req.params[:key]
+
+		finish_with( HTTP::NOT_FOUND, "No such object." ) unless self.metastore.include?( uuid )
+		finish_with( HTTP::FORBIDDEN, "Protected metadata." ) if
+			OPERATIONAL_METADATA_KEYS.include?( key )
+
+		self.metastore.remove( uuid, key )
+
+		res = req.response
+		res.status = HTTP::NO_CONTENT
+
+		return res
+	end
+
+
+	#########
+	protected
+	#########
+
+
+	### Overridden from the base handler class to allow spooled uploads.
+	def handle_async_upload_start( request )
+		self.log.info "Starting asynchronous upload: %s" %
+			[ request.headers.x_mongrel2_upload_start ]
+	end
+
+
+	### Return a Hash of default metadata extracted from the given +request+.
+	def extract_default_metadata( request )
+		return {
+			'useragent'     => request.headers.user_agent,
+			'extent'        => request.headers.content_length,
+			'uploadaddress' => request.remote_ip,
+			'format'        => request.content_type,
+			'created'       => Time.now.getgm,
+		}
+	end
+
+
+	### Extract and validate supplied metadata from the +request+.
+	def extract_metadata( req )
+		new_metadata = req.params.fields.dup
+		new_metadata.delete( :uuid )
+
+		protected_keys = OPERATIONAL_METADATA_KEYS & new_metadata.keys
+
+		unless protected_keys.empty?
+			finish_with HTTP::FORBIDDEN,
+				"Unable to alter protected metadata. (%s)" % [ protected_keys.join(', ') ]
+		end
+
+		return new_metadata
+	end
+
+
+	### Extract metadata from X-ThingFish-* headers from the given +request+ and return
+	### them as a Hash.
+	def extract_header_metadata( request )
+		self.log.debug "Extracting metadata from headers: %p" % [ request.headers ]
+		metadata = {}
+		request.headers.each do |header, value|
+			name = header.downcase[ /^x_thingfish_(?<name>[[:alnum:]\-]+)$/i, :name ] or next
+			self.log.debug "Found metadata header %p" % [ header ]
+			metadata[ name ] = value
+		end
+
+		return metadata
+	end
+
+
+	### Send an event of +type+ with the given +msg+ over the zmq event socket.
+	def send_event( type, msg )
+		self.event_socket.send( type.to_s, ZMQ::SNDMORE )
+		self.event_socket.send( Yajl.dump(msg) )
+	end
+
+
+end # class Thingfish::Handler
+
+# vim: set nosta noet ts=4 sw=4:

File lib/thingfish/mixins.rb

View file
 require 'thingfish' unless defined?( Thingfish )
 
 
-class Thingfish
+module Thingfish
 
 	# Hides your class's ::new method and adds a +pure_virtual+ method generator for
 	# defining API methods. If subclasses of your class don't provide implementations of

File lib/thingfish/processor.rb

View file
+# -*- ruby -*-
+#encoding: utf-8
+
+require 'pluggability'
+
+require 'thingfish' unless defined?( Thingfish )
+
+
+# Thingfish asset processor base class.
+class Thingfish::Processor
+	extend Pluggability
+
+
+	plugin_prefixes 'processor'
+
+
+	### Process the data and/or metadata in the +request+.
+	def process_request( request )
+		# No-op by default
+	end
+
+
+	### Process the data and/or metadata in the +response+.
+	def process_response( response )
+		# No-op by default
+	end
+
+end # class Thingfish::Processor
+

File lib/thingfish/processordaemon.rb

View file
+# -*- ruby -*-
+#encoding: utf-8
+
+require 'zmq'
+require 'configurability'
+require 'loggability'
+
+require 'thingfish' unless defined?( Thingfish )
+
+
+# 
+class Thingfish::ProcessorDaemon
+
+	
+
+end # class Thingfish::ProcessorDaemon
+
+

File spec/thingfish/handler_spec.rb

View file
+#!/usr/bin/env ruby
+
+require_relative '../helpers'
+
+require 'rspec'
+require 'thingfish/handler'
+require 'thingfish/processor'
+
+
+describe Thingfish::Handler do
+
+	before( :all ) do
+		setup_logging()
+		Thingfish::Handler.configure
+	end
+
+	before( :each ) do
+		@png_io = StringIO.new( TEST_PNG_DATA.dup )
+		@text_io = StringIO.new( TEST_TEXT_DATA.dup )
+		@handler = described_class.new( TEST_APPID, TEST_SEND_SPEC, TEST_RECV_SPEC )
+	end
+
+
+	after( :each ) do
+		@handler.shutdown
+	end
+
+	# let( :handler ) { described_class.new(TEST_APPID, TEST_SEND_SPEC, TEST_RECV_SPEC) }
+
+
+	#
+	# Shared behaviors
+	#
+
+	it_should_behave_like "an object with Configurability"
+
+
+	#
+	# Examples
+	#
+
+	context "misc api" do
+
+		let( :factory ) do
+			Mongrel2::RequestFactory.new(
+				:route => '/',
+				:headers => {:accept => '*/*'})
+		end
+
+		it 'returns interesting configuration info' do
+			req = factory.get( '/serverinfo',  content_type: 'text/plain' )
+			res = @handler.handle( req )
+
+			expect( res.status_line ).to match( /200 ok/i )
+			expect( res.headers ).to include( 'x-thingfish' )
+		end
+
+	end
+
+
+	context "datastore api" do
+
+		let( :factory ) do
+			Mongrel2::RequestFactory.new(
+				:route => '/',
+				:headers => {:accept => '*/*'})
+		end
+
+		it 'accepts a POSTed upload' do
+			req = factory.post( '/', TEST_TEXT_DATA, content_type: 'text/plain' )
+			res = @handler.handle( req )
+
+			expect( res.status_line ).to match( /201 created/i )
+			expect( res.headers.location.to_s ).to match( %r:/#{UUID_PATTERN}$: )
+		end
+
+
+		it "allows additional metadata to be attached to uploads via X-Thingfish-* headers" do
+			headers = {
+				content_type: 'text/plain',
+				x_thingfish_title: 'Muffin the Panda Goes To School',
+				x_thingfish_tags: 'rapper,ukraine,potap',
+			}
+			req = factory.post( '/', TEST_TEXT_DATA, headers )
+			res = @handler.handle( req )
+
+			expect( res.status_line ).to match( /201 created/i )
+			expect( res.headers.location.to_s ).to match( %r:/#{UUID_PATTERN}$: )
+
+			uuid = res.headers.location.to_s[ %r:/(?<uuid>#{UUID_PATTERN})$:, :uuid ]
+			expect( @handler.metastore.fetch_value(uuid, 'title') ).
+				to eq( 'Muffin the Panda Goes To School' )
+			expect( @handler.metastore.fetch_value(uuid, 'tags') ).to eq( 'rapper,ukraine,potap' )
+		end
+
+		it 'replaces content via PUT' do
+			uuid = @handler.datastore.save( @text_io )
+			@handler.metastore.save( uuid, {'format' => 'text/plain'} )
+
+			req = factory.put( "/#{uuid}", @png_io, content_type: 'image/png' )
+			res = @handler.handle( req )
+
+			expect( res.status ).to eq( HTTP::NO_CONTENT )
+			expect( @handler.datastore.fetch(uuid).read ).to eq( TEST_PNG_DATA )
+			expect( @handler.metastore.fetch(uuid) ).to include( 'format' => 'image/png' )
+		end
+
+
+		it "doesn't case about the case of the UUID when replacing content via PUT" do
+			uuid = @handler.datastore.save( @text_io )
+			@handler.metastore.save( uuid, {'format' => 'text/plain'} )
+
+			req = factory.put( "/#{uuid.upcase}", @png_io, content_type: 'image/png' )
+			res = @handler.handle( req )
+
+			expect( res.status ).to eq( HTTP::NO_CONTENT )
+			expect( @handler.datastore.fetch(uuid).read ).to eq( TEST_PNG_DATA )
+			expect( @handler.metastore.fetch(uuid) ).to include( 'format' => 'image/png' )
+		end
+
+
+		it 'can fetch all uploaded data' do
+			text_uuid = @handler.datastore.save( @text_io )
+			@handler.metastore.save( text_uuid, {
+				'format' => 'text/plain',
+				'extent' => @text_io.string.bytesize
+			})
+			png_uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( png_uuid, {
+				'format' => 'image/png',
+				'extent' => @png_io.string.bytesize
+			})
+
+			req = factory.get( '/' )
+			res = @handler.handle( req )
+			content = Yajl::Parser.parse( res.body.read )
+
+			expect( res.status_line ).to match( /200 ok/i )
+			expect( res.headers.content_type ).to eq( 'application/json' )
+			expect( content ).to be_a( Array )
+			expect( content[0] ).to be_a( Hash )
+			expect( content[0]['uri'] ).to eq( "#{req.base_uri}#{text_uuid}" )
+			expect( content[0]['format'] ).to eq( "text/plain" )
+			expect( content[0]['extent'] ).to eq( @text_io.string.bytesize )
+			expect( content[1] ).to be_a( Hash )
+			expect( content[1]['uri'] ).to eq( "#{req.base_uri}#{png_uuid}" )
+			expect( content[1]['format'] ).to eq( 'image/png' )
+			expect( content[1]['extent'] ).to eq( @png_io.string.bytesize )
+		end
+
+
+		it "can fetch an uploaded chunk of data" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {'format' => 'image/png'} )
+
+			req = factory.get( "/#{uuid}" )
+			result = @handler.handle( req )
+
+			expect( result.status_line ).to match( /200 ok/i )
+			expect( result.body.read ).to eq( @png_io.string )
+			expect( result.headers.content_type ).to eq( 'image/png' )
+		end
+
+
+		it "doesn't care about the case of the UUID when fetching uploaded data" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {'format' => 'image/png'} )
+
+			req = factory.get( "/#{uuid.upcase}" )
+			result = @handler.handle( req )
+
+			expect( result.status_line ).to match( /200 ok/i )
+			expect( result.body.read ).to eq( @png_io.string )
+			expect( result.headers.content_type ).to eq( 'image/png' )
+		end
+
+
+		it "can remove everything associated with an object id" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			req = factory.delete( "/#{uuid}" )
+			result = @handler.handle( req )
+
+			expect( result.status_line ).to match( /200 ok/i )
+			expect( @handler.metastore ).to_not include( uuid )
+			expect( @handler.datastore ).to_not include( uuid )
+		end
+
+
+		it "returns a 404 Not Found when asked to remove an object that doesn't exist" do
+			req = factory.delete( "/#{TEST_UUID}" )
+			result = @handler.handle( req )
+
+			expect( result.status_line ).to match( /404 not found/i )
+		end
+
+	end
+
+
+	context "metastore api" do
+
+		let( :factory ) do
+			Mongrel2::RequestFactory.new(
+				:route => '/',
+				:headers => {:accept => 'application/json'})
+		end
+
+		it "can fetch the metadata associated with uploaded data" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'  => 'image/png',
+				'extent'  => 288,
+				'created' => Time.at(1378313840),
+			})
+
+			req = factory.get( "/#{uuid}/metadata" )
+			result = @handler.handle( req )
+			content = result.body.read
+
+			content_hash = Yajl::Parser.parse( content )
+
+			expect( result.status ).to eq( 200 )
+			expect( result.headers.content_type ).to eq( 'application/json' )
+			expect( content_hash ).to be_a( Hash )
+			expect( content_hash['extent'] ).to eq( 288 )
+			expect( content_hash['created'] ).to eq( Time.at(1378313840).to_s )
+		end
+
+
+		it "returns a 404 Not Found when fetching metadata for an object that doesn't exist" do
+			req = factory.get( "/#{TEST_UUID}/metadata" )
+			result = @handler.handle( req )
+
+			expect( result.status_line ).to match( /404 not found/i )
+		end
+
+
+		it "can fetch a value for a single metadata key" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			req = factory.get( "/#{uuid}/metadata/extent" )
+			result = @handler.handle( req )
+			content = Yajl::Parser.parse( result.body.read )
+
+			expect( result.status ).to eq( 200 )
+			expect( result.headers.content_type ).to eq( 'application/json' )
+			expect( content ).to be( 288 )
+		end
+
+
+		it "returns a 404 Not Found when fetching a single metadata value for a uuid that doesn't exist" do
+			req = factory.get( "/#{TEST_UUID}/metadata/extent" )
+			result = @handler.handle( req )
+
+			expect( result.status_line ).to match( /404 not found/i )
+		end
+
+
+		it "doesn't error when fetching a non-existent metadata value" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			req = factory.get( "/#{uuid}/metadata/hururrgghh" )
+			result = @handler.handle( req )
+
+			content = Yajl::Parser.parse( result.body.read )
+
+			expect( result.status ).to eq( 200 )
+			expect( result.headers.content_type ).to eq( 'application/json' )
+
+			expect( content ).to be_nil
+		end
+
+
+		it "can merge in new metadata for an existing resource with a POST" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			body_json = Yajl.dump({ 'comment' => 'Ignore me!' })
+			req = factory.post( "/#{uuid}/metadata", body_json, 'Content-type' => 'application/json' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NO_CONTENT )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to eq( 'Ignore me!' )
+		end
+
+
+		it "returns FORBIDDEN when attempting to merge metadata with operational keys" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			body_json = Yajl.dump({ 'format' => 'text/plain', 'comment' => 'Ignore me!' })
+			req = factory.post( "/#{uuid}/metadata", body_json, 'Content-type' => 'application/json' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::FORBIDDEN )
+			expect( result.body.string ).to match( /unable to alter protected metadata/i )
+			expect( result.body.string ).to match( /format/i )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to be_nil
+			expect( @handler.metastore.fetch_value(uuid, 'format') ).to eq( 'image/png' )
+		end
+
+
+		it "can create single metadata values with a POST" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			req = factory.post( "/#{uuid}/metadata/comment", "urrrg", 'Content-type' => 'text/plain' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::CREATED )
+			expect( result.headers.location ).to match( %r|#{uuid}/metadata/comment$| )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to eq( 'urrrg' )
+		end
+
+
+		it "returns NOT_FOUND when attempting to create metadata for a non-existent object" do
+			req = factory.post( "/#{TEST_UUID}/metadata/comment", "urrrg", 'Content-type' => 'text/plain' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NOT_FOUND )
+			expect( result.body.string ).to match( /no such object/i )
+		end
+
+
+		it "returns CONFLICT when attempting to create a single metadata value if it already exists" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'  => 'image/png',
+				'extent'  => 288,
+				'comment' => 'nill bill'
+			})
+
+			req = factory.post( "/#{uuid}/metadata/comment", "urrrg", 'Content-type' => 'text/plain' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::CONFLICT )
+			expect( result.body.string ).to match( /already exists/i )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to eq( 'nill bill' )
+		end
+
+
+		it "can create single metadata values with a PUT" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format' => 'image/png',
+				'extent' => 288,
+			})
+
+			req = factory.put( "/#{uuid}/metadata/comment", "urrrg", 'Content-type' => 'text/plain' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NO_CONTENT )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to eq( 'urrrg' )
+		end
+
+
+		it "can replace a single metadata value with a PUT" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'  => 'image/png',
+				'extent'  => 288,
+				'comment' => 'nill bill'
+			})
+
+			req = factory.put( "/#{uuid}/metadata/comment", "urrrg", 'Content-type' => 'text/plain' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NO_CONTENT )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to eq( 'urrrg' )
+		end
+
+
+		it "returns FORBIDDEN when attempting to replace a operational metadata value with a PUT" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'  => 'image/png',
+				'extent'  => 288,
+				'comment' => 'nill bill'
+			})
+
+			req = factory.put( "/#{uuid}/metadata/format", "image/gif", 'Content-type' => 'text/plain' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::FORBIDDEN )
+			expect( result.body.string ).to match( /protected metadata/i )
+			expect( @handler.metastore.fetch_value(uuid, 'format') ).to eq( 'image/png' )
+		end
+
+
+		it "can replace all metadata with a PUT" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'    => 'image/png',
+				'extent'    => 288,
+				'comment'   => 'nill bill',
+				'ephemeral' => 'butterflies',
+			})
+
+			req = factory.put( "/#{uuid}/metadata", %[{"comment":"Yeah."}],
+			                   'Content-type' => 'application/json' )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NO_CONTENT )
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to eq( 'Yeah.' )
+			expect( @handler.metastore.fetch_value(uuid, 'format') ).to eq( 'image/png' )
+			expect( @handler.metastore ).to_not include( 'ephemeral' )
+		end
+
+
+		it "can remove all non-default metadata with a DELETE" do
+			timestamp = Time.now.getgm
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'        => 'image/png',
+				'extent'        => 288,
+				'comment'       => 'nill bill',
+				'useragent'     => 'Inky/2.0',
+				'uploadaddress' => '127.0.0.1',
+				'created'       => timestamp,
+			})
+
+			req = factory.delete( "/#{uuid}/metadata" )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NO_CONTENT )
+			expect( result.body.string ).to be_empty
+			expect( @handler.metastore.fetch_value(uuid, 'format') ).to eq( 'image/png' )
+			expect( @handler.metastore.fetch_value(uuid, 'extent') ).to eq( 288 )
+			expect( @handler.metastore.fetch_value(uuid, 'uploadaddress') ).to eq( '127.0.0.1' )
+			expect( @handler.metastore.fetch_value(uuid, 'created') ).to eq( timestamp )
+
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to be_nil
+			expect( @handler.metastore.fetch_value(uuid, 'useragent') ).to be_nil
+		end
+
+
+		it "can remove a single metadata value with DELETE" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'  => 'image/png',
+				'comment' => 'nill bill'
+			})
+
+			req = factory.delete( "/#{uuid}/metadata/comment" )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::NO_CONTENT )
+			expect( result.body.string ).to be_empty
+			expect( @handler.metastore.fetch_value(uuid, 'comment') ).to be_nil
+			expect( @handler.metastore.fetch_value(uuid, 'format') ).to eq( 'image/png' )
+		end
+
+
+		it "returns FORBIDDEN when attempting to remove a operational metadata value with a DELETE" do
+			uuid = @handler.datastore.save( @png_io )
+			@handler.metastore.save( uuid, {
+				'format'  => 'image/png'
+			})
+
+			req = factory.delete( "/#{uuid}/metadata/format" )
+			result = @handler.handle( req )
+
+			expect( result.status ).to eq( HTTP::FORBIDDEN )
+			expect( result.body.string ).to match( /protected metadata/i )
+			expect( @handler.metastore.fetch_value(uuid, 'format') ).to eq( 'image/png' )
+		end
+	end
+
+
+	context "processors" do
+
+		after( :each ) do
+			described_class.processors.clear
+		end
+
+		let( :factory ) do
+			Mongrel2::RequestFactory.new(
+				:route => '/',
+				:headers => {:accept => '*/*'})
+		end
+
+		let!( :test_filter ) do
+			klass = Class.new( Thingfish::Processor ) do
+				def self::name; 'Thingfish::Processor::Test'; end
+				def initialize
+					@requests = []
+					@responses = []
+				end
+				attr_accessor :requests, :responses
+				def process_request( request )
+					self.requests << request
+				end
+				def process_response( response )
+					self.responses << response
+				end
+			end
+			# Re-call inherited so it associates it with its name
+			Thingfish::Processor.inherited( klass )
+			klass
+		end
+
+
+		it "loads configured processors when it is instantiated" do
+			described_class.configure( :processors => %w[test] )
+
+			expect( described_class.processors ).to be_an( Array )
+
+			processor = described_class.processors.first
+			expect( processor ).to be_an_instance_of( test_filter )
+		end
+
+
+		it "processes requests and responses" do
+			described_class.configure( :processors => %w[test] )
+
+			req = factory.post( '/', TEST_TEXT_DATA, content_type: 'text/plain' )
+			res = @handler.handle( req )
+
+			processor = described_class.processors.first
+			expect( processor.requests ).to eq([ req ])
+			expect( processor.responses ).to eq([ res ])
+		end
+
+	end
+
+
+	context "event hook" do
+
+		let( :factory ) do
+			Mongrel2::RequestFactory.new(
+				:route => '/',
+				:headers => {:accept => '*/*'})
+		end
+
+		before( :each ) do
+			@subsock = Mongrel2.zmq_context.socket( ZMQ::SUB )
+			@subsock.setsockopt( ZMQ::LINGER, 0 )
+			@subsock.setsockopt( ZMQ::SUBSCRIBE, '' )
+			@subsock.connect( @handler.class.event_socket_uri )
+		end
+
+		after( :each ) do
+			@subsock.close
+		end
+
+
+		it "publishes notifications about uploaded assets to a PUBSUB socket" do
+			req = factory.post( '/', TEST_TEXT_DATA, content_type: 'text/plain' )
+			res = @handler.handle( req )
+
+			handles = ZMQ.select( [@subsock], nil, nil, 0 )
+			expect( handles ).to be_an( Array )
+			expect( handles[0] ).to have( 1 ).socket
+			expect( handles[0].first ).to be( @subsock )
+
+			event = @subsock.recv
+			expect( @subsock.getsockopt( ZMQ::RCVMORE ) ).to be_true
+			expect( event ).to eq( 'created' )
+
+			resource = @subsock.recv
+			expect( @subsock.getsockopt( ZMQ::RCVMORE ) ).to be_false
+			expect( resource ).to match( /^\{"uuid":"#{UUID_PATTERN}"\}$/ )
+		end
+
+	end
+
+end
+
+# vim: set nosta noet ts=4 sw=4 ft=rspec:

File spec/thingfish/processor_spec.rb

View file
+#!/usr/bin/env ruby
+
+require_relative '../helpers'
+
+require 'rspec'
+require 'thingfish/processor'
+
+
+describe Thingfish::Processor do
+
+	before( :all ) do
+		setup_logging()
+	end
+
+
+	it "has pluggability" do
+		expect( described_class.plugin_type ).to eq( 'Processor' )
+	end
+
+
+	it "defines a (no-op) method for handling requests" do
+		expect {
+			described_class.new.process_request( nil )
+		}.to_not raise_error
+	end
+
+
+	it "defines a (no-op) method for handling responses" do
+		expect {
+			described_class.new.process_response( nil )
+		}.to_not raise_error
+	end
+
+
+end
+
+# vim: set nosta noet ts=4 sw=4 ft=rspec:

File spec/thingfish_spec.rb

-#!/usr/bin/env ruby
-
-require_relative 'helpers'
-
-require 'rspec'
-require 'thingfish'
-
-
-describe Thingfish do
-
-	before( :all ) do
-		setup_logging()
-		Thingfish.configure
-	end
-
-	before( :each ) do
-		@png_io = StringIO.new( TEST_PNG_DATA.dup )
-		@text_io = StringIO.new( TEST_TEXT_DATA.dup )
-	end
-
-	let( :handler ) { described_class.new(TEST_APPID, TEST_SEND_SPEC, TEST_RECV_SPEC) }
-
-
-	#
-	# Shared behaviors
-	#
-
-	it_should_behave_like "an object with Configurability"
-
-
-	#
-	# Examples
-	#
-
-	it "returns a version string if asked" do
-		Thingfish.version_string.should =~ /\w+ [\d.]+/
-	end
-
-
-	it "returns a version string with a build number if asked" do
-		Thingfish.version_string(true).should =~ /\w+ [\d.]+ \(build [[:xdigit:]]+\)/
-	end
-
-
-	context "misc api" do
-
-		let( :factory ) do
-			Mongrel2::RequestFactory.new(
-				:route => '/',
-				:headers => {:accept => '*/*'})
-		end
-
-		it 'returns interesting configuration info' do
-			req = factory.get( '/serverinfo',  content_type: 'text/plain' )
-			res = handler.handle( req )
-
-			expect( res.status_line ).to match( /200 ok/i )
-			expect( res.headers ).to include( 'x-thingfish' )
-		end
-	end
-
-
-	context "datastore api" do
-
-		let( :factory ) do
-			Mongrel2::RequestFactory.new(
-				:route => '/',
-				:headers => {:accept => '*/*'})
-		end
-
-		it 'accepts a POSTed upload' do
-			req = factory.post( '/', TEST_TEXT_DATA, content_type: 'text/plain' )
-			res = handler.handle( req )
-
-			expect( res.status_line ).to match( /201 created/i )
-			expect( res.headers.location.to_s ).to match( %r:/#{UUID_PATTERN}$: )
-		end
-
-
-		it "allows additional metadata to be attached to uploads via X-Thingfish-* headers" do
-			headers = {
-				content_type: 'text/plain',
-				x_thingfish_title: 'Muffin the Panda Goes To School',
-				x_thingfish_tags: 'rapper,ukraine,potap',
-			}
-			req = factory.post( '/', TEST_TEXT_DATA, headers )
-			res = handler.handle( req )
-
-			expect( res.status_line ).to match( /201 created/i )
-			expect( res.headers.location.to_s ).to match( %r:/#{UUID_PATTERN}$: )
-
-			uuid = res.headers.location.to_s[ %r:/(?<uuid>#{UUID_PATTERN})$:, :uuid ]
-			expect( handler.metastore.fetch_value(uuid, 'title') ).
-				to eq( 'Muffin the Panda Goes To School' )
-			expect( handler.metastore.fetch_value(uuid, 'tags') ).to eq( 'rapper,ukraine,potap' )
-		end
-
-		it 'replaces content via PUT' do
-			uuid = handler.datastore.save( @text_io )
-			handler.metastore.save( uuid, {'format' => 'text/plain'} )
-
-			req = factory.put( "/#{uuid}", @png_io, content_type: 'image/png' )
-			res = handler.handle( req )
-
-			expect( res.status ).to eq( HTTP::NO_CONTENT )
-			expect( handler.datastore.fetch(uuid).read ).to eq( TEST_PNG_DATA )
-			expect( handler.metastore.fetch(uuid) ).to include( 'format' => 'image/png' )
-		end
-
-
-		it "doesn't case about the case of the UUID when replacing content via PUT" do
-			uuid = handler.datastore.save( @text_io )
-			handler.metastore.save( uuid, {'format' => 'text/plain'} )
-
-			req = factory.put( "/#{uuid.upcase}", @png_io, content_type: 'image/png' )
-			res = handler.handle( req )
-
-			expect( res.status ).to eq( HTTP::NO_CONTENT )
-			expect( handler.datastore.fetch(uuid).read ).to eq( TEST_PNG_DATA )
-			expect( handler.metastore.fetch(uuid) ).to include( 'format' => 'image/png' )
-		end
-
-
-		it 'can fetch all uploaded data' do
-			text_uuid = handler.datastore.save( @text_io )
-			handler.metastore.save( text_uuid, {
-				'format' => 'text/plain',
-				'extent' => @text_io.string.bytesize
-			})
-			png_uuid = handler.datastore.save( @png_io )
-			handler.metastore.save( png_uuid, {
-				'format' => 'image/png',
-				'extent' => @png_io.string.bytesize
-			})
-
-			req = factory.get( '/' )
-			res = handler.handle( req )
-			content = Yajl::Parser.parse( res.body.read )
-
-			expect( res.status_line ).to match( /200 ok/i )
-			expect( res.headers.content_type ).to eq( 'application/json' )
-			expect( content ).to be_a( Array )
-			expect( content[0] ).to be_a( Hash )
-			expect( content[0]['uri'] ).to eq( "#{req.base_uri}#{text_uuid}" )
-			expect( content[0]['format'] ).to eq( "text/plain" )
-			expect( content[0]['extent'] ).to eq( @text_io.string.bytesize )
-			expect( content[1] ).to be_a( Hash )
-			expect( content[1]['uri'] ).to eq( "#{req.base_uri}#{png_uuid}" )
-			expect( content[1]['format'] ).to eq( 'image/png' )
-			expect( content[1]['extent'] ).to eq( @png_io.string.bytesize )
-		end
-
-
-		it "can fetch an uploaded chunk of data" do
-			uuid = handler.datastore.save( @png_io )
-			handler.metastore.save( uuid, {'format' => 'image/png'} )
-
-			req = factory.get( "/#{uuid}" )
-			result = handler.handle( req )
-
-			expect( result.status_line ).to match( /200 ok/i )
-			expect( result.body.read ).to eq( @png_io.string )
-			expect( result.headers.content_type ).to eq( 'image/png' )
-		end
-
-
-		it "doesn't care about the case of the UUID when fetching uploaded data" do
-			uuid = handler.datastore.save( @png_io )
-			handler.metastore.save( uuid, {'format' => 'image/png'} )
-
-			req = factory.get( "/#{uuid.upcase}" )
-			result = handler.handle( req )
-
-			expect( result.status_line ).to match( /200 ok/i )
-			expect( result.body.read ).to eq( @png_io.string )
-			expect( result.headers.content_type ).to eq( 'image/png' )
-		end
-
-
-		it "can remove everything associated with an object id" do
-			uuid = handler.datastore.save( @png_io )
-			handler.metastore.save( uuid, {
-				'format' => 'image/png',
-				'extent' => 288,
-			})
-
-			req = factory.delete( "/#{uuid}" )
-			result = handler.handle( req )
-
-			expect( result.status_line ).to match( /200 ok/i )
-			expect( handler.metastore ).to_not include( uuid )
-			expect( handler.datastore ).to_not include( uuid )
-		end
-
-
-		it "returns a 404 Not Found when asked to remove an object that doesn't exist" do
-			req = factory.delete( "/#{TEST_UUID}" )
-			result = handler.handle( req )
-
-			expect( result.status_line ).to match( /404 not found/i )
-		end
-
-	end
-
-
-	context "metastore api" do
-
-		let( :factory ) do
-			Mongrel2::RequestFactory.new(
-				:route => '/',
-				:headers => {:accept => 'application/json'})
-		end
-
-		it "can fetch the metadata associated with uploaded data" do
-			uuid = handler.datastore.save( @png_io )
-			handler.metastore.save( uuid, {
-				'format'  => 'image/png',
-				'extent'  => 288,
-				'created' => Time.at(1378313840),
-			})
-
-			req = factory.get( "/#{uuid}/metadata" )
-			result = handler.handle( req )
-			content = result.body.read
-
-			content_hash = Yajl::Parser.parse( content )
-
-			expect( result.status ).to eq( 200 )
-			expect( result.headers.content_type ).to eq( 'application/json' )
-			expect( content_hash ).to be_a( Hash )
-			expect( content_hash['extent'] ).to eq( 288 )
-			expect( content_hash['created'] ).to eq( Time.at(1378313840).to_s )
-		end
-
-
-		it "returns a 404 Not Found when fetching metadata for an object that doesn't exist" do
-			req = factory.get( "/#{TEST_UUID}/metadata" )
-			result = handler.handle( req )
-
-			expect( result.status_line ).to match( /404 not found/i )
-		end
-
-
-		it "can fetch a value for a single metadata key" do
-			uuid = handler.datastore.save( @png_io )
-			handler.metastore.save( uuid, {