Commits

Jasper Op de Coul committed 4b70d00

Added better README, removed unused zope.interface references

Comments (0)

Files changed (8)

 lib
+build
 bin
 MOAI.egg-info
 include
+
+MOAI, an Open Access Server Platform for Institutional Repositories
+===================================================================
+
+MOAI is a platform for aggregating content from different sources, and publishing it through the Open Archive Initiatives protocol for metadata harvesting.
+It's been built for academic institutional repositories dealing with relational metadata and asset files.
+
+What does it do?
+----------------
+
+The MOAI software can aggregate content from different sources, transform it and store it in a database. The contents of this database can then be published in many separate OAI feeds, each with its own configuration. 
+
+The MOAI software has a very flexible system for combining records into sets, and can use these sets in the feed configuration. It also comes with a simple yet flexible authentication scheme that can be easily customized. Besides providing authentication for the feeds, the authentication also controls the access to the assets. 
+
+Why MOAI
+--------
+
+MOAI has been specifically developed for universities, and contains a lot of hard-earned wisdom. The software has been in production use since 2007, and new features have been continually added. In late 2008, the software was completely refactored and packaged under the name "MOAI". You can read more about this on the `MOAI History`_ page.
+
+MOAI is a standalone system, so it can be used in combination with any repository software that comes with an OAI feed such as `Fedora Commons`_, `EPrints`_ or `DSpace`_. It can also be used directly with an SQL database or just a folder of XML files.
+
+The MOAI project takes the philosophy that every repository is different and unique, and that an institutional repository is a living thing. It is therefore never finished. Metadata is always changes, improving, and evolves. We think this is healthy. 
+
+Because of this viewpoint, the MOAI software makes it as easy as possible to add or modify parts of your repository (OAI) services stack. It tries to do this without sacrificing power, and encouraging the re-use of components.
+
+Features
+--------
+
+MOAI has some interesting features not found in most OAI servers. 
+Besides serving OAI, it can also harvest OAI. This makes it possible for MOAI to work as a pipe, where the OAI data can be reconfigured, cached, and enriched while it passes through the MOAI processing.
+
+
+More specifically MOAI has the ability to:
+
+- Harvest data from different kinds of sources
+- Serve many OAI feeds from one MOAI server, each with their own configuration
+- Turn metadata values into OAI sets on the fly, creating new collections
+- Use OAI sets to filter records shown in a feed, configurable for each feed
+- Work easily with relational data (e.g. if an author changes, the publication should also change)
+- Simple and robust authentication through integration with the Apache webserver
+- Serve assets via Apache while still using configurable authentication rules
+
+In the coming period we will be adding more features and updating this page accordingly. 
+
+.. _MOAI History: http://moai.infrae.com/history.html
+.. _Fedora Commons: http://www.fedora.info
+.. _EPrints: http://www.eprints.org
+.. _DSpace: http://www.dspace.org/
 
-MOAI, an Open Access Server Platform for Institutional Repositories
-===================================================================
+Installing MOAI
+===============
 
-MOAI is a platform for aggregating content from different sources, and publishing it through the Open Archive Initiatives protocol for metadata harvesting.
-It's been built for academic institutional repositories dealing with relational metadata and asset files.
+The MOAI Software can be run in any wsgi compliant server. 
 
-What does it do?
-----------------
+MOAI comes with a development server that can be used for testing. In production mod_wsgi can be used to run Moai in the apache webserver.
 
-The MOAI software can aggregate content from different sources, transform it and store it in a database. The contents of this database can then be published in many separate OAI feeds, each with its own configuration. 
+Installation Steps
+==================
 
-The MOAI software has a very flexible system for combining records into sets, and can use these sets in the feed configuration. It also comes with a simple yet flexible authentication scheme that can be easily customized. Besides providing authentication for the feeds, the authentication also controls the access to the assets. 
+MOAI is a normal python package. It is tested with python2.5 and 2.6. 
+I recommend creating a virtualenv to install the package in.
 
-Why MOAI
---------
+http://pypi.python.org/pypi/virtualenv/
 
-MOAI has been specifically developed for universities, and contains a lot of hard-earned wisdom. The software has been in production use since 2007, and new features have been continually added. In late 2008, the software was completely refactored and packaged under the name "MOAI". You can read more about this on the `MOAI History`_ page.
+This makes development and deployment easier.
+Instructions below are for unix, but MOAI should also work on Windows
 
-MOAI is a standalone system, so it can be used in combination with any repository software that comes with an OAI feed such as `Fedora Commons`_, `EPrints`_ or `DSpace`_. It can also be used directly with an SQL database or just a folder of XML files.
+Go into the MOAI directory with the setup.py, and run the virtualenv command
 
-The MOAI project takes the philosophy that every repository is different and unique, and that an institutional repository is a living thing. It is therefore never finished. Metadata is always changes, improving, and evolves. We think this is healthy. 
+> cd moai
+> virtualenv .
 
-Because of this viewpoint, the MOAI software makes it as easy as possible to add or modify parts of your repository (OAI) services stack. It tries to do this without sacrificing power, and encouraging the re-use of components.
+Now, activate the virtualenv
 
-Features
---------
+> source bin/activate
 
-MOAI has some interesting features not found in most OAI servers. 
-Besides serving OAI, it can also harvest OAI. This makes it possible for MOAI to work as a pipe, where the OAI data can be reconfigured, cached, and enriched while it passes through the MOAI processing.
+Install MOAI in the virtualenv using pip
 
+> pip install -e .
 
-More specifically MOAI has the ability to:
+(this will take a while)
 
-- Harvest data from different kinds of sources
-- Serve many OAI feeds from one MOAI server, each with their own configuration
-- Turn metadata values into OAI sets on the fly, creating new collections
-- Use OAI sets to filter records shown in a feed, configurable for each feed
-- Work easily with relational data (e.g. if an author changes, the publication should also change)
-- Simple and robust authentication through integration with the Apache webserver
-- Serve assets via Apache while still using configurable authentication rules
+When this process finishes, Moai and all its dependencies will be installed.
 
-In the coming period we will be adding more features and updating this page accordingly. 
+Running in development mode
+===========================
 
-.. _MOAI History: http://moai.infrae.com/history.html
-.. _Fedora Commons: http://www.fedora.info
-.. _EPrints: http://www.eprints.org
-.. _DSpace: http://www.dspace.org/
+The development server should never be used for production, it is convenient for testing and development. Note that you should always activate the virtualenv otherwise the dependecies will not be found
+
+> cd moai
+> source bin/activate
+> ./bin/paster serve settings.ini
+
+This will print something like:
+
+  Starting server in PID 7306.
+  Starting HTTP server on http://127.0.0.1:8080
+
+You can now visit localhost:8080/oai to view the moai oaipmh feed. 
+
+Configuring MOAI
+================
+
+Configuration is done in the settings.ini file. The default settings file uses the Paste#urlmap application to map wsgi applications to a url.
+
+In the `composite:main` section is a line:
+
+/oai = moai_example
+
+Which maps the /oai url to a Moai instance. 
+This makes it easy to run many Maoi instances in one server, each with it's own configuration.
+
+The app:moai_example configuration let's you specify the following options:
+
+name:              The name of the oai feed (returned in Identify verb)
+url:               The url of the oai feed (returned in oaipmh xml output)
+admin_email:       The email adress of the amdin  (returned in Identify verb)
+
+formats:           Available metadata formats
+disallow_sets:     List of setspecs that are not allowed in the output of this
+                   feed
+allow_sets:        If used, only sets listed here will be returned
+
+database:          SQLAlchemy uri to identify the database for used for storage
+provider:          Provider identifier where moai retrieves content from
+content:           Class that maps metadata from provider format to moai format
+
+Adding Content
+==============
+
+The Moai system is designed to fetch periodically fetch content from a `provider`, and convert that to Moais internal format, which can then be translated in the different metadata formats from the oaipmh feed.
+
+Moai comes with an example that shows this principle:
+
+In the moai/moai directory are two XML files. Let's pretend these files are from a remote system, and we want to publish them with MOAI.
+
+In the settings.ini file, the following option is specified:
+
+provider = file://moai/example-*.xml
+
+This tells moai that we want to use a file provider, with some files located in moai/example-*.xml. 
+
+The following option points to the class that we want to use for converting the example content xml data to Moais internal format.
+
+content = moai_example
+
+The last option tells Moai where to store it's data, this is usually a sqlite database:
+
+database = sqlite:///moai-example.db
+
+Now let's try to add these two xml files, let's first visit the oaipmh feed to make sure nothing is allready being served:
+
+http://localhost:8080/oai?verb=ListRecords&metadataPrefix=oai_dc
+
+This should return a nowRecordsMatch error.
+
+To add the content, run the update_content script, with the section name from the settings.ini as argument
+
+> ./bin/update_content moai_example
+
+This will produce the following output:
+
+/ Updating content provider: example-2345.xml                                   
+Content provider returned 2 new/modified objects
+
+100.0%[====================================================================>] 2
+Updating database with 2 objects took 0 seconds
+
+Now when you visit the oaipmh feed again you should see the two records:
+
+http://localhost:8080/oai?verb=ListRecords&metadataPrefix=oai_dc
+
+When you run the update_moai script again, it will create a new database with all the records (in this case moai_example.db). It is also possible to specify a data with the --date switch. When a data is specified, only records that were modified after this date will be added. 
+The update_moai script can be run from a daily or hourly cron job to update the database
+
+Adding your own Provider / Content and Metadata Classes
+=======================================================
+
+It's possible and most of the time, needed, to extend Moai for your use-cases.
+The Provider and Content classes from the example might be a good starting point if you want to do that. All your customizations should be registered with Moai through `entry_points`. Have a look at Moais setup.py for more information.
+The best approach would be to create your own python package with setup.py and install this in the same environment as Moai. This will let Moai find your customizations. Note that when you change something in your setup.py, you have to reinstall the package, for Moai to pick up the changes.
+
+Note that the moai.interfaces file contains documentation about the different classes that you can implement.
+
+Adding your own Database
+========================
+
+Instead of writing your own provider/content classes, you can also register your own custom database. Implementing a replacement for moai.database.SQLDatabase can be more complicated then writing a provider/content class, but it has the advantage that Moai is always up to date, and you don't need a second sqlite database.
+
+
+
+
+
+

moai/provider/fedora.py

 import hashlib
 
 from lxml import etree
-from zope.interface import implements
 
 from moai.provider.oai import OAIBasedContentProvider
-from moai.interfaces import IContentProvider
 
 class FOXMLFile(object):
 
     Implements the :ref:`IContentProvider` interface
     """
     
-    implements(IContentProvider)
-    
-    
     def __init__(self, fedora_url, output_path,
                  datastream_name=None, username=None, password=None):
         oai_url = '%s/oai' % fedora_url
             self._log.warning('HTTP %s -> Can not get Fedora data: %s' % (err.code, url))
             return False
 
-        directory = md5.new(fedora_id).hexdigest()[:3]
+        directory = hashlib.md5(fedora_id).hexdigest()[:3]
 
         path = os.path.join(self._path, directory)
         if not os.path.isdir(path):

moai/provider/file.py

 import time
 import fnmatch
 
-from zope.interface import implements
-
-from moai.interfaces import IContentProvider
 
 class FileBasedContentProvider(object):
     """Provides content by reading directories of files
     Implements the :ref:`IContentProvider` interface
     """
-    implements(IContentProvider)
-
     def __init__(self, uri, content_filter="*"):
         assert uri.startswith('file://'), 'unknown uri format'
         path = uri[7:]

moai/provider/list.py

 import time
 
-from zope.interface import implements
-
-from moai.interfaces import IContentProvider
-
 class ListBasedContentProvider(object):
     """Provides content from a python list,
     implementation of :ref:`IContentProvider`"""
-    
-    implements(IContentProvider)
 
     def __init__(self, content):
         self._content = content
 import os
 import tempfile
 
-from zope.interface import implements
 import oaipmh.error
 
-from moai.interfaces import IServer, IFeedConfig
 from moai.oai import OAIServerFactory, OAIServer
 
 class Server(object):
     Developers might want to subclass this, to provide custom
     asset handling in their implementation.
     """
-
-    implements(IServer)
-
     def __init__(self, base_url, db, config):
         self.base_url = base_url
         self._db = db
     """The feedconfig object contains all the settings for a specific
     feed. It implements the :ref:`IFeedConfig` interface.
     """
-
-    implements(IFeedConfig)
-
     def __init__(self,
                  repository_name,
                  base_url,
 from moai.utils import (get_duration,
                         get_moai_log,
                         ProgressBar)
-from moai.database import Database
+from moai.database import SQLDatabase
 
 VERSION = pkg_resources.working_set.by_key['moai'].version
                  
     else:
         from_date = None
         
-    database = Database(config['database'])
+    database = SQLDatabase(config['database'])
     for content_point in iter_entry_points(group='moai.content',
                                            name=config['content']):
         ContentClass = content_point.load()
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.