Commits

Thomas Waldmann committed 392f649

move most docs from README.txt to package/module docstrings

Comments (0)

Files changed (13)

 It is intended as a incompatible replacement for the current stuff you see there:
 https://bitbucket.org/thomaswaldmann/moin-2.0/src/tip/MoinMoin/storage/
 
-
-Storage Layers
-==============
-We use a layered approach like this::
-
- Indexing Middleware   does complex stuff like indexing, searching, listing,
-  |                    lookup by name, ACL checks, ...
-  v
- Router Middleware     dispatches to multiple backends based on the name,
-  |          |         cares about absolute and relative names
-  v          v
- Backend1   Backend2   simple stuff: store, get, destroy revisions
-
-
-Indexing Middleware
-===================
-This is the level that does complex stuff with the help of Whoosh (a fast
-pure-Python indexing and search library).
-
-Using Whoosh we build, maintain and use 2 indexes:
-
-* "all revisions" index (big, needed for history search)
-* "latest revisions" index (smaller, just the current revisions)
-
-When creating or destroying revisions, indexes are automatically updated.
-
-There is also code to do a full index rebuild in case it gets damaged, lost
-or needs rebuilding for other reasons. There is also index update code to
-do a quick "intelligent" update of a "mostly ok" index, that just adds,
-updates, deletes stuff that is different in backend compared to current index.
-
-Indexing is the only layer that can easily deal with **names** (it can
-easily translate names to UUIDs and vice versa) and with **items** (it
-knows current revision, it can easily list and order historial revisions),
-using the index.
-
-The layers below are using UUIDs to identify revisions meta and data:
-
-* revid (metaid) - a UUID identifying a specific revision (revision metadata)
-* dataid - a UUID identifying some specific revision data (optional), it is
-  just stored into revision metadata.
-* itemid - a UUID identifying an item (== a set of revisions), it is just
-  stored into revision metadata. itemid is only easily usable on indexing
-  level.
-
-Many methods provided by the indexing middleware will be fast, because they
-will not access the layers below (like the backend), but just the index files,
-usually it is even just the small and thus quick latest-revs index.
-
-Indexing Middleware also checks ACLs, so a user will not see items in search
-results that he is not allowed to read. Also, trying to access a revision
-without read permission will give an AccessDenied exception.
-
-Note: The currently used "ACL implementation" is just a proof-of-concept
-quickhack to enable testing the acl checks and will be replaced by a sane
-one later.
-
-
-Router Middleware
-=================
-Just think of UNIX fstab and mount.
-
-Lets you mount backends that store items belonging to some specific part
-of the namespace. Router middleware has same API as a backend.
-
-
-"stores" Backend
-================
-This is a backend that ties together 2 key/value stores:
-
-* meta store
-  - key = revid UUID (bytes, ascii)
-  - value = bytes (bytes, utf-8)
-* data store
-  - key = dataid UUID (bytes, ascii)
-  - value = file (gets/returns open file instances, to read/write binary data)
-
-Already implemented key/value stores:
-
-* fs (stores into filesystem)
-* sqlite (stores into sqlite3 table, single db file in the filesystem)
-* kc (kyoto cabinet, single db file in the filesystem, local and single process ONLY!)
-* kt (kyoto tycoon, network server for kyoto cabinet, remote or multi-process usage possible)
-* memory (stores into RAM, non-persistent!)
-* memcached (talks to a memcached server, non-persistent!)
-
-Likely one can also use many other existing k/v stores with very little code.
-
-
-"fileserver" Backend
-====================
-This is a read-only backend that exposes a part of the filesystem:
-
-* files show as single revision items
-
-  - metadata is made up from fs metadata + mimetype guessing
-  - data is read from the file
-
-* directories create a virtual directory item
-
+See the package/module docstring for more information.

storage/__init__.py

 
 """
 MoinMoin - storage subsystem
+============================
+
+We use a layered approach like this::
+
+ Indexing Middleware               does complex stuff like indexing, searching,
+ |                                 listing, lookup by name, ACL checks, ...
+ v
+ Routing  Middleware               dispatches to multiple backends based on the
+ |                 |               name, cares about absolute and relative names
+ v                 v
+ "stores" Backend  Other Backend   simple stuff: store, get, destroy revisions
+ |           |
+ v           v
+ meta store  data store            simplest stuff: store, get, destroy and iterate
+                                   over key/value pairs
 """
 

storage/backends/fileserver.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - backend exposing part of the filesystem (read-only)
+MoinMoin - fileserver backend, exposing part of the filesystem (read-only)
+
+Files show as single revision items.
+
+  - metadata is made up from fs metadata + mimetype guessing
+  - data is read from the file
+
+Directories create a virtual directory item, listing the files in that
+directory.
 """
 
 

storage/backends/stores.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - backend using 2 stores
+MoinMoin - backend that ties together 2 key/value stores
 
-Usually, this will be a ByteStore for meta and a FileStore for data.
+A meta store (a ByteStore):
 
-But, you could also use other combinations, you just need to be prepared
-for the revision data datatype it returns (that is exactly what the data_store
-returns) and also feed it with the correct revision data datatype (which
-is what the data_store accepts).
+- key = revid UUID (bytes, ascii)
+- value = bytes (bytes, utf-8)
+
+A data store (a FileStore):
+
+- key = dataid UUID (bytes, ascii)
+- value = file (gets/returns open file instances, to read/write binary data)
+
+See the stores package for already implemented key/value stores.
 """
 
 

storage/middleware/indexing.py

 
 We also check ACLs here. Index has ALL content, so we must be careful not
 to show data from index to a user that is not allowed to read that data.
+
+Using Whoosh (a fast pure-Python indexing and search library), we build,
+maintain and use 2 indexes:
+
+* "all revisions" index (big, needed for history search)
+* "latest revisions" index (smaller, just the current revisions)
+
+When creating or destroying revisions, indexes are automatically updated.
+
+There is also code to do a full index rebuild in case it gets damaged, lost
+or needs rebuilding for other reasons. There is also index update code to
+do a quick "intelligent" update of a "mostly ok" index, that just adds,
+updates, deletes stuff that is different in backend compared to current index.
+
+Indexing is the only layer that can easily deal with **names** (it can
+easily translate names to UUIDs and vice versa) and with **items** (it
+knows current revision, it can easily list and order historial revisions),
+using the index.
+
+The layers below are using UUIDs to identify revisions meta and data:
+
+* revid (metaid) - a UUID identifying a specific revision (revision metadata)
+* dataid - a UUID identifying some specific revision data (optional), it is
+  just stored into revision metadata.
+* itemid - a UUID identifying an item (== a set of revisions), it is just
+  stored into revision metadata. itemid is only easily usable on indexing
+  level.
+
+Many methods provided by the indexing middleware will be fast, because they
+will not access the layers below (like the backend), but just the index files,
+usually it is even just the small and thus quick latest-revs index.
+
+Indexing Middleware also checks ACLs, so a user will not see items in search
+results that he is not allowed to read. Also, trying to access a revision
+without read permission will give an AccessDenied exception.
 """
 
 
         return self.itemid is not None
 
     def allows(self, capability):
-        # just a temporary hack to be able to test this without real ACL code
+        # TODO: this is just a temporary hack to be able to test this without real ACL code,
+        # replace it by a sane one later.
         # e.g. acl = "joe:read"  --> user joe may read
         if not self.indexer.acl_support:
             return True

storage/middleware/routing.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - router middleware
+MoinMoin - routing middleware
 
 Routes requests to different backends depending on the item name.
+
+Just think of UNIX filesystems, fstab and mount.
+
+This middleware lets you mount backends that store items belonging to some
+specific part of the namespace. Routing middleware has same API as a backend.
 """
 
 

storage/stores/__init__.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - store base classes
+MoinMoin - simple key/value stores.
+
+If some kvstore implementation you'ld like to use is missing from this package,
+you can likely implement it adding very little and rather easy code.
 """
 
 

storage/stores/fs.py

 
 """
 MoinMoin - filesystem store
+
+Store into filesystem, one file per k/v pair.
 """
 
 

storage/stores/kc.py

 """
 MoinMoin - kyoto cabinet store
 
+Stores k/v pairs into a single kyoto cabinet file in the filesystem.
+
 Note: only ONE process can open a kyoto cabinet in OWRITER (writable) mode.
       Multithreading is allowed, but not multi-process.
       

storage/stores/kt.py

 
 """
 MoinMoin - kyoto tycoon store
+
+Stores k/v pairs into a Kyoto Tycoon server. Kyoto Tycoon is a network server
+for kyoto cabinet, remote or multi-process usage is possible).
 """
 
 

storage/stores/memcached.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - memcached "store" (rather a CACHE, non-persistent, in RAM)
+MoinMoin - memcached store
 
-Note: does not support iteration.
+This stores k/v pairs into a server using the memcached protocol.
+
+Note:
+
+- does not support iteration.
+- if the remote server uses memory (RAM), this is non-persistent!
+  memcached uses RAM, but there might be other servers using same protocol
+  using a persistent storage.
 """
 
 

storage/stores/memory.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - memory store (based on a dict)
+MoinMoin - memory store
+
+Stores k/v pairs into memory (RAM, non-persistent!).
+
+Note: likely this is mostly useful for unit tests.
 """
 
 

storage/stores/sqlite.py

 # License: GNU GPL v2 (or any later version), see LICENSE.txt for details.
 
 """
-MoinMoin - sqlite3 key/value store (optionally with zlib/"gzip" compression)
+MoinMoin - sqlite3 key/value store
+
+This store stores into sqlite3 table, using a single db file in the filesystem.
+You can use the same db file for multiple stores, just using a different table
+name.
+
+Optionally, you can use zlib/"gzip" compression.
 """