Commits

Thomas Waldmann  committed 490222d

update some docs

  • Participants
  • Parent commits f3b7fe1
  • Branches storage-ng

Comments (0)

Files changed (5)

File docs/admin/backup.rst

 To create a dump of all data stored in moin (wiki items, user profiles), do
 this::
 
- moin maint_xml --save --file backup.xml
+ moin save --file backup.moin
 
-Please note that backup.xml contains sensitive data (like user profiles, wiki
-contents). Wiki item revision data is stored encoded, but it is **not**
-encrypted, so store your backups at a safe place and make sure no unauthorized
+Please note that this file contains sensitive data (like user profiles, wiki
+contents), so store your backups at a safe place and make sure no unauthorized
 persons can access them.
 
 Selective Restore
 =================
 
 Restore all software and configuration files (see above) to their original
-place. Make sure your (empty) wiki works as expected and the moin version is
-the same as when you made the backup.
+place. Make sure your (empty) wiki works as expected::
 
-To load the xml dump into your empty wiki, do this::
+ moin moin -s -i  # -s = create new storage
+                  # -i = create new index
 
- moin maint_xml --load --file backup.xml
+To load the backup file into your empty wiki, do this::
 
+ moin load --file backup.moin
+

File docs/admin/configure.rst

   parts of the namespace:
 
   - content
-  - trash
   - userprofiles
 * to configure ACLs protecting these parts of the namespace
 * to setup a router middleware that dispatches to these parts of the namespace
 
 Call it like::
 
-    from MoinMoin.storage.backends import create_simple_mapping
+    from MoinMoin.storage import create_simple_mapping
 
-    namespace_mapping = create_simple_mapping(
-        backend_uri=...,
+    namespace_mapping, acl_mapping = create_simple_mapping(
+        uri=...,
         content_acl=dict(before=...,
                          default=...,
                          after=...,
                               after=..., ),
     )
 
-The `backend_uri` depends on the kind of storage backend you want to use (see
-below). Usually it is a URL-like string that looks like::
+The `uri` depends on the kind of storage backend and stores you want to use
+(see below). Usually it is a URL-like string that looks like::
 
-    fs2:/srv/mywiki/%(nsname)s
+    stores:fs:/srv/mywiki/%(nsname)s/%%(kind)s
     
-`fs2` is the name of the backend, followed by a colon, followed by a backend
-specific part that may include a `%(nsname)s` placeholder which gets replaced
-by 'content', 'trash' or 'userprofiles' for the respective backend.
+`stores` is the name of the backend, followed by a colon, followed by a store
+specification. `fs` is the name of the store, followed by a specification
+that makes sense for the fs (filesystem) store (== a path with placeholders).
+
+`%(nsname)s` placeholder will be replaced 'content' or 'userprofiles' for
+the respective backend. `%%(kind)s` will be replaced by 'meta' or 'data'
+later.
 
 In this case, the mapping created will look like this:
 
 +----------------+-----------------------------+
 | /              | /srv/mywiki/content/        |
 +----------------+-----------------------------+
-| /Trash/        | /srv/mywiki/trash/          |
-+----------------+-----------------------------+
 | /UserProfiles/ | /srv/mywiki/userprofiles/   |
 +----------------+-----------------------------+
 
-`content_acl` is a dictionary specifying the ACLs for this part of the
-namespace (the normal content). See the docs about ACLs.
+`content_acl` and `user_profile_acl` are dictionaries specifying the ACLs for
+this part of the namespace (normal content, user profiles).
+See the docs about ACLs.
 
-acl middleware
---------------
+protecting middleware
+---------------------
 Features:
 
-* protects access to lower storage layers by Access Control Lists
+* protects access to lower storage layers by ACLs (Access Control Lists)
 * makes sure there won't be ACL security issues, even if upper layers have bugs
 * if you use create_simple_mapping, you just give the ACL parameters, the
   middleware will be set up automatically by moin.
 
-router middleware
------------------
+routing middleware
+------------------
 Features:
 
 * dispatches storage access to different backends depending on the item name
 * if you use create_simple_mapping, the router middleware will be set up
   automatically by moin.
 
-indexing mixin
---------------
+indexing middleware
+-------------------
 Features:
 
 * maintains an index for important metadata values
 * speeds up looking up / selecting items
 * makes it possible that lower storage layers can be simpler
-* if you use create_simple_mapping, the indexing will be set up automatically
-  by moin.
+* the indexing middleware will be set up automatically by moin.
 
-fs2 backend
------------
+stores backend
+--------------
+This is a backend that ties together 2 stores (one for meta, one for data) to
+form a backend.
+
+fs store
+--------
 Features:
 
 * stores into the filesystem
 * store metadata and data into separate files/directories
-* uses content-hash addressing for revision data files
-
-  - this automatically de-duplicates revision data with same content within the
-    whole backend!
 
 Configuration::
 
-    from MoinMoin.storage.backends import create_simple_mapping
+    from MoinMoin.storage import create_simple_mapping
 
     data_dir = '/srv/mywiki/data'
-    namespace_mapping = create_simple_mapping(
-        backend_uri='fs2:%s/%%(nsname)s' % data_dir,
+    namespace_mapping, acl_mapping = create_simple_mapping(
+        uri='stores:fs:%s/%%(nsname)s/%%%%(kind)s' % data_dir,
         content_acl=dict(before=u'WikiAdmin:read,write,create,destroy',
                          default=u'All:read,write,create',
                          after=u'', ),
     )
 
 
-fs backend
+sqla store
 ----------
 Features:
 
-* stores into the filesystem
-* stores meta and data of a revision into single file
-
-`backend_uri` for `create_simple_mapping` looks like::
-
-    fs:/srv/mywiki/data/%(nsname)s
-
-hg backend
-----------
-Features:
-
-* stores data into Mercurial DVCS (hg) - you need to have Mercurial installed
-
-`backend_uri` for `create_simple_mapping` looks like::
-
-    hg:/srv/mywiki/data/%(nsname)s
-
-sqla backend
-------------
-Features:
-
-* stores data into a (SQL) database
-* uses slqalchemy ORM as database abstraction
+* stores data into a (SQL) database / table
+* uses slqalchemy (without ORM) as database abstraction
 * supports multiple types of databases, like:
  
   - sqlite (default, comes built-into Python)
   - mysql
   - and others (see sqlalchemy docs).
 
-`backend_uri` for `create_simple_mapping` looks like e.g.::
+`uri` for `create_simple_mapping` looks like e.g.::
 
-    sqla:sqlite:////srv/mywiki/data/mywiki_%(nsname)s.db
-    sqla:mysql://myuser:mypassword@localhost/mywiki_%(nsname)s
-    sqla:postgres://myuser:mypassword@localhost/mywiki_%(nsname)s
+    stores:sqla:sqlite:////srv/mywiki/data/mywiki_%(nsname)s.db
+    stores:sqla:mysql://myuser:mypassword@localhost/mywiki_%(nsname)s
+    stores:sqla:postgres://myuser:mypassword@localhost/mywiki_%(nsname)s
 
 Please see the sqlalchemy docs about the part after `sqla:`.
 
-In case you use some DBMS (like postgresql or mysql) that does not allow
-creation of new databases on an as-needed basis, you need to create the
-databases 'mywiki_content', 'mywiki_trash', 'mywiki_userprofiles' yourself
-manually.
-
 Grant 'myuser' (his password: 'mypassword') full access to these databases.
 
-.. todo::
 
-   The sqla backend needs more love, more tuning.
+kc store
+--------
+Features:
+
+* uses a Kyoto Cabinet file to store
+* very fast
+* single-process only, local only
+
+.. todo:
+
+   add kc store configuration example
+
+
+kt store
+--------
+Features:
+
+* uses a Kyoto Tycoon server to store
+* fast
+* multi-process, local or remote.
+
+.. todo:
+
+   add kt store configuration example
+
+
+memory store
+--------------
+Features:
+
+* keeps everything in RAM
+* definitely not for production use
+* mostly intended for testing
+* if your system or the moin process crashes, you'll lose everything
+* single process only
+
+.. todo:
+
+   add memory store configuration example
+
 
 fileserver backend
 ------------------
   + directories will show up as index items, listing links to their contents
 * might be useful together with SMBMount pseudo-authenticator
 
-flatfile backend
-----------------
-Features:
-
-* uses flat files for item storage
-* no revisioning
-* no separate metadata, just some stuff at top of the (text) data
-* thus, only suitable for text items
-
-memory backend
---------------
-Features:
-
-* keeps everything in RAM
-* definitely not for production use
-* mostly intended for testing
-* if your system or the moin process crashes, you'll lose everything
-* single process only
-* maybe not threadsafe
-
-fs19 backend
-------------
-Features:
-
-* reads moin 1.9 content and users from the filesystem
-* read-only, only provided for data migration from moin 1.9.x
-* not optimized for speed or resource usage
-
-For more details please see the chapter about upgrading from moin 1.9.
-
-
-.. todo:
-
-   add more backends / more configuration examples
-
 
 Mail configuration
 ==================

File docs/admin/index.rst

 
 General
 =======
-moin strongly relies on indexes that accelerate access to item metadata and data.
+moin strongly relies on indexes that accelerate access to item metadata and
+data and makes it possible to have simple backends, because the index layer
+is doing all the hard and complex work.
+
 Indexes are used internally for many operations like item lookup, history,
 iterating over items, search (also for interactive searches), etc..
 
-So, you need to configure indexing correctly first.
+moin won't be able to start with damaged, inaccessible or non-existing indexes.
+
+So, you need to configure and initialize indexing correctly first.
 
 moin will automatically update the index when items are created, updated, deleted,
-destroyed or renamed (via the storage api of moin):
-* when users change content via the user interface
-* when xml is unserialized to load items into the backend
-
-In case you have items changing without usage of the backend api or in case
-your index gets damaged or lost, you need to manually do an index build -
-or moin won't be able to work correctly.
+destroyed or renamed (via the storage api of moin, indexing layer or above).
 
 Configuration
 =============
-Your wiki config needs ``index_dir`` and ``index_dir_tmp`` to point to different
-directories. They have default values and most likely you don't need to change
-them.
+Your wiki config needs ``index_dir`` to point to a writable directory - a fast,
+local filesystem is preferred.
+Use something like::
 
-But if you want, try something like::
+    index_dir = "/path/to/moin-2.0/wiki/index"
 
-      index_dir = "/path/to/moin-2.0/wiki/index"
-      index_dir_tmp = "/path/to/moin-2.0/wiki/tmp_build"
-
-**Note:** Paths MUST BE absolute.
+**Note:**
+* The path MUST be absolute.
+* Moin will use `index_dir`.temp location also, if you build an index at
+the so-called `temporary location`.
 
 
 moin index script reference
 ===========================
-You can use the ``moin index`` script to build, update, clean, move and monitor
-indexes.
+You can use the ``moin index-*`` group of script commands to manage indexes.
 
-MoinMoin uses 2 indexes: ``latest-revs`` (index stores only current revisions)
-and ``all-revs`` (index stores all revisions).
+Many of the script commands for index management support a `--tmp` option to use
+the temporary index location. This is useful if you want to do index operations
+in parallel to a running wiki which is still using the index at the normal
+index location.
 
-**Note:** If you see <indexname> below, use one of ``latest-revs``, ``all-revs``
-or ``both`` 
+moin index-create
+-----------------
+Creates an empty, but valid index.
 
-Let's see what you can do with that stuff.
+**Note:** the moin wsgi application needs an index to successfully start up.
+As the moin index-* script commands are also based on the moin wsgi application,
+this can lead to "which came first, the chicken or the egg?" like problems.
 
-Build
------
-Index all revisions of all items to the index located in ``index_dir_tmp`` (we
-use this separate location so that index building does not affect the index
-your wiki engine is currently using).
+To solve this, the moin command has a ``-i`` (``--index-create``) option that
+will trigger index creation on startup.
 
-If there is no index at that location yet, a new index will be built there.
-If there is already an index at that location, that index will be extended.
+Additionally, if the storage is also non-existant yet, one might also need
+``-s`` (``--storage-create``) to create an empty storage on startup.
 
-Example::
+moin index-build
+----------------
+Process all revisions of this wiki and add the indexable documents to the index.
 
-    moin index --for <indexname> --action build
+**Note:**
+* For big wikis, this can take rather long, consider using --tmp.
+* index-build does NOT clear the index at the beginning.
+* index-build does not check the current contents of the index, you must not run
+  index-build multiple times for the same data / same wiki.
 
-**Note:** moin won't use this index until you have moved it to ``index_dir``.
+moin index-update
+-----------------
+Compare an index to the current storage contents and update the index as
+needed (add, remove, update stuff) to reflect the current storage contents.
 
-Move
-----
-Move indexes from ``index_dir_tmp`` to ``index_dir``.
+**Note:** You can use this after building at the tmp location, to also get
+the changes that happened to the wiki while building the index. You can run
+index-update multiple times to increasingly catch up.
 
-Example::
+moin index-destroy
+------------------
+Destroy an index (nothing left at the respective location).
 
-    moin index --for <indexname> --action move
+moin index-move
+---------------
+Move the index from the temporary location to the normal location.
 
-Update
-------
-Update the index located in ``index_dir`` to reflect the current backend
-contents. Add new stuff, remove outdated stuff.
+moin index-optimize
+-------------------
+Optimize an index, see Whoosh docs for more details.
 
-Example::
+moin index-dump
+---------------
+Output index contents in human readable form, e.g. for debugging purposes.
 
-    moin index --for <indexname> --action update
-
-Clean
------
-Create empty index in ``index_dir`` for given index (previous will be erased).
-
-Example::
-
-    moin index --for <indexname> --action clean
-
-Show
-----
-Show contents of the index located in ``index_dir`` in human readable form.
-This is mostly used for debugging.
-
-Example::
-
-    moin index --for indexname --action show
-
-**Note:** field length limited to 40 chars.
-
-**Note:** fields without attribute ``stored=True`` are not displayed.
+**Note:** only fields with attribute ``stored=True`` can be displayed.
 
 
 Building an index for a single wiki
 ===================================
-Build index at separate place, move it at right place:
 
-     moin index --for both --action build
-     moin index --for both --action move
+If your wiki is fresh and empty
+-------------------------------
+Use::
+
+    moin index-create --storage-create --index-create
+    moin index-create -s -i  # same, but shorter
+
+Storage and index is now initialized (both empty).
+
+If you add stuff to your wiki, the index will get updated on the fly.
+
+
+If your wiki has data and is shut down
+--------------------------------------
+If index needs a rebuild for some reason (e.g. index lost, index damaged,
+incompatible upgrade, ...), use::
+
+    moin index-create -i
+    moin index-build  # can take a while...
+
+
+If your wiki has data and should stay online
+--------------------------------------------
+Use::
+
+     moin index-create -i --tmp
+     moin index-build --tmp  # can take a while...
+     moin index-update --tmp  # should be quicker, make sure we have 99.x%
+     # better shut down the wiki now (or at least make sure it is not changed)
+     moin index-update --tmp  # make sure we have indexed ALL content - should be even quicker.
+     moin index-move  # instantaneously
+     # start the wiki again (or allow changes now again)
+
+**Note:** Indexing puts load onto your server, so if you like to do regular
+index rebuilds, schedule them at some time when your server is not too busy
+otherwise.
 
 
 Building an index for a wiki farm
 
       interwikiname = u"Sales"
       index_dir = "/path/to/wiki/index"
-      index_dir_tmp = "/path/to/wiki/index_tmp"
 
 wiki config for ``Engineering``::
 
       interwikiname = u"Engineering"
       index_dir = "/path/to/wiki/index"
-      index_dir_tmp = "/path/to/wiki/index_tmp"
 
-Now do the initial index building:
+Now do the initial index building::
 
-     moin index --for both --action build # in Sales virtual env
-     moin index --for both --action build # in Engineering virtual env
-     moin index --for both --action move # you can run it from any virtual env
+     moin index-create -i  # create an empty index
+     # now ADD stuff from all your wikis:
+     moin index-build  # with Sales wiki configuration
+     moin index-build  # with Engineering wiki configuration
 
 Now you should have a shared index for all these wikis.
 
 **Note:** Do not build indexes for multiple wikis in parallel, this is not
 supported.
 
-Building indexes while your wiki is running
-===========================================
-If you want to build an index while your wiki is running, you have to be
-careful not to miss any changes that happen while you build the index.
-
-``moin index --action build`` is made to not interfere with your running wiki.
-So you can run this in parallel without taking your wiki offline.
-Depending on the size of your wiki, index build can take rather long - but it
-doesn't matter as you don't have to take your wiki offline for this.
-
-But: if indexing takes rather long, it can easily happen that content that was
-already put into the index is updated afterwards in the online wiki. So we need
-to do a quick index update while the wiki is offline:
-
-Offline your wiki (or at least make it read-only, so no data in it changes).
-
-``moin index --action move`` to move indexes into place.
-
-``moin index --action update`` to add anything we might have missed otherwise.
-As this is not as much as doing a full index build, this should be rather quick
-(but still: it has to look at every item in your wiki, whether it has been
-updated after the initial index build).
-
-Put your wiki back online again.
-
-**Note:** Indexing puts load onto your server, so if you like to do regular
-index rebuilds, schedule them at some time when your server is not too busy
-otherwise.
-

File docs/admin/install.rst

 That way, you can have all sorts of Pythons in different virtualenv directories
 within your moin2 workdir.
 
-Loading some items
-------------------
-In case you do not want to have a completely empty wiki, you may want to load
-some items into it. We provide some in `contrib/xml` directory and you can load
-them like this::
+
+Entering the virtual env
+------------------------
 
  # enter your virtual environment:
  source env/bin/activate
 
+
+Initializing the storage and the index
+--------------------------------------
+
+ moin create-index -s -i
+
+The -s and -i switches make sure moin creates an (empty) storage and an (empty)
+index.
+
+
+Loading some items
+------------------
+In case you do not want to have a completely empty wiki, you may want to load
+some items into it. We provide some in `contrib/serialized` directory and you
+can load them like this::
+
  # load some example items:
- moin maint_xml --load --file=contrib/xml/preloaded_items.xml
+ moin load --file contrib/serialized/preloaded_items.moin
+
 
 Installing PIL
 ~~~~~~~~~~~~~~

File docs/admin/upgrade.rst

 Configuration::
 
     from os.path import join
-    from MoinMoin.storage.backends import create_simple_mapping
+    from MoinMoin.storage import create_simple_mapping
 
     interwikiname = ... # critical, make sure it is same as in 1.9!
     sitename = ... # same as in 1.9
 
     # think about which backend you will use in the end and configure
     # it here (this is NOT the fs19 backend!):
-    namespace_mapping = \
-        create_simple_mapping(
-            backend_uri='fs2:/some/path/%%(nsname)s',
-            content_acl=dict(before=u'', # acl_rights_before in 1.9
-                             default=u'', # acl_rights_default
-                             after=u'', # acl_rights_after
-                             hierarchic=False), # acl_hierarchic
-            user_profile_acl=dict(before=u'',
-                                  default=u'',
-                                  after=u'',
-                                  hierarchic=False),
-        )
+    # TODO
 
-Exporting your moin 1.9 data to an XML file
--------------------------------------------
-moin2 can use the `fs19` storage backend to access your moin 1.9 content
-(pages, attachments and users). The fs19 backend is a bit more than just
-a backend - it also makes the moin 1.9 data look like moin2 data when
-moin accesses them. To support this, it is essential that you adjust your
-wiki config first, see previous section.
 
-Then, use a **copy** of your 1.9 content, do not point moin2 it at your
-original data::
+Exporting your moin 1.9 data to a file
+--------------------------------------
 
-    moin maint_xml --moin19data=/your/moin19/data --save --file=moin19.xml
+TBD
 
-This will serialize all your moin 1.9 data into moin19.xml.
 
-Note: depending on the size of your wiki, this can take rather long and consume
-about the same amount of additional disk space.
+Importing the file into moin2
+------------------------------
 
-Importing the XML file into moin2
----------------------------------
-Just load moin19.xml into the storage backend you have already configured::
+TBD
 
-    moin maint_xml --load --file=moin19.xml
-
-Note: depending on the size of your wiki, this can take rather long and consume
-about the same amount of additional disk space.
 
 Testing
 -------
 If you find issues with data migration from moin 1.9 to 2, please check the
 moin2 issue tracker.
 
+
 Cleanup
 -------
 If you made a **copy** of your 1.9 content, you can remove that copy now.