codernity avatar codernity committed 0a9365a

Documentation updated

Comments (0)

Files changed (7)

 .settings/
 .*\.swp
 htmlcov_.*
+coverage.*\.xml
 docs/_build/*
-coverage.*\.xml
 junit-.*\.xml
 .*\.orig
Add a comment to this file

docs/CodernityDB_HTTP.png

Added
New image

docs/codes/tables_like_indexes.py

+class AllUsers(HashIndex):
+
+    def __init__(self, *args, **kwargs):
+        kwargs['key_format'] = '16s'
+        super(AllUsers, self).__init__(*args, **kwargs)
+
+    def make_key(self, key):
+        return md5(key).digest()
+
+    def make_key_value(self, data):
+        if data.get('_t') == 'user':
+            return md5(data.get('name')).digest(), None
+
+
+class AllItems(HashIndex):
+
+    def __init__(self, *args, **kwargs):
+        kwargs['key_format'] = '16s'
+        super(AllUsers, self).__init__(*args, **kwargs)
+
+    def make_key(self, key):
+        return md5(key).digest()
+
+    def make_key_value(self, data):
+        if data.get('_t') == 'item':
+            return md5(data.get('owner')).digest(), None

docs/database_indexes.rst

 Database indexes
 ================
 
+.. note::
+
+    At first you should read :ref:`what's index in design section<database_design_index>`. 
+
 
 Currently there are two main Index implementations.
 You need to know the difference between them before you will choose the base for your Index.
 
 .. seealso::
 
-    :ref:`internal_index_functions` for description of powerfull mechanizm of inside index (database) functions.
+    :ref:`internal_index_functions` for description of powerful mechanism of inside index (database) functions.
 
 Using custom index is quite simple. You need to decide if your index needs to be Hash based (:ref:`details <internal_hash_index>`) or Tree based (:ref:`details <internal_tree_index>` for
 pros and cons)
 
 custom_header
     It's string that will be inserted to final index file. It's
-    usefull to pass the custom imports there. You will find an example
+    useful to pass the custom imports there. You will find an example
     in :ref:`Examples - secure storage <secure_storage_example>`
 
 storage_class
     into list, then traversed when needed.
 
 duplicate keys
-   For duplicate keys the same mechanizm is used as for
+   For duplicate keys the same mechanism is used as for
    :ref:`conflict resolution <conflict_resolution>`. All indexes different than *id* one can
    accept more than one record with the same key
    (:py:meth:`~CodernityDB.database.Database.get_many`).
     An example code for Md5 based index can be found :ref:`design`,
     more examples in :ref:`examples`
 
-    .. note:: For format specification and explaination please visit
+    .. note:: For format specification and explanation please visit
         `Python struct documentation`_
 
 
     Current default is ``0xfffff`` which means ``1048575`` different
     hash function results.
 
-    .. hint:: In perfect conditions you will be abble to store those
+    .. hint:: In perfect conditions you will be able to store those
         number of unique records without conflicts, in practice you
-        will be abble to store like ``1200`` records without conflict
+        will be able to store like ``1200`` records without conflict
         with 50% probability (for example `birthday problem`_). Lookup
         when conflict occurs is slower because linked list is
         traversed. More informations about conflicts :ref:`Hash Index
     That function is called by database when inserting new or updating
     objects in database.  It **has** to return ``None`` if index is
     not matched (not required to operate on it) and 2 values if index
-    is mached. That 2 values are in order: *key* and *value*. Please
+    is matched. That 2 values are in order: *key* and *value*. Please
     remember that key must fit your :py:attr:`entry_line_format`.
 
 
 ---------------
 
 
-Quite important thing in CodernityDB are index functions. You can do with them anything you want they have access to database object, so they can perform operations on multiple indexes. If you want join like operation, you should write function. Then you will be abble to run that function database side when using |CodernityDB-HTTP-link|. The only mandatory argument for that kind of function is ``db``, the rest are function arguments.
+Quite important thing in CodernityDB are index functions. You can do with them anything you want they have access to database object, so they can perform operations on multiple indexes. If you want join like operation, you should write function. Then you will be able to run that function database side when using |CodernityDB-HTTP-link|. The only mandatory argument for that kind of function is ``db``, the rest are function arguments.
 
 
-Writting function is easy see an example there:
+Writing function is easy see an example there:
 
 .. code-block:: python
 
 
 
 .. _BPlus tree: http://en.wikipedia.org/wiki/B%2B_tree
+
+
+
+.. _simple_index:
+
+Easier way of creating indexes
+------------------------------
+
+    Do I really need to code indexes in Python, is there an easier way to create indexes?
+
+You bet there is! We prepared special, simplified mode for creating indexes. That code will be translated to Python code, and then used exactly in the same way as the rest indexes (so that simple indexes are exactly the same fast as pure Python ones). They are just simple by name not by possibilities.
+
+.. note:: 
+    Don't be surprised, if we will name it SimpleIndex in several places there.
+
+Usage of this mode is really basic, you just need to provide 2 (or optionally 3) things: 
+
+* properties of the index like this:
+    
+::
+    
+    name = MyTestIndex
+    type = HashIndex
+    key_format = I
+    etc. etc. ...
+
+* body for *make_key_value* containing our simplified syntax:
+   
+::
+ 
+    make_key_value:
+    a > 1: 1, None
+    a, None
+
+* and optionally body for *make_key* (if you don't provide it, it will be generated automaticlly and set to return key value as it is):
+    
+::
+
+    make_key:
+    key > 1: 1
+    key
+
+Syntax of function body is really basic, every line preceded by a statement with colon at the end means that 
+if conditions before colon are met, function will return everything that follows colon. If there is no colon, 
+value will be always returned. Of course the order of lines actually matters, so if you provide body like that:
+
+::
+
+    a > 1: 1, None
+    a > 2: 2, None
+
+The second value will be never returned (because if *a* is less then 1 it's for sure less than 2).
+Every name will be look for in dictionaries given to functions, so body like:
+
+::
+    
+    a > 1: a, None
+
+will generate python code like that:
+
+.. code-block:: python
+    
+    if data["a"] > 1:
+        return data["a"], None
+
+That's everything you need to know to work with our simplified index creator, you just need to alway provide *name* and *type* in index properties
+and provide body for make_key_value, which has to return always two values (the 2nd has to be a dictionary or None). 
+Here you have some examples and their equivalents in python. Remember that this simplified creator doesn't provide python power,
+so if you want to write more sophisticated index, you will have to learn python.
+
+
+::
+
+    name = Test
+    type = HashIndex
+    key_format = 16s
+
+    make_key_value:
+    md5(a),None
+
+    make_key:
+    md5(key)
+
+is an equivalent to: 
+
+.. code-block:: python
+
+    class Test( HashIndex ):
+        def __init__(self,*args,**kwargs):         
+            kwargs["key_format"] = '16s' 
+            super( Test ,self).__init__(*args,**kwargs)
+        def make_key_value(self,data):         
+            return md5 ( data["a"] ) .digest() , None 
+        def make_key(self,key): 
+            return md5 ( key ) .digest()
+
+
+Keywords & Helpers in simple index
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Please note that Python ``None`` value can be written as: ``none``, ``None``, ``null``.
+
+Supported logic operators:
+
+* ``and``
+* ``&&``
+* ``or``
+* ``||``
+
+Functions that you can use in ``make_key`` and ``make_key_value``:
+
+.. method:: str(value)
+
+    it will convert value to string
+
+    :param value: value to convert to string
+    :returns: string value of ``value``
+    :rtype: string
+
+.. method:: len(value) 
+
+    it will return length of a value
+
+    :param value: a value to check length
+    :returns: length of value
+    :rtype: integer
+
+.. method:: md5(value)
+
+    it will return md5 value of a string
+    
+    :param string value: string value to get md5 from it
+    :returns: md5 sum of value
+    :rtype: string
+
+.. method::  fix_r(value, length)
+
+    it will return fixed string length. ``Value`` shorter than ``length``
+    will be right filled with ``_``.
+
+    :param string value: value to adjust string length
+    :param integer length: how long should be output string
+    :returns: fixed size string
+    :rtype: string
+
+.. note::
+    Obviously you can use that simple indexes in |CodernityDB-HTTP-link| without any problem.
+
+.. note::
+    Error reporting / handling system in that mode will tell you exactly what's wrong with your code.
+
+
+.. _tables_colections_q:
+
+Tables, collections...?
+-------------------------
+
+    Ok, I got it, but can I store more than one data type in Database. Is there something like table or collection ?
+
+.. note::
+
+   In |CodernityDB-demos| you can find minitwit example which is rewrite from Sqlite application.
+
+Sure! You can use Index mechanism do to it. As it has been mentioned before, Index mechanism in CodernityDB is like read only Table in SQL databases (see :ref:`index design <database_design_index>`). So all you need is to define how your records will differ each other.
+
+Let's assume that you want to users and users and some data that belongs to user. You will probably want to be able to get all users, and all things that belongs to him, right? So.
+
+.. literalinclude:: codes/tables_like_indexes.py
+   :linenos:
+
+
+Having that indexes in your database will allow you to query for single user and for items of that user. Isn't it simple?
+As you can see, index in CodernityDB is not an index that you probably get used to. It's much more.
+
+How an index code is processed by CodernityDB?
+-----------------------------------------------
+
+    When you provide CodernityDB with an index, it uses `getsource <http://docs.python.org/2/library/inspect.html#inspect.getsource>`_ function from `inspect module <http://docs.python.org/2/library/inspect.html>`_ to get index code. This means, that after you call add_index function, it will look for the index class in current scope, take the whole class code as it is (including intends) and place it inside it's own code. Hence there are few cons you have to bear in mind:
+
+* You can not generate class code on the fly inside, let's say, a function, like that:
+    
+.. code-block:: python
+
+    def create_index(self,a):
+        class MyIndex(HashIndex):
+            def __init__(self, *args, **kwargs):
+                kwargs['key_format'] = 'I'
+                super(WithXIndex, self).__init__(*args, **kwargs)
+            def make_key_value(self,data):
+                if data['a'] == a:
+                    return data['b'], None
+            def make_key(self,key):
+                return key
+        return MyIndex
+
+Despite of code being correct in python terms, it will produce an error in CodernityDB, since class isn't defined in proper scope. 
+
+    * You can not provide index class code with a variable defined outside this class:
+
+.. code-block:: python
+    
+    a = 5
+    class MyIndex(HashIndex):
+        def __init__(self, *args, **kwargs):
+            kwargs['key_format'] = 'I'
+            super(WithXIndex, self).__init__(*args, **kwargs)
+        def make_key_value(self,data):
+            if data['a'] == a:
+                return data['b'], None
+        def make_key(self,key):
+                return key
+
+Even if now class is in proper scope, the example won't work, because variable *a* isn't known to CodernityDB.
 
 
 
+.. _database_design_index:
+
 Index
 ^^^^^
 
 
     If you want to search / query database for given parameters /
     values etc., you have to specify the index that will return
-    key/value from it. Please see :ref:`examples` and :ref:`database_indexes`
+    key/value from it. Please see :ref:`examples` and :ref:`database_indexes`.
 
 By index we call the class in Python language that was added to the
-Database. It can be compared to SQL table (but read only), to update
+Database. It can be compared to SQL table (read only), to update
 you always need to pass full object to database, our Indexes can be
-compared also with CouchDB views mechanizm.
+compared also with CouchDB views mechanizm. (you would like probably to see :ref:`simple_index`). You can have as much indexes as you want and single record in database can "exists" in more than one index.
 
 Index itself does not store any information except it's
 *metadata*. You don't have to copy full data everytime in indexes,
 Can I add index to existing DB ?
     Yes you can, but you will need to reindex that index to have in it data that were in database already before you add that index. (see :ref:`database_indexes` for details)
 
+What about tables or collections ?
+    Everything can be done through our Index mechanism see :ref:`tables_colections_q`.
+
+How does it compare to MongoDB, CouchDB and other "big" NoSQL databases ?
+    It does not compare. Different purposes, different design, different use case. For some projects it makes sense to use CodernityDB instead of MongoDB, CouchDB and others, for some other not.
+
 Why Python 3 is not supported ?
-    Python 3 introduced many incompatible changes. In case of CodernityDB having working version for 2.x and 3.x series in the same codebase without ugly hacks (big monkey-patching etc) is almost impossible. If you're interested Python 3 version of CodernityDB contact us. Porting CodernityDB to Python 3.x is not hard. Python 3.x support in fact was never needed. That's why there is no support for it (yet?).
+    Python 3 introduced many incompatible changes. In case of CodernityDB having working version for 2.x and 3.x series in the same codebase without ugly hacks (big monkey-patching etc.) is almost impossible. If you're interested Python 3 version of CodernityDB contact us. Porting CodernityDB to Python 3.x is not hard. Python 3.x support in fact was never needed. That's why there is no support for it (yet?).
 
 I want to contribute, what can I do?
     Just fork and create pull request |bitbucket_link|.
 
 I want to have a bit customized CodernityDB
     No problem, just contact us to get more details about that.
+
+I want to use CodernityDB in commercial project, do I have to pay for it?
+    CodernityDB is released on `Apache 2.0 license`_, it allows you to freely use it even for commercial purposes without any need to pay for it. IT'S FREE FOR COMMERCIAL USE. 
+
+
+.. _Apache 2.0 license: http://www.apache.org/licenses/LICENSE-2.0.html
 * Native Python database
 * Multiple indexes
 * Fast (more than 50 000 insert operations per second see :ref:`speed` for details)
-* Embeded mode (default) and :ref:`Server<server>`, with client library that aims to be 100% compatible with embedded one.
+* Embedded mode (default) and :ref:`Server<server>`, with client library that aims to be 100% compatible with embedded one.
 * Easy way to implement custom Storage (see :ref:`example_storage`)
 
 
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.