Commits

Mike Bayer committed 568de1e

- new dialect development README

Comments (0)

Files changed (2)

+========================
+Developing new Dialects
+========================
+
+.. note::
+
+   When studying this file, it's probably a good idea to also
+   familiarize with the  README.unittests.rst file, which discusses
+   SQLAlchemy's usage and extension of the Nose test runner.
+
+While SQLAlchemy includes many dialects within the core distribution, the
+trend for new dialects should be that they are published as external
+projects.   SQLAlchemy has since version 0.5 featured a "plugin" system
+which allows external dialects to be integrated into SQLAlchemy using
+standard setuptools entry points.  As of version 0.8, this system has
+been enhanced, so that a dialect can also be "plugged in" at runtime.
+
+On the testing side, SQLAlchemy as of 0.8 also includes a "dialect
+compliance suite" that is usable by third party libraries.  There is no
+longer a strong need for a new dialect to run through SQLAlchemy's full
+testing suite, as a large portion of these tests do not have
+dialect-sensitive functionality.  The "dialect compliance suite" should
+be viewed as the primary target for new dialects, and as it continues
+to grow and mature it should become a more thorough and efficient system
+of testing new dialects.
+
+The file structure of a dialect is typically similar to the following::
+
+    sqlalchemy-<dialect>/
+                         setup.py
+                         setup.cfg
+                         run_tests.py
+                         sqlalchemy_<dialect>/
+                                              __init__.py
+                                              base.py
+                                              <dbapi>.py
+                         test/
+                                              __init__.py
+                                              requirements.py
+                                              test_suite.py
+                                              test_<dialect_specific_test>.py
+                                              ...
+
+An example of this structure can be seen in the Access dialect at
+https://bitbucket.org/zzzeek/sqlalchemy-access/.
+
+Key aspects of this file layout include::
+
+* setup.py - should specify setuptools entrypoints, allowing the
+  dialect to be usable from create_engine(), e.g.:
+
+        entry_points={
+         'sqlalchemy.dialects': [
+              'access = sqlalchemy_access.pyodbc:AccessDialect_pyodbc',
+              'access.pyodbc = sqlalchemy_access.pyodbc:AccessDialect_pyodbc',
+              ]
+        }
+
+Above, the two entrypoints ``access`` and ``access.pyodbc`` allow URLs to be
+used such as::
+
+    create_engine("access://user:pw@dsn")
+
+    create_engine("access+pyodbc://user:pw@dsn")
+
+* setup.cfg - this file contains the traditional contents such as [egg_info]
+  and [nosetests] directives, but also contains new directives that are used
+  by SQLAlchemy's testing framework.  E.g. for Access::
+
+    [egg_info]
+    tag_build = dev
+
+    [nosetests]
+    with-sqla_testing = true
+    where = test
+    cover-package = sqlalchemy-access
+    with-coverage = 1
+    cover-erase = 1
+
+    [sqla_testing]
+    requirement_cls=test.requirements:Requirements
+    profile_file=.profiles.txt
+
+    [db]
+    default=access+pyodbc://admin@access_test
+    sqlite=sqlite:///:memory:
+
+  Above, the ``[sqla_testing]`` section contains configuration used by
+  SQLAlchemy's test plugin.The ``[nosetests]`` section includes the
+  directive ``with-sql_testing = true``, which indicates to Nose that
+  the SQLAlchemy nose plugin should be used.
+
+* run_tests.py - The plugin is provided with SQLAlchemy, however is not
+  plugged into Nose automatically; instead, a ``run_tests.py`` script
+  should be composed as a front end to Nose, such that SQLAlchemy's plugin
+  will be correctly installed.
+
+  run_tests.py has two parts.  One optional, but probably helpful, step
+  is that it installs your third party dialect into SQLAlchemy without
+  using the setuptools entrypoint system; this allows your dialect to
+  be present without any explicit setup.py step needed.  The other
+  step is to import SQLAlchemy's nose runner and invoke it.  An
+  example run_tests.py file looks like the following::
+
+    from sqlalchemy.dialects import registry
+
+    registry.register("access", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc")
+    registry.register("access.pyodbc", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc")
+
+    from sqlalchemy.testing import runner
+
+    runner.main()
+
+  Where above, the ``registry`` module, introduced in SQLAlchemy 0.8, provides
+  an in-Python means of installing the dialect entrypoints without the use
+  of setuptools, using the ``registry.register()`` function in a way that
+  is similar to the ``entry_points`` directive we placed in our ``setup.py``.
+  The call to ``runner.main()`` then runs the Nose front end, which installs
+  SQLAlchemy's testing plugins.   Invoking our custom runner looks like the
+  following::
+
+    $ python run_tests.py -v
+
+* requirements.py - The ``requirements.py`` file is where directives
+  regarding database and dialect   capabilities are set up.
+  SQLAlchemy's tests are often annotated with decorators   that mark
+  tests as "skip" or "fail" for particular backends.  Over time, this
+  system   has been refined such that specific database and DBAPI names
+  are mentioned   less and less, in favor of @requires directives which
+  state a particular capability.   The requirement directive is linked
+  to target dialects using a ``Requirements`` subclass.   The custom
+  ``Requirements`` subclass is specified in the ``requirements.py`` file
+  and   is made available to SQLAlchemy's test runner using the
+  ``requirement_cls`` directive   inside the ``[sqla_testing]`` section.
+
+  For a third-party dialect, the custom ``Requirements`` class can
+  usually specify a simple yes/no answer for a particular system. For
+  example, a requirements file that specifies a database that supports
+  the RETURNING construct but does not support reflection of tables
+  might look like this::
+
+      # test/requirements.py
+
+      from sqlalchemy.testing.requirements import SuiteRequirements
+
+      from sqlalchemy.testing import exclusions
+
+      class Requirements(SuiteRequirements):
+          @property
+          def table_reflection(self):
+              return exclusions.closed()
+
+          @property
+          def returning(self):
+              return exclusions.open()
+
+  The ``SuiteRequirements`` class in
+  ``sqlalchemy.testing.requirements`` contains a large number of
+  requirements rules, which attempt to have reasonable defaults. The
+  tests will report on those requirements found as they are run.
+
+* test_suite.py - Finally, the ``test_suite.py`` module represents a
+  Nose test suite, which pulls   in the actual SQLAlchemy test suite.
+  To pull in the suite as a whole, it can   be imported in one step::
+
+      # test/test_suite.py
+
+      from sqlalchemy.testing.suite import *
+
+  That's all that's needed - the ``sqlalchemy.testing.suite`` package
+  contains an ever expanding series of tests, most of which should be
+  annotated with specific requirement decorators so that they can be
+  fully controlled. To specifically modify some of the tests, they can
+  be imported by name and subclassed::
+
+      from sqlalchemy.testing.suite import *
+
+      from sqlalchemy.testing.suite import ComponentReflectionTest as _ComponentReflectionTest
+
+      class ComponentReflectionTest(_ComponentReflectionTest):
+          @classmethod
+          def define_views(cls, metadata, schema):
+              # bypass the "define_views" section of the
+              # fixture
+              return
+
+
 SQLALCHEMY UNIT TESTS
 =====================
 
-SQLAlchemy unit tests by default run using Python's built-in sqlite3 
+SQLAlchemy unit tests by default run using Python's built-in sqlite3
 module.  If running on Python 2.4, pysqlite must be installed.
 
-Unit tests are run using nose.  Nose is available at:
+Unit tests are run using nose.  Nose is available at::
 
     http://pypi.python.org/pypi/nose/
 
 This plugin is invoked when the test runner script provided with
 SQLAlchemy is used.
 
-**NOTE:** - the nose plugin is no longer installed by setuptools as of 
+**NOTE:** - the nose plugin is no longer installed by setuptools as of
 version 0.7 !  Use "python setup.py test" or "./sqla_nose.py".
 
 RUNNING TESTS VIA SETUP.PY
 
     $ python setup.py test
 
-The -v flag also works here:
+The -v flag also works here::
 
     $ python setup.py test -v
 
-RUNNING ALL TESTS 
+RUNNING ALL TESTS
 ------------------
-To run all tests:
+To run all tests::
 
     $ ./sqla_nose.py
 
 If you're running the tests on Microsoft Windows, then there is an additional
-argument that must be passed to ./sqla_nose.py:
+argument that must be passed to ./sqla_nose.py::
 
     > ./sqla_nose.py --first-package-wins
 
 sys.modules if it sees a package with the same name in a different location.
 Setting this argument disables that behavior.
 
-Assuming all tests pass, this is a very unexciting output.  To make it more 
-interesting:
+Assuming all tests pass, this is a very unexciting output.  To make it more
+interesting::
 
     $ ./sqla_nose.py -v
 
 RUNNING INDIVIDUAL TESTS
 -------------------------
 Any directory of test modules can be run at once by specifying the directory
-path:
+path::
 
     $ ./sqla_nose.py test/dialect
 
-Any test module can be run directly by specifying its module name:
+Any test module can be run directly by specifying its module name::
 
     $ ./sqla_nose.py test.orm.test_mapper
 
-To run a specific test within the module, specify it as module:ClassName.methodname:
+To run a specific test within the module, specify it as module:ClassName.methodname::
 
     $ ./sqla_nose.py test.orm.test_mapper:MapperTest.test_utils
 
 
 COMMAND LINE OPTIONS
 --------------------
-Help is available via --help:
+Help is available via --help::
 
     $ ./sqla_nose.py --help
 
-The --help screen is a combination of common nose options and options which 
-the SQLAlchemy nose plugin adds.  The most commonly SQLAlchemy-specific 
+The --help screen is a combination of common nose options and options which
+the SQLAlchemy nose plugin adds.  The most commonly SQLAlchemy-specific
 options used are '--db' and '--dburi'.
 
 
 ----------------
 
 Tests will target an in-memory SQLite database by default.  To test against
-another database, use the --dburi option with any standard SQLAlchemy URL:
+another database, use the --dburi option with any standard SQLAlchemy URL::
 
     --dburi=postgresql://user:password@localhost/test
 
 are used to test dotted-name access scenarios.  On some databases such
 as Oracle or Sybase, these are usernames, and others such as Postgresql
 and MySQL they are schemas.   The requirement applies to all backends
-except SQLite and Firebird.  The names are:
+except SQLite and Firebird.  The names are::
 
     test_schema
     test_schema_2 (only used on Postgresql)
 
-Please refer to your vendor documentation for the proper syntax to create 
+Please refer to your vendor documentation for the proper syntax to create
 these namespaces - the database user must have permission to create and drop
 tables within these schemas.  Its perfectly fine to run the test suite
 without these namespaces present, it only means that a handful of tests which
 expect them to be present will fail.
 
-Additional steps specific to individual databases are as follows:
+Additional steps specific to individual databases are as follows::
 
     MYSQL: Default storage engine should be "MyISAM".   Tests that require
     "InnoDB" as the engine will specify this explicitly.
         grant dba to scott;
 
     SYBASE: Similar to Oracle, "test_schema" is created as a user, and the
-    primary test user needs to have the "sa_role". 
- 
+    primary test user needs to have the "sa_role".
+
     It's also recommended to turn on "trunc log on chkpt" and to use a
-    separate transaction log device - Sybase basically seizes up when 
+    separate transaction log device - Sybase basically seizes up when
     the transaction log is full otherwise.
 
-    A full series of setup assuming sa/master: 
+    A full series of setup assuming sa/master:
 
         disk init name="translog", physname="/opt/sybase/data/translog.dat", size="10M"
         create database sqlalchemy on default log on translog="10M"
         sp_role "grant", sa_role, scott
 
     Sybase will still freeze for up to a minute when the log becomes
-    full.  To manually dump the log:
+    full.  To manually dump the log::
 
         dump tran sqlalchemy with truncate_only
 
     properly pass the URL arguments into the nose test runner.
 
 If you'll be running the tests frequently, database aliases can save a lot of
-typing.  The --dbs option lists the built-in aliases and their matching URLs:
+typing.  The --dbs option lists the built-in aliases and their matching URLs::
 
     $ ./sqla_nose.py --dbs
     Available --db options (use --dburi to override)
             postgresql    postgresql://scott:tiger@127.0.0.1:5432/test
     [...]
 
-To run tests against an aliased database:
+To run tests against an aliased database::
 
     $ ./sqla_nose.py --db=postgresql
 
-To customize the URLs with your own users or hostnames, make a simple .ini
-file called `test.cfg` at the top level of the SQLAlchemy source distribution
-or a `.satest.cfg` in your home directory:
+To customize the URLs with your own users or hostnames, create a file
+called `test.cfg` at the top level of the SQLAlchemy source distribution.
+This file is in Python config format, and contains a [db] section which
+lists out additional database configurations::
 
     [db]
     postgresql=postgresql://myuser:mypass@localhost/mydb
 -------------------
 SQLAlchemy logs its activity and debugging through Python's logging package.
 Any log target can be directed to the console with command line options, such
-as:
+as::
 
     $ ./sqla_nose.py test.orm.unitofwork --log-info=sqlalchemy.orm.mapper \
       --log-debug=sqlalchemy.pool --log-info=sqlalchemy.engine
 
 BUILT-IN COVERAGE REPORTING
 ------------------------------
-Coverage is tracked using Nose's coverage plugin.   See the nose 
-documentation for details.  Basic usage is:
+Coverage is tracked using Nose's coverage plugin.   See the nose
+documentation for details.  Basic usage is::
 
     $ ./sqla_nose.py test.sql.test_query --with-coverage
 
 store the incorrect filepaths, which will break the coverage system.  If
 coverage numbers are coming out as low/zero, try deleting all .pyc files.
 
-TESTING NEW DIALECTS
---------------------
-You can use the SQLAlchemy test suite to test any new database dialect in
-development.  All possible database features will be exercised by default.
-Test decorators are provided that can exclude unsupported tests for a
-particular dialect.  You'll see them all over the source, feel free to add
-your dialect to them or apply new decorations to existing tests as required.
-
-It's fine to start out with very broad exclusions, e.g. "2-phase commit is not
-supported on this database" and later refine that as needed "2-phase commit is
-not available until server version 8".
-
-To be considered for inclusion in the SQLAlchemy distribution, a dialect must
-be integrated with the standard test suite.  Dialect-specific tests can be
-placed in the 'dialects/' directory.  Comprehensive testing of
-database-specific column types and their proper reflection are a very good
-place to start.
-
-When working through the tests, start with 'engine' and 'sql' tests.  'engine'
-performs a wide range of transaction tests that might deadlock on a brand-new
-dialect- try disabling those if you're having problems and revisit them later.
-
-Once the 'sql' tests are passing, the 'orm' tests should pass as well, modulo
-any adjustments needed for SQL features the ORM uses that might not be
-available in your database.  But if an 'orm' test requires changes to your
-dialect or the SQLAlchemy core to pass, there's a test missing in 'sql'!  Any
-time you can spend boiling down the problem to it's essential sql roots and
-adding a 'sql' test will be much appreciated.
-
-The test suite is very effective at illuminating bugs and inconsistencies in
-an underlying DB-API (or database!) implementation.  Workarounds are almost
-always possible.  If you hit a wall, join us on the mailing list or, better,
-IRC!
+DEVELOPING AND TESTING NEW DIALECTS
+-----------------------------------
 
+See the new file README.dialects.rst for detail on dialects.