add more explicit documentation/flags for the many gaerdbms DBAPI options

[rodrigo moraes]

I added support to gaerdbms to use a local database. A working dialect is here:

http://stackoverflow.com/a/16198395/125967

Basically, when running the development server, it uses a local MySql database through this snippet:

    # Development mode with local database.
    from google.appengine.api import rdbms_mysqldb
    return rdbms_mysqldb

I suppose the os.environ way to configure won't be welcome, so maybe this should be an option passed to dialect constructor (or something, sorry I'm really new to SqlAlchemy).

For backwards compatibility, the default setting for development could be remote access to the database, configurable to use a local database. However, a local database may be more desirable for development, so perhaps local mode should be the default.

[Mike Bayer]

Adding to the list of options we have in #2649, we have:

            # Production mode.
            from google.storage.speckle.python.api import rdbms_apiproxy

            # Development mode with remote database.
            from google.storage.speckle.python.api import rdbms_googleapi

            # Development mode with local database.
            from google.appengine.api import rdbms_mysqldb

We can document right off that any of these APIs can be selected as follows (this is your workaround for now):

from google.xyz.qpr import some_api

engine = create_engine("mysql+gaerdbms://...", module=some_api)

we should add a simple flag to support selecting among these APIs explicitly:

engine = create_engine("mysql+gaerdbms://...", googleapi="apiproxy|mysqldb|googleapi")

[rodrigo moraes]

Sounds good. This way the dialect leaves to the developer to detect which environment is currently running. I only wonder if two ways to do it are necessary; maybe the string based approach is enough?

One note: there's an special case in create_connect_args() when using the mysqldb api: the 'dsn' argument should not be set as it will cause an error ("DBAPIError: (TypeError) 'dsn' is an invalid keyword argument for this function None None"). The 'instance' argument is fine.

[Mike Bayer]

06a2ce55a1043 0.8

diff:@d5363fc:c8f9831 master

[Martijn Pieters]

Heads-up: If the SDK cannot find the MySQLdb library (you need to install it manually), then the rdbms_mysqldb module will not have a paramstyle attribute, leading to:

# initial lines omitted to protect the easily bored..
  File "/Users/mj/Development/Projects/zagat_personalisation/app/distlib/sqlalchemy/engine/__init__.py", line 332, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/mj/Development/Projects/zagat_personalisation/app/distlib/sqlalchemy/engine/strategies.py", line 69, in create
    dialect = dialect_cls(**dialect_args)
  File "/Users/mj/Development/Projects/zagat_personalisation/app/distlib/sqlalchemy/dialects/mysql/base.py", line 1986, in __init__
    default.DefaultDialect.__init__(self, **kwargs)
  File "/Users/mj/Development/Projects/zagat_personalisation/app/distlib/sqlalchemy/engine/default.py", line 124, in __init__
    self.paramstyle = self.dbapi.paramstyle
AttributeError: 'module' object has no attribute 'paramstyle'

Tested with GAE SDK 1.8.8.

This is not a problem with SQLAlchemy itself, but the error ''is'' confusing. It certainly caught me.

[Mike Bayer]

so what do you suggest?

[Martijn Pieters]

I initially mistook this as a bug with either the GAE SDK or SQLAlchemy bug, but at best it is an unfortunate interaction. In the rdbms_mysql module, if import MySQLdb fails, connect is defined as:

  def connect(instance=None, database=None):
    logging.error('The rdbms API (Google Cloud SQL) is not available because '
                  'the MySQLdb library could not be loaded. Please see the SDK '
                  'documentation for installation instructions.')

    raise NotImplementedError('Unable to find the MySQLdb library')

so connecting will raise an exception, but paramstyle and other parameters, normally imported with from MySQLdb import *, are naturally missing.

If you are feeling generous, perhaps the sqlalchemy.dialects.mysql.gaerdbms:MySQLDialect_gaerdbms.dbapi() class method could verify paramstyle is present and raise an ImportError exception if not available.

[Mike Bayer]

but then if gaerdbms fixes that bug it's sort of legacy garbage lying around then, can we determine more specifically what ".paramstyle" missing means and if there's a more idiomatic way to detect this failure ?

[Martijn Pieters]

The full implementation of that module is essentially:

try:
  import google
  import MySQLdb

  from MySQLdb import *



  __import__('MySQLdb.constants', globals(), locals(), ['*']('*'))
except ImportError:

  def connect(instance=None, database=None):
    logging.error('The rdbms API (Google Cloud SQL) is not available because '
                  'the MySQLdb library could not be loaded. Please see the SDK '
                  'documentation for installation instructions.')

    raise NotImplementedError('Unable to find the MySQLdb library')
else:


  def connect(instance=None, database=None, **kwargs):
    merged_kwargs = _connect_kwargs.copy()
    if database:
      merged_kwargs['db']('db') = database
    merged_kwargs.update(kwargs)
    if 'password' in merged_kwargs:
      merged_kwargs['passwd']('passwd') = merged_kwargs.pop('password')
    host = merged_kwargs.get('host')
    if ((not host or host == 'localhost') and
        not merged_kwargs.get('unix_socket') and
        _OS_NAME == 'posix'):

(with the famed Google 2-space indent).

So essentially, all but the MySQLdb.connect() method is taken directly from the MySQLdb module. I doubt Google will ever add in a paramstyle attribute here, that is not the bug.

[Mike Bayer]

OK so if we put a check around .paramstyle, and just let it default to something like "format", when the program goes to connect theyll get the NotImplementedError. lets just do that can you send me a PR ?

[Martijn Pieters]

Hrm, not sure how far to go with this; altering the paramstyle test with:

        if paramstyle is not None:
            self.paramstyle = paramstyle
        elif self.dbapi is not None and hasattr(self.dbapi, 'paramstyle'):
            self.paramstyle = self.dbapi.paramstyle
        else:
            self.paramstyle = self.default_paramstyle

inevitably leads to more problems.

What happens in that case is that the DefaultEngineStrategy.create() method tries to call dialect.connect() which fails by raising NotImplementedError, as expected. We'd like to propagate that error if it is not a disconnection exception.

But the MySQL dialect is_disconnected() method tries to test exceptions against dbapi.OperationalError, which is of course missing here. If you override is_disconnected() on MySQLDialect_gaerdbms, to return False for NotImplementedError exceptions, then DefaultEngineStrategy.create() will try and use dbapi.Error to re-raise the exception. Nope, not present either.

So, short of poking in a dummy dbapi.Error class, there is no real clean way to handle this corner case. Instead, I've documented the on Stack Overflow. The solution is simple anyway, just install the MySQLdb package, as instructed in the SDK.

[Mike Bayer]

OK well the NotImplementedError should not be getting fed into is_disconnect(), nor the wrapping of Error, that's a bug. The normal place this happens is in engine/base.py in _handle_dbapi_exception(), but I see the one in strategies.py isn't doing that, so that's broke. that's #2881, so it'll be fixed.

[Mike Bayer]

Hm well, yes, we really need dbapi.Error at least to handle this. So I guess that makes it very hard to work around this without putting all kinds of hardcoded assumptions in the dialect. Google's API should be raising ImportError immediately when it's imported.

[Mike Bayer]

Removing milestone: 0.8.xx (automated comment)