TypeError: issubclass() arg 1 must be a class

[Antony Oduor]

In implementing a multi-tenant system using PostreSQL schemas... When A new customer signs-up I create a new schema when they activate their account. Here is the code for the function that creates the schema and even put's in a dummy record.

settings = request.registry.settings
engine = get_engine(settings)

Session = sessionmaker(engine, expire_on_commit=False)
Session.configure(bind=engine)
session = Session()

session.begin(subtransactions=True)
try:
    engine.execute(CreateSchema(tenant_id))

    metadata = MetaData(schema=tenant_id)
    declared_base = declarative_base(bind=engine, name='NewTenantBase', metadata=metadata)

    NewTenantBase = automap_base(declarative_base=declared_base)
    NewTenantBase.metadata.schema = tenant_id
    # NewTenantBase.metadata.reflect(bind=engine)

    NewTenantBase.prepare(engine, reflect=False)
    configure_mappers()

    class MyBooks(BooksMixin, NewTenantBase):
        __tablename__ = 'mybooks'

    # mapped classes are ready
    MyBooks = NewTenantBase.classes.mybooks

    NewTenantBase.metadata.create_all()

    try:
        # set the search path
        session.execute("SET search_path TO %s" % tenant_id)

        session.add(MyBooks(name='Fiction', description='You know what fiction are'))
        session.flush()
        session.execute("commit")
    except Exception as e:
        traceback.print_exc()
        session.rollback()

    session.commit()
except Exception as e:
    traceback.print_exc()
    session.rollback()

This using Pyramid-Websauna framework. The first signup goes well and everything is setup properly in the new schema without an error. The problem is subsequent signups don't go as well. I get the error below which happens after the new schema has been created.

TypeError('issubclass() arg 1 must be a class',)
Traceback (most recent call last):
  File "/opt/mapylons/myproject/myapp/myapp/tasks.py", line 80, in create_schema
    NewTenantBase.prepare(engine, reflect=False)
  File "/opt/mapylons/myproject/venv3.5/lib/python3.5/site-packages/sqlalchemy/ext/automap.py", line 787, in prepare
    generate_relationship)
  File "/opt/mapylons/myproject/venv3.5/lib/python3.5/site-packages/sqlalchemy/ext/automap.py", line 894, in _relationships_for_fks
    local_cls, referred_cls):
TypeError: issubclass() arg 1 must be a class

The interesting bit is that is when I restart the server between each signup, they will all go well and the customer schema setup will go on without a hitch. How can I make this multi-tenancy work properly without literally restarting pserve after each customer signup? ...which would be quite a hack job

[Mike Bayer]

this error is occuring because automap is being used in a way in which it was not desgined. Some of the classes that were used from a previous request are creating artifacts that are hanging around after the class has been garbage collected and producing errors.

It's possible to try to work out the failure here but the above code is very confusing, and I dont understand what it's trying to do. The class-on-the-fly thing is a pretty bad way to do things though, you should use the schema translation feature with just one set of classes and be done with it.

[Mike Bayer]

this test can sort of hit a similar issue, not the same stack trace but the general idea where concurrent prepare() is being called and might hit a mapper_config.cls that was garbage collected but the config still present

from sqlalchemy.ext.automap import automap_base
from sqlalchemy import *
from sqlalchemy.orm import configure_mappers
import time


class SomeOtherBase(object):
    pass

def worker():
    while True:
        m = MetaData()
        Table(
            'user', m,
            Column('id', Integer, primary_key=True)
        )

        Table(
            'address', m,
            Column('id', Integer, primary_key=True),
            Column('user_id', ForeignKey('user.id'))
        )

        Base = automap_base(metadata=m)


        Base.prepare(reflect=False)

        class Foo(Base, SomeOtherBase):
            __tablename__ = 'foo'

            id = Column(Integer, primary_key=True)
            user_id = Column(ForeignKey('user.id'))

        configure_mappers()

        time.sleep(.1)


import threading

threads = [threading.Thread(target=worker) for i in range(10)]
for t in threads:
    t.start()
for t in threads:
    t.join()

we would assume something similar to that is happening.

[Mike Bayer]

well now that test is failing on just not being able to find "user". So, I really can't reproduce this. I'll commit a few checks and that's it, the pattern here is likely going to continue not working.

[Antony Oduor]

Thank you for the insights. I totally agree with the conclusion, would there be a way to manually clear out the remnants from the previous requests.

What the function is trying to do is:

1. Create a new schema for a new user.
2. Insert a few records into the tables of the new schema.

Initially I was using plain declarative_base to setup classes-on-the-fly but it started getting a bit hairy with table relationship between public schema and the customers' schema. I came across automap and config_mappers to resolve the issue with table relationships.

FYI, there is a subset of tables that need to be created on each new schema. The public schema has a larger set and this was the way I ended up having it. It's really not ideal at all but I adopted it from the automap documentation

Based on the above expectations, using declarative_base, what would be the best approach to avoid running into uncollected garbage issue?

[Mike Bayer]

OK here is how you do that:

1. make a single, persistent model, of just the tables you need to deal with

2. the model can be just fixed, I wouldn't really bother with automap which is more about ad-hoc queries on databases where you don't necessarily know much about the structure. if you did want to use automap, just automap once, as a startup function, from just one database/schema.

3. when you want to run a query / insert for a certain user's new schema, use the schema translation feature.

session = Session(engine)
session.connection(execution_options={"schema_translate_map": {None: "users_schema"}})
# work with session
session.commit()
session.close()

as far as the uncollected garbage issue, as I said there are changes coming here to try to guard against that which will close this ticket.

[Mike Bayer]

22570c3181ef4e98c548c2f6254a0c7585568f06

[Antony Oduor]

Awesome! I have been able to resolve my problem thanks especially to the snippet you shared above. It helped a lot.