AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'

Issue #347 new
Brandon Jones created an issue

I can’t seem to get Celery 3.1.26.post2 to play nice with Kallithea 0.4.1. I have tried multiple different configurations for the backend (rpc, amqp, db) to no avail. Here is my stack trace:

Sep 17 00:17:01 wti-repos kallithea[2218]: 2019-09-17 00:17:01.678 DEBUG [amqp] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL.  See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright$
Sep 17 00:17:01 wti-repos kallithea[2218]: 2019-09-17 00:17:01.679 DEBUG [amqp] Open OK!
Sep 17 00:17:01 wti-repos kallithea[2218]: 2019-09-17 00:17:01.679 DEBUG [amqp] using channel_id: 1
Sep 17 00:17:01 wti-repos kallithea[2218]: 2019-09-17 00:17:01.680 DEBUG [amqp] Channel open
Sep 17 00:17:03 wti-repos kallithea[2218]: Traceback (most recent call last):
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/appwrappers/session.py", line 71, in __call__
Sep 17 00:17:03 wti-repos kallithea[2218]:     response = self.next_handler(controller, environ, context)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/appwrappers/i18n.py", line 71, in __call__
Sep 17 00:17:03 wti-repos kallithea[2218]:     return self.next_handler(controller, environ, context)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/wsgiapp.py", line 285, in _dispatch
Sep 17 00:17:03 wti-repos kallithea[2218]:     return controller(environ, context)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/kallithea/lib/base.py", line 539, in __call__
Sep 17 00:17:03 wti-repos kallithea[2218]:     return super(BaseController, self).__call__(environ, context)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/controllers/dispatcher.py", line 119, in __call__
Sep 17 00:17:03 wti-repos kallithea[2218]:     response = self._perform_call(context)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/controllers/dispatcher.py", line 108, in _perform_call
Sep 17 00:17:03 wti-repos kallithea[2218]:     r = self._call(action, params, remainder=remainder, context=context)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/controllers/decoratedcontroller.py", line 131, in _call
Sep 17 00:17:03 wti-repos kallithea[2218]:     output = controller_caller(context_config, bound_controller_callable, remainder, params)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/decorators.py", line 44, in _decorated_controller_caller
Sep 17 00:17:03 wti-repos kallithea[2218]:     return application_controller_caller(tg_config, controller, remainder, params)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/tg/configuration/app_config.py", line 127, in call_controller
Sep 17 00:17:03 wti-repos kallithea[2218]:     return controller(*remainder, **params)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "</srv/kallithea/venv/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-15>", line 2, in repo_check
Sep 17 00:17:03 wti-repos kallithea[2218]:
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/kallithea/lib/auth.py", line 797, in __wrapper
Sep 17 00:17:03 wti-repos kallithea[2218]:     return func(*fargs, **fkwargs)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "</srv/kallithea/venv/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-14>", line 2, in repo_check
Sep 17 00:17:03 wti-repos kallithea[2218]:
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/kallithea/lib/base.py", line 627, in jsonify
Sep 17 00:17:03 wti-repos kallithea[2218]:     data = func(*args, **kwargs)
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/kallithea/controllers/admin/repos.py", line 189, in repo_check
Sep 17 00:17:03 wti-repos kallithea[2218]:     if task.failed():
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/celery/result.py", line 267, in failed
Sep 17 00:17:03 wti-repos kallithea[2218]:     return self.state == states.FAILURE
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/celery/result.py", line 394, in state
Sep 17 00:17:03 wti-repos kallithea[2218]:     return self._get_task_meta()['status']
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/celery/result.py", line 339, in _get_task_meta
Sep 17 00:17:03 wti-repos kallithea[2218]:     return self._maybe_set_cache(self.backend.get_task_meta(self.id))
Sep 17 00:17:03 wti-repos kallithea[2218]:   File "/srv/kallithea/venv/lib/python2.7/site-packages/celery/backends/base.py", line 307, in get_task_meta
Sep 17 00:17:03 wti-repos kallithea[2218]:     meta = self._get_task_meta_for(task_id)
Sep 17 00:17:03 wti-repos kallithea[2218]: AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'

Comments (15)

  1. Thomas De Schampheleire

    Could you share the part of your .ini file relating to celery?

    The error is a result of the backend not being correctly configured.

    What did you change since before? Is it the first time you enable Celery?

  2. Brandon Jones reporter

    Here are the relevant entries in my INI file - however, I have tried multiple different configurations to no avail - all configurations would return the same error

    celery.imports = kallithea.lib.celerylib.tasks
    celery.accept.content = pickle
    celery.result.backend = db+sqlite:///results.db
    

    I have also tried rpc:// and amqp:// for the celery.result.backend entry. The celery.result.backend entry above is used in an example in the documentation for Celery 3.1.2.5 - https://docs.celeryproject.org/en/3.1/configuration.html

  3. Thomas De Schampheleire

    The values provided in e.g. development.ini are:

    ## Example: connect to the virtual host 'rabbitmqhost' on localhost as rabbitmq:
    broker.url = amqp://rabbitmq:qewqew@localhost:5672/rabbitmqhost
    
    celery.imports = kallithea.lib.celerylib.tasks
    celery.accept.content = pickle
    celery.result.backend = amqp
    celery.result.dburi = amqp://
    celery.result.serialier = json
    

    Here the backend and dburi are separate values, and backend is a simple word. The doc you refer to says:

    CELERY_RESULT_DBURI
    
    This setting is no longer used as it’s now possible to specify the database URL directly in the CELERY_RESULT_BACKEND setting.
    

    but I have never used it this way.

    Do you still have a (perhaps empty) ‘celery.result.dburi’ setting in your ini file, or did you remove it?

    I’m using ‘backend = amqp’ and ‘dburi = amqp://' (i.e. empty) with a slightly older Kallithea version and that works. The amqp broker is rabbitmq in my case.

    Could you first try with a configuration as in development.ini?

  4. Brandon Jones reporter

    I have celery.result.dburi commented out in my configuration. I started out with the default configuration which didn’t work, which is why I continued to make changes to see if there was some combination that worked.

    Using the exact values in the default development.ini (swapped out for my local rabbitmq server and credentials), I am presented with the same error when trying to create a repository through the web interface. Here are my log entries in graylog:

    2019-09-18 07:56:53.631 DEBUG [amqp] Channel open
    2019-09-18 07:56:53.630 DEBUG [amqp] using channel_id: 1
    2019-09-18 07:56:53.629 DEBUG [amqp] Open OK!
    2019-09-18 07:56:53.628 DEBUG [amqp] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2017 Pivotal Software, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'direct_reply_to': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@wti-repos', u'platform': u'Erlang/OTP', u'version': u'3.6.10'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
    
    File "/srv/kallithea/venv/lib/python2.7/site-packages/celery/result.py", line 267, in failed
    return self.state == states.FAILURE
    File "/srv/kallithea/venv/lib/python2.7/site-packages/celery/result.py", line 339, in _get_task_meta
    AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'
    

  5. Thomas De Schampheleire

    Ok, I verified that celery with amqp/rabbitmq works correctly at my end. Celery version 3.1.26.post2. Rabbitmq 3.7.17 (docker container).

    Let’s take a step back to fill in the details, I will be assuming amqp below as this is what I verified. Once this works we could go back to whatever you really want.

    When do you get the reported stack trace? Is it as soon as you start up celery? Or later when an actual task is run?

    How do you start celery? It should be with: kallithea-cli celery-run -c <inifile>

    In case of rabbitmq, which version are you using?

    Are you using docker at all, or is this a native installation?

    I suggest adding some traces in the celery code to understand the issue better. In your virtualenv, edit file: lib/python2.7/site-packages/celery/backends/__init__.py , specifically in methods get_backend_cls and get_backend_by_url. As you can see in the first function, if backend passed as argument is None, then the disabled backend is used, which maps on DisabledBackend.

  6. Brandon Jones reporter

    I get the stack trace when I try to create a repository, specifically.

    I start Celery the way just as you mentioned, running as a service in Ubuntu.

    I am not using Docker - this is a native installation. Last time I tried to setup Kallithea with Docker I just couldn’t find enough useful documentation to make the process worth my time.

    I was at RabbitMQ version 3.6.10, but I went ahead and moved to RabbitMQ version 3.7.17 (and Erlang 22.0.7) to match your configuration. I am still getting this error every time I try to create a repository.

    I will try to do some stack tracing when I get a chance to see if I can figure anything else out. I’m not opposed to moving to a docker container if RabbitMQ and Celery work out-of-the-box, but again, I felt like there was a lack of information on how to set that up.

  7. Thomas De Schampheleire

    You shouldn’t need docker for this. I could explain it more later, but it is not necessary at all, it won’t fix things. I think there is a configuration issue.

    Curiously, the first time I created a repo I did get the same error as you. At that time my config file had a bogus setting “backend=amqp://foobar” . Then I tried out different settings and things worked. However, later I could no longer reproduce it even with the foobar setting.

    However, during the testing I found differences depending on whether only celery-run was restarted after changing a setting, or both kallithea and celery-run. Both of them should be restarted when the celery settings are changed, are you doing that?

  8. Brandon Jones reporter

    Right, I'm aware of what Docker is and that I don't need it…. It might just work better than it has been for me because I'm assuming the container has RabbitMQ and Celery

    What does your amqp:// url look like for the back end?

    And yes I've been restarting both services everytime

  9. Thomas De Schampheleire

    These are the settings that work for me:

    ####################################
    ###        CELERY CONFIG        ####
    ####################################
    
    use_celery = true
    
    ## Example: connect to the virtual host 'rabbitmqhost' on localhost as rabbitmq:
    broker.url = amqp://rabbitmq:qewqew@localhost:5672/rabbitmqhost
    
    celery.imports = kallithea.lib.celerylib.tasks
    celery.accept.content = pickle
    celery.result.backend = amqp
    celery.result.dburi = amqp://
    celery.result.serialier = json
    
    #celery.send.task.error.emails = true
    #celery.amqp.task.result.expires = 18000
    
    celeryd.concurrency = 2
    celeryd.max.tasks.per.child = 1
    
    ## If true, tasks will never be sent to the queue, but executed locally instead.
    celery.always.eager = false
    

    The docker image I was referring to is only for rabbitmq, i.e. I just did:

    docker run --rm -d --hostname my-rabbit --name some-rabbit -e RABBITMQ_DEFAULT_USER=rabbitmq -e RABBITMQ_DEFAULT_PASS=qewqew -e RABBITMQ_DEFAULT_VHOST=rabbitmqhost -p 5672:5672 rabbitmq
    

    Outside of docker, I just ran celery-run normally.

    In my work instance, we have everything in docker, also in separate containers: one for postgresql, one for rabbitmq, one for kallithea, one for celery.

  10. Brandon Jones reporter

    What all is Celery used for? Because besides getting an error when trying to create a repo, it seems as if most other things work fine. I am able to clone just fine, which I assume would be using Celery. Perhaps it is a bug specific to create repository, as you saw it on your initial try as well?

  11. Thomas De Schampheleire

    Celery is used to handle some tasks asynchronously, outside of the web server process. Repository creation and forking, email sending are some examples. See kallithea/lib/celerylib/tasks.py for a full list. The web server process will continue immediately, the user will get a new web page, and the task continues in the background.

    When Celery is not configured or not available, the tasks are handled in the main web server process, which means it will wait until the task is completed. In this sense, Celery is not mandatory.

    But, for real deployment with more than a few users, it is recommended to configure Celery for performance.

    Could you get any more info by adding traces in the Celery files as suggested?

  12. Mads Kiilerich

    It seems like a celery bug that https://github.com/celery/celery/blob/master/celery/backends/base.py reference _get_task_meta_for in the base Backend class, but only implement it in BaseKeyValueStoreBackend and not in DisabledBackend.

    I guess https://github.com/celery/celery/blob/master/celery/backends/base.py#L842 should have disabled get_task_meta instead?

    But https://github.com/celery/celery/issues/4471#event-1783002644 was closed as not a bug … perhaps worth following up on …

    I can however only reproduce the problem if I explicitly set celery.result.backend = disabled .

    I wonder how your setup ends up with DisabledBackend. Can you try to debug your site-packages/celery/backends/__init__.py and figure out what comes in and out of these functions? (Perhaps just by adding print backend .)

    The following works for me for testing with celery==3.1.26.post2:

    use_celery = true
    broker.url = sqla+sqlite:///broker.db
    
    celery.imports = kallithea.lib.celerylib.tasks
    celery.accept.content = pickle
    celery.result.backend = db+sqlite:///results.db
    

    (Note: sqla broker is not recommended - http://docs.celeryproject.org/en/3.1/getting-started/brokers/sqlalchemy.html#using-sqlalchemy )

  13. Brandon Jones

    I added print backend to the appropriate place but I’m not sure where that is printing to, or if I have to setup some debug flag…

    However, if I look at my Celery service, I can clearly see that Celery is getting the correct configuration (specifically looking at the results: sqlite:///results.db

    Mads - what happens if you omit the celery.result.backend line (or just comment it out)? Do you still get the error? I’m starting to wonder if somehow kallithea-cli is not getting the appropriate configuration.

  14. Thomas De Schampheleire

    The print should end up in the console where you started celery (via kallithea-cli). To make sure you are changing the right celery file, I suggest adding a syntax error. If things still start up, then the file you are changing is not the right one.

  15. Brandon Jones reporter

    OK, so I believe I am beyond the DisabledBackend error (although it seems like a bug in Celery that happens the first time I try to create a repo), but I am now presented with the following error:

    [2019-10-04 14:52:16,648: ERROR/MainProcess] Task kallithea.lib.celerylib.create_repo[95a16e47-48d7-40ab-8519-cc668118d241] raised unexpected: OperationalError('(psycopg2.OperationalError) SSL error: decryption failed or bad record mac\n',)
    Traceback (most recent call last):
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
        R = retval = fun(*args, **kwargs)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
        return self.run(*args, **kwargs)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/celerylib/__init__.py", line 69, in f_async
        f_org(*args, **kwargs)
      File "</srv/kallithea/venv/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-6>", line 2, in create_repo
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/celerylib/__init__.py", line 130, in __wrapper
        ret = func(*fargs, **fkwargs)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/celerylib/tasks.py", line 333, in create_repo
        cur_user = User.guess_instance(cur_user)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/model/db.py", line 551, in guess_instance
        return super(User, cls).guess_instance(value, User.get_by_username)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/model/db.py", line 141, in guess_instance
        return cls.get(value)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/model/db.py", line 122, in get
        return cls.query().get(id_)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 936, in get
        return self._get_impl(ident, loading.load_on_pk_identity)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 1026, in _get_impl
        return db_load_fn(self, primary_key_identity)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 282, in load_on_pk_identity
        return q.one()
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3039, in one
        ret = self.one_or_none()
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3008, in one_or_none
        ret = list(self)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/kallithea/lib/caching_query.py", line 81, in __iter__
        return Query.__iter__(self)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
        return self._execute_and_instances(context)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
        result = conn.execute(querycontext.statement, self._params)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute
        return meth(self, multiparams, params)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
        return connection._execute_clauseelement(self, multiparams, params)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
        distilled_params,
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
        e, statement, parameters, cursor, context
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1458, in _handle_dbapi_exception
        util.raise_from_cause(sqlalchemy_exception, exc_info)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb, cause=cause)
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
        cursor, statement, parameters, context
      File "/srv/kallithea/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
        cursor.execute(statement, parameters)
    OperationalError: (psycopg2.OperationalError) SSL error: decryption failed or bad record mac
     [SQL: 'SELECT users.firstname AS users_firstname, users.email AS users_email, users.user_data AS users_user_data, users.user_id AS users_user_id, users.username AS users_username, users.password AS users_password, users.active AS users_active, users.admin AS users_admin, users.lastname AS users_lastname, users.last_login AS users_last_login, users.extern_type AS users_extern_type, users.extern_name AS users_extern_name, users.api_key AS users_api_key, users.inherit_default_permissions AS users_inherit_default_permissions, users.created_on AS users_created_on \nFROM users \nWHERE users.user_id = %(param_1)s'] [parameters: {'param_1': 3L}] (Background on this error at: http://sqlalche.me/e/e3q8)
    

    SSL error: decryption failed or bad record mac

    Doing some research online it sounds like it might be related to how WSGI is forking in waitress?

    https://stackoverflow.com/questions/22752521/uwsgi-flask-sqlalchemy-and-postgres-ssl-error-decryption-failed-or-bad-reco

  16. Log in to comment