Error 500 when trying to access several pages with admin user

Issue #909 resolved
Pablo A. Revert created an issue

When I try to access several pages with admin user I get the error pasted at the end of the issue.

At least this pages

  • /root
  • /_admin/repos
  • /_admin/repo_groups

Exceptions Tracker - Exception ID: 140373394319232

Exception `DBPageNotFoundError` generated on UTC date: 2019-10-14T13:23:34.629653

Traceback (most recent call last):
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/view.py", line 657, in _call_view
    response = view_callable(context, request)
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/opt/rhodecode/store/98nqg9sak7d1if134kfldiz507dkfild-python2.7-pyramid-1.10.1/lib/python2.7/site-packages/pyramid/viewderivers.py", line 132, in _class_view
    response = getattr(inst, attr)()
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2352, in local_wrapper
    return wrapper(func, *args, **kwds)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1665, in __wrapper
    return func(*fargs, **fkwargs)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/apps/home/views.py", line 548, in main_page
    repo_data, repo_group_data = self._get_groups_and_repos()
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/apps/home/views.py", line 529, in _get_groups_and_repos
    repo_group_list=repo_group_list_acl, admin=False)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/model/repo_group.py", line 723, in get_repo_groups_as_dict
    for group in repo_group_list:
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/model/scm.py", line 142, in __iter__
    if not checker(name, self.__class__.__name__, **self.extra_kwargs):
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2082, in __call__
    check_location, user)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1986, in __call__
    if self.check_permissions(user):
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 2085, in check_permissions
    perms = user.permissions
  File "/opt/rhodecode/store/rq6n4kdw9z8d44km9njnr5rxxyh01p4h-python2.7-zope.cachedescriptors-4.3.1/lib/python2.7/site-packages/zope/cachedescriptors/property.py", line 106, in __get__
    value = func(inst)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1069, in permissions
    return self.get_perms(user=self, cache=None)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 1230, in get_perms
    calculate_super_admin)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/rc_cache/utils.py", line 79, in decorate
    return self.get_or_create(key, creator, timeout, should_cache_fn)
  File "/opt/rhodecode/store/f20ysi76vr526ibh0sqv6cv47kk6cxa3-python2.7-dogpile.cache-0.7.1/lib/python2.7/site-packages/dogpile/cache/region.py", line 879, in get_or_create
    async_creator) as value:
  File "/opt/rhodecode/store/f20ysi76vr526ibh0sqv6cv47kk6cxa3-python2.7-dogpile.cache-0.7.1/lib/python2.7/site-packages/dogpile/lock.py", line 186, in __enter__
    return self._enter()
  File "/opt/rhodecode/store/f20ysi76vr526ibh0sqv6cv47kk6cxa3-python2.7-dogpile.cache-0.7.1/lib/python2.7/site-packages/dogpile/lock.py", line 86, in _enter
    value = value_fn()
  File "/opt/rhodecode/store/f20ysi76vr526ibh0sqv6cv47kk6cxa3-python2.7-dogpile.cache-0.7.1/lib/python2.7/site-packages/dogpile/cache/region.py", line 826, in get_value
    value = self.backend.get(key)
  File "/opt/rhodecode/store/khmw04fzpjqmckrn03ndzkl6zykd5zbi-python2.7-rhodecode-enterprise-ce-4.16.2/lib/python2.7/site-packages/rhodecode/lib/rc_cache/backends.py", line 150, in get
    value = dbm.get(key, NO_VALUE)
  File "/opt/rhodecode/store/6rjyfgq8yrnzwsc4x9b6mlwi6h9qhay6-python-2.7.15/lib/python2.7/_abcoll.py", line 382, in get
    return self[key]
  File "/opt/rhodecode/store/6rjyfgq8yrnzwsc4x9b6mlwi6h9qhay6-python-2.7.15/lib/python2.7/bsddb/__init__.py", line 270, in __getitem__
    return _DeadlockWrap(lambda: self.db[key])  # self.db[key]
  File "/opt/rhodecode/store/6rjyfgq8yrnzwsc4x9b6mlwi6h9qhay6-python-2.7.15/lib/python2.7/bsddb/dbutils.py", line 68, in DeadlockWrap
    return function(*_args, **_kwargs)
  File "/opt/rhodecode/store/6rjyfgq8yrnzwsc4x9b6mlwi6h9qhay6-python-2.7.15/lib/python2.7/bsddb/__init__.py", line 270, in <lambda>
    return _DeadlockWrap(lambda: self.db[key])  # self.db[key]
DBPageNotFoundError: (-30986, 'BDB0075 DB_PAGE_NOTFOUND: Requested page not found')

System Info

SYSTEM INFO
-----------

  rhodecode_version: RhodeCode Community Edition, version 4.16.2
  latest_version: 4.17.3
        ALERT_WARNING New version available: 4.17.3
  rhodecode_base_url: http://rhodecode.local
  rhodecode_server_ip: 127.0.0.1:10020
  rhodecode_server_id: ip-172-31-6-155-23218
  rhodecode_configuration: /home/ubuntu/.rccontrol/community-1/rhodecode.ini
  rhodecode_certificate: /home/ubuntu/.rccontrol-profile/etc/ca-bundle.crt
  workers: 2
  worker_type: gevent
  ---
  database: mysql://rhodecode:***@127.0.0.1:3306/RHODECODE @ migration version: 95
  database_version: mysql 5.7.27.0ubuntu0.18.4.1
  ---
  platform: Linux-4.15.0-1043-aws-x86_64-with-debian-buster-sid
  platform_uuid: e137f37afa745cd7f1b7e863cd305f04984b72f536fc819d10e2b2a01c57a08c
  lang: LANG: en_US.UTF-8, locale LC_ALL: (None, None),  Default locales: ('en_US', 'UTF-8')
  python_version: CPython 2.7.15   default Apr 29 2018 23:18:59 GCC 7.3.0
  python_path: /opt/rhodecode/store/6rjyfgq8yrnzwsc4x9b6mlwi6h9qhay6-python-2.7.15/bin/python2.7
  ---
  cpu: 2 cores at 0.0 %
  load: 1min: 0.0, 5min: 0.0, 15min: 0.0
  memory: 1.0 GiB/1.9 GiB, 53.6% used
  uptime: Server started 2 months and 28 days ago
  ---
  ulimit: cpu time (seconds):(-1, -1), file size:(-1, -1), stack size:(8388608, -1), core file size:(0, -1), address space size:(-1, -1), locked in mem size:(16777216, 16777216), heap size:(-1, -1), rss size:(-1, -1), number of processes:(7727, 7727), open files:(1024, 4096)
  storage_location: /repos
  storage_info: 11.4 GiB/29.4 GiB, 38.9% used
  storage_inodes: 111465/1966080, 5.7% used
  gist_storage_location: /repos/.rc_gist_store
  gist_storage_info: 0 B (0 items)
  archive_cache_storage_location: Enable this by setting archive_cache_dir=/path/to/cache option in the .ini file
  archive_cache_info: 0 B (0 items)
  temp_storage_location: /tmp
  temp_storage_info: 8.7 GiB/19.3 GiB, 45.0% used
  search_info: backend:`rhodecode.lib.index.whoosh`
  search_location: /home/ubuntu/.rccontrol/community-1/data/index
  ---
  vcs_backends: Enabled backends in order: hg,git,svn
  vcs_server: 127.0.0.1:10010@ver:4.16.2 via http mode[workers:2], connection:connected
  git: version reported from VCSServer: 2.19.2
  hg: version reported from VCSServer: 4.6.2
  svn: version reported from VCSServer: 1.10.2 (r1835932)

PYTHON PACKAGES
---------------

alembic: 1.0.5
amqp: 2.3.1
appenlight-client: 0.6.26
atomicwrites: 1.2.1
attrs: 18.2.0
Authomatic: 0.1.0.post1
Babel: 1.3
backports.shutil-get-terminal-size: 1.0.0
Beaker: 1.9.1
beautifulsoup4: 4.6.3
billiard: 3.5.0.3
bleach: 3.0.2
bumpversion: 0.5.3
celery: 4.1.1
Chameleon: 2.24
channelstream: 0.5.2
Click: 7.0
colander: 1.5.1
configobj: 5.0.6
configparser: 3.7.1
cov-core: 1.15.0
coverage: 4.5.1
cssselect: 1.0.3
decorator: 4.1.2
deform: 2.0.7
defusedxml: 0.5.0
dm.xmlsec.binding: 1.3.7
docutils: 0.14
dogpile.cache: 0.7.1
dogpile.core: 0.4.1
ecdsa: 0.13
elasticsearch: 6.3.1
elasticsearch-dsl: 6.3.1
elasticsearch1: 1.10.0
elasticsearch1-dsl: 0.0.12
elasticsearch2: 2.5.0
entrypoints: 0.2.2
enum34: 1.1.6
FormEncode: 1.2.4
funcsigs: 1.0.2
functools32: 3.2.3.post2
future: 0.14.3
futures: 3.0.2
gevent: 1.4.0
gnureadline: 6.3.8
gprof2dot: 2017.9.19
greenlet: 0.4.15
gunicorn: 19.9.0
hupper: 1.4.2
infrae.cache: 1.0.1
invoke: 0.13.0
ipaddress: 1.0.22
ipdb: 0.11
ipython: 5.1.0
ipython-genutils: 0.2.0
iso8601: 0.1.12
isodate: 0.6.0
itsdangerous: 0.24
Jinja2: 2.9.6
jsonschema: 2.6.0
jupyter-client: 5.0.0
jupyter-core: 4.4.0
kombu: 4.2.1
lxml: 4.2.5
Mako: 1.0.7
Markdown: 2.6.11
MarkupSafe: 1.1.0
meld3: 1.0.2
mistune: 0.8.4
mock: 1.0.1
more-itertools: 5.0.0
msgpack-python: 0.5.6
MySQL-python: 1.2.5
nbconvert: 5.3.1
nbformat: 4.4.0
packaging: 15.2
pandocfilters: 1.4.2
Paste: 3.0.5
PasteDeploy: 2.0.1
PasteScript: 3.0.0
pathlib2: 2.3.3
peppercorn: 0.6
pexpect: 4.6.0
pickleshare: 0.7.5
pip: 9.0.1
plaster: 1.0
plaster-pastedeploy: 0.6
pluggy: 0.8.1
prompt-toolkit: 1.0.15
psutil: 5.4.8
psycopg2: 2.7.5
ptyprocess: 0.6.0
py: 1.6.0
py-bcrypt: 0.4
py-gfm: 0.1.4
pyasn1: 0.4.5
pyasn1-modules: 0.2.4
pycrypto: 2.6.1
pycurl: 7.43.0.2
pyflakes: 0.8.1
Pygments: 2.3.1
PyMySQL: 0.8.1
pyotp: 2.2.7
pyparsing: 2.3.0
pyramid: 1.10.1
pyramid-beaker: 0.8
pyramid-debugtoolbar: 4.5
pyramid-jinja2: 2.7
pyramid-mailer: 0.15.1
pyramid-mako: 1.0.2
pysqlite: 2.8.3
pytest: 3.8.2
pytest-cov: 2.6.0
pytest-profiling: 1.3.0
pytest-runner: 4.2
pytest-sugar: 0.9.1
pytest-timeout: 1.3.2
Python: 2.7.15
python-dateutil: 2.7.5
python-editor: 1.0.4
python-ldap: 3.1.0
python-memcached: 1.59
python-pam: 1.8.4
python-saml: 2.4.2
pytz: 2018.4
pyzmq: 14.6.0
redis: 2.10.6
repoze.lru: 0.7
repoze.sendmail: 4.4.1
requests: 2.9.1
rhodecode-enterprise-ce: 4.16.2
rhodecode-tools: 1.2.1
Routes: 2.4.1
scandir: 1.9.0
setproctitle: 1.1.10
setuptools: 38.4.0
setuptools-scm: 2.1.0
simplegeneric: 0.8.1
simplejson: 3.16.0
six: 1.11.0
SQLAlchemy: 1.1.18
sshpubkeys: 2.2.0
subprocess32: 3.5.3
supervisor: 3.3.5
Tempita: 0.5.2
termcolor: 1.1.0
testpath: 0.4.2
traitlets: 4.3.2
transaction: 2.4.0
translationstring: 1.3
tzlocal: 1.5.1
urllib3: 1.24.1
URLObject: 2.4.3
venusian: 1.2.0
vine: 1.2.0
waitress: 1.1.0
wcwidth: 0.1.7
webencodings: 0.5.1
WebError: 0.10.3
WebHelpers: 1.3
WebHelpers2: 2.0
WebOb: 1.8.4
WebTest: 2.0.32
wheel: 0.30.0
Whoosh: 2.7.4
ws4py: 0.5.1
wsgiref: 0.1.2
zope.cachedescriptors: 4.3.1
zope.deprecation: 4.3.0
zope.event: 4.3.0
zope.interface: 4.5.0

SYSTEM SETTINGS
---------------

cert_path: /home/ubuntu/.rccontrol-profile/etc/ca-bundle.crt

[config]appenlight.environ_keys_whitelist: 
[config]license_token: 3d34-e18a-6108-11e9
[config]rc_cache.cache_repo_longterm.max_size: 10000
[config]celery.broker_url: amqp://rabbitmq:qweqwe@localhost:5672/rabbitmqhost
[config]startup.import_repos: False
[config]mako.imports: from rhodecode.lib.base import h_filter
[config]labs_settings_active: true
[config]sqlalchemy.db1.pool_recycle: 3600
[config]vcs.hooks.host: 127.0.0.1
[config]ssh.wrapper_cmd: ~/.rccontrol/community-1/profile/bin/rc-ssh-wrapper
[config]rc_cache.cache_perms.arguments.filename: /home/ubuntu/.rccontrol/community-1/data/rc_cache_1
[config]vcs.hooks.direct_calls: False
[config]ssh.enable_debug_logging: false
[config]vcs.connection_timeout: 3600
[config]svn.proxy.config_file_path: /home/ubuntu/.rccontrol/community-1/mod_dav_svn.conf
[config]rc_cache.cache_license.backend: dogpile.cache.rc.file_namespace
[config]rc_cache.cache_license.expiration_time: 300
[config]vcs.server: 127.0.0.1:10010
[config]supervisor.uri: 127.0.0.1:10000
[config]exception_tracker.store_path: /tmp
[config]rc_cache.cache_repo.backend: dogpile.cache.rc.file_namespace
[config]lang: en
[config]channelstream.server: 127.0.0.1:9800
[config]svn.proxy.generate_config: false
[config]appenlight.logging.level: WARNING
[config]vcs.start_server: False
[config]appenlight.api_public_key: YOUR_API_PUBLIC_KEY
[config]server:main: {'max_requests': '1000', 'workers': '2', 'debug': 'false', 'worker_class': 'gevent', 'use': 'egg:gunicorn#main', 'max_requests_jitter': '30', 'host': '127.0.0.1', 'timeout': '21600', 'proc_name': 'gunicorn-community-1', 'port': '10020', 'worker_connections': '10'}
[config]vcs.server.log_level: info
[config]allow_repo_location_change: true
[config]ssh.generate_authorized_keyfile: false
[config]appenlight.slow_request_time: 1
[config]beaker.session.httponly: true
[config]ssh.executable.git: ~/.rccontrol/vcsserver-1/profile/bin/git
[config]gzip_responses: False
[config]rc_cache.sql_cache_short.max_size: 10000
[config]instance_id: ip-172-31-6-155-23218
[config]channelstream.proxy_path: /_channelstream
[config]channelstream_config: {'server': '127.0.0.1:9800', 'secret': 'secret', 'enabled': False}
[config]channelstream.enabled: false
[config]app.base_url: http://rhodecode.local
[config]vcs.server.enable: True
[config]use_htsts: false
[config]mako.directories: ['rhodecode:templates']
[config]rss_cut_off_limit: 256000
[config]svn.proxy.list_parent_path: true
[config]rss_items_per_page: 10
[config]vcs_full_cache: true
[config]celery.task_always_eager: false
[config]is_test: False
[config]rhodecode.api.url: /_admin/api
[config]rc_cache.cache_license.arguments.filename: /home/ubuntu/.rccontrol/community-1/data/rc_cache_3
[config]appenlight.logging_on_error: false
[config]use_celery: false
[config]ssh.executable.svn: ~/.rccontrol/vcsserver-1/profile/bin/svnserve
[config]sqlalchemy.db1.echo: false
[config]beaker.session.timeout: 2592000
[config]supervisor.group_id: community-1
[config]generate_js_files: false
[config]cache_dir: /home/ubuntu/.rccontrol/community-1/data
[config]ssh.wrapper_cmd_allow_shell: false
[config]rc_cache.cache_repo.arguments.filename: /home/ubuntu/.rccontrol/community-1/data/rc_cache_2
[config]allow_custom_hooks_settings: true
[config]force_https: false
[config]file_store.enabled: true
[config]search.module: rhodecode.lib.index.whoosh
[config]svn.proxy.location_root: /
[config]lock_ret_code: 423
[config]cut_off_limit_diff: 512000
[config]file_store.backend: local
[config]rhodecode.edition: Community Edition
[config]vcs.server.protocol: http
[config]__file__: /home/ubuntu/.rccontrol/community-1/rhodecode.ini
[config]app_instance_uuid: 250aa0f3b67f4f70871e3b2fa03c2804
[config]custom.conf: 1
[config]git_rev_filter: --branches --tags
[config]appenlight.slow_requests: true
[config]beaker.session.key: community-1
[config]rc_cache.sql_cache_short.expiration_time: 30
[config]file_store.storage_path: /home/ubuntu/.rccontrol/community-1/data/file_store
[config]rc_cache.cache_repo_longterm.backend: dogpile.cache.rc.memory_lru
[config]rc_cache.cache_perms.backend: dogpile.cache.rc.file_namespace
[config]rss_include_diff: false
[config]vcs.scm_app_implementation: http
[config]channelstream.history.location: /home/ubuntu/.rccontrol/community-1/channelstream_history
[config]default_encoding: ['cp1252', 'UTF-8']
[config]ssh.authorized_keys_file_path: ~/.ssh/authorized_keys_rhodecode
[config]appenlight.report_404: false
[config]beaker.session.type: file
[config]ssh.executable.hg: ~/.rccontrol/vcsserver-1/profile/bin/hg
[config]mako.default_filters: h_filter
[config]git_update_server_info: false
[config]appenlight.log_namespace_blacklist: 
[config]beaker.session.data_dir: /home/ubuntu/.rccontrol/community-1/data/sessions
[config]gist_alias_url: 
[config]rc_cache.cache_repo_longterm.expiration_time: 345600
[config]rc_cache.sql_cache_short.backend: dogpile.cache.rc.memory_lru
[config]api_access_controllers_whitelist: 
[config]vcs.backends: ['hg', 'git', 'svn']
[config]vcs.svn.compatible_version: 
[config]cut_off_limit_file: 128000
[config]beaker.session.lock_dir: /home/ubuntu/.rccontrol/community-1/data/sessions/lock
[config]vcs.hooks.protocol: http
[config]rc_cache.cache_repo.expiration_time: 2592000
[config]channelstream.ws_url: ws://rhodecode.yourserver.com/_channelstream
[config]appenlight.logging: true
[config]appenlight: False
[config]search.location: /home/ubuntu/.rccontrol/community-1/data/index
[config]rc_cache.cache_perms.expiration_time: 300
[config]appenlight.request_keys_blacklist: 
[config]beaker.session.auto: false
[config]here: /home/ubuntu/.rccontrol/community-1
[config]debug: false
[config]celery.max_tasks_per_child: 100
[config]appenlight.server_url: https://api.appenlight.com
[config]sqlalchemy.db1.convert_unicode: true
[config]auth_ret_code: 
[config]beaker.session.secure: false
[config]auth_ret_code_detection: false
[config]ssh.enable_ui_key_generator: true
[config]pyramid.includes: ['rhodecode.lib.middleware.request_wrapper']

path: /home/ubuntu/.rccontrol/community-1/rhodecode.ini

Comments (3)

  1. RhodeCode GmbH

    Looks like cache db got corrupted, there is no valuable information stored there except cache that will be rebuild. I think best way to fix it is to delete the cachedb and it will be simply re-created.

    From the logs you’d need to find which one is it, or you but it’s one of the

    [config]rc_cache.cache_perms.arguments.filename: /home/ubuntu/.rccontrol/community-1/data/rc_cache_1

    [config]rc_cache.cache_repo.arguments.filename: /home/ubuntu/.rccontrol/community-1/data/rc_cache_2

    [config]rc_cache.cache_license.arguments.filename: /home/ubuntu/.rccontrol/community-1/data/rc_cache_3

  2. Pablo A. Revert reporter

    It works like a charm!!! Thanks a lot.

    I’ve stopped rhodecode, clear log files (rc_cache* & cache*) and everything works great now.

  3. Log in to comment