1. Marcin Kuzminski
  2. RhodeCode
  3. Issues


Issue #122 resolved

Database is locked

created an issue

I get this error once in a while when the statistics are being created.



Task rhodecode.lib.celerylib.tasks.get_commits_stats[2bac406f-e74c-480c-a3ae-42afc80bde4b] raised exception: SQLAlchemyError('(OperationalError) database is locked',) Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/celery-2.2.4-py2.6.egg/celery/execute/trace.py", line 34, in trace return cls(states.SUCCESS, retval=fun(args, kwargs)) File "/usr/local/lib/python2.6/dist-packages/celery-2.2.4-py2.6.egg/celery/task/base.py", line 227, in call return self.run(*args, kwargs) File "/usr/local/lib/python2.6/dist-packages/celery-2.2.4-py2.6.egg/celery/app/init.py", line 141, in run return fun(args, kwargs) File "<string>", line 2, in get_commits_stats File "/usr/local/lib/python2.6/dist-packages/RhodeCode-1.1.4-py2.6.egg/rhodecode/lib/celerylib/__init__.py", line 91, in __wrapper ret = func(*fargs, fkwargs) File "/usr/local/lib/python2.6/dist-packages/RhodeCode-1.1.4-py2.6.egg/rhodecode/lib/celerylib/tasks.py", line 210, in get_commits_stats stats.languages = json.dumps(get_codes_stats(repo_name)) File "/usr/local/lib/python2.6/dist-packages/RhodeCode-1.1.4-py2.6.egg/rhodecode/lib/celerylib/tasks.py", line 382, in get_codes_stats repos_path = get_repos_path() File "/usr/local/lib/python2.6/dist-packages/RhodeCode-1.1.4-py2.6.egg/rhodecode/lib/celerylib/tasks.py", line 74, in get_repos_path q = sa.query(RhodeCodeUi).filter(RhodeCodeUi.ui_key == '/').one() File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/query.py", line 1646, in one ret = list(self) File "/usr/local/lib/python2.6/dist-packages/RhodeCode-1.1.4-py2.6.egg/rhodecode/model/caching_query.py", line 79, in iter return Query.iter(self) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/query.py", line 1688, in iter self.session._autoflush() File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/session.py", line 862, in _autoflush self.flush() File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/session.py", line 1388, in flush self._flush(objects) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/session.py", line 1469, in _flush flush_context.execute() File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 302, in execute rec.execute(self) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/unitofwork.py", line 446, in execute uow File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/orm/mapper.py", line 1845, in _save_obj execute(statement, params) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/engine/base.py", line 1191, in execute params) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement return self.execute_context(context) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/engine/base.py", line 1302, in execute_context context.parameters[0], context=context) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/engine/base.py", line 1401, in _cursor_execute context) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/engine/base.py", line 1394, in _cursor_execute context) File "/usr/local/lib/python2.6/dist-packages/SQLAlchemy-0.6.6-py2.6.egg/sqlalchemy/engine/default.py", line 299, in do_execute cursor.execute(statement, parameters) OperationalError: (OperationalError) database is locked


Any ideas?


Comments (12)

  1. Marcin Kuzminski repo owner

    Well, this is sqlite limitations of open connection/transactions to sqlite database, when transaction is open the db file is locked for writing. Some race conditions may occur, did You browse the app while statistics build ? I don't know why this happened, the demo version in under quite a load, people tend to play with it, forking large repos and browsing app, while statistics are build and i never had this.Is it happening just when You generate statistics ?

    You can use postgres for db engine if you whish, and if you have a large number of users concurrently accessing rhodecode that would also give You better performance than sqlite

  2. striker69 reporter

    I had modify the line for alchemy to use mysql:

    sqlalchemy.db1.url = mysql://user:pwd!@mysqlserver/DBName

    But that didn't work right away. I had to modify db.py as well as db_manage.py (well not sure db_manage.py modifications were required). I attached the modifications.

  3. Marcin Kuzminski repo owner

    One more thing. You might want to keep class UserLog(Base): action = Column("action", String(length=100*1000*12...

    since action keeps a string of pushed commits hashes, and 250 will limit to 250/(12+1+4)=14 commits keept in log so if You'll push more than 14 revisions it will be lost.

  4. striker69 reporter

    Did not work...

    Column length too big for column 'action' (max = 65535); use BLOB or TEXT instead

    I had to change its type to Text for this to work. Is it ok?

  5. striker69 reporter

    Just got that error while the statistics are building...

    DataError: (DataError) (1406, "Data too long for column 'commit_activity' at row 1") 'UPDATE statistics SET stat_on_revision=%s, commit_activity=%s, commit_activity_combined=%s WHERE statistics.stat_id = %s' (1799, ...

    The type of commit_activity is LargeBinary. I had to change the type in MySQL to LONGBLOB. Not sure what type to use in SQLAlchemy. I found this:

    class sqlalchemy.dialects.mysql.base.LONGBLOB(length=None)
    Bases: sqlalchemy.types._Binary
    MySQL LONGBLOB type, for binary data up to 2^32 bytes.
  6. Log in to comment