Migrate a 0.2.9.1 db to 0.3.0

Issue #312 resolved
Cody Crewson created an issue

I'm not sure that migration has been properly configured.

1) I know I'm jumping the gun so no I am not expecting progress, only hoping

2) I understand that this might be known, if it is please ignore

Here's the stack trace for two migrate attempts. The first was using just migrate, the second was with the --fake-initial argument.

Thank you all

(qatrackPy3) crcrewso@Sparrow:~/web/qatrackplus$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, attachments, auth, contacts, contenttypes, django_comments, issue_tracker, notifications, parts, qa, service_log, sessions, sites, tastypie, units
Running migrations:
  Applying contenttypes.0001_initial...Traceback (most recent call last):
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: table "django_content_type" already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/operations/models.py", line 97, in database_forwards
    schema_editor.create_model(model)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 303, in create_model
    self.execute(sql, params or None)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 120, in execute
    cursor.execute(sql, params)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: table "django_content_type" already exists
(qatrackPy3) crcrewso@Sparrow:~/web/qatrackplus$ python manage.py migrate --fake-initial
Operations to perform:
  Apply all migrations: admin, attachments, auth, contacts, contenttypes, django_comments, issue_tracker, notifications, parts, qa, service_log, sessions, sites, tastypie, units
Running migrations:
  Applying contenttypes.0001_initial... FAKED
  Applying auth.0001_initial... FAKED
  Applying admin.0001_initial... FAKED
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying units.0001_initial... FAKED
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying qa.0001_initial... FAKED
  Applying sites.0001_initial... FAKED
  Applying django_comments.0001_initial... OK
  Applying django_comments.0002_update_user_email_field_length... OK
  Applying django_comments.0003_add_submit_date_index... OK
  Applying qa.0002_029_to_030_first...Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/home/crcrewso/web/qatrackplus/qatrack/qa/migrations/0002_029_to_030_first.py", line 54, in set_utc_name
    utc.name = TestListCycle.objects.get(pk=utc.object_id).name
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
__fake__.DoesNotExist: TestListCycle matching query does not exist.

Comments (19)

  1. Cody Crewson reporter

    I've reread this. Would it help if I removed any TestListCycles that we have in the original before migrating?

  2. Randle Taylor

    Thanks Cody. It is extremely helpful to have people like you testing this before release!

    The --fake-initial is needed for updating a 0.2.9 db (as you discovered).

    I took a quick look at the failing migration and it looks correct to me so I can't explain this failure off the top of my head (it was written by @ryanbottema so maybe he has some insight) . For some reason the migration can't tell if it is dealing with a test list or test list cycle. Deleting your TestListCycles may or may not allow you to continue on (assuming this isn't your production db!).

    This is not a great way to handle it but you could do the following:

    In /qatrack/qa/migrations/0002_029_to_030_first.py

    Make the following change:

             utc_ct = utc.content_type
    
    -        if utc_ct.id == test_list_type.id:
    -            utc.name = TestList.objects.get(pk=utc.object_id).name
    -        else:
    -            utc.name = TestListCycle.objects.get(pk=utc.object_id).name
    +        try:
    +            if utc_ct.id == test_list_type.id:
    +                utc.name = TestList.objects.get(pk=utc.object_id).name
    +            else:
    +                utc.name = TestListCycle.objects.get(pk=utc.object_id).name
    +        except:
    +            utc.name = "Unable to determine"
    
             utc.save()
    

    (i.e. wrap lines 51-54 in a try/except block)

  3. Randle Taylor

    @crcrewso Actually rather than the above change, if you just change line 51 to if utc_ct.name == "test list": it might work correctly.

  4. Cody Crewson reporter

    Randle

    I'll try to find it but it's buried in my history, according to documentation --fake-initial was default in 1.7, and only became explicit in 1.8.

    The 'test list' change didn't work, retrying the try except paying more attention to spaces

  5. Cody Crewson reporter

    The try block worked

    (qatrackPy3) crcrewso@Sparrow:~/web/qatrackplus$ python manage.py migrate --fake-initial
    Operations to perform:
      Apply all migrations: admin, attachments, auth, contacts, contenttypes, django_comments, issue_tracker, notifications, parts, qa, service_log, sessions, sites, tastypie, units
    Running migrations:
      Applying qa.0002_029_to_030_first... OK
      Applying qa.0002_auto_20161218_1851... OK
      Applying attachments.0001_initial... OK
      Applying attachments.0002_auto_20161218_1859... OK
      Applying contacts.0001_initial... FAKED
      Applying issue_tracker.0001_initial... OK
      Applying issue_tracker.0002_auto_20170505_1334... OK
      Applying issue_tracker.0003_auto_20170505_1341... OK
      Applying issue_tracker.0004_auto_20170505_1453... OK
      Applying issue_tracker.0005_auto_20170505_1457... OK
      Applying issue_tracker.0006_issuetag_description... OK
      Applying issue_tracker.0007_auto_20170505_1522... OK
      Applying issue_tracker.0008_auto_20170505_1549... OK
      Applying issue_tracker.0009_auto_20170505_1623... OK
      Applying issue_tracker.0010_auto_20170531_1149... OK
      Applying issue_tracker.0011_auto_20170531_1255... OK
      Applying issue_tracker.0012_auto_20170531_1453... OK
      Applying issue_tracker.0013_auto_20170531_1516... OK
      Applying issue_tracker.0014_auto_20171027_1151... OK
      Applying notifications.0001_initial... FAKED
      Applying notifications.0002_auto_20161003_2213... OK
      Applying service_log.0001_initial... OK
      Applying parts.0001_initial... OK
      Applying parts.0002_auto_20171027_1151... OK
      Applying qa.0003_auto_20171103_1256... OK
      Applying qa.0004_auto_20171103_1615... OK
      Applying qa.0005_auto_20171103_2158... OK
      Applying qa.0006_auto_20171120_2126... OK
      Applying qa.0007_auto_20171124_1102... OK
      Applying qa.0008_sublist_to_children... OK
      Applying qa.0009_remove_testlist_sublists... OK
      Applying qa.0010_auto_20171208_1411... OK
      Applying service_log.0002_auto_20171101_1115... OK
      Applying service_log.0003_auto_20171204_1232... OK
      Applying service_log.0004_auto_20180110_1257... OK
      Applying service_log.0005_auto_20180206_1730... OK
      Applying sessions.0001_initial... FAKED
      Applying sites.0002_alter_domain_unique... OK
      Applying tastypie.0001_initial... FAKED
    
  6. Randle Taylor

    OK thanks. If you could also do me a favour when you are up and running, go into the admin, to the Assign Test Lists to Units section, and take a look to see how many of the UnitTestCollection objects have the name set to "Unable to determine".

    No need to go digging through your notes, I was just confirming that I knew about the --fake-initial flag being required.

  7. Randle Taylor

    Even better, I notice you're using a sqlite db here...are you able to share (email) the (pre migratied) sqlite file with me so I can reproduce the issue? No worries if you can't, but it would make fixing the issue easier.

  8. Cody Crewson reporter

    I just got there, the short answer is all of them!!! So while the migration completed it wasn't successful :(

    Also I can't perform anything

    AttributeError at /qa/utc/perform/131/
    'TestListCycleMembership' object has no attribute 'next_list'
    Request Method: GET
    Request URL:    http://127.0.0.1:8000/qa/utc/perform/131/?day=next&next=/qa/
    Django Version: 1.11.9
    Exception Type: AttributeError
    Exception Value:    
    'TestListCycleMembership' object has no attribute 'next_list'
    Exception Location: /home/crcrewso/web/qatrackplus/qatrack/qa/models.py in next_list, line 1311
    Python Executable:  /home/crcrewso/venvs/qatrackPy3/bin/python
    Python Version: 3.5.2
    Python Path:    
    ['/home/crcrewso/web/qatrackplus',
     '/home/crcrewso/venvs/qatrackPy3/src/django-formtools',
     '/home/crcrewso/venvs/qatrackPy3/src/django-genericdropdown-master',
     '/home/crcrewso/venvs/qatrackPy3/src/django-listable',
     '/usr/lib/python35.zip',
     '/usr/lib/python3.5',
     '/usr/lib/python3.5/plat-x86_64-linux-gnu',
     '/usr/lib/python3.5/lib-dynload',
     '/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages']
    Server time:    Tue, 6 Feb 2018 21:06:48 -0500
    
  9. Cody Crewson reporter

    No, it's totally not sharable, it's a copy of prod on a very very test system. I was hoping to get our semester intern a working full db so he didn't have to create everything from scratch.

    Speaking of, as a separate issue will the default db be updated now that we have 1M+ new features?

  10. Randle Taylor

    Ok thanks anyways.

    Based on that strange traceback, I think maybe something got screwed up with the content types application, possibly during your botched initial migration attempt. I don't think Sqlite can roll back a transaction in the middle of a failed migration like "real" db's and can get left in an ugly state.

    Do you have a copy of the 0.2.9 SQLite file that you could try the migrations on again?

  11. Cody Crewson reporter

    It's getting late. I can redo it from scratch tomorrow, not to worry.

    Next time I'll remember to back up my sqlite files.

    Thank you for your help :)

  12. Cody Crewson reporter

    I did some troubleshooting and it turns out that my DB wasn't quite right before the migration. I can get screenshots later but something is wrong with the db. In any Unit's list in 0.2.9 I can see all the test lists, but there's no names, Additionally, in Admin in the Assign to Unit view, no test lists are visible.

    This DB was generated by creating a json from prod using the command from here

    manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json
    

    So it seems to be a source problem not a migration problem. Is it possible that a table wasn't included in the dumpdata result?

  13. Randle Taylor

    Yeah that explains it. When you create your new database the content type ids don't necessarily match the ones in the old database. To solve that, you should should add '--natural' to your dumpdata command.

    manage.py dumpdata --natural --exclude auth.permission --exclude contenttypes > db.json

  14. Cody Crewson reporter

    Thank you

    Natural takes forever.

    It looks like all this talk has solved another issue, as I now understand more about using Django to restore a db from file (our backups are all using SQL server so I've never had to question it)

    I'll test this out a bit more and reply to this thread if there are any further issues.

  15. Log in to comment