Migrate a 0.2.9.1 db to 0.3.0

[Cody Crewson]

I'm not sure that migration has been properly configured.

1) I know I'm jumping the gun so no I am not expecting progress, only hoping

2) I understand that this might be known, if it is please ignore

Here's the stack trace for two migrate attempts. The first was using just migrate, the second was with the --fake-initial argument.

Thank you all

(qatrackPy3) crcrewso@Sparrow:~/web/qatrackplus$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, attachments, auth, contacts, contenttypes, django_comments, issue_tracker, notifications, parts, qa, service_log, sessions, sites, tastypie, units
Running migrations:
  Applying contenttypes.0001_initial...Traceback (most recent call last):
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: table "django_content_type" already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/operations/models.py", line 97, in database_forwards
    schema_editor.create_model(model)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 303, in create_model
    self.execute(sql, params or None)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/base/schema.py", line 120, in execute
    cursor.execute(sql, params)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: table "django_content_type" already exists
(qatrackPy3) crcrewso@Sparrow:~/web/qatrackplus$ python manage.py migrate --fake-initial
Operations to perform:
  Apply all migrations: admin, attachments, auth, contacts, contenttypes, django_comments, issue_tracker, notifications, parts, qa, service_log, sessions, sites, tastypie, units
Running migrations:
  Applying contenttypes.0001_initial... FAKED
  Applying auth.0001_initial... FAKED
  Applying admin.0001_initial... FAKED
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying units.0001_initial... FAKED
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying qa.0001_initial... FAKED
  Applying sites.0001_initial... FAKED
  Applying django_comments.0001_initial... OK
  Applying django_comments.0002_update_user_email_field_length... OK
  Applying django_comments.0003_add_submit_date_index... OK
  Applying qa.0002_029_to_030_first...Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/home/crcrewso/web/qatrackplus/qatrack/qa/migrations/0002_029_to_030_first.py", line 54, in set_utc_name
    utc.name = TestListCycle.objects.get(pk=utc.object_id).name
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
__fake__.DoesNotExist: TestListCycle matching query does not exist.

[Randle Taylor]

Added code tags to traceback.

[Cody Crewson]

I've reread this. Would it help if I removed any TestListCycles that we have in the original before migrating?

[Randle Taylor]

Thanks Cody. It is extremely helpful to have people like you testing this before release!

The --fake-initial is needed for updating a 0.2.9 db (as you discovered).

I took a quick look at the failing migration and it looks correct to me so I can't explain this failure off the top of my head (it was written by @ryanbottema so maybe he has some insight) . For some reason the migration can't tell if it is dealing with a test list or test list cycle. Deleting your TestListCycles may or may not allow you to continue on (assuming this isn't your production db!).

This is not a great way to handle it but you could do the following:

In /qatrack/qa/migrations/0002_029_to_030_first.py

Make the following change:

         utc_ct = utc.content_type

-        if utc_ct.id == test_list_type.id:
-            utc.name = TestList.objects.get(pk=utc.object_id).name
-        else:
-            utc.name = TestListCycle.objects.get(pk=utc.object_id).name
+        try:
+            if utc_ct.id == test_list_type.id:
+                utc.name = TestList.objects.get(pk=utc.object_id).name
+            else:
+                utc.name = TestListCycle.objects.get(pk=utc.object_id).name
+        except:
+            utc.name = "Unable to determine"

         utc.save()

(i.e. wrap lines 51-54 in a try/except block)

[Randle Taylor]

@crcrewso Actually rather than the above change, if you just change line 51 to if utc_ct.name == "test list": it might work correctly.

[Cody Crewson]

Randle

I'll try to find it but it's buried in my history, according to documentation --fake-initial was default in 1.7, and only became explicit in 1.8.

The 'test list' change didn't work, retrying the try except paying more attention to spaces

[Cody Crewson]

The try block worked

(qatrackPy3) crcrewso@Sparrow:~/web/qatrackplus$ python manage.py migrate --fake-initial
Operations to perform:
  Apply all migrations: admin, attachments, auth, contacts, contenttypes, django_comments, issue_tracker, notifications, parts, qa, service_log, sessions, sites, tastypie, units
Running migrations:
  Applying qa.0002_029_to_030_first... OK
  Applying qa.0002_auto_20161218_1851... OK
  Applying attachments.0001_initial... OK
  Applying attachments.0002_auto_20161218_1859... OK
  Applying contacts.0001_initial... FAKED
  Applying issue_tracker.0001_initial... OK
  Applying issue_tracker.0002_auto_20170505_1334... OK
  Applying issue_tracker.0003_auto_20170505_1341... OK
  Applying issue_tracker.0004_auto_20170505_1453... OK
  Applying issue_tracker.0005_auto_20170505_1457... OK
  Applying issue_tracker.0006_issuetag_description... OK
  Applying issue_tracker.0007_auto_20170505_1522... OK
  Applying issue_tracker.0008_auto_20170505_1549... OK
  Applying issue_tracker.0009_auto_20170505_1623... OK
  Applying issue_tracker.0010_auto_20170531_1149... OK
  Applying issue_tracker.0011_auto_20170531_1255... OK
  Applying issue_tracker.0012_auto_20170531_1453... OK
  Applying issue_tracker.0013_auto_20170531_1516... OK
  Applying issue_tracker.0014_auto_20171027_1151... OK
  Applying notifications.0001_initial... FAKED
  Applying notifications.0002_auto_20161003_2213... OK
  Applying service_log.0001_initial... OK
  Applying parts.0001_initial... OK
  Applying parts.0002_auto_20171027_1151... OK
  Applying qa.0003_auto_20171103_1256... OK
  Applying qa.0004_auto_20171103_1615... OK
  Applying qa.0005_auto_20171103_2158... OK
  Applying qa.0006_auto_20171120_2126... OK
  Applying qa.0007_auto_20171124_1102... OK
  Applying qa.0008_sublist_to_children... OK
  Applying qa.0009_remove_testlist_sublists... OK
  Applying qa.0010_auto_20171208_1411... OK
  Applying service_log.0002_auto_20171101_1115... OK
  Applying service_log.0003_auto_20171204_1232... OK
  Applying service_log.0004_auto_20180110_1257... OK
  Applying service_log.0005_auto_20180206_1730... OK
  Applying sessions.0001_initial... FAKED
  Applying sites.0002_alter_domain_unique... OK
  Applying tastypie.0001_initial... FAKED

[Randle Taylor]

OK thanks. If you could also do me a favour when you are up and running, go into the admin, to the Assign Test Lists to Units section, and take a look to see how many of the UnitTestCollection objects have the name set to "Unable to determine".

No need to go digging through your notes, I was just confirming that I knew about the --fake-initial flag being required.

[Randle Taylor]

Even better, I notice you're using a sqlite db here...are you able to share (email) the (pre migratied) sqlite file with me so I can reproduce the issue? No worries if you can't, but it would make fixing the issue easier.

[Cody Crewson]

I just got there, the short answer is all of them!!! So while the migration completed it wasn't successful :(

Also I can't perform anything

AttributeError at /qa/utc/perform/131/
'TestListCycleMembership' object has no attribute 'next_list'
Request Method: GET
Request URL:    http://127.0.0.1:8000/qa/utc/perform/131/?day=next&next=/qa/
Django Version: 1.11.9
Exception Type: AttributeError
Exception Value:    
'TestListCycleMembership' object has no attribute 'next_list'
Exception Location: /home/crcrewso/web/qatrackplus/qatrack/qa/models.py in next_list, line 1311
Python Executable:  /home/crcrewso/venvs/qatrackPy3/bin/python
Python Version: 3.5.2
Python Path:    
['/home/crcrewso/web/qatrackplus',
 '/home/crcrewso/venvs/qatrackPy3/src/django-formtools',
 '/home/crcrewso/venvs/qatrackPy3/src/django-genericdropdown-master',
 '/home/crcrewso/venvs/qatrackPy3/src/django-listable',
 '/usr/lib/python35.zip',
 '/usr/lib/python3.5',
 '/usr/lib/python3.5/plat-x86_64-linux-gnu',
 '/usr/lib/python3.5/lib-dynload',
 '/home/crcrewso/venvs/qatrackPy3/lib/python3.5/site-packages']
Server time:    Tue, 6 Feb 2018 21:06:48 -0500

[Cody Crewson]

No, it's totally not sharable, it's a copy of prod on a very very test system. I was hoping to get our semester intern a working full db so he didn't have to create everything from scratch.

Speaking of, as a separate issue will the default db be updated now that we have 1M+ new features?

[Randle Taylor]

Ok thanks anyways.

Based on that strange traceback, I think maybe something got screwed up with the content types application, possibly during your botched initial migration attempt. I don't think Sqlite can roll back a transaction in the middle of a failed migration like "real" db's and can get left in an ugly state.

Do you have a copy of the 0.2.9 SQLite file that you could try the migrations on again?

[Cody Crewson]

It's getting late. I can redo it from scratch tomorrow, not to worry.

Next time I'll remember to back up my sqlite files.

Thank you for your help :)

[Randle Taylor]

Sounds good. Thanks!

[Cody Crewson]

I did some troubleshooting and it turns out that my DB wasn't quite right before the migration. I can get screenshots later but something is wrong with the db. In any Unit's list in 0.2.9 I can see all the test lists, but there's no names, Additionally, in Admin in the Assign to Unit view, no test lists are visible.

This DB was generated by creating a json from prod using the command from here

manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json

So it seems to be a source problem not a migration problem. Is it possible that a table wasn't included in the dumpdata result?

[Randle Taylor]

Yeah that explains it. When you create your new database the content type ids don't necessarily match the ones in the old database. To solve that, you should should add '--natural' to your dumpdata command.

manage.py dumpdata --natural --exclude auth.permission --exclude contenttypes > db.json

[Cody Crewson]

Thank you

Natural takes forever.

It looks like all this talk has solved another issue, as I now understand more about using Django to restore a db from file (our backups are all using SQL server so I've never had to question it)

I'll test this out a bit more and reply to this thread if there are any further issues.

[Cody Crewson]

Fixed, correctly exporting a good database solved my problems

[Randle Taylor]

Thanks for the update Cody! Keep us posted if you run into any other issues with 0.3.0.