Browser out of memory on tasks page

[David Platten]

When viewing the tasks summary page on my test system there are so many “Older tasks” that my browser crashes (Chrome, out of memory error) and is unable to display the page. This is on my laptop with 32 GB RAM.

I have just checked in the database and there are 158366 rows in the backgroundtask table.

[Ed McDonagh]

Any ideas as to how to deal with this? (Configurable) limit of number of old tasks before they are automatically purged? Some sort of archiving?

I think we can also reduce the impact on the database of the auto-refresh by doing that in a different way.

[David Platten]

Hi Ed. Yes, I think we need to configure a limit on the number of task records kept.

‌

I’ve just deleted all the rows from the table that were completed successfully using PgAdmin4:

DELETE FROM public.remapp_backgroundtask
WHERE completed_successfully = TRUE AND complete = TRUE;

[David Platten]

This page: https://www.appsloveworld.com/django/100/34/limit-the-number-of-rows-in-a-django-table

has a snippet of code that removes the oldest entry in a table if the number of entries exceeds some value:

def my_handler(sender, instance, **kwargs):
    qs = MyModel.objects.order_by('created') #ensure ordering.
    if qs.count() > 10:
        qs[0].delete() #remove the oldest element

class MyModel(models.Model):
    title = models.CharField('title', max_length=200)
    created = models.DateTimeField(auto_add_now=True, editable=False)

post_save.connect(my_handler, sender=MyModel)

‌

Perhaps we could use something like that for the backgroundtask table, where the threshold for deleting is something like 2000 entries?

‌

[Ed McDonagh]

So it just deletes it from the queryset, not the database?

[David Platten]

I’ve not tested it. I thought it deleted from the database table.

‌

[Ed McDonagh]

Yes, looks like it

[David Platten]

We could either keep the database table rows below a certain number, or only retrieve the most recent x rows in the appropriate view when populating the task table.

[Ed McDonagh]

My preference would be to only keep so many records in the table, as long as we can have an admin config of how many that is?

[David Platten]

Added a custom save method to the BackgroundTask table to prevent the number of rows exceeding the value in the max_background_task_rows field in the new BackgroundTaskMaximumRows singleton table. Need to create a simple page for an administrator to modify this value. Refs issue #964

→ <<cset ca6b2b1dc65e>>

[David Platten]

Added a form, view and entry in the Config menu for an OpenREM administrator to be able to change the BackgroundTaskMaximumRows singleton table. Refs issue #964

→ <<cset a4f57aef9ca9>>

[David Platten]

Adding form to update size of stored task history. Updated task documentation. Refs issue #964

→ <<cset 2fd1d38fc1c8>>

[David Platten]

Using post_save signal rather than overriding the save method of the BackgroundTask. The Internet tells me this is more elegant, and Codacy may be happier with this. Refs issue #964

→ <<cset f4742ea38e0e>>

[David Platten]

Making use of the sender argument. Refs issue #964

→ <<cset 6882e056c135>>

[David Platten]

Trying to make Codacy ignore the unsed arguments. Refs issue #964

→ <<cset 261f16adba9f>>

[David Platten]

Re-worded a sentence in the docs to avoid the need for another comma. Added specific link to the task management docs in the task history limit form, and updated the docs link for the task_admin page. Refs issue #964

→ <<cset d8e4791a3d5b>>

[David Platten]

Updating title of documentation links [skip ci]. Refs issue #964

→ <<cset 6a7ebbb488cf>>

[David Platten]

Made config menu screen shot smaller [skip ci]. Refs issue #964

→ <<cset ebc799683628>>

[David Platten]

Captured new, smaller, config menu screen shot [skip ci]. Refs issue #964

→ <<cset 5524c5d8d219>>

[David Platten]

Merged in issue964limitSizeOfStoredBackgroundTaskHistory (pull request #549)

Issue964limitSizeOfStoredBackgroundTaskHistory

Approved-by: Ed McDonagh

Fixes issue #964

→ <<cset b8cfba4d95d6>>