Display name page is slow to populate

[David Platten]

The population of the How many studies and Date of last study columns when viewing the openrem/viewdisplaynames/ page is very slow on my live system. The data to populate these columns is obtained from individual queries for each display name. I think they could all be done at once for each modality.

For example, for CT line 383 of views_admin.py could be replaced with:

from django.db.models import Count, Max
name_set = f.filter(generalequipmentmoduleattr__general_study_module_attributes__modality_type="CT").distinct().annotate(num_entries=Count("generalequipmentmoduleattr__general_study_module_attributes__modality_type"), lates
t_entry_date=Max("generalequipmentmoduleattr__general_study_module_attributes__study_date"))

This annotates a field called num_entries which will contain the number of studies for each name, and annotates another field called latest_entry_date containing the date of the most recent study from that entry.

This would avoid the need for the many individual queries for each modality and the associated Ajax calls.

‌

A question: why does the How many studies column have two values in? Presumably what I am suggesting won’t quite cope with whatever that reason is (dual modality, perhaps?).

[David Platten]

Retrieving count and date of latest study for all systems of a given modality all at once, rather than one at a time. This update has removed the 'xxx (of xxxx)' display of the number of studies for each modality. The 'How many studies' column now just shows the total number of studies for that modality. Refs issue #983

→ <<cset 00bad6cfbeec>>

[David Platten]

I’ve made these changes because I’ve been using the viewdisplaynames page quite a bit on my live system recently, and it is really slow to populate. It can take several minutes for the page to fully populate. When using the code from this branch it takes a few tens of seconds.

On my high-end test system (16 core AMD Ryzen 5950X, 64 GB RAM) I have used the Django debug toolbar to time things: the develop branch takes 21s to fully populate. On the same system this branch takes 3s to fully populate.

‌

[David Platten]

Another set of timings from the Django debug toolbar with OpenREM, this time running on my laptop:

131s to fully populate the page (develop branch)

33s to fully populate the page (issue983speedUpDisplayNamePage branch)

‌

These are the number of display name entries and studies in my test system:

Modality	Number of display name entries	Number of studies
CT	118	359298
Fluoroscopy	53	22164
Mammography	64	186068
Radiography	163	996401
Nuclear medicine	1	1

‌

‌

[Ed McDonagh]

HI David

The x (of x) does seem to relate to dual modality devices. What does your branch do with those - count just the studies of that modality of that device, or count all the studies of that device and display it twice (once in each table)?

[David Platten]

For dual-modalities my branch just counts all the studies of that device and displays the value in the radiographic and fluoroscopy entry.

[David Platten]

Added a method to obtain the count of failed imports for all modalities in a single query, rather than running a query per modality. Now only needs to be called once. Refs issue #983

→ <<cset 0912ccc5388d>>

[David Platten]

For the OT modality, replaced the multiple joined ~Q statements in the filter with a single __in Q statement - this is much faster. I've also changed the Count annotations to use Count(generalequipmentmoduleattr__pk) which may be a little quicker than Count(generalequipmentmoduleattr). Refs issue #983

→ <<cset 43cab27877dc>>

[David Platten]

Replaced chained Q statements with a single Q(__in) statement where possible in views_admin.py. Removed all __exact in views_admin.py because __exact is implied, so not required. Refs issue #983

→ <<cset d06553e8dd68>>

[David Platten]

Replaced chained Q statements with a single Q(__in) statement where possible in views.py. Removed all __exact in views.py because __exact is implied, so not required. Refs issue #983

→ <<cset 0cc5dbd447bf>>

[David Platten]

Replaced all occurences of chained Q statements with a single Q(__in) statement. Removed some __exact because __exact is implied, so not required. Refs issue #983

→ <<cset 43379d706bbc>>

[David Platten]

Merging develop into this branch and fixing conflicts with views.py due to earlier merge of issue #984 branch. Refs issue #983

→ <<cset 9354c70fbaa8>>

[David Platten]

Adding an index to display_name which should speed things up a little. Refs issue #983

→ <<cset 9acd7d4565be>>

[David Platten]

Merged the two definitions of Meta. Refs issue #983

→ <<cset 1d2222db5262>>

[David Platten]

I’d really like this update to be in version 1.0 because it improves the usability of the display name page.

[Ed McDonagh]

View Display Names page, fresh server installs with same database imported. 670 entries across CT, Fluoro, Mammo, Radiography, Other.

1.0.0b2	This branch
53.56 s 54.76 s	13.07 s 14.34 s

For systems that are dual modality, as suggested this branch presents the total number of studies in both the Fluoro and Radiography tables, but if you Review you can see the of which x are DX etc numbers.

[David Platten]

Merged in issue983speedUpDisplayNamePage (pull request #584)

Issue983speedUpDisplayNamePage

Approved-by: Ed McDonagh

Fixes issue #983

→ <<cset 30236cf24b89>>