1. OpenREM Avatar OpenREM
  2. OpenREM
  3. OpenREM
  4. Issues

Mapping request, study and acquisition names to standard names

Issue #862 resolved
David Platten created an issue

Add the ability to map each request, study and acquisition name to a user-defined name.

This will enable easy dose comparison between systems where the same clinical exposures use different names.

Charts of user-defined names vs. dose metrics will have to be produced.

Required to enable work on #598

I would like charts of standard acquisition name and also of standard study name. I would like standard study / request / procedure name pooled together into one. So if a particular study description and requested procedure name share the same standard name, I would like the data from both to be included in a chart which shows that standard name

So far:

  • An admin user can enable or disable the standard name mapping feature
  • An admin user can define standard names for each modality, each associated with a study description, procedure name, requested procedure name or acquisition protocol
  • Multiple study descriptions, procedure names, requested procedure names or acquisition protocols can be mapped to a standard name all at once
  • If the standard name mapping feature is enabled then users can filter the data in the summary views by standard study, procedure, request and acquisition names
  • CT, radiography, mammography and fluoroscopy charts of standard acquisition / study name
  • When clicking on “Modify” for a standard name entry, make the modification page pick up all study descriptions, procedure names, requested procedure names and acquisition protocols associated with that name
  • Import routines associate any relevant standard name with the study on import for all modalities
  • Standard name data now included in exports (not tested)
  • xlsx exports now include a worksheet for each standard acquisition protocol, prefixed with “[standard]“

To do:

  • Add some tests
  • Write some documentation

‌

Comments (154)

  2. David Platten reporter

    Added skeleton form to enable a user to enter a standard name. Need to populate the other fields with drop-down options based on existing database contents. May need a form for each modality. Also need a view to show the user all the existing standard names (probably a bit like the display name page). Refs issue #862

    → <<cset f579a6122f9c>>

  7. David Platten reporter

    Adding modality as a hidden field so that I can use the modality to customise the URL, like /add_new/ct; /add_new/dx etc (not implemented yet). Refs issue #862

    → <<cset 5e84ba982f72>>

  14. David Platten reporter

    Added views to show the current standard names. Based on the display name code - needs updating, but does show the standard names. Need to make it possible to edit or delete a current entry. Refs issue #862

    → <<cset 112295ef5a0e>>

  17. David Platten reporter

    Added ability to modify a standard name entry. Added links from standard name table to the delete and modify pages. Added a link in the Config menu to the standard name page. Refs issue #862

    → <<cset f1a75ad6dd83>>

  19. David Platten reporter

    Added a system-level setting to configure if standard names should be used. Added display of standard names in the CT summary page. Modified standard name table so that a requested procedure name, study description, procedure name and acquisition protocol can only appear in the table once - not sure if this will cause issues. It may be that these fields should be unique with modality, rather than just unique on their own. Refs issue #862

    → <<cset 7c39ab6e3850>>

  22. David Platten reporter

    Added display of standard acquisition protocol in the detailed CT view. Created check for empty fields in the standard name forms to ensure that None is returned, otherwise an empty string is stored in the database and causes issues with the unique field constraints. Refs issue #862

    → <<cset efb0864afe36>>

  23. David Platten reporter

    Added a standard name base form, and using this as the basis for the modality-specific ones. Any existing entries in the standard name table are removed from the form options. For example, if the requested procedure name CT head already appears in a standard name, then CT head will not appear in the drop down list of requested proecdure names when creating a new CT standard name. Refs issue #862

    → <<cset 9cffaec69294>>

  26. Ed McDonagh

    Looking good @David Platten

    It would be good to have something other than None appear in the menus and tables - not sure if it is Any maybe? Though that wouldn’t apply in the filter list pages.

    Regarding RadLex, how would this be surfaced? My concern about RadLex, and I’m not sure it is a valid concern or not, is whether for a UK audience it is too American, and for a non-English audience whether it is useful at all. But either way, I presume you could just ignore it? Would there be a facility to load up an alternative localised version, either in a future release or at an institution level?

  33. David Platten reporter

    Making study description, procedure name, requested procedure name and acqisition protocol unique per modality in the standard name table. They were globally unique, so if a DX study description matched a CT study description then you could only use it for one of the two modalities. Refs issue #862

    → <<cset b3d8b3a646ee>>

  37. David Platten reporter

    Added skelton CT bar chart of standard study name - no data provided for this yet. Replaced the standard study, procedure and requested procedure name filters with one standard study filter. Refs issue #862

    → <<cset 13694c0443c1>>

  38. David Platten reporter

    Added CT bar chart of standard study name - mean and median implemented. There's something wrong with the standard name filtering at the moment. Refs issue #862

    → <<cset 62f7244c634c>>

  39. David Platten reporter

    The filtering does work - it just provided a strange result due to the way I have configured my standard names at the moment.

  41. David Platten reporter

    Refactored out a method to create a DataFrame of standard_study data - will be useful for other chart routines such as boxplot and histogram. Refs issue #862

    → <<cset 66a407ba90bd>>

  42. David Platten reporter

    Reverted create_dataframe_aggregates to original code. Using create_standard_study_df in the view_charts code and passing this data frame to the chart function. Boxplot and histogram CT standard name charts now working. Refs issue #862

    → <<cset 726f200a017a>>

  43. David Platten reporter

    Added CT standard study frequency chart. Moved creation of standard study dataframe so that it is only calculated once, rather than for each chart. Refs issue #862

    → <<cset f0bee192586e>>

  44. David Platten reporter

    Added CT standard study workload chart. Rearranged order of CT charts so they appear on the screen in the same order as they appear in the chart options. Refs issue #862

    → <<cset 90b485abe4ae>>

  46. David Platten reporter

    Reordered the CT standard name charts so that the workload chart is at the bottom, the same as for the study description chart order. Refs issue #862

    → <<cset f947dfe39e19>>

  48. David Platten reporter

    Only include standard study charts in the required list if the standard names feature is enabled. Added a horizontal bar at the start of the standard name chart options on the CT filtered page and the general chart options page. Refs issue #862

    → <<cset 6a560a723583>>

  52. David Platten reporter

    Collapsed the CT chart options on the CT filtered page into categories. This tidies up the options part of the page, I think. Refs issue #862

    → <<cset 85dccc600bc8>>

  57. David Platten reporter

    Added CT chart of standard acquisition name frequency. Solved issue of blank still appearing - needed to remove the unused category. Refs issue #862

    → <<cset 381ee5d5ee95>>

  60. David Platten reporter

    Correcting typo from an earlier commit where I used 'field=' instead of 'field_name=' - this caused the DX, RF and MG filtered pages to break. Refs issue #862

    → <<cset 57eaf82f24dd>>

  68. David Platten reporter

    Using a many to many relationship for the standard names. Massive speed increase. Not finished yet, but CT should work (not sure if the tests will pass). Sort of refs issue #862

    → <<cset ad36b350800f>>

  71. David Platten reporter

    Updated acqusition protocol standard names to match method of study-level ones. Need to update CT standard protocol charts to use the new data. Refs issue #862

    → <<cset 026f47502b26>>

  72. David Platten reporter

    Updated the CT acquisition filters and charts to use the many to many field. Removed redundant checks for presence of acquisition level filters which related to the old HighCharts plots in CT and DX. Refs issue #862

    → <<cset 8d0417a4b5fa>>

  75. David Platten reporter

    Added JavaScript to add, update and delete form submit buttons so that the user is warned that database is being updated and it may take a while. Refs issue #862

    → <<cset f45cc85c4307>>

  76. David Platten reporter

    Updated acquisition-level standard name updating and adding to ensure that only the required modality items in IrradEventXRayData.objects are changed. Updated MG and RF pages to use the manytomany standard names. Refs issue #862

    → <<cset 8aae76076379>>

  77. Ed McDonagh

    Is it possible to select multiple names for mapping simultaneously? When I last installed this branch, trying to select each of the CT TAP protocols one at a time was a bit frustrating! Worth doing, but could be nicer.

  78. David Platten reporter

    Corrected bug where if a user cancelled the form it caused an error. Disabled cancel button on form once a user has clicked submit. Refs issue #862

    → <<cset 5303d373b3dc>>

  81. David Platten reporter

    Changed the form for adding and modifying standard names. Uses a layout similar to the Django user admin pages. Making use this multiselect JavaScript library: http://loudev.com/ and this JavaScript quicksearch library: https://github.com/riklomas/quicksearch. Need to modify the display of standard names so that there is one row per standard name showing all study descriptions etc that are associated with it. Also need to modify the updating page so that it shows all the currently selected items for that standard name. Refs issue #862

    → <<cset 8d926865702c>>

  84. David Platten reporter

    @Ed McDonagh I didn’t answer your other question. Django isn’t using the method I have implemented on the user pages. That method is specific to the admin view, and I cannot get it to work on a non-admin page. I’ve done a fair bit of searching and haven’t found an example that uses the admin view’s method on a non-admin form that works for me.

  86. David Platten reporter

    Updated standard study name addition and modify pages to use exactly the same code as that used by the Django user admin page. Refs issue #862

    → <<cset 6742b4fc1d09>>

  88. David Platten reporter

    Standard name view now groups entries so each standard name appears one. The Modify and Delete buttons now work as expected. It's possible to increase the efficiency of the Modify button code so that only changed data is updated / added - at the moment a blanket approach is used. Refs issue #862

    → <<cset cc1f23d44f8e>>

  94. David Platten reporter

    Grouped fluoroscopy chart options on the filtered view. Made all filtered view chart options visible by default (they can then be collapsed by the user by clicking on the heading). Refs issue #862

    → <<cset 67d3cfce6a65>>

  96. David Platten reporter

    Added standard study name charts for radiography. Also changed check for empty queryset from 'if not f.qs' to 'if not f.qs.exists()' - this is much quicker to execute. Refs issue #862

    → <<cset 9975b6c68aea>>

  98. David Platten reporter

    Some optimisation on chart queries when just standard name charts are selected: any None standard name entries are excluded from the queryset before creation of the dataframe. This increases speed. Refs issue #862

    → <<cset 836098536101>>

  99. David Platten reporter

    Check for required standard name charts before trying to exclude data from the standard name dataframe. Fixes the failed test (on my local machine at least). Refs issue #862

    → <<cset 219c0aac8270>>

  103. David Platten reporter

    Made overflow auto for the value selection boxes on the add / modify standard name mapping page. Some of my fluoro names are long so it's good to be able to scroll to read them. Refs issue #862

    → <<cset 4abd5e072b56>>

  112. David Platten reporter

    Adding standard acquisition names to exports. Tested CT, but not the other modalities. Need to add an option to enable a worksheet per standard acquisition name, rather than per protocol. Refs issue #862

    → <<cset 9115b75df8de>>

  116. David Platten reporter

    Trying to fix RF exports when standard names activated. Frequency of study descriptions and requested procedures remains incorrect on the summary page of the xlsx file. Refs issue #862

    → <<cset 163a40500df6>>

  117. David Platten reporter

    Solved the issue of study descriptions, requested procedure names and standard names being counted multiple times in the export summary sheet whenever a values was entered in the standard acquisition name filter. This was affecting all modality exports. Refs issue #862

    → <<cset 0576ff14aac0>>

  118. David Platten reporter

    Adding this line to my local_settings.py has been really useful when debugging the export routines:

    CELERY_TASK_ALWAYS_EAGER  = True

    It forces the delayed celery tasks to run in the same thread as the main python, making it easy to insert breakpoints and debug any code that is usually a delayed action.

  120. David Platten reporter

    Removing code that excluded studies with null standard acquisition names from the chart queryset if just standard acquisition protocol charts were being asked for. This didn't work as I expected because the exclusion removes studies if any acqusition in that study has a null standard acquistion name, rather than exlcuding the study only if all of the acquisitions have a null standard acquisition name. Thanks to my colleage Rory for testing the system and pointing out this problem. Refs issue #862

    → <<cset 0f460618b279>>

  121. David Platten reporter

    Moved aggregated dataframe creation within the mean and median if statement - only need to create it for these. Will speed up code when just boxplots or histograms are required. Refs issue #862

    → <<cset ebd7e1a30623>>

  124. David Platten reporter

    Improving table titles, replacing NaN with - in the table, switching off escape of special characters so that CTDIvol is correctly subscripted in the table headings. Sort-of refs issue #862

    → <<cset 00c9f1528349>>

  126. David Platten reporter

    @Ed McDonagh I’m unsure of how to update the export feature. I have already included standard study names and standard acquisition names in the existing export documents.

    The existing xlsx export routines create a worksheet for each acquisition protocol name. I would like the user to be able to select an export where a worksheet is created for each standard acquisition name, but am unsure about how to do this. Do you have any guidance?

  127. Ed McDonagh

    I guess there’s three options?

    1. Where you configure standard names on/off, also determine whether standard names are used instead of original names where available in exports
    2. Duplicate set of export buttons to export with standard names instead of original
    3. Export both somehow!

    I think what is probably want is tabs with standard names where they exist, but a standard name and an original name column in the sheets.

    Does this help in any way? Or am I barking up the wrong tree?

    ‌

  128. David Platten reporter

    Thanks @Ed McDonagh that’s helpful.

    I think I’ll go with creating worksheets for both if the standard names option is switched on. I’ll have to prefix the standard name sheets with something in case a standard name is exactly the same as a regular acquisition name. Perhaps “Std name: “.

  132. David Platten reporter

    Removing the max_length of the procedure_code_meaning field. The restriction is causing problems with my test system where I have updated this field with longer-than-usual names. Refs issue #862

    → <<cset 5a5ae7ef9a58>>

  133. David Platten reporter

    @Ed McDonagh we have been using this branch on a test server to come up with local DRLs using the standard name mapping feature. We think it works well. The feature needs documentation writing, and perhaps some tests, otherwise it is good to go.

  135. Ed McDonagh

    Hmm. I just tried migrating my database to the new develop merged branch and it is not working. makemigrations seemed happy, but migrate is complaining:

    Running migrations:
  Applying remapp.0016_auto_20220519_0903...Traceback (most recent call last):
  File "/home/mcdonaghe/research/vep38/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
psycopg2.errors.DuplicateTable: relation "remapp_standardnames" already exists

    I don’t know yet if that means the table is already in the database (in which case why didn’t makemigrations say so) or if there is an issue with the inter-table relationships…

  136. Ed McDonagh

    I do have a StandardNames table in my database (via PGAdmin), and it has one row in. I wonder why makemigrations thinks it needs to be added again? I’ll delete all my migration files and try again.

  150. Ed McDonagh

    @David Platten can you take a look at dev.openrem.org and tell me if there is a reason I get nothing in the “Chart of standard study name mean DLP” (CT, filtering and chart options are in the link)

  155. Log in to comment
Assignee
Type
enhancement
Priority
major
Status
resolved
Component
Models
Milestone
1.1.0
Votes
0
Watchers
1
Jira: the preferred issue tracker for Bitbucket. Join the team!