Add DICOM QR SCU functionality

Issue #207 resolved
Ed McDonagh created an issue

From #41, refs #206

Comments (91)

  1. Ed McDonagh reporter

    Corrected format syntax, corrected multiple if or statements syntax, tested with SR, CR and DX, seems to work so far. No CT/PET/RF/XA to test at the moment. Refs #207

    → <<cset d8999ead49ed>>

  2. Ed McDonagh reporter

    Added ImplementationClassUID for 0.7.0 b1, though this section is likely to be removed! Changed PatientRootFind and Move to StudyRoot in the declared SOP Class SCUs. Refs #207

    → <<cset d5cd6aa7b84f>>

  3. Ed McDonagh reporter

    Removed storeSCP - will use existing Store rather than creating a second for QR. Changed all the PatientRoot finds etc to StudyRoot. Added in the Move requests and tested successfully against dcmtk QR SCP containging CR, DR, CT SRs on their own and CT with an SR series within the study. Refs #207

    → <<cset 93688f7ef2b5>>

  4. Ed McDonagh reporter

    Removed option to specify local port for QR as the store SCP element is provided by storescp.py. Used the ephemeral port (0) in the creation of the AE. Refs #207

    → <<cset 64f62938e827>>

  5. Ed McDonagh reporter

    Added link on the admin menu to the DICOM networking configuration page. Removed references to the font awesome icons that are a hangover from the website dropdown menus these were copied from! refs #207

    → <<cset 52d83fea2a6a>>

  6. Ed McDonagh reporter

    Added dicomsummary and dicomconfig under admin in urls, started creating the views for them and created a basic dicomsummary template. Refs #207

    → <<cset 6f29ee23330b>>

  7. Ed McDonagh reporter

    Corrected the name of the template. Need to create links from summary, and add in the security, as well as making everything pretty. Refs #207

    → <<cset 85542552717b>>

  8. Ed McDonagh reporter

    Added process for adding/modifying/deleting remote QR node condigs. Corrected some closing div errors, and put the configs into basic tables. Refs #207

    → <<cset 92556b387372>>

  9. Ed McDonagh reporter

    Modified the context of the DICOM Store Create class to include the group and version information to pass to the template. Modified the Store create template to only display if in the admin group. Refs #207

    → <<cset de28bbdd819c>>

  10. Ed McDonagh reporter

    Added import of pk to storescp to allow database settings to be used. Added URL link to template to start the server. Added urlpattern to launch storescp. Refs #207

    → <<cset ce005fee0c5e>>

  11. Ed McDonagh reporter

    Adding the daemon flag means quitting runserver now exits both the runserver and the dicom store server - without daemon neither actually quit. Refs #207

    → <<cset 5b8911f9c875>>

  12. Ed McDonagh reporter

    Added a while sleep loop to web_store with a SystemExit exeption catch to call MyAE.Quit(). Storescp now works in a shell and the process including the thread that holds the port open can be closed with ctype_async_raise(t,SystemExit) - the problem is getting hold of t when not running in the shell. Propose that this is passed to Celery, and try and raise it on the task? Refs #207

    → <<cset 2bd9162a53a0>>

  13. Ed McDonagh reporter

    Added stop-store function - works, but is inelegant. Removed some print statements. To run more than two stores at once, or for anything else to happen with Celery when stores are running, it is neccesary to increase the concurrency. Adding --concurrency=10 to the end of the celery launch will achieve this. Refs #207

    → <<cset 223b39bc010f>>

  14. Ed McDonagh reporter

    Functioning, clean way to kill DICOM store SCP now works in shell, works using Celery from shell, but doesn't work using Celery from web interface. Port does't get cleared until the web server is killed. Refs #207

    → <<cset 542380dc5532>>

  15. Ed McDonagh reporter

    Clean kill method is better, merging in again. Didn't work completely from the web interface in debug mode, has done in non-debug mode, does when not using web interface. Needs more testing. Refs #207

    → <<cset 930d3440d6ce>>

  16. Ed McDonagh reporter

    Moved the argparse into the main function, and explicitly passed all variables to the qrscu function as keywords. Works as before. Refs #207

    → <<cset 9b476b876b44>>

  17. Ed McDonagh reporter

    Added models to hold the study and series level responses. Series is currently foreign key linked to study - not sure yet if that is the right way to do it. Refs #207

    → <<cset 3a0f8e88fd5e>>

  18. Ed McDonagh reporter

    Moved the netdicom imports into the qrscu routine else it doesn't get imported when calling qrscu from python. Changed the submodule * import to explicit as that isn't allowed. Reduced remotehost and port to initials to save typing. Refs #207

    → <<cset b63787f464fd>>

  19. Ed McDonagh reporter

    Removed many of the print statements and put all the data for study level queries in the database. For automated running, need to be able to pass a move=True flag, for interactive running, need to keep the data to display to the user and only move when approved, accessing the _moveStudy function directly. Refs #207

    → <<cset eecb62a613cd>>

  20. Ed McDonagh reporter

    Added move flag to automatically prompt the c-move, which would be the norm for scheduled QR jobs. Moved the print statements to inside the move condition. Series level query now populates the database, using the foreign key. Series Description doesn't work - need to establish if this is peculiar to this QR SCP (OFFIS), or standard, and if the same issue could occur with other DICOM QR SCPs. Refs #207

    → <<cset 38abe51e68a6>>

  21. Ed McDonagh reporter

    Added patient age field to series and study level query models. Added in conversion to decimal age from study date minus birth date or conversion from DICOM age. Changed names of _moveStudy to _move_req and _querySeriesCT to _query_series (PEP8 and drop CT specific). Added option to pass in the study request id value, so that it can be found in the database later. Refs #207

    → <<cset db43efaf26a1>>

  22. Ed McDonagh reporter

    Added class to models to hold query_id and status, study level responses now FK to query level model. Started process of using database contents to inform status and messages, currently in infinite loop... Refs #207

    → <<cset d03e4fde94c3>>

  23. Ed McDonagh reporter

    Issue was that the query_id was not being passed back through when test3 returned. Now successfully repeats until the query is finished. Refs #207

    → <<cset 05a96ec8bec4>>

  24. Ed McDonagh reporter

    Made the remote QR node 'name' unique, so it can be usefully used to select a node in the form. Added a unicode function to return the node name - didn't use it in the form in the end but it could be. Refs #207

    → <<cset 810031e20602>>

  25. Ed McDonagh reporter

    Now deals appropriately with an unsuccessful association and with no modalities being selected. Date and modality choices are still ignored. Refs #207

    → <<cset c53388f3abff>>

  26. Ed McDonagh reporter

    Added various fields to the query that are either mandatory on some PACS study level queries, or look to be useful. Created new date functions to supply me with the DICOM style date strings. Refs #207

    → <<cset a5ec2de73bb5>>

  27. Ed McDonagh reporter

    Tested date filter with postgres database, and it works. Added test to modality as on some queries the modality type wasn't listed, leading to an attempt to concatonate a NoneType into a string. Refs #207

    → <<cset 054c68ab5da0>>

  28. Ed McDonagh reporter

    Switched to using form.cleaned_data.get rather than request.POST. This gives all the modalities as a list, and doesn't crash if missing. This is then passed through, but probably not added to modalities in study correctly. Cleaned_data also normalises the dates to datetime objects, which then need to be converted to iso-format again to be serialized. Refs #207

    → <<cset 931f76db38fa>>

  29. Ed McDonagh reporter

    Suspended the series query for now, as some CR returns didn't have sensible information in them - not sure what was happening! Added modalities_in_series to the study level database table. Now have a rough outline for the process to go through if modalities have been chosen. Refs #207

    → <<cset 44acbb2963f5>>

  30. Ed McDonagh reporter

    Added attributes to tick the modalities by defualt, and require at least one. As with other errors, the invalid form is not dealt with yet. Refs #207

    → <<cset 58ab400028a5>>

  31. Ed McDonagh reporter

    Currently non-functioning. Added in a break to the query series for each study return - if it is not there, then the CR seems to get stuck in a loop infinitely querying the same study. If it is there, then my plans for dealing with QR SCPs that don't match on modalities_in_study don't work as there are no series. Refs #207

    → <<cset 0da2d43014f5>>

  32. Ed McDonagh reporter

    It works! Split series and study associations - using the same association to make interleaving queries on different levels didn't work and led to very strange outcomes. Added a break from the study modality query section that actually works. Lots of additional print statements currently. Not tested on a QR server that has ModalitiesInStudy as a match. Refs #207

    → <<cset 2b8bbc7491f8>>

  33. Ed McDonagh reporter

    It didn't quite work - it was deleting too much as the different modalities that CT, MG, FL and DX could be weren't in the test. So now there is a new dict of dict and lists with all the details, and it seems to work. The old mods list can be retired once all_mods has replaced it in the study query loop. Refs #207

    → <<cset 10bf7b046b54>>

  34. Ed McDonagh reporter

    Increased size of modalities_in_study model CharField, and added functions to store and get jsonified lists. Lists and sets were previously being stored as strings, making them difficult to work with when retrieved. Modalities in study in the html table is now pretty. Refs #207

    → <<cset 78e538a26c56>>

  35. Ed McDonagh reporter

    Added form field to ask if SR only studies should be included. This works for Find SCPs that don't filter on ModalitiesInStudy, but hasn't been incorporated into the initial query yet for those that do. Refs #207.

    → <<cset c9b85f1cb059>>

  36. Ed McDonagh reporter

    Combined all the individual modality finds into one loop through all_mods. This makes it easier to follow, and much easier to add sub-modalities and modality choices in the future. Refs #207

    → <<cset d956720f0602>>

  37. Ed McDonagh reporter

    Added in the bulk of the series weeding logic. Started tidying up the series and study query sections, removing irrelevant tags and database fields, adding in some more useful ones, separating the mandatory and the optional. Removed PatientName and ID etc, hoping they are not required for a query at any level. Not in a testable state. Refs #207

    → <<cset 7ee74737b4ab>>

  38. Ed McDonagh reporter

    Completed provisional series weeding. Todo: improve data fed back to interface, provide method of triggering the retrieve, provide way of storing query so it can be scheduled, lots of other stuff. Refs #207

    → <<cset cac6ef2fd85d>>

  39. Ed McDonagh reporter

    Added in duplicates question to avoid sending a C-Move request for any studies that are already in the database. Can't establish if it works till I start populating the database again. Refs #207

    → <<cset fa5941a029ca>>

  40. Ed McDonagh reporter

    Now has modalities_in_study whilst in progress, but modality when complete. Sensible headings are fed through. Table has column titles. Refs #207

    → <<cset f3b1d91db7b7>>

  41. Ed McDonagh reporter

    Added storeSCP choice to QR form. Added QR SCP and Store SCP as foreign keys to the Query model so that it can be passed to the move request. Added name to the StoreSCP. Added StoreSCP to the dicomviews processing of the form, pass database pk of SCPs rather than values to qrscu. Added launch of movescu to r_start in dicomviews. Made various changes in qr_scu to PEP8 the variable names (lowercase not camel). Converted to using db entries for remote QR SCP and local Store SCP. Ensured AETitles were in ASCII as they fail in Unicode (default for db). Added movescu function. Works, but with no indication of progress. Added StoreSCP choice to QR template. Changed dicomsummary template to be two columns and removed initial AJAX testing code from the bottom. Better, but needs more work. And Store status needs to be AJAX'd. Dropped 'enabled' from the various SCP create/update views, added name to Store SCP. Refs #207

    → <<cset fcda3a3e10e3>>

  42. Ed McDonagh reporter

    Added move request progress reporting. Added move_complete to model. Added r_update to dicomviews. Added status updates to movescu. Added retrieve_progress to js. Added the move status into the template. Added the view to the urls. Basically works. Doesn't feed back any indication that request has succeeded but the move operation hasn't. Don't think that is available. Doesn't offer method of saving query to run automatically. Refs #207

    → <<cset f2ff124b157a>>

  43. Ed McDonagh reporter

    Merged in issue207revamp (pull request #29)

    • Brought DICOM Store SCP configuration and management within the web interface.
    • Added DICOM QR SCU configuration and management.

    Still todo:

    • Add QR Scheduling with Celery Beat
    • Improve monitoring of DICOM Store SCP
    • Better logic for sorting out what we want to retrieve and what we don't
    • Documentation
    • Better error handling
    • Logging

    Refs #207

    → <<cset f4e373c746cd>>

  44. Ed McDonagh reporter

    Added in logging.info statements to reflect the updates stored in the database, and logging.debug for trying to trackdown exactly was going on with a particular issue. The test for assoc_series was due to an AttributeError that I've had a couple of times where NoneType has no StudyRootFindSOPClass, and therefore the association presumably failed. New code not tested in the wild. Refs #207"

    → <<cset 06cdfef883d3>>

  45. Log in to comment