Rework Q-R duplicate flag for multi RDSR
Need to start recording Series Instance UID etc
Comments (37)
-
reporter -
reporter Current behaviour for MG and DX images is as follows (extractor modules):
- Check
StudyInstanceUID
against database - If existing, check
SOPInstanceUID
against the database in the fieldprojectionxrayradiationdose__irradeventxraydata__irradiation_event_uid
(not limited to the sameStudyInstanceUID
!) - When importing new images,
SOPInstanceUID
is recorded as though it wereIrradiationEventUID
Therefore for
C-Find
filtering we need to something similar.- If
StudyInstanceUID
in database is true: - For each
series_rsp
ifmodality is not SR
(or isMG or DX or CR
): - For each
images_rsp
check ifSOPInstanceUID
is in database
Next I need to see what we do for legacy Philips CT...
- Check
-
reporter For legacy Philips CT (
ct_philips.py
), we do not consider the possibility of there being more than one 'Dose Info' object for the same study, so we don't need to here either.For legacy Toshiba CT (
rdsr_toshiba_fct_from_dose_images.py
), we are creating the RDSR so there won't be multiple ones. If we come across the sameStudyInstanceUID
again, I don't think we want to fetch it again to see if it has changed. -
reporter So, to bring it together:
Do all the normal filtering, then
if remove_duplicates:
if StudyInstanceUID in database:
for series in series_rsp:
if series.modality == "SR":
if SeriesInstanceUID in database:
thendelete
elseif SeriesTime earlier than series_time in database
thendelete
if series.modality in ['MG', 'DX', 'CR']:
for image in images_rsp:
if SOPInstanceUID in database:
(inIrradiationEventUID
field), delete
I'm now thinking it would be best to remove all but the latest date RDSR of a study at the same time? Leave them all in
if not remove_duplicates
. -
reporter Slight modification - will need to
_query_images
in order to cater for mammo and DX. Currently this only happens if in_get_toshiba_dose_images
or _check_sr_type_in_study
. -
reporter Added SeriesInstanceUID to GeneralStudyModuleAttr model. Added SeriesTime to DicomQRRspSeries model. Added SeriesInstanceUID to rdsr extractor. Moved remove_duplicates to much later in process to ensure series level data is available. Started to implement ref
#610→ <<cset 252d8259631e>>
-
reporter Added asking for series time in C-find, added MG/DX logic. Should work now, need to add tests. ref
#610→ <<cset 59d2219a217f>>
-
reporter Factored remove duplicates code into separate function to make it testable. Refs
#610→ <<cset 7d45f37f0c4b>>
-
reporter Added tests for RDSR in remove duplicates. Refs
#610→ <<cset 2abb5e226949>>
-
reporter Added tests for DX in remove duplicates. Refs
#610→ <<cset f04d6074926d>>
-
reporter Adding refs
#506,#610to changes.→ <<cset dcf39de06f08>>
-
reporter - changed status to resolved
Merged in issue506documentQR (pull request #176)
New duplicates code fixes
#610, update QR docs fixes#506→ <<cset 435a93149947>>
-
reporter - changed status to open
Rethink this now that ref
#625has changed the way duplicates are dealt with. I don't think the logic here allows for continued studies. -
reporter Now appears to function for RDSR. Need to modify the tests, but works with current tests locally. WIP, refs
#610. Need also to look at other import objects. [skip ci]→ <<cset 51a89a0ac629>>
-
reporter Added sop instance UID tests for continued studies. Instance UID now added just to applicable study in db. Now to look at other imports. [skip ci] because tested locally. Refs
#610→ <<cset f1c27cb8ad92>>
-
reporter Added checking of SOP instance UID to DX _create_event. Also changed logic for multiple studies with the same UID - previous code would have fallen over. Need to add UID stuff to new DX study function. [skip ci] because tested locally. Refs
#610→ <<cset 6cda0dfa4cdd>>
-
reporter Now adds SOP UID to database for new studies too. [skip ci] because tested locally. Refs
#610→ <<cset f2a7e7a333f5>>
-
reporter Added a couple of debug messages; added check that duplicate image import reulsts in debug message, and check uid list is created correctly. [skip ci] because tested locally. Refs
#610→ <<cset 5b51d9523763>>
-
reporter Reverted sample local_settings. Added explicit setting of log level for each log capture. Refs
#610,#628→ <<cset 5259c6b09128>>
-
reporter Adding a sed regex to replace all Info levels with Debug in local settings. Sledgehammer... Refs
#610,#628Also removed pip list.→ <<cset 502919e5c20c>>
-
reporter Adding the mod to set the logging level to debug for all the testing. Refs
#610,#628→ <<cset 5b83dff4a34d>>
-
reporter Refactored _create_event into separate module so I don't have to re-code it for mammo. Works for DX, haven't started modifying for mammo yet. Refs
#610,#628→ <<cset 3cb339f36eb9>>
-
reporter Substituted each logging message 'DX' with modality. Added requirement to check object is not the secondary capture Hologic object.. Refs
#610,#628→ <<cset d038e4b4f940>>
-
reporter Rename image_import_create_event to get_study_check_dup. Removed _create_event from each module. Set mam to use the new function. Refs
#610,#628→ <<cset 61ca11fe05f9>>
-
reporter Adding test files to enable testing of mammo import with duplicates etc. Started on tests. Refs
#610,#628→ <<cset ac0249a123c6>>
-
reporter Removed try except and corrected an error it was hiding. Initial test works. Refs
#610,#628→ <<cset bba06fddc29f>>
-
reporter Explicitly named mam log for consistency. Added record_sop for new studies. Added tests for importing new event and duplicate vent a second time. Lots of duplication still... for another time! Refs
#610,#628→ <<cset 3bf064fa0fa6>>
-
reporter qrscu
routine now needs to make use of the newobjectuidsprocessed_set
field... -
reporter Replaced check duplicates QR code - was based on series instance UID and series time, now based on SOPInstanceUID. Modified tests to provide the new information, and split them up. Refs
#610→ <<cset ac09001e187f>>
-
reporter Fixing minor codacy issue. Refs
#610,#628→ <<cset 26f0f8eadf5a>>
-
reporter Fixing import issue in test flagged by @dplatten's codacy integration. Refs
#610,#628→ <<cset 5edb87d524b1>>
-
reporter Adding ref
#628to changes. Ref#610is already there.→ <<cset 594e6746ee03>>
-
reporter - changed status to resolved
Merged in issue610duplicatesagain (pull request #193)
Issue610duplicatesagain
→ <<cset 9162ebc7d1ba>>
-
reporter Tested in production. check_uid.check_uid was preventing UIDs to be collected, and isn't needed in extract_common. Removed. Added debug logging to duplicates QR code. Added query stage updates for web interface. Refs
#610,#628→ <<cset 05e0457273b4>>
-
reporter Updated test due to duplicate message appearing at different point after removing check_uid.check_uid use. Refs
#610,#628→ <<cset 381e8aec20ca>>
-
reporter Modifying interface duplicates help text as we are not using study instance UID as the duplicate marker. Refs
#610,#628. [skip ci]→ <<cset b335851ad132>>
-
reporter Adding space to log message. Refs
#610→ <<cset b2fdf48cf757>>
- Log in to comment
Writing thoughts to myself as much as anything:
For RDSRs (of any flavour):
SeriesInstanceUID
Therefore:
series_instance_uid
toGeneralStudyModuleAttr
model along side the series time and content time already added for ref#114SeriesInstanceUID
inrdsr.py
SeriesInstanceUID
inrdsr.py
because it is already covered by the times and the UID doesn't tell you which one is laterStudyInstanceUID
matches, then if found look forSeriesInstanceUID
SeriesTime
to be added to the seriesC-Find
and to theDicomQRRspSeries
modelSeriesInstanceUID
recorded, rely on times?SeriesInstanceUID
values, preventing a repeat next time!I'll do another comment later to consider how this will affect DX and MG images before implementing it.