DX extractor error: total_number_of_radiographic_frames is NoneType

[David Platten]

A radiographic object (DX modality) caused the following error to be thrown by the dx.py extractor:

Traceback (most recent call last): File "e:\openrem-src\openrem\remapp\tools\background.py", line 137, in run_as_task func(*args, **kwargs) File "e:\openrem-src\openrem\remapp\extractors\dx.py", line 943, in dx _dx2db(dataset) File "e:\openrem-src\openrem\remapp\extractors\dx.py", line 783, in _dx2db _irradiationeventxraydata( File "e:\openrem-src\openrem\remapp\extractors\dx.py", line 532, in _irradiationeventxraydata _accumulatedxraydose_update(event) File "e:\openrem-src\openrem\remapp\extractors\dx.py", line 550, in _accumulatedxraydose_update accumint.total_number_of_radiographic_frames + 1 TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

[David Platten]

Added if...else statement to cover situation where total_number_of_radiographic_frames is None. Refs issue #957

→ <<cset 79f0a38ba689>>

[Ed McDonagh]

How can there be no radiographic frames in a DX image?

[David Platten]

This is a code extract from dx.py from line 779 which is used when importing an exposure when the overall study already exists:

‌

    if study_in_db:
        sleep(
            2.0
        )  # Give initial event a chance to get to save on _projectionxrayradiationdose
        this_study = get_study_check_dup(dataset, modality="DX")
        if this_study:
            _irradiationeventxraydata(
                dataset, this_study.projectionxrayradiationdose_set.get()
            )
            populate_dx_rf_summary(this_study)
            this_study.number_of_events = (
                this_study.projectionxrayradiationdose_set.get().irradeventxraydata_set.count()
            )
            this_study.save()
        else:
            error = f"Study {study_uid.replace('.', '. ')} already in DB"
            logger.error(error)
            record_task_error_exit(error)
            return

‌

I think the problem is rooted in the sleep command. I suspect that 2 seconds wasn’t long enough for my system to have saved the _projectionradiationdose.

[David Platten]

I wonder if we need to make the database transactions atomic for the extractors:

https://stackoverflow.com/questions/40059955/django-delay-in-creating-database-entry

‌

[Ed McDonagh]

Ah, that makes sense.

Was this on 0.10? I think Janis has solved this for 1.0. The script ensures that only one dx import can take place at any one time, so the sleep function shouldn't be triggered.

[Ed McDonagh]

But then if it was 0.10 you wouldn’t have got the trace ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

[David Platten]

The error happened on my 1.0 beta installation.

[David Platten]

I suspect that some errors I have from large mammography objects are related to the same sort of thing in the mam.py extractor:

if study_in_db:
    sleep(
        2.0    )  # Give initial event a chance to get to save on _projectionxrayradiationdose    this_study = get_study_check_dup(dataset, modality="MG")
    if this_study:
        _irradiationeventxraydata(
            dataset, this_study.projectionxrayradiationdose_set.get()
        )
        populate_mammo_agd_summary(this_study)
        this_study.number_of_events = (
            this_study.projectionxrayradiationdose_set.get().irradeventxraydata_set.count()
        )
        this_study.save()

Traceback (most recent call last): File "e:\openrem-src\openrem\remapp\tools\background.py", line 137, in run_as_task func(*args, **kwargs)
File "e:\openrem-src\openrem\remapp\extractors\mam.py", line 664, in mam _mammo2db(dataset)
File "e:\openrem-src\openrem\remapp\extractors\mam.py", line 548, in _mammo2db this_study = get_study_check_dup(dataset, modality="MG")
File "e:\openrem-src\openrem\remapp\extractors\extract_common.py", line 163, in get_study_check_dup this_study.projectionxrayradiationdose_set.get().irradeventxraydata_set.all()
File "E:\venv\lib\site-packages\django\db\models\manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs)
File "E:\venv\lib\site-packages\django\db\models\query.py", line 435, in get raise self.model.DoesNotExist( remapp.models.ProjectionXRayRadiationDose.DoesNotExist: ProjectionXRayRadiationDose matching query does not exist.

‌

[Ed McDonagh]

Are these definitely using the scripts, eg

if __name__ == "__main__":
    from openrem.remapp.extractors.dx import dx
    import openrem.remapp.tools.default_import as default_import

    default_import.default_import(
        dx, "import_dx", "the DICOM radiography image file", 0, {"import_dx": 1}
    )

This should prevent more than one import_dx import being run simultaneously.

[David Platten]

I’ve just checked. No, my imports are not using that code. I installed my test OpenREM system using the -e pip option to run it from the source code folder. The import scripts in my venv\scripts folder are different to the ones in my openrem\scripts folder.

I’ve just changed the orthanc lua file to use the scripts in my source code tree so that it will use the correct scripts:

local python_scripts_path = 'E:\\openrem-src\\openrem\\scripts\\

Hopefully that sorts out the problem!

[Ed McDonagh]

Ok that’s hopefully it then! When I’ve done that install I’ve done it without the -e so it installs to the normal place.

[David Platten]

Using atomic transaction to try and avoid orphan generalstudymoduleattr entry if a DX import fails part way through, preventing further re-imports of the failed DX object. Refs issue #957

→ <<cset 3c93f29175e2>>

[David Platten]

Resolving merge conflict in changes.rst. Refs issue #957

→ <<cset 3015d39a8d4b>>

[David Platten]

Removed the atomic transaction from the dx extractor as I think this may be responsible for the issues I have had with my live test system. Refs issue #957 and refs issue #963

→ <<cset f6f477da9d82>>

[David Platten]

Merged in issue957DXextractorErrorFramesNoneType (pull request #547)

Issue957DXextractorErrorFramesNoneType

Approved-by: Ed McDonagh

Fixes issue #957

→ <<cset 4678656fd3b7>>

[Ed McDonagh]

Black. Refs #957

→ <<cset bd79eb423979>>