On a Windows server skin dose maps are not calculated on import when this option is selected

[David Platten]

On Windows the skin dose map data is not calculated on import when this option is selected. The study itself is successfully imported into OpenREM, it is just that the skin dose map data is not calculated and put in the appropriate folder in media.

If you click on the study in OpenREM to view its details then the skin dose map data is successfully calculated. This makes me think that there is nothing wrong with the make_skin_map routine, and that it is working OK with celery.

I suspect that the delay on make_skin_map in rdsr.py is causing the problem, because changing

make_skin_map.delay(g.pk)

to

make_skin_map(g.pk)

solves the problem. Perhaps the delay in the rdsr.py call to make_skin_map is not needed because make_skin_map is already being run as a celery task?

I have tested the above change by importing a fluoro rdsr, and whilst it is importing and the skin dose data being calculated I have imported several DX studies. The fluoro data and skin dose map data were calculated correctly, and I imported nine DX studies at the same time. I think the above is what is needed.

Any thoughts?

[Ed McDonagh]

Off the top of my head, the .delay is what causes it to be a celery task, otherwise make_skin_map will be called directly and you won't exit rdsr.py until the calculation is complete.

[David Platten]

Hi @edmcdonagh. Thanks for that. rdsr itself is a celery task, so does it matter if it isn't exited until the skin dose calculations are complete?

As an aside, is the skin dose map data calculated on import with your Linux system? I'm wondering if this is a Windows issue.

[Ed McDonagh]

How are you calling rdsr? If you are calling the openrem_rdsr.py script from a conquest script, that isn't usually the case is it? It does call rdsr as a celery task when the built in QR function is used.

[David Platten]

I've sorted this. It is a Windows issue that other users have had too. The celery tasks are never executed. The solution is to add the --pool=solo worker switch to the celery command when using Windows. @edmcdonagh, would you like me to update the documentation?

See

https://github.com/celery/celery/issues/2146

and

http://stackoverflow.com/questions/26636033/trouble-getting-result-from-celery-queue

[Ed McDonagh]

Awesome, yes please. I need to work out a way of having documentation that can change without a new release, with a reference that can remain the same so that it can be linked to from the installed version...

Have other OpenREM users hit this? If not, why not?

[David Platten]

OK. I'll update the current documentation.

This seems to only affect the calculation of skin dose map data, so it may be that I'm the only one that has seen this.

[David Platten]

@edmcdonagh, there is a better solution to this. Rather than using --pool=solo with the celery command, adding ignore_result=True to make_skin_map's @shared_task solves the problem.

I have concerns that using --pool=solo will result in blocking of some commands (see the note here: http://docs.celeryproject.org/en/latest/userguide/workers.html)

[David Platten]

Added ignore_result=True to the make_skin_map shared task. This ensures that the skin map data is calculated when using a Windows system. Fixes issue #407

→ <<cset 29bfc95d4369>>

[David Platten]

My "fix" didn't fix this on my live system. I've reverted to using "--pool=solo" in the celery call, which does work.

[David Platten]

Strange. "--pool=solo" isn't working either.

The "ignore_results=True" addition to "make_skin_map" works fine on my local install. I think that this is the solution, and that there is something else going on with my live system.

[David Platten]

Running make_skin_map.delay(some_pk) from a python console on my live system for a study that is already in the database results in the skin dose map data being calculated. I don't understand why this command isn't working when it is called as part of rdsr.py during the import of a DICOM RDSR file.

[Ed McDonagh]

Remind me again what your setup is?

1. Conquest as DICOM Store SCP?
2. Conquest executes a script?
3. Script calls python path\to\openrem_rdsr.py path\to\rdsr.dcm?
4. With no virtualenv?

[David Platten]

Yes, that's the setup.

My local copy (that successfully creates skin dose map data on import) does use virtualenv...

[David Platten]

An update. If I run

python c:\Python27\Lib\site-packages\openrem\scripts\openrem_rdsr.py c:\ProgramData\OpenREM\test.dcm

the study is imported into OpenREM but no skin dose map data is calculated.

However, if I run

python c:\Python27\Lib\site-packages\openrem\remapp\extractors\rdsr.py c:\ProgramData\OpenREM\test.dcm

then the study is imported into OpenREM and the skin dose map data is calculated.

[Ed McDonagh]

Why aren't you running python openrem_rdsr.py c:\ProgramData\OpenREM\test.dcm?

Do you not have the python bin directory on the path? The version in the scripts folder should never be run - the version of that that is copied to the python bin directory (or whatever it's called on Windows) has the shebang line edited on install to suit your environment.

[Ed McDonagh]

C:\Python27\Scripts\openrem_rdsr.py

[David Platten]

Thanks for the update.

My earlier post hasn't fixed the issue: the creation of skin dose map data on my system is intermittent...

When a skin dose map fails my rabbitmq log has

=WARNING REPORT==== 24-Jun-2016::09:57:30 ===
closing AMQP connection <0.16945.0> (127.0.0.1:63474 -> 127.0.0.1:5672):
client unexpectedly closed TCP connection

in it.

[Ed McDonagh]

Firewall issue? #clutchingatstraws

Did you try using the installed version of the script to see if it works like the direct call?

[David Platten]

I tried using the installed version. Sometimes it works. Sometimes it doesn't (pulling virtual hair out of my bald head at this point...)

Is skin dose map data calculated on import for your test system?

[David Platten]

Could be something to do with this: http://stackoverflow.com/questions/20955293/celery-closes-unexpectedly-after-longer-inactivity

[David Platten]

I've set the heartbeat time to 60s on my rabbitmq install and will see how this affects things. I've done this by setting the contents of the c:\rabbitmq\rabbitmq.config file to:

[
 {rabbit,
  [%%
   %% Network Connectivity
   %% ====================
   %% Set the default AMQP heartbeat delay (in seconds).
   %%
   {heartbeat, 60}
  ]}
].

[David Platten]

The heartbeat configuration seems to have fixed the issue. I will check again at the end of Monday.

[David Platten]

Heartbeat hasn't solved the problem. I have now added

BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}

to the celery config in settings.py and restarted celery, apache and rabbitmq (http://www.lshift.net/blog/2015/04/30/making-celery-play-nice-with-rabbitmq-and-bigwig/)

I'll see how it goes now.

[David Platten]

Five cardiac studies have arrived in OpenREM since adding BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True}. All have had skin dose map data calculated automatically on import. I'll check again tomorrow morning.

[David Platten]

I have removed the heartbeat configuration from the rabbitmq.config file, but left BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True} in the OpenREM settings.py file. Skin dose map data continues to be calculated on import.

This is good, as we can control the contents of settings.py, whereas the user would have to alter rabbitmq.config themselves.

[David Platten]

Added celery BROKER_TRANSPORT_OPTIONS line to settings.py to ensure that skin dose maps are calculated on import when using Windows Server. Fixes issue #407

→ <<cset f3042f1b4b7e>>