Add export format for PHE (UK) CT Survey
Public Health England is conducting a CT Survey in the UK. They have published an Excel spreadsheet proforma. It would be useful for OpenREM to be able to export with the right column format to make it easy to copy and paste.
Comments (63)
-
reporter -
reporter Added filters for event numbers by type, but not currently working. Refs
#737→ <<cset 776e5a148ce3>>
-
reporter Start of export function. Refs
#737→ <<cset 887886e7c5c9>>
-
reporter Nearly working now, something wrong when used together. Refs
#737→ <<cset 419111b1a1cb>>
-
reporter Added topograms, stationary; dropdown list. Work individually, not together. Refs
#737→ <<cset 48cd052ef173>>
-
reporter Nearly all works for filtering by combined number of spiral/axial/stationary/spr. Nearly. Refs
#737→ <<cset fb44dc7a735a>>
-
reporter Minor modification to relabel 0 as None. Seems to be working now... refs
#737→ <<cset 83e8a039306f>>
-
reporter Removing the print statements. refs
#737→ <<cset c0581c96f6e3>>
-
reporter Added tests of various combinations of specific event numbers with test dataset. refs
#737→ <<cset 4b66236b28d5>>
-
reporter First run-through of export to xlsx. Not tested. Refs
#737→ <<cset b7e085dd8a67>>
-
reporter Yay! Works! Will add some docs then we need to get beta out to try. Refs
#737→ <<cset abe5b4b92a75>>
-
reporter @dplatten - I'd appreciate your opinion on this branch. Basically, you filter your data for one of the exams PHE 2019 CT survey wants, then use the new specific number of event types drop downs to make sure you don't have any exams where an extra series has been added in, then use the new export button.
Export columns are as per the template, with topograms ignored. Spiral, Axial and Stationary will be exported. The assumption is that there are four events or fewer (not including topograms), but not checked for. Yet.
The data can then be copied and pasted into the template.
-
reporter Not sure why the test failed. Will need to look into it. Don't let that stop you from trying it though - at worst the specific event type numbers filter won't get there right result!
-
Hmmm
[2019-03-27 16:37:11,549: ERROR/MainProcess] Task remapp.exports.ct_export.ct_phe_2019[18ef67da-81fe-47f9-b983-87066fda31b2] raised unexpected: IndexError('list index out of range',) Traceback (most recent call last): File "c:\pythonvirtualenvs\openrem-dev\lib\site-packages\celery\app\trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "c:\pythonvirtualenvs\openrem-dev\lib\site-packages\celery\app\trace.py", line 438, in __protected_call__ return self.run(*args, **kwargs) File "c:\pythonvirtualenvs\openrem-dev\lib\site-packages\openrem\remapp\exports\ct_export.py", line 561, in ct_phe_2019 kv = source_parameters[0].kvp File "c:\pythonvirtualenvs\openrem-dev\lib\site-packages\django\db\models\query.py", line 201, in __getitem__ return list(qs)[0] IndexError: list index out of range
-
About to try running it again in debug mode in PyCharm.
-
reporter Looks like you have studies with no x-ray source parameters. Can fix with try except
-
Added exception for missing kVp - prevents the error I initially experienced. References issue
#737→ <<cset 2136791e98e2>>
-
I've just exported 21473 studies from a several month period - no further hitches encountered with the export. I haven't tested the filters yet.
-
reporter Adding print statements to attempt to narrow down failure. Refs
#737→ <<cset 441ff39d2d55>>
-
reporter Checking if fourth test would work. Refs
#737→ <<cset ab9b83d274b4>>
-
reporter Not sure why the third filter, stationary, is tripping up the third and fourth tests, and difficult to troubleshoot as it works fine locally.
Still to do:
- If more than four series are exported, leave a couple of cells for Total DLP and comment, then carry on
- Work out if there are other cells that could be usefully filled in
- Add documentation
- Consider making exports configurable (as in which ones are presented) - almost certainly a future feature request though
- Fix this issue!
-
reporter Adding in an index to check when we have four events. Effectively stashing so I can switch! Refs
#737→ <<cset 494787250c20>>
-
reporter Modified PHE export to function correctly if more than 4 events are exported. Refs
#737→ <<cset f2f7fc1ce60a>>
-
reporter Minor change to ensure events are listed in event order in CT detail view. Refs
#737→ <<cset a6b22f33a96b>>
-
reporter Reducing the scope of the tests to try and troubleshoot the pipeline failures. Refs
#737→ <<cset 836e5bf58013>>
-
reporter Adding in more print statements to see if gives any clues, also further restricting the testing. Refs
#737→ <<cset 83a52450f122>>
-
reporter Printing event types in remaining and how many match filter. Refs
#737→ <<cset 9b5e1bbe3dcf>>
-
reporter Finding SQLite3 version. Refs
#737→ <<cset c7cf0b24014f>>
-
reporter More logging. Refs
#737→ <<cset 02da2377fd7f>>
-
reporter Minor change to logging. Refs
#737→ <<cset 34dfb8bbf35f>>
-
reporter Adding test with just the final study. Refs
#737→ <<cset f0d1d6eefca1>>
-
reporter Testing if creating a copy of the queryset makes any difference. Refs
#737→ <<cset 527d28aabae0>>
-
reporter Last commit surprisingly worked. Don't know why. Refs
#737→ <<cset 60ff7a05c271>>
-
reporter And then it didn't work finding the text, but did give the right numbers... Refs
#737→ <<cset 3e86660cd7e5>>
-
reporter Checking contents of filtered, also printing html response! Refs
#737→ <<cset ca0a959bf836>>
-
reporter Html response unnecessary. Removing duplicate check. Refs
#737→ <<cset d9f7e4465db9>>
-
reporter Confused why that worked... by printing it am I forcing it to be evaluated? refs
#737→ <<cset 18ea4f7aa5aa>>
-
reporter Worked again. Why? Adding test three back in refs
#737→ <<cset 129a3e2cda42>>
-
reporter Back to fourth test. Added query print. refs
#737→ <<cset 751218e6cf79>>
-
reporter See if clearing annotations helps... Refs
#737→ <<cset d1306947e7e5>>
-
reporter Might have done the trick... added to each of the filters, added third test back in. Refs
#737→ <<cset d36e4925fd1f>>
-
reporter Third test didn't work - check the others with the extra annotation removals. Refs
#737→ <<cset 671cc4f9ccfa>>
-
reporter Removed distinct from the other functions. Refs
#737→ <<cset c47a9333c49a>>
-
reporter Putting distinct back into all four functions. Refs
#737→ <<cset b09377b59e0f>>
-
reporter Split acquisition filter tests up, commented out other tests for now. Initial commit to check nothing has changed. Refs
#737→ <<cset c4ab0c4220f5>>
-
reporter Forgot to change the test name in tox. Refs
#737→ <<cset dcbc92dd4635>>
-
reporter Building up the number of tests. Not complete, not expecting any revelations at this point. Refs
#737→ <<cset 2722fda85b10>>
-
reporter Added new permutations. All pass locally. Refs
#737→ <<cset 3e87299f9b76>>
-
reporter Created a new queryset each time. All pass locally. Refs
#737→ <<cset a9847ea938c5>>
-
reporter Added in commented out tests, removed print statements. Refs
#737→ <<cset 85a577164148>>
-
reporter Put the event number tests back into one function. Refs
#737→ <<cset 784270ddba22>>
-
reporter Undoing tox changes. Hopefully. Refs
#737→ <<cset 91dbce941ebf>>
-
reporter Undoing tox more changes. Tidying up the filter code. Refs
#737→ <<cset 6f763331eece>>
-
reporter Adding ref
#737to changes. Still need to do docs and review etc→ <<cset 10930723f898>>
-
reporter Silence unused argument 'x' PyLint issue. Refs
#737→ <<cset 177027a2c4fa>>
-
reporter Set both references to SPRs to be localiser. Refs
#737→ <<cset 0db94ef76762>>
-
reporter Had the wrong PyLint disable comment. Refs
#737→ <<cset 69edf944f3f8>>
-
reporter Removing the fixed height and scrolling of the ct acquisition type selection. Refs
#737→ <<cset 43abff7a2a59>>
-
reporter Updating CT filter image with filtering by number of acquisitions. Refs
#737[skip ci] docs only→ <<cset cee9c34996c5>>
-
reporter Updated text to go with updated diagram. Refs
#737[skip ci] docs only→ <<cset 8ca08707a7fe>>
-
reporter Added instructions for using the PHE 2019 CT export. Refs
#737[skip ci] docs only→ <<cset 5518390b9a26>>
-
reporter Minor grammar changes. Refs
#737[skip ci] docs only→ <<cset c2e275f33c6c>>
-
reporter - changed status to resolved
- Log in to comment
Auto code reformat before trying to work out how to add specifying number of axial/helical series to condition dataset before PHE export. Refs
#737→ <<cset 90834c291175>>