Reduce skin map pickle sizes
Issue #402 resolved
Is it possible to reduce the precision stored in the skin map pickles and use gzip compression?
I'm contemplating adding each row so you can get an animated dose but don't want 16MB pickle files per case.
I'll look into that and get back to you.
Added gzip compression for skin dose map pickle files. Users of beta code who already have skin dose map pickle files will need to delete them, as there is no provision to cope with uncompressed pickle files. Fixes issue
→ <<cset 4457e994a863>>
Is it possible to reduce the precision of the stored numbers as well? At the moment they go to many, many decimal places but probably only the first few are required.
I'll take a look. We may need a consensus on how many decimal places to store. If we reduce it too much then things like pacing procedures may appear blank, which would be a shame.
@jacole, the skin dose map data that is saved in the pickle files is just the data contained in openSkin's geomclass.SkinDose.totalDose. Would it be more sensible to address the decimal places in openSkin rather than in OpenREM?
Yes, that would be much more sensible.
I will do that with my other changes.
Anything less than that is likely to be nonsense anyway. To be honest, even 10 or 100 mGy might be within the uncertainly.
1 mGy is good with me.
No functional changes. Renamed variable
fin commented out code that can be used to create a pickle file for every RF study in the database. Now named
pickle_fileso that if the code is uncommented it works OK (
fwas already in use elsewhere in the view). References issue
→ <<cset f841d3a78145>>
I may be teaching my grandmother to suck eggs, but if you were to use
Decimalin openSkin as the datatype for the skin dose map values then it's easy to control the number of decimal places, and therefore how much file space the pickle files take up:
pickleSmall.pfile is 36 bytes vs. 85 bytes for
pickeLarge.p. Note that both of these files occupy 4096 bytes on my Windows PC - there will be a tangible space saving when many values are stored in a single pickle file.
I did initially look at Decimal but didn't use it. I can't remember why now, I think maybe it didn't work with something from numpy?
I will try using it just for the final dose maps and see.