Reduce skin map pickle sizes
Is it possible to reduce the precision stored in the skin map pickles and use gzip compression?
I'm contemplating adding each row so you can get an animated dose but don't want 16MB pickle files per case.
Comments (12)
-
-
- changed status to resolved
Added gzip compression for skin dose map pickle files. Users of beta code who already have skin dose map pickle files will need to delete them, as there is no provision to cope with uncompressed pickle files. Fixes issue
#402.→ <<cset 4457e994a863>>
-
-
assigned issue to
- changed milestone to 0.7.0
-
assigned issue to
-
reporter Is it possible to reduce the precision of the stored numbers as well? At the moment they go to many, many decimal places but probably only the first few are required.
-
I'll take a look. We may need a consensus on how many decimal places to store. If we reduce it too much then things like pacing procedures may appear blank, which would be a shame.
-
@jacole, the skin dose map data that is saved in the pickle files is just the data contained in openSkin's geomclass.SkinDose.totalDose. Would it be more sensible to address the decimal places in openSkin rather than in OpenREM?
-
reporter Yes, that would be much more sensible.
I will do that with my other changes.
-
reporter 1 mGy?
Anything less than that is likely to be nonsense anyway. To be honest, even 10 or 100 mGy might be within the uncertainly.
-
1 mGy is good with me.
-
No functional changes. Renamed variable
f
in commented out code that can be used to create a pickle file for every RF study in the database. Now namedpickle_file
so that if the code is uncommented it works OK (f
was already in use elsewhere in the view). References issue#402→ <<cset f841d3a78145>>
-
I may be teaching my grandmother to suck eggs, but if you were to use
Decimal
in openSkin as the datatype for the skin dose map values then it's easy to control the number of decimal places, and therefore how much file space the pickle files take up:# Many decimal places print Decimal(1.0155) 1.0155000000000000692779167366097681224346160888671875 pickle.dump( Decimal(1.0155), open("c:\\temp\\pickleSmall.p", "wb")) print pickle.load( open("c:\\temp\\pickleLarge.p", "rb") ) 1.0155000000000000692779167366097681224346160888671875 # Fewer decimal places print Decimal(1.0155).quantize(Decimal('0.001')) 1.016 pickle.dump( Decimal(1.0155).quantize(Decimal('0.001')), open("c:\\temp\\pickleSmall.p", "wb")) print pickle.load( open("c:\\temp\\pickleSmall.p", "rb") ) 1.016
The
pickleSmall.p
file is 36 bytes vs. 85 bytes forpickeLarge.p
. Note that both of these files occupy 4096 bytes on my Windows PC - there will be a tangible space saving when many values are stored in a single pickle file. -
reporter I did initially look at Decimal but didn't use it. I can't remember why now, I think maybe it didn't work with something from numpy?
I will try using it just for the final dose maps and see.
- Log in to comment
I'll look into that and get back to you.