Copying symmetry expanded particle sets to scratch takes up a lot of space
Issue #58
resolved
Hi,
Currently, if I run a refinement or classification with a symmetry expanded particle set, relion copies each particle (the "original" particle and all the symmetry expanded copies) to the scratch drive, taking up n-times more space than the original dataset.
This becomes problematic with large datasets or high order symmetry, and it doesn't seem necessary, since the original stacks are all that is referenced by the star file (with different Euler angles). Would there be any way to alter this behavior to only take up as much space as the original dataset?
Cheers Oli
Comments (7)
-
-
reporter
Thanks Takanori - also for 3.1 then (maybe?), it would be great if relion kept track of what is already on scratch, so if I run three jobs in a row with the same settings but different input (e.g three different classification runs), relion does not have to recopy the data each time. For large datasets the copying can take quite a while.
Cheers Oli
-
Do you know
--keep_scratchand--reuse_scratch? -
reporter
No I do not... but I will investigate - thanks!
-
At LMB, every job goes into a random node in a cluster so RELION deletes all cache before and after a job by default (the two options above change this). And if one owns a personal dedicated workstation, they can copy particles to the cache and replace Extract/jobXXX by a symbolic link to it. Thus, I am afraid to say that the priority for persistent cache system is low here (but I welcome patches).
-
reporter
Ah I see - thank you
-
- changed status to resolved
Discussion continues in https://github.com/3dem/relion/issues/462
- Log in to comment
Unfortunately no, but this is indeed a good suggestion for 3.1.