Make job-specific directories on scratch space
Issue #35
resolved
Conflicts arise when running more than one job at the same time, both using scratch space. Seems like the later jobs overwrite the same file name.
Comments (4)
-
-
-
We were resorting to giving each job a subfolder within the scratch folder manually at first to prevent this from happening
This is what we do.
-
- changed status to resolved
I close this since workaround is available on user's side and the best solution depends on how clusters are managed. Job specific scratch space can cause problems when a job crashes (non-deleted cache will proliferate in the scratch).
- Log in to comment
Indeed. We were resorting to giving each job a subfolder within the scratch folder manually at first to prevent this from happening, and later simply made the submission script so that it overrides the scratch_dir input and uses a job-specific scratch instead.
Would be nicer if this was handled within the program though, e.g. having an option to go for "scratch: no, auto, specify", where "specify" option behaves like the existing one, and the "auto" one creates a job-specific folder to use in parent scratch folder.
Cheers
J