Make job-specific directories on scratch space

[Reza Behrouzi]

Conflicts arise when running more than one job at the same time, both using scratch space. Seems like the later jobs overwrite the same file name.

[Juraj Ahel]

Indeed. We were resorting to giving each job a subfolder within the scratch folder manually at first to prevent this from happening, and later simply made the submission script so that it overrides the scratch_dir input and uses a job-specific scratch instead.

Would be nicer if this was handled within the program though, e.g. having an option to go for "scratch: no, auto, specify", where "specify" option behaves like the existing one, and the "auto" one creates a job-specific folder to use in parent scratch folder.

Cheers

J

[Takanori Nakane]

We were resorting to giving each job a subfolder within the scratch folder manually at first to prevent this from happening

This is what we do.

[Takanori Nakane]

I close this since workaround is available on user's side and the best solution depends on how clusters are managed. Job specific scratch space can cause problems when a job crashes (non-deleted cache will proliferate in the scratch).