Make job-specific directories on scratch space

Issue #35 resolved
Reza Behrouzi created an issue

Conflicts arise when running more than one job at the same time, both using scratch space. Seems like the later jobs overwrite the same file name.

Comments (4)

  1. Juraj Ahel

    Indeed. We were resorting to giving each job a subfolder within the scratch folder manually at first to prevent this from happening, and later simply made the submission script so that it overrides the scratch_dir input and uses a job-specific scratch instead.

    Would be nicer if this was handled within the program though, e.g. having an option to go for "scratch: no, auto, specify", where "specify" option behaves like the existing one, and the "auto" one creates a job-specific folder to use in parent scratch folder.

    Cheers

    J

  2. Takanori Nakane

    We were resorting to giving each job a subfolder within the scratch folder manually at first to prevent this from happening

    This is what we do.

  3. Takanori Nakane

    I close this since workaround is available on user's side and the best solution depends on how clusters are managed. Job specific scratch space can cause problems when a job crashes (non-deleted cache will proliferate in the scratch).

  4. Log in to comment