The mdb in SimFactory has an entry for scratchdir. If scratchdir is a relative path, it is assumed to be relative to the restart output directory. It's not clear exactly what the purpose of this is, but SimFactory does the following things with it:
- Create the directory at the start of a simulation if it doesn't exist.
- Delete the entire directory using rmtree(scratchdir) during "cleanup". This happens, for example, when an existing simulation is restarted.
The second of these seems very dangerous to me and I think it should be removed. In particular, if you happen to have scratchdir set to be the same for more than one simulation (by setting it to, e.g., /scratch), this scrachdir for all simulations will be erased as soon as one simulation does a cleanup.
I recently had an even worse experience where I was using /scratch as my basedir and /scratch also happened to be set as the scratchdir in the mdb for that machine. All of my simulations on that machine had only a single restart, so all was well until I tried to restart one simulation and instead of doing so SimFactory erased all of my simulations on the machine.
While I can admit to user error in this case (I also wrote the machine entry), I think the goal of SimFactory should be to err on the side of caution and shouldn't do this blind rmtree of whatever scratchdir is set to.