cannot touch temporary snakemake file --- directory not present

Create issue
Issue #273 resolved
Kristoffer Sahlin created an issue

touch: cannot touch /nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.IOE948/0.jobfinished': No such file or directory touch: cannot touch/nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.IOE948/0.jobfailed': No such file or directory

Also see this old issue: https://groups.google.com/d/msg/snakemake/96yyOEz_r44/BIov2uJ8EQEJ Apparently this can still happen in later versions.

Command creating this: [ksahlin@brubeck scripts]$ snakemake --keep-going -j 999 --cluster "sbatch -c {cluster.cpus-per-task} -N {cluster.Nodes} -t {cluster.runtime} -J {cluster.jobname} --mail-type={cluster.mail_type} --mail-user={cluster.mail}" --cluster-config cluster.json without_quality_values

[ksahlin@brubeck scripts]$ snakemake -v 3.5.4

I'm unsure if this is a bug or if I have specified something incorrectly. The target files are however generated so the code in the rules seems to be working. Please let me know if you need other information to solve this.

Thank you, Kristoffer

Comments (21)

  1. Johannes Köster

    Hi,
    thanks for reporting! Can you give me the absolute path of the folder in which you invoke snakemake? Also, can you tell me if the folder /nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.IOE948/ is present?

    Thanks!

  2. Kristoffer Sahlin reporter

    Hi,

    Thank you for the quick reply.

    absolute path from where I'm issuing snakemake: [ksahlin@brubeck scripts]$ pwd /galaxy/home/ksahlin/prefix/source/pactrans-denovo/scripts

    The folder /nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.IOE948/ is not present, but there are other tmp folders there, this is how it looks like:

    [ksahlin@brubeck scripts]$ ls /nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/ code_tracking incomplete_files input_tracking locks params_tracking rule_tracking shadow shellcmd_tracking tmp.07MNS3 tmp.2URCFK tmp.365DQC tmp.9YOBNV tmp.ATSOFJ tmp.CPZJOS tmp.M6PH7Q tmp.S5A2EI tmp.UYC7J3 tmp.XHV6WS tmp.ZXKPQB version_tracking

    So I'm issuing snakemake from /galaxy/home/ksahlin/ , but this seems to be linked to /nfs/thumper.galaxyproject.org/home/ksahlin/

    Thank you, K

  3. Johannes Köster

    Thanks. Sounds very much like an NFS latency issue. That temp folder is created in the main thread on the host node. But the touch happens on the job node. If the job is very fast, it might not have the temp folder yet. I will fix this soon.

  4. Kristoffer Sahlin reporter

    Hi, After starting to use the shadow: "shallow", i still get this error. Sorry :(

    Changing to shadow directory: /nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/shadow/tmpghytoo3u
    touch: cannot touch `/nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.JYVIZG/1.jobfinished': No such file or directory
    touch: cannot touch `/nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.JYVIZG/1.jobfailed': No such file or directory
    
  5. Kristoffer Sahlin reporter

    Hi, I just ran the new master version: I run into the following error (before jobs are submitted)

    Traceback (most recent call last):
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/__init__.py", line 392, in snakemake
        no_hooks=no_hooks)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/workflow.py", line 436, in execute
        success = scheduler.schedule()
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/scheduler.py", line 216, in schedule
        self.run(job)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/scheduler.py", line 238, in run_cluster_or_local
        error_callback=self._error)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/executors.py", line 443, in run
        jobfailed=jobfailed)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/executors.py", line 383, in spawn_jobscript
        target=target, wait_for_files=" ".join(wait_for_files),
    TypeError: sequence item 2: expected str instance, NoneType found
    
  6. Johannes Köster

    Hi, Sorry for the problems. I have just committed a potential fix. Does it work now? Best, Johannes

  7. Kristoffer Sahlin reporter

    Hi, no worries. I don't get the above error with the new commit. However, when I launch snakemake something weird happens. It does not create any slurm-<jobid>.out file with makes me believe it does not submit any job. I do get a mail "SLURM Job_id=112007 Name=pbtranscript_classify-Is Failed, Run time 00:00:00" As I have specified to get mail of status. I don't know if its on my side (i.e. the rule) or the new commit. But I have never experienced not getting the slurm-<jobid>.out file in the directory I'm launching snakemake from, even with shadow or other parameters invoked. Any Idea what is going on?

  8. Kristoffer Sahlin reporter

    Hi again,

    I reverted to commit 8a0e0a8071cced89cf9bd04edba97704da8a9a7f git branch new_branch_name 56bca53787687b0f97a2098c43088a0db782e105 HEAD is now at 8a0e0a8... Added -C shortcut for --config. And the job is submitted and works fine, so it seems that it is indeed later commits that causes this weird behaviour

  9. Johannes Köster

    Thanks for your help to track this down. I currently have no cluster access, so it is hard to do the debugging myself. Can you try to subsequently add the following commits and see which commit breaks it?

    That would be awesome! Sorry for the inconvenience!

  10. Kristoffer Sahlin reporter

    Well this is confusing, maybe it was something temporary on my end that caused the lack of the slurm-<jobid>.out file --- because now it seems to work. I can confirm that current head commit: d6ee3824cc5e46992ed758c1b9593c47e4314328 works without error. This conclusion after I tried earlier commits: 0644362bc098b6d835bbb7442c8157bfc40c1b97, 56bca53787687b0f97a2098c43088a0db782e105 and 56d3737022ef29afb987ac4a90387675e2025acf that gave me

    Traceback (most recent call last):
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/__init__.py", line 392, in snakemake
        no_hooks=no_hooks)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/workflow.py", line 436, in execute
        success = scheduler.schedule()
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/scheduler.py", line 216, in schedule
        self.run(job)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/scheduler.py", line 238, in run_cluster_or_local
        error_callback=self._error)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/executors.py", line 443, in run
        jobfailed=jobfailed)
      File "/galaxy/home/ksahlin/.pyenv/versions/3.4.2/lib/python3.4/site-packages/snakemake-3.5.4-py3.4.egg/snakemake/executors.py", line 383, in spawn_jobscript
        target=target, wait_for_files=" ".join(wait_for_files),
    TypeError: sequence item 5: expected str instance, NoneType found
    

    and 1928eaac9975fb7564e17a26c2dc4fa8ee91be41 worked fine without error.

    I will probably have an answer to the original issue (cannot touch) here when my rule finishes, as I'm running it with the master head commit now.

  11. Kristoffer Sahlin reporter

    To clarify, current head commit works well with respect to the error I reported yesterday (no slurm-output), I will confirm if it also solves the original error reprooted later.

    Thanks for your time and sorry for the mess.

  12. Kristoffer Sahlin reporter

    I still get the error touch: cannot touch '/nfs/thumper.galaxyproject.org/home/ksahlin/prefix/source/pactrans-denovo/scripts/.snakemake/tmp.V36RTP/0.jobfailed': No such file or directory'

    the directory /tmp.V36RTP/ does not exist. I ran with shadow: "shallow"

  13. Log in to comment