`syntax error near unexpected token` when using '--cluster-status' with 'bsub'

Create issue
Issue #1185 new
kpj created an issue

I am using Snakemake version 5.4.4 with Python 3.6.

The script specified with --cluster-status receives the output of the command given for --cluster as its argument. In the case of bsub this leads to an error, because bsub does not only return the job id but also more text:

Generic job.
Job <1883954> is submitted to queue <foo>

Conceptually, this is not problem as one can extract the job id in the cluster-status script (eg jobid="$(echo $1 | perl -n -e '/<(\d+)>/ && print $1')").

However, the bug lies in the way that Snakemake passes the bsub return-value to the cluster-status script: since <, > are special characters (in bash) this leads to the following error:

/bin/sh: -c: line 0: syntax error near unexpected token `1883954'
/bin/sh: -c: line 0: `./cluster_status.sh Job <1883954> is submitted to queue <foo>.'
Exception in thread Thread-1:
Traceback (most recent call last):
  File "foo/python3.6/site-packages/snakemake/executors.py", line 733, in job_status
    shell=True).decode().split("\n")[0]
  File "foo/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "foo/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command './cluster_status.sh Job <1883954> is submitted to queue <foo>.' returned non-zero exit status 1.

This can be fixed by adding quotation marks around the script’s argument:

# in snakemake/executors.py
return subprocess.check_output(
  '{statuscmd} "{jobid}"'.format(
    jobid=job.jobid,
    statuscmd=self.statuscmd),
  shell=True).decode().split("\n")[0]

Comments (2)

  1. kpj reporter
    • edited description

    I am using Snakemake version 5.4.4.

    The script specified with --cluster-status receives the output of the command given for --cluster as its argument. In the case of bsub this leads to an error, because bsub does not only return the job id but also more text:

    Generic job.
    Submitted job 23 with external jobid 'Job <1883954> is submitted to queue <normal.24h>.'.
    

    Conceptually, this is not problem as one can extract the job id in the cluster-status script (eg jobid="$(echo $1 | perl -n -e '/<(\d+)>/ && print $1')").

    However, the bug lies in the way that Snakemake passes the bsub return-value to the cluster-status script: since <, > are special characters (in bash) this leads to the following error:

    /bin/sh: -c: line 0: syntax error near unexpected token `1883954'
    /bin/sh: -c: line 0: `./cluster_status.sh Job <1883954> is submitted to queue <foo>.'
    Exception in thread Thread-1:
    Traceback (most recent call last):
      File "foo/python3.6/site-packages/snakemake/executors.py", line 733, in job_status
        shell=True).decode().split("\n")[0]
      File "foo/python3.6/subprocess.py", line 336, in check_output
        **kwargs).stdout
      File "foo/python3.6/subprocess.py", line 418, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command './cluster_status.sh Job <1883954> is submitted to queue <foo>.' returned non-zero exit status 1.
    

    This can be fixed by changing the affected line to '{statuscmd} "{jobid}"', i.e. adding quotation marks around the script’s argument.

  2. kpj reporter
    • edited description

    I am using Snakemake version 5.4.4.

    The script specified with --cluster-status receives the output of the command given for --cluster as its argument. In the case of bsub this leads to an error, because bsub does not only return the job id but also more text:

    Generic job.
    Submitted job 23 with external jobid &#39;Job &lt;1883954&gt; is submitted to queue &lt;normal.24h&gt;.&#39;.
    

    Conceptually, this is not problem as one can extract the job id in the cluster-status script (eg jobid=&#34;$(echo $1 | perl -n -e &#39;/&lt;(\d+)&gt;/ &amp;&amp; print $1&#39;)&#34;).

    However, the bug lies in the way that Snakemake passes the bsub return-value to the cluster-status script: since &lt;, &gt; are special characters (in bash) this leads to the following error:

    /bin/sh: -c: line 0: syntax error near unexpected token `1883954&#39;
    /bin/sh: -c: line 0: `./cluster_status.sh Job &lt;1883954&gt; is submitted to queue &lt;foo&gt;.&#39;
    Exception in thread Thread-1:
    Traceback (most recent call last):
      File &#34;foo/python3.6/site-packages/snakemake/executors.py&#34;, line 733, in job_status
        shell=True).decode().split(&#34;\n&#34;)[0]
      File &#34;foo/python3.6/subprocess.py&#34;, line 336, in check_output
        **kwargs).stdout
      File &#34;foo/python3.6/subprocess.py&#34;, line 418, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command &#39;./cluster_status.sh Job &lt;1883954&gt; is submitted to queue &lt;foo&gt;.&#39; returned non-zero exit status 1.
    

    This can be fixed by adding quotation marks around the script’s argument:

    # in snakemake/executors.py
    return subprocess.check_output(
      '{statuscmd} "{jobid}"'.format(
        jobid=job.jobid,
        statuscmd=self.statuscmd),
      shell=True).decode().split("\n")[0]
    
  3. Log in to comment