Jobs submitted on the cluster cannot be run in multiple threads

[yun jian li]

Hello, my local computer has 18 cores and 32 threads, which can run perfectly. But I want to increase the calculation speed, so I submitted it to the cluster using the command line. But I found that I can't call multiple cores on the cluster. My command indicates that 128 cores are used, but in fact only 7 cores are working. It becomes very slow.

This is a fasta containing 1.6 million sequences. There are no errors when running on the cluster, but it is very slow.

The local system has 18 cores and 32 threads, and 17 min have completed the operation. However, the cluster has only completed 50% of the operation after using 128 cores for 1 hour.

vsearch was installed using conda, and presto was installed using pip. Python 3.10.14, 3.8, and 3.12 were all tested.

Here are the commands I used:

source /public/home/zuot1/software/anaconda3/etc/profile.d/conda.sh
conda activate li
ClusterSets.py set -s test.fastq -f BARCODE -k CLUSTER --cluster vsearch --ident 0.85 --length 0.98 --outname test --log test.log --nproc 128

‌

‌

[ssnn]

It could be that your data has 5 really large barcode groups that are taking a long time to process (7 minus 2 I/O processes). If you have such large groups, you can downsample them.

You can check if that could be the issue by looking at the barcode group size distribution. Fow a quick view, maybe a command line similar to this can help (change my.fastq): grep "BARCODE=" <my.fastq> | sed 's/.*BARCODE=//' | awk '{print $1}' | sort | uniq -c | sort -s -rn -k 1,1 | head

If you prefer to make a plot, you can use ParseHeaders (https://presto.readthedocs.io/en/stable/tools/ParseHeaders.html#parseheaders-py-table) to create a tabulated file with the barcodes. Then you can easily load the barcodes to R or python.

[yun jian li]

Thanks for your reply, I will reduce the number of threads for testing. Thanks