error in adding test MAGs to the standard database: IndexError
Hello,
Thank your for this great tool. I had no problem running my viral ncontig files with the standard database. When I tried to add test MAGs (Data_test_add_to_db) to the standard database, I am having the following error:
" . . [7] Add new genomes to PHP database... counting kmer ... Preparing output file done. [8] Now build the new host genome metadata file... Traceback (most recent call last): File "/Users/yanack/anaconda3/envs/iphop/bin/iphop", line 10, in <module> sys.exit(cli()) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/iphop.py", line 122, in cli args"func" File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/modules/master_add_to_db.py", line 222, in main add_to_genome_file(args,logger) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/modules/master_add_to_db.py", line 55, in add_to_genome_file args['taxo_a'] = glob.glob(os.path.join(args['gtdb_dir'],"infer","gtdbtk.ar[0-9]*.decorated.tree-taxonomy"))[0] IndexError: list index out of range "
I would be grateful, If you could help with solving this issue.
Thanks,
Kadir
Comments (7)
-
repo owner -
repo owner Hi Kadir: looking into it, it’s almost certainly the same issue as https://bitbucket.org/srouxjgi/iphop/issues/22/error-in-adding-mags-to-standard-db , so the same “fix” should help (i.e. adding the test MAGs to your MAG set).
Let us know if this works !
Best,
Simon
-
Hi Simon,
Thank you for your quick respond. I will try it once I am done with the runs with standard database.
I have another question: is there any way of speeding up the runs? I have several hundred thousand contigs. It takes so long to process one single contigs file with the standard database (-t 100).
Best,
Kadir
-
repo owner So the short answer is “not really”. iPHoP is optimized for batches of ~ 1,000 sequences, and running batches will be much quicker than running sequences one by one. However, it is still relatively slow, and the only way around it is to distribute the compute around different nodes / compute infrastructures.
-
Hi Simon,
I am getting the error below while running some contigs files. I was not able to fix it. Can you please help me with solving this issue?
2023-02-08 01:53:50.232708: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at sparse_to_dense_op.cc:227 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[500,61,21,31] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/Users/yanack/anaconda3/envs/iphop/bin/iphop", line 10, in <module> sys.exit(cli()) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/iphop.py", line 122, in cli args["func"](args) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/modules/master_predict.py", line 98, in main runmodels.run_individual_models(args) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/modules/runmodels.py", line 111, in run_individual_models full_predicted = run_single_classifier(classifier,tensors,args) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/iphop/modules/runmodels.py", line 238, in run_single_classifier predict = best_model.predict([tf.sparse.to_dense(tensors[i])]) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/tensorflow/python/ops/sparse_ops.py", line 1715, in sparse_tensor_to_dense return gen_sparse_ops.sparse_to_dense( File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/tensorflow/python/ops/gen_sparse_ops.py", line 3162, in sparse_to_dense _ops.raise_from_not_ok_status(e, name) File "/Users/yanack/anaconda3/envs/iphop/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 7107, in raise_from_not_ok_status raise core._status_to_exception(e) from None # pylint: disable=protected-access tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[500,61,21,31] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:SparseToDense]
Thanks,
Kadir
-
repo owner Hi Kadir,
“OOM when allocating tensor” is basically an out-of-memory error: you can either try to re-run the job with more memory, or you can split your contig files in smaller batches and run them separately, that should help too.
Best,
Simon
-
repo owner - changed status to closed
Answered
- Log in to comment
Hi Kadir,
This looks like the same error as https://bitbucket.org/srouxjgi/iphop/issues/22/error-in-adding-mags-to-standard-db . If you ran GTDB-tk wf in both bacteria and archaea modes in the same folder, then something happens that iPHoP did not anticipate and I need to look into it.