FileNotFoundError:[...]prediction.list

[Alyzza]

Hello, I get a segmentation fault after WIsH starts running and a FileNotFoundError:[...]prediction.list'.

I tried the recommended solution which was to create the files needed to update the LD_LIBRARY_PATH variable, but it still gives me the same error. I was only doing the quick test: iphop predict --fa_file test/test_input_phages.fna --db_dir iphop_db/Test_db/ --out_dir iphop_test_results/test_input_phages_iphop

[3/1/Run] Running WIsH...
/bin/sh: line 1: 907205 Segmentation fault      /gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/iphop/utils/WIsH -c predict -g iphop_test_results/test_input_phages_iphop/Wdir/split_input/ -m iphop_db/Test_db/db/wish_models -n iphop_db/Test_db/db_infos/Wish_negFits.csv -r iphop_test_results/test_input_phages_iphop/Wdir/wish_results/ -t 1 -b > iphop_test_results/test_input_phages_iphop/Wdir/wish.log 2>&1
[3/2/Run] Get relevant WIsH hits...
Traceback (most recent call last):
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/bin/iphop", line 10, in <module>
    sys.exit(cli())
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/iphop/iphop.py", line 121, in cli
    args["func"](args)
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/master_predict.py", line 79, in main
    wish.run_and_parse_wish(args)
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 48, in run_and_parse_wish
    get_wish_results(args["fasta_file"],args["wishrawresult"],args["wishparsed"],args['messages'])
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 61, in get_wish_results
    with open(pred_file, newline='') as csvfile:
FileNotFoundError: [Errno 2] No such file or directory: 'iphop_test_results/test_input_phages_iphop/Wdir/wish_results/prediction.list'

Are there other solutions to fix this error?

[Simon Roux]

Hi,

Unfortunately, this is an issue with WIsH, and can be difficult to troubleshoot. Was your LD_LIBRARY_PATH successful , i.e. do you see the expected library path when you try:

\$ echo $LD_LIBRARY_PATH

?

[Alyzza]

Hi Simon,

I got WIsH to run by exporting LD_LIBRARY_PATH=/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/x86_64-conda-linux-gnu/lib/:/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/: and then running the iphop predict with the test data:

export LD_LIBRARY_PATH
iphop predict --fa_file test/test_input_phages.fna --db_dir iphop_db/Test_db/ --out_dir iphop_test_results/test_input_phages_iphop

But now I’m getting an error at [8/1.2]:

### Welcome to iPHoP ###
Looks like everything is now set up, we will first clean up the input file, and then we will start the host prediction steps themselves
[1/1/Skip] Skipping computation of blastn against microbial genomes...
[1/3/Skip] Skipping blast parsing...
[2/1/Skip] Skipping computation of blastn against CRISPR...
[2/2/Skip] Skipping crispr parsing...
[3/1/Skip] Skipping computation of WIsH scores...
[3/2/Skip] Skipping WIsH parsing...
[4/1/Skip] Skipping computation of VHM s2 similarities...
[4/2/Skip] Skipping VHM parsing...
[5/1/Skip] Skipping computation of PHP scores...
[5/2/Skip] Skipping PHP parsing...
[6/1/Skip] Skipping RaFAH...
[6/2/Skip] Skipping RaFAH parsing...
[6.5/1/Skip] Skipping diamond search against RaFAH refs...
[6.5/2/Skip] Skipping calculation of AAI to RaFAH refs...
[7/Skip] We already found all the expected files, we skip...
[7.5/Skip] We already found all the expected files, we skip...
[8] Running the convolution networks...
[8/1] Loading data as tensors..
[8/1.1] Getting blast-based scores..
[8/1.2] Run blast classifier Model_blast_Conv-87 (by batch)..
Traceback (most recent call last):
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/bin/iphop", line 10, in <module>
    sys.exit(cli())
  File "/gxfs_work1/geomar/smomw535/iphop/iphop/iphop.py", line 122, in cli
    args["func"](args)
  File "/gxfs_work1/geomar/smomw535/iphop/iphop/modules/master_predict.py", line 98, in main
    runmodels.run_individual_models(args)
  File "/gxfs_work1/geomar/smomw535/iphop/iphop/modules/runmodels.py", line 58, in run_individual_models
    full_predicted = run_single_classifier(classifier,tensors,args)
  File "/gxfs_work1/geomar/smomw535/iphop/iphop/modules/runmodels.py", line 229, in run_single_classifier
    best_model = keras.models.load_model(h5_file, compile=False)
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/gxfs_home/geomar/smomw535/miniconda3/envs/iphop_env/lib/python3.8/site-packages/tensorflow/python/saved_model/loader_impl.py", line 118, in parse_saved_model
    raise IOError(
OSError: SavedModel file does not exist at: /gxfs_work1/geomar/smomw535/iphop/iphop/classifiers/Model_blast_Conv-87.h5/{saved_model.pbtxt|saved_model.pb}

‌

[Simon Roux]

Hi Alyzza,

So this is an unusual error, and it suggests that the original installation of iPHoP was not successful (the missing file in “/gxfs_work1/geomar/smomw535/iphop/iphop/classifiers/Model_blast_Conv-87.h5/” should have been downloaded when iPHoP was first set up).

Can you share the content of the folders in “/gxfs_work1/geomar/smomw535/iphop/iphop/classifiers/” ?

[Alyzza]

Hi Simon,

This is what’s inside /classifiers:

ls /gxfs_work1/geomar/smomw535/iphop/iphop/classifiers/
Empirical_FDRs.pkl  Model_blast_Conv-87.h5  Model_crispr_Conv-85.h5   Model_php_Conv-90.h5   Model_vhm_Conv-92.h5   Model_wish_Conv-2.h5    RaFAH_exclusion_list
Model_Final_RF-24   Model_blast_RF-39       Model_crispr_Dense-15.h5  Model_php_Dense-74.h5  Model_vhm_Dense-97.h5  Model_wish_Dense-54.h5

‌

[Simon Roux]

Ok, so all the files seem to be here. I guess it could be a question of tensorflow version, or some of these files are incomplete. Can you check the size of all these files in this “classifiers” folder and compared to https://bitbucket.org/srouxjgi/iphop/src/main/iphop/classifiers/ ?

[Bryan Brown]

running into the same IO issue for the saved blast model as Alyzza, was anyone able to get it figured out?

‌

thanks,

bryan

[Simon Roux]

I have not heard yet about a fix unfortunately. I am wondering if this may be linked to h5py (see e.g. https://stackoverflow.com/a/70649154). So I wonder if, in the iphop_env, it may be worth trying to uninstall and reinstall h5py (as suggested in the stack overflow link) ?

[Bryan Brown]

Thanks for the suggestion, Simon. Unfortunately, that didn’t fix things for me but what did was downloading the actual model! I completely brainfarted here and assumed that git-lfs was installed on our hpc and it was not so git installed some small placeholder files that, of course, didn’t contain the models. I pulled the actual files and it worked just fine. So, TLDR: for folks installing via github, check that git-lfs is installed, it wont throw an error if you pull the repo without git-lfs!

[Simon Roux]

Ha, the good old git-lfs (I thought so clever when I decided to use it, only to regret it in a number of instances later on :-D ). And yes, I think it may throw a small warning but nothing really clear, and it will put placeholder files in there. Anyway, glad you figured it out, and thanks for sharing the solution that worked for you !

[Simon Roux]

Solved