iPHoP v1.3.3: raise ValueError('Unexpected result of `predict_function` '

[muchun zhu]

Hi Simon,

I find your Comments about ValueError on the Issue section. But the ValueErrorh is till existed when I excute iPHoP v1.3.3.

raise ValueError('Unexpected result of predict_function '
ValueError: Unexpected result of predict_function (Empty batch_outputs). Please use Model.compile(..., run_eagerly=True), or tf.config.run_functions_eagerly(True) for more information of where went wrong, or file a issue/bug to tf.keras.

[Simon Roux]

Hi Muchun,

“ValueError” may mean many different things, so these are not necessarily the same errors as before (likely are not).
Could you share the log and/or output folder so I can look into this one ?

[muchun zhu]

The following code block is the output log.

[1/1/Skip] Skipping computation of blastn against microbial genomes...
[1/3/Skip] Skipping blast parsing...
[2/1/Skip] Skipping computation of blastn against CRISPR...
[2/2/Skip] Skipping crispr parsing...
[3/1/Skip] Skipping computation of WIsH scores...
[3/2/Skip] Skipping WIsH parsing...
[4/1/Skip] Skipping computation of VHM s2 similarities...
[4/2/Skip] Skipping VHM parsing...
[5/1/Skip] Skipping computation of PHP scores...
[5/2/Skip] Skipping PHP parsing...
[6/1/Skip] Skipping RaFAH...
[6/2/Skip] Skipping RaFAH parsing...
[6.5/1/Skip] Skipping diamond search against RaFAH refs...
[6.5/2/Skip] Skipping calculation of AAI to RaFAH refs...
[7/Skip] We already found all the expected files, we skip...
[7.5/Skip] We already found all the expected files, we skip...
[8] Running the convolution networks...
[8/1] Loading data as tensors..
[8/1.1] Getting blast-based scores..
[8/1.2] Run blast classifier Model_blast_RF-39 (by batch)..
TF Parameter Server distributed training not available (this is expected for the pre-build release).
[INFO kernel.cc:1153] Loading model from path
[INFO decision_forest.cc:617] Model loaded with 1000 root(s), 637446 node(s), and 15 input feature(s).
[INFO abstract_model.cc:1063] Engine "RandomForestOptPred" built
[INFO kernel.cc:1001] Use fast generic engine
Traceback (most recent call last):
  File "/aimigene/zhumuchun/iphop/iphop_environment/bin/iphop", line 8, in <module>
      sys.exit(cli())
  File "/aimigene/zhumuchun/iphop/iphop_environment/lib/python3.8/site-packages/iphop/iphop.py", line 128, in cli
    args["func"](args)
  File "/aimigene/zhumuchun/iphop/iphop_environment/lib/python3.8/site-packages/iphop/modules/master_predict.py", line 107, in main
    runmodels.run_individual_models(args)
  File "/aimigene/zhumuchun/iphop/iphop_environment/lib/python3.8/site-packages/iphop/modules/runmodels.py", line 55, in run_individual_models
    full_predicted = run_single_classifier_rf(classifier,args["matrix_blast_rf"],args,full_matrix_labels)
  File "/aimigene/zhumuchun/iphop/iphop_environment/lib/python3.8/site-packages/iphop/modules/runmodels.py", line 265, in run_single_classifier_rf
    batch_predicted = best_model.predict(tfdf.keras.pd_dataframe_to_tf_dataset(feature_matrix))
  File "/aimigene/zhumuchun/iphop/iphop_environment/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/aimigene/zhumuchun/iphop/iphop_environment/lib/python3.8/site-packages/keras/engine/training.py", line 1804, in predict
    raise ValueError('Unexpected result of `predict_function` '
ValueError: Unexpected result of `predict_function` (Empty batch_outputs). Please use `Model.compile(..., run_eagerly=True)`, or ` 
tf.config.run_functions_eagerly(True)` for more information of where went wrong, or file a issue/bug to `tf.keras`.

Another thing to note is that I inpute Test_db_rw database.

[Simon Roux]

Thanks. I am actually not sure whether this is an iPHoP issue or an issue with Keras (the underlying library we use for the automated classifiers). One thing is that it seems like you are re-running on an existing directory. Did you always see this issue, or was your run cancelled/stopped midway through, and you are trying to resume it ? Can you also list the files currently existing in the “Wdir” output folder ?

[muchun zhu]

I can always see this problem when I re-run on an existing directory. The follow picture is “Wdir” output folder.

Some fasta files will run successfully, some will have this problem.

[Simon Roux]

Right, so if this is always when re-running on existing directory, I think this is most likely because of the step at which iPHoP was stopped and you are trying to resume from. One possibility would be to remove all the files starting with “matrix…” and “Prediction…”, and try to re-run on this directory.

[muchun zhu]

Unfortunately，I remove all the files starting with “matrix…” and “Prediction…”, and try to re-run on this directory. The bug remain as showing the following picture.

‌

[Simon Roux]

Could you share an input file for which you get this error ? And is it using the default database ?

[muchun zhu]

The input file has been e-mailed to you

[Simon Roux]

I’m sorry I can’t find an email with this input file. Could you try to re-send it at sroux . at . lbl.gov ? Thanks

[muchun zhu]

I have ben re-sent input file to sroux . at . lbl.gov. Please check！

[Simon Roux]

Got it, thanks ! I’ll look into it and let you know what I can find.

[Simon Roux]

So I looked at the input file, and I think the issue is that you are trying to use the database “Test_db_rw”. This is a test database that was only meant to provide a quick test when used alongside the test input file (test_input_phages.fna), but it’s a very limited database and I believe iPHoP is essentially complaining here that there are not enough results.

If you use the latest default database (“Aug_2023_pub_rw”), you should not see this error anymore (at least I did not see it on my side).

Let me know if that also fixes things for you.

[muchun zhu]

Thanks for you works. This input files was executed successfully by selecting the dafault database. I just worried that some input files may encounter this bug again because  I have many of contigs file as input files  can bu executed. Of course，I hope it can’t heppend forever.