An error occurred while running the WIsH in step 3

[linzhl]

Hi Simon,

Thanks for providing this powerful tool!

When the program runs to the third step, it gives me the errors as below:

[3/1/Run] Running (recoded)WIsH...
### Welcome to iPHoP ###
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'normalized'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/frame.py", line 3751, in _set_item_mgr
    loc = self._info_axis.get_loc(key)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
    raise KeyError(key) from err
KeyError: 'normalized'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/wish.py", line 214, in process_batch
    rewish_results = add_pvalues(rewish_results,ref_file)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/wish.py", line 227, in add_pvalues
    rewish_results["normalized"] = rewish_results.apply(lambda x: transform(x['LL'],x['Host'],ref_mat), axis=1)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/frame.py", line 3602, in __setitem__
    self._set_item_frame_value(key, value)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/frame.py", line 3742, in _set_item_frame_value
    self._set_item_mgr(key, arraylike)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/frame.py", line 3754, in _set_item_mgr
    self._mgr.insert(len(self._info_axis), key, value)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1162, in insert
    block = new_block(values=value, ndim=self.ndim, placement=slice(loc, loc + 1))
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1937, in new_block
    check_ndim(values, placement, ndim)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1979, in check_ndim
    raise ValueError(
ValueError: Wrong number of items passed 3, placement implies 1
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/public1/group_lwj/software/conda_envs/iphop/bin/iphop", line 10, in <module>
    sys.exit(cli())
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/iphop.py", line 128, in cli
    args["func"](args)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/master_predict.py", line 87, in main
    wish.run_and_parse_wish(args)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/wish.py", line 44, in run_and_parse_wish
    run_rewish(args["fasta_file"],args["wishrawresult"],args["rewish_db_dir"],args["wish_negfit"],args["tmp"],threads_tmp,n_host_by_phage)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/wish.py", line 159, in run_rewish
    async_parallel(process_batch, args_list, threads)
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/wish.py", line 251, in async_parallel
    return [r.get() for r in results]
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/site-packages/iphop/modules/wish.py", line 251, in <listcomp>
    return [r.get() for r in results]
  File "/public1/group_lwj/software/conda_envs/iphop/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
ValueError: Wrong number of items passed 3, placement implies 1

Do you have any idea how to fix this?

Best regards,

Zhiliang

[Simon Roux]

Hi Zhiliang,

I’m not sure what is happening here, there seems to be an issue with multithreading, but it’s not clear what is happening. How many threads do you ask iPhoP to use, and if more than 1, can you try with only 1 thread, and see if it helps ?

Best,

Simon

[linzhl]

Hi Simon,

Using one thread did not make any difference when I reran the program. However, it worked well after I split my sequences into 100 small files and tested one. It seems that the problem is either the large file size or some weird sequences. I will report back after testing all files.

‌

[linzhl]

Hi Simon,

I've got all of my results with no errors. A file containing too many sequences may cause that fault.

Thanks,

Zhiliang

‌

[Simon Roux]

Oh interesting, I wonder if maybe some of the threads were running out of memory, and then it caused issues when iPHoP was trying to gather all the results (which is the step at which you saw the error the first time).

Anyway, thanks a lot for letting us know, and I will add this information to the FAQ !

[Simon Roux]

Likely due to known issues when input file has 1,000 sequences or an exact multiple of 1,000. Solved now