wish KeyError
Issue #36
closed
Thank you for your nice work . I made some custom MAGs and added them into iphop_db . I got de_novo_wf result of GTDBrelaese 214 using gtdbktk-2.3.0
The MAG added database was created without any problem.
When I tried with two sequence to identify host, They generated wish error . I 'll attach log below line.
/home01/k052a02/sra_data/virus/from_genomad/pan88269_analysis/Sept_2021_pub_rw_SR3952/db/rewish_models_extra/Batch_extra.pkl LL processed, now we calculate the p-values and export
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'SAMEA2467029_bin.11_sub'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 208, in process_batch
rewish_results = add_pvalues(rewish_results,ref_file)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 221, in add_pvalues
rewish_results["normalized"] = rewish_results.apply(lambda x: transform(x['LL'],x['Host'],ref_mat), axis=1)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/frame.py", line 8740, in apply
return op.apply()
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/apply.py", line 688, in apply
return self.apply_standard()
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/apply.py", line 812, in apply_standard
results, res_index = self.apply_series_generator()
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/apply.py", line 828, in apply_series_generator
results[i] = self.f(v)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 221, in <lambda>
rewish_results["normalized"] = rewish_results.apply(lambda x: transform(x['LL'],x['Host'],ref_mat), axis=1)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 227, in transform
ref_row = ref_mat.loc[host,['Average','Stdev']]
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexing.py", line 925, in getitem
return self._getitem_tuple(key)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexing.py", line 1100, in _getitem_tuple
return self._getitem_lowerdim(tup)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexing.py", line 838, in _getitem_lowerdim
section = self._getitem_axis(key, axis=i)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexing.py", line 1164, in _getitem_axis
return self._get_label(key, axis=axis)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexing.py", line 1113, in _get_label
return self.obj.xs(label, axis=axis)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/generic.py", line 3776, in xs
loc = index.get_loc(key)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
raise KeyError(key) from err
KeyError: 'SAMEA2467029_bin.11_sub'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/bin/iphop", line 10, in <module>
sys.exit(cli())
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/iphop.py", line 128, in cli
args"func"
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/master_predict.py", line 87, in main
wish.run_and_parse_wish(args)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 48, in run_and_parse_wish
run_rewish(args["fasta_file"],extra_raw_results,args["wish_db_dir_extra"],extra_negfit,extra_out_tmpdir,threads_tmp)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 156, in run_rewish
async_parallel(process_batch, args_list, threads)
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 245, in async_parallel
return [r.get() for r in results]
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/site-packages/iphop/modules/wish.py", line 245, in <listcomp>
return [r.get() for r in results]
File "/home01/k052a02/conda_envs/k026a02/mambaforge/envs/iphop_env/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
KeyError: 'SAMEA2467029_bin.11_sub'
# db made log
…
[5] Add new genomes to WIsH database...
[6] Add new genomes to VHM database...
[7] Add new genomes to PHP database...
counting kmer ...
Preparing output file
done.
[8] Now build the new host genome metadata file...
We added 3930 additional bacteria genomes and 22 additional archaea genomes
[9] All done
I installed your tool using mamba.
here is the conda env information
Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
_r-mutex 1.0.1 anacondar_1 conda-forge
abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge
absl-py 1.4.0 pyhd8ed1ab_0 conda-forge
aiohttp 3.7.4.post0 py38h497a2fe_1 conda-forge
alsa-lib 1.2.8 h166bdaf_0 conda-forge
astunparse 1.6.3 pyhd8ed1ab_0 conda-forge
async-timeout 3.0.1 py_1000 conda-forge
attrs 23.1.0 pyh71513ae_1 conda-forge
binutils_impl_linux-64 2.40 hf600244_0 conda-forge
biopython 1.79 py38h0a891b7_3 conda-forge
blast 2.12.0 hf3cf87c_4 bioconda
blinker 1.6.2 pyhd8ed1ab_0 conda-forge
boost-cpp 1.74.0 h75c5d50_8 conda-forge
bottleneck 1.3.7 py38h7e4f40d_0 conda-forge
brotlipy 0.7.0 py38h0a891b7_1005 conda-forge
bwidget 1.9.14 ha770c72_1 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.0 hd590300_0 conda-forge
ca-certificates 2023.5.7 hbcca054_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cachetools 4.2.4 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
certifi 2023.5.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py38h4a40e3a_3 conda-forge
chardet 4.0.0 py38h578d9bd_3 conda-forge
charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge
click 8.0.4 py38h578d9bd_0 conda-forge
crisper_recognition_tool 1.2 hdfd78af_2 bioconda
cryptography 39.0.0 py38h1724139_0 conda-forge
cuda-version 11.1 hdbd7af8_2 conda-forge
cudatoolkit 11.1.1 ha002fc5_11 conda-forge
cudnn 8.8.0.121 h0800d71_0 conda-forge
curl 7.86.0 h7bff187_1 conda-forge
diamond 2.0.15 hb97b32f_1 bioconda
entrez-direct 16.2 he881be0_1 bioconda
expat 2.5.0 hcb278e6_1 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
gast 0.4.0 pyh9f0ad1d_0 conda-forge
gcc_impl_linux-64 12.2.0 hcc96c02_19 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gfortran_impl_linux-64 12.2.0 h55be85b_19 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
google-auth 1.35.0 pyh6c4a22f_0 conda-forge
google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge
google-pasta 0.2.0 pyh8c360ce_0 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
grpc-cpp 1.43.2 h9e046d8_3 conda-forge
grpcio 1.43.0 py38hdd6454d_0 conda-forge
gsl 2.7 he838d99_0 conda-forge
gxx_impl_linux-64 12.2.0 hcc96c02_19 conda-forge
h5py 3.8.0 nompi_py38hd5fa8ee_100 conda-forge
harfbuzz 6.0.0 h8e241bc_0 conda-forge
hdf5 1.12.2 nompi_h2386368_100 conda-forge
hmmer 3.3.2 hdbdd923_4 bioconda
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.6.0 pyha770c72_0 conda-forge
iphop 1.3.1 pyhdfd78af_0 bioconda
joblib 1.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keras 2.7.0 pyhd8ed1ab_0 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lcms2 2.14 h6ed2654_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcups 2.3.3 h3e49a29_2 conda-forge
libcurl 7.86.0 h7bff187_1 conda-forge
libdeflate 1.14 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 7.2.0 h69d50b8_2 conda-forge
libgcc-devel_linux-64 12.2.0 h3b97bd3_19 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.76.2 hebfc3b9_0 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libidn2 2.3.4 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
hdf5 1.12.2 nompi_h2386368_100 conda-forge
hmmer 3.3.2 hdbdd923_4 bioconda
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.6.0 pyha770c72_0 conda-forge
iphop 1.3.1 pyhdfd78af_0 bioconda
joblib 1.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keras 2.7.0 pyhd8ed1ab_0 conda-forge
keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.19.3 h3790be6_0 conda-forge
lcms2 2.14 h6ed2654_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcups 2.3.3 h3e49a29_2 conda-forge
libcurl 7.86.0 h7bff187_1 conda-forge
libdeflate 1.14 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 7.2.0 h69d50b8_2 conda-forge
libgcc-devel_linux-64 12.2.0 h3b97bd3_19 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.76.2 hebfc3b9_0 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libidn2 2.3.4 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libprotobuf 3.19.6 h3eb15da_0 conda-forge
libsanitizer 12.2.0 h46fd767_19 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx-devel_linux-64 12.2.0 h3b97bd3_19 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libtiff 4.4.0 h82bc61c_5 conda-forge
libunistring 0.9.10 h7f98852_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.10.3 hca2bb57_4 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
make 4.3 hd18ef5c_1 conda-forge
markdown 3.4.3 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.2 py38h1de0b5d_0 conda-forge
multidict 6.0.4 py38h1de0b5d_0 conda-forge
nccl 2.15.5.1 h12f7317_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nomkl 1.0 h5ca1d4c_0 conda-forge
numexpr 2.8.4 py38h69a160b_100 conda-forge
numpy 1.23.5 py38h7042d01_0 conda-forge
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
openjdk 17.0.3 h58dac75_5 conda-forge
openssl 1.1.1t h0b41bf4_0 conda-forge
opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py38h8c16a72_0
pango 1.50.14 hd33c08f_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
perl 5.22.2.1 0 conda-forge
perl-archive-tar 2.18 1 bioconda
perl-bioperl 1.6.924 4 bioconda
perl-common-sense 3.74 0 bioconda
perl-exporter-tiny 0.042 1 bioconda
perl-json 2.90 1 bioconda
perl-json-xs 2.34 0 bioconda
perl-list-moreutils 0.413 1 bioconda
perl-threaded 5.32.1 hdfd78af_1 bioconda
perl-yaml 1.24 pl522_1 bioconda
piler-cr 1.06 h4ac6f70_4 bioconda
pip 23.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
platformdirs 3.5.1 pyhd8ed1ab_0 conda-forge
pooch 1.7.0 pyha770c72_3 conda-forge
prodigal 2.6.3 h031d066_6 bioconda
protobuf 3.19.6 py38h8dc9893_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pyasn1 0.4.8 py_0 conda-forge
pyasn1-modules 0.2.7 py_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyjwt 2.7.0 pyhd8ed1ab_0 conda-forge
pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.8.15 h257c98d_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-flatbuffers 2.0 pyhd8ed1ab_0 conda-forge
python_abi 3.8 3_cp38 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
r-base 4.0.5 hb87df5d_8 conda-forge
r-lattice 0.20_45 r40hcfec24a_0 conda-forge
r-matrix 1.4_1 r40h0154571_0 cond
r-rcpp 1.0.9 r40h7525677_1 conda-forge
r-rcppeigen 0.3.3.9.2 r40h43535f1_0 conda-forge
re2 2022.02.01 h9c3ff4c_0 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.29.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge
rsa 4.9 pyhd8ed1ab_0 conda-forge
scikit-learn 0.22.2.post1 py38hcdab131_0 conda-forge
scipy 1.10.1 py38h59b608b_3 conda-forge
sed 4.8 he412f7d_0 conda-forge
setuptools 67.7.2 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sqlite 3.42.0 h2c6b66d_0 conda-forge
sysroot_linux-64 2.12 he073ed8_15 conda-forge
tensorboard 2.6.0 pyhd8ed1ab_1 conda-forge
tensorboard-data-server 0.6.1 py38h2b5fc30_4 conda-forge
tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge
tensorflow 2.7.0 pypi_0 pypi
tensorflow-base 2.7.1 cuda111py38h13b88b6_0 conda-forge
tensorflow-decision-forests 0.2.2 pypi_0 pypi
tensorflow-estimator 2.7.1 cuda111py38h6353a60_0 conda-forge
termcolor 2.3.0 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tktable 2.10 hb7b940f_3 conda-forge
typing-extensions 4.5.0 hd8ed1ab_0 conda-forge
typing_extensions 4.5.0 pyha770c72_0 conda-forge
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
werkzeug 2.3.4 pyhd8ed1ab_0 conda-forge
wget 1.20.3 ha56f1ee_1 conda-forge
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
wrapt 1.15.0 py38h1de0b5d_0 conda-forge
xorg-fixesproto 5.0 h7f98852_1002 conda-forge
xorg-inputproto 2.3.2 h7f98852_1002 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.8.4 h0b41bf4_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge
xorg-libxi 1.7.10 h7f98852_0 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-libxt 1.2.1 h7f98852_2 conda-forge
xorg-libxtst 1.2.3 h7f98852_1002 conda-forge
xorg-recordproto 1.14.2 h7f98852_1002 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yarl 1.9.1 py38h01eb140_0 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
Let me know how to handle this issue.
Thank you
Comments (8)
-
repo owner
-
reporter
I found out that only about 1400 genomes were integrated into wish DB.
3952 genomes were used for database creation.
When I opened iphop dbcreation log file, there is no warning or error massage in wish database part.
when I ran iphop again on failed run, all steps works well execept Wish part ( they recognize files and skip wish part).How could I handle it ?
# db creation log (iphop add_to_db)
[5] Add new genomes to WIsH database...
[6] Add new genomes to VHM database...
[7] Add new genomes to PHP database...
counting kmer ...
Preparin
Thank you very much .
-
repo owner
Right, there seems to be something happening when too many genomes are added to the iPHoP database (or rather, something seems to be wrong with the WIsH database when trying to add more than ~ 1,000 genomes). I am looking into it right now (see similar issue https://bitbucket.org/srouxjgi/iphop/issues/37/key-error-with-custom-db), and will update this issue when we have a fix in place.
-
reporter
Thank you for your quick reply.
I cannot access issue 37 you mentioned .
Anyway I’ll wait for it. Thank you !
-
repo owner
Hi,
There is a new version on bioconda (iPHoP v1.3.2) in which this bug should be fixed. Please update your iPHoP install (“conda install iphop=1.3.2”), and rebuild your custom database (you will unfortunately need to start from scratch here, i.e. re-run the “add_to_db” part). With the new custom database built with iPHoP v1.3.2, you should not see this error anymore.
Let me know if it works !
Thanks,
Simon
-
reporter
Thank you for your update.
I made database with iPHoP v1.3.2 with custom genomes. Now, there are all custom genome accession numbers in db/db_infos/Wish_extra_negFits.csv.I tested the database with one sequence, there was no error during operation .
- (except one warning
- WARNING:tensorflow:5 out of the last 14 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f84eb088e50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
5 out of the last 14 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f84eb088e50> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.)
Based on this, I believe that the issue has been resolved.
I appreciate for your maintaining this tool.
Thank you.
-
repo owner
Perfect, thanks for the follow up!
-
repo owner
- changed status to closed
Bug fixed in latest version
- Log in to comment
Hi,
Can you check the content of the file /home01/k052a02/sra_data/virus/from_genomad/pan88269_analysis/Sept_2021_pub_rw_SR3952/db_infos/Wish_extra_negFits.csv ? And specifically whether there is a row for SAMEA2467029_bin.11_sub ?
Best,
Simon