numpy.core._exceptions.MemoryError

[Former user]

I got the following error. Does it mean my data is too big and no way to solve it ?

^[[1;42mINFO^[[1;0m:vcontact2: Building the cluster and profiles (this may take some time...) If it fails, try re-running using --blast-fp flag and specifying merged.self-diamond.tab (or merged.self-blastp.tab) ^[[1;42mINFO^[[1;0m:vcontact2: Saving intermediate files... ^[[1;42mINFO^[[1;0m:vcontact2: Read 15531014 entries (dropped 253419 singletons) from vConTACT2_output/vConTACT_profiles.csv Traceback (most recent call last): File "/home/u3007815/.local/bin/vcontact2", line 834, in <module> main(options) File "/home/u3007815/.local/bin/vcontact2", line 668, in main pcp = vcontact2.pcprofiles.PCProfiles(merged_df, pcs_csv_df, profiles_csv, cpus, name, args.sig, args.max_sig, File "/home/u3007815/.local/lib/python3.8/site-packages/vcontact2/pcprofiles.py", line 71, in init self.ntw = self.network(self.matrix, self.singletons, thres=sig, max_sig=max_sig, threads=self.threads) File "/home/u3007815/.local/lib/python3.8/site-packages/vcontact2/pcprofiles.py", line 156, in network S += S.T # Symmetry File "/share1/anaconda/3.8/lib/python3.8/site-packages/scipy/sparse/lil.py", line 132, in iadd self[:,:] = self + other File "/share1/anaconda/3.8/lib/python3.8/site-packages/scipy/sparse/lil.py", line 333, in setitem IndexMixin.setitem(self, key, x) File "/share1/anaconda/3.8/lib/python3.8/site-packages/scipy/sparse/_index.py", line 116, in setitem self._set_arrayXarray_sparse(i, j, x) File "/share1/anaconda/3.8/lib/python3.8/site-packages/scipy/sparse/lil.py", line 319, in _set_arrayXarray_sparse x = np.asarray(x.toarray(), dtype=self.dtype) File "/share1/anaconda/3.8/lib/python3.8/site-packages/scipy/sparse/coo.py", line 321, in toarray B = self._process_toarray_args(order, out) File "/share1/anaconda/3.8/lib/python3.8/site-packages/scipy/sparse/base.py", line 1202, in _process_toarray_args return np.zeros(self.shape, dtype=self.dtype, order=order) numpy.core._exceptions.MemoryError: Unable to allocate 8.02 TiB for an array with shape (1049616, 1049616) and data type float64