Error occurred when running hard negative mining

[wongbokhe1]

python hn_mine.py --model_name_or_path /data/bxhuang/model_folder/xiaobu-embedding-v2 --input_file /data/bxhuang/Reranking/train_data/text_pairs.json --output_file /data/bxhuang/Reranking/train_data/text_pairs_HN.json --range_for_sampling 50-80 --negative_number 15 --use_gpu_for_searching

Traceback (most recent call last):
File "/data/bxhuang/FlagEmbedding/scripts/hn_mine.py", line 116, in
find_knn_neg(model,
File "/data/bxhuang/FlagEmbedding/scripts/hn_mine.py", line 83, in find_knn_neg
index = create_index(p_vecs, use_gpu=use_gpu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/bxhuang/FlagEmbedding/scripts/hn_mine.py", line 29, in create_index
co = faiss.GpuMultipleClonerOptions()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'faiss' has no attribute 'GpuMultipleClonerOptions'

[hanhainebula]

Hello, @wongbokhe1. You can remove --use_gpu_for_searching from the command, or refer to here to install faiss-gpu.

[wongbokhe1]

Hello, @wongbokhe1. You can remove --use_gpu_for_searching from the command, or refer to here to install faiss-gpu.

Thanks a lot! After solving the faiss-gpu problem, a new error related to 'AbsEmbedder.py' occur:
Exception ignored in: <function AbsEmbedder.del at 0x7f6bfdd130a0>
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/site-packages/FlagEmbedding/abc/inference/AbsEmbedder.py", line 260, in del
File "/root/miniconda3/lib/python3.10/site-packages/FlagEmbedding/abc/inference/AbsEmbedder.py", line 352, in stop_multi_process_pool
File "/root/miniconda3/lib/python3.10/multiprocessing/process.py", line 133, in terminate
File "/root/miniconda3/lib/python3.10/multiprocessing/popen_fork.py", line 57, in terminate
AttributeError: 'NoneType' object has no attribute 'SIGTERM'
/root/miniconda3/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

[hanhainebula]

Hello, @wongbokhe1. You can remove --use_gpu_for_searching from the command, or refer to here to install faiss-gpu.

Thanks a lot! After solving the faiss-gpu problem, a new error related to 'AbsEmbedder.py' occur: Exception ignored in: <function AbsEmbedder.del at 0x7f6bfdd130a0> Traceback (most recent call last): File "/root/miniconda3/lib/python3.10/site-packages/FlagEmbedding/abc/inference/AbsEmbedder.py", line 260, in del File "/root/miniconda3/lib/python3.10/site-packages/FlagEmbedding/abc/inference/AbsEmbedder.py", line 352, in stop_multi_process_pool File "/root/miniconda3/lib/python3.10/multiprocessing/process.py", line 133, in terminate File "/root/miniconda3/lib/python3.10/multiprocessing/popen_fork.py", line 57, in terminate AttributeError: 'NoneType' object has no attribute 'SIGTERM' /root/miniconda3/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

Please refer to: #1187 (comment)