OSError: [Errno 24] Too many open files during training

[HsuWanTing]

@MhLiao I tried to pretrain a model on SynthText dataset.
I followed your training script:

python -m torch.distributed.launch --nproc_per_node=4  tools/train_net.py --config-file configs/pretrain/seg_rec_poly_fuse_feature.yaml

But I always got the OSError after training a few iterations.

OSError: [Errno 24] Too many open files

How to fix this bug? Is there a way to reduce the number of opened files?

[samanthawyf]

@HsuWanTing @MhLiao hello, have you solved the problem and how? I met the same error. Thanks.

[gw00295652]

@HsuWanTing 你这个问题可能是分布式训练（多进程）导致的问题。

[HsuWanTing]

@samanthawyf I just used the command line ulimit -n 65535 and the error was gone.

[lanfeng4659]

In my situation, I solve this error by inserting the following two lines code in the train_net.py file.

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')