lower mIoU than the result of the paper

[minwang-ai]

Hi,
I used the .yml file to create conda environment and default setting to train NUYv2 and the result 47. 48 is much lower than the result of the paper 50.3.

Does anyone get a similar result as follows?
mIoU test kv1: 0.4748908361165817
mIoU test: 0.4748908361165817
0.4748908361165817

What do you think is the reason for this?

Do you use the maximum or average in the paper ? Do you use different hyperparameters?

# or use the default arguments python train.py \ --dataset nyuv2 \ --dataset_dir ./datasets/nyuv2 \ --pretrained_dir ./trained_models/imagenet \ --results_dir ./results

[danielS91]

Note that the labels in NYUv2 are not perfect and the results may vary a bit. Therefore, we repeated each training several times and used a grid search for the learning rate ranging from 0.005 to 0.04 for NYUv2. Finally, we picked the highest value.

[minwang-ai]

Thanks for your information. I thought you used grid search and set the best hyperparameters as default values, right? That's why I feel confused.

[danielS91]

The best hyperparameters vary depending on the actual network configuration (network depth, block type, fusion, upsampling, ...). For the default hyperparameters, we picked the values that were best for most of the runs.

We have run the default configuration once again on one of our GPU nodes used for the paper (2080TI, CUDA 10.1.243, driver 460.39) during the night and got a mIoU of 49.93 at epoch 452.
This result is much better than yours. Are you sure that your environment is correctly set up and you are evaluating the best checkpoint?

[minwang-ai]

Hi Daniel Thank you for your reply.

Both of us used the same network configuration (network depth, block type, fusion, upsampling, ...) since we run default configuration.
I just followed the README file to create conda env using one command. The cuda version is slightly different I use 10.0 before and cuda/10.1.105 now. The GPU and cuda version (2080TI, CUDA 10.1.243, driver 460.39) might not affect the result that much.
The result 47.48% is not the best checkpoint but I just checked the best one is about 47.9% (however, none of result reaches 48%)

Please find the attached output files.
output.zip

I try to run multiple times now and see if there is a performance boost.

[minwang-ai]

I try to run multiple times now and see if there is a performance boost.

I just checked the results of another runs - epoch 257. It seems that the cuda version might affect the results.
Validation on test
kv1: 654 samples
mIoU test kv1: 0.4837290586722024
mIoU test: 0.4837290586722024
0.4837290586722024
The performance degradation might be caused by this issue:

However the best result within epoch 500 is still around 49.0 or 49.3%, which is less than 49.9% and 50.3%. I attached the output files output_49.zip

The only difference is the cuda version :

• You used 2080TI, CUDA 10.1.243, driver 460.39
• I used Tesla V100, CUDA 10.1.105, driver 460.32.03

Do you use other tricks which are not included in the source code or paper?