Train the NYU dataset, but the result is not good.

[BayMaxBHL]

I attempted to test the NYU dataset using the provided weight file and obtained a result that was consistent with 85.17 in the paper.

But when I tried to train the NYU dataset, it was difficult to achieve the same accuracy and the gap was significant.

I have not made any changes to the training configuration of train, using a 3090TI.

[BayMaxBHL]

I am not sure if there are any configuration conditions that have not been set as default options in the code besides those mentioned in the paper. If you still remember some details, I would greatly appreciate it.

[BayMaxBHL]

There are few articles on estimating table plane normal vectors from a single image.
Your article is still a benchmark in this direction even today. The success of reproducing the accuracy of the paper greatly affects other students who wish to study in this field.

[BayMaxBHL]

exp_dir: ./experiments/exp00_test/
exp_name: exp00_test
visible_gpus: 01
architecture: GN
use_baseline: False
sampling_ratio: 0.4
importance_ratio: 0.7
loss_fn: UG_NLL_ours
n_epochs: 5
batch_size: 4
validate_every: 5000
visualize_every: 1000
distributed: True
workers: 12
weight_decay: 0.01
lr: 0.000357
same_lr: False
grad_clip: 0.1
div_factor: 25.0
final_div_factor: 10000.0
dataset_name: nyu
input_height: 480
input_width: 640
data_augmentation_color: True
data_augmentation_hflip: True
data_augmentation_random_crop: False
num_threads: 12
mode: train
exp_model_dir: ./experiments/exp00_test//models/
exp_vis_dir: ./experiments/exp00_test//vis/
exp_log_dir: ./experiments/exp00_test//log/

[BayMaxBHL]

total_iter: 5000
mean median rmse 5 7.5 11.25 22.5 30
38.050 35.639 44.162 3.527 6.796 12.270 30.207 41.788

total_iter: 10000
mean median rmse 5 7.5 11.25 22.5 30
37.014 34.177 43.714 4.668 9.045 15.647 33.464 44.225

total_iter: 15000
mean median rmse 5 7.5 11.25 22.5 30
35.905 31.601 43.314 4.855 9.957 17.958 36.793 47.755

total_iter: 20000
mean median rmse 5 7.5 11.25 22.5 30
33.596 29.393 40.789 7.073 12.327 19.922 39.492 50.873

total_iter: 25000
mean median rmse 5 7.5 11.25 22.5 30
30.600 24.442 38.644 9.411 16.359 25.794 46.973 57.809

total_iter: 30000
mean median rmse 5 7.5 11.25 22.5 30
27.787 20.209 36.379 11.813 20.074 30.835 53.938 64.413

total_iter: 35000
mean median rmse 5 7.5 11.25 22.5 30
25.923 17.471 34.673 12.125 21.840 34.493 59.016 68.453

total_iter: 40000
mean median rmse 5 7.5 11.25 22.5 30
24.833 15.984 34.082 14.782 24.670 37.487 62.002 70.908

total_iter: 45000
mean median rmse 5 7.5 11.25 22.5 30
23.196 14.173 32.517 17.344 28.517 41.961 65.105 73.120

total_iter: 50000
mean median rmse 5 7.5 11.25 22.5 30
22.580 13.074 32.308 19.077 31.062 44.797 66.849 74.086

total_iter: 55000
mean median rmse 5 7.5 11.25 22.5 30
22.276 13.061 31.848 18.954 30.767 44.799 67.221 74.752

total_iter: 60000
mean median rmse 5 7.5 11.25 22.5 30
22.371 13.480 31.842 18.548 30.219 43.702 67.058 74.760

total_iter: 65000
mean median rmse 5 7.5 11.25 22.5 30
21.183 12.596 30.262 19.146 31.754 46.001 68.717 76.282

total_iter: 70000
mean median rmse 5 7.5 11.25 22.5 30
20.808 12.078 29.944 21.470 33.815 47.596 68.965 76.379

total_iter: 75000
mean median rmse 5 7.5 11.25 22.5 30
20.328 11.331 29.803 22.613 35.818 49.755 70.425 77.408

total_iter: 80000
mean median rmse 5 7.5 11.25 22.5 30
20.401 11.620 29.554 20.897 34.330 48.873 70.140 77.281

total_iter: 85000
mean median rmse 5 7.5 11.25 22.5 30
20.059 11.298 29.339 23.348 36.078 49.856 70.490 77.533

total_iter: 90000
mean median rmse 5 7.5 11.25 22.5 30
19.746 10.673 29.418 25.366 38.233 51.694 71.325 78.140

total_iter: 95000
mean median rmse 5 7.5 11.25 22.5 30
19.476 10.660 28.776 24.273 37.805 51.772 71.517 78.454

total_iter: 100000
mean median rmse 5 7.5 11.25 22.5 30
19.137 10.321 28.540 26.420 39.461 52.761 72.026 78.773

total_iter: 105000
mean median rmse 5 7.5 11.25 22.5 30
19.244 10.469 28.610 24.954 38.532 52.352 72.156 78.914

total_iter: 110000
mean median rmse 5 7.5 11.25 22.5 30
19.027 10.308 28.347 25.833 39.239 52.799 72.235 79.080

total_iter: 115000
mean median rmse 5 7.5 11.25 22.5 30
18.652 9.862 28.041 27.609 40.954 54.154 72.876 79.456

total_iter: 120000
mean median rmse 5 7.5 11.25 22.5 30
18.546 9.805 27.941 28.195 41.376 54.201 72.972 79.622

total_iter: 125000
mean median rmse 5 7.5 11.25 22.5 30
18.567 9.785 27.950 28.098 41.399 54.253 72.857 79.541

total_iter: 130000
mean median rmse 5 7.5 11.25 22.5 30
18.445 9.632 27.951 29.051 42.102 54.705 73.074 79.707

total_iter: 135000
mean median rmse 5 7.5 11.25 22.5 30
18.365 9.522 27.847 29.200 42.398 55.026 73.235 79.766

total_iter: 140000
mean median rmse 5 7.5 11.25 22.5 30
18.451 9.656 27.882 28.579 41.888 54.664 73.122 79.694

total_iter: 145000
mean median rmse 5 7.5 11.25 22.5 30
18.339 9.501 27.833 29.475 42.558 55.075 73.275 79.780

total_iter: 150000
mean median rmse 5 7.5 11.25 22.5 30
18.345 9.553 27.780 29.217 42.347 54.920 73.202 79.767

total_iter: 154560
mean median rmse 5 7.5 11.25 22.5 30
18.319 9.507 27.768 29.469 42.577 55.031 73.219 79.774

[baegwangbin]

Hi, have you tried training on the full NYUv2 dataset which consists of 30K images?

[BayMaxBHL]

I am really excited to receive your reply.@baegwangbin
Yes, I used GeoNet's 3W+ image dataset.
I have tried using it directly and have also tried using it after converting it into an image.

[BayMaxBHL]

I have also tried to expand the number of epochs.

[BayMaxBHL]

But it has never been reproduced successfully.
Do you need me to provide anything? For example, code?
I'm worried that I won't be able to reproduce it due to my own changes, so I haven't changed anything except the address.

[zhangshaos]

I meet the same problem in Scannet dataset. The loss seems not converging well.
But when I train the model in my own dataset, the result is pretty good.

After visualizing the predicted result in Scannet, I found the model learn the data bias in early training phase. The bias is not previous in my own dataset.

[BayMaxBHL]

@zhangshaos
Can you train the accuracy of the article using the ScanNet dataset?
I saw at the time that the Scannet dataset was too large and my computing resources were not very large, so I have been trying to reproduce the accuracy of NYU.
But I'm curious about the accuracy of the ScanNet dataset.

[BayMaxBHL]

@zhangshaos
I don't quite understand what you're saying about data bias？

[BayMaxBHL]

@baegwangbin
I have been thinking about why the accuracy of 85.17 cannot be reproduced.
I guess the problem may be with the data. Would it be convenient for you to publicly disclose the processed dataset? Baidu Netdisk or other Netdisk can be used.
The dataset I am currently using is inferred from your processing steps. The data provided by GeoNet is a floating-point precision label, but your label data is an uint8 type image. I processed the annotation to the image in reverse based on the data in your paper.
Or did you include test data in the training txt in the final training?

[baegwangbin]

@BayMaxBHL Hi, sorry for the delayed reply. I took GeoNet's data and saved them into uint8 png type by doing ((n+1) * 0.5) * 255.

[BayMaxBHL]

[zhangshaos]

I choose a random subset from the total ScanNet dataset. I split it into 23228 pictures for training and 5639 pictures for testing.
I make other hyper parameters default except visualization params.
In early phase, the NN learns nothing and just output a direct result:

In middle phase, it output over smooth result:

In last phase:

---

But for my own dataset, the NN converge in the early phase. And output pretty good result in last phase.

I guess the slow converging in early phase ruins the final output.

[BayMaxBHL]

@baegwangbin
Do you still have the two log files for training NYU? I really want to get them.

[baegwangbin]

@BayMaxBHL

1. For obvious reasons, test images were not included in the training set
2. Yes I also save two log files
3. When evaluating on NYUv2, which image/GT are you using? Are the images in .png format or .jpg? Is the GT defined for pixels with invalid depth? For some versions, the holes in the depth map are filled and the GT normal is computed from the filled depth map. This shouldnt be the case and the GT normal should only be defined for the pixels with valid depth.

[BayMaxBHL]

@baegwangbin
Firstly, I should apologize to you. I should not question the test data during training. I was also too anxious.
Regarding data, both testing and training data are saved in png format. Excluded invalid information from deep data.
I am sure there is no problem with the test data because I used the weight file you provided for evaluation, and the accuracy can reach 85.17 (consistent with the paper).
The main problem is that I am unable to train weights with the same accuracy through training code and default configuration, and even the difference is far (the accuracy can only reach 79).
So I really hope to obtain two log files from your training at that time. So examine what issues I have with my reproduction.
In the end, I would like to apologize to you again for the harm caused by my impulsive words before.