Testing accuracy low even after successful training

[Karlinik]

Hi all,

first of all, great repo, good job!

I have an issue with training/testing and I would like to ask you for an advice.

I am trying to train the model for custom data. Everything is running I don't have any issues it that area but results on the test set are weird. During training it looks like the model is detecting all the images correctly, loss and lr looks great but when I want to test the model the result are completely off.
I downloded the Tensorflow pretrain weights as described in the instructions, using r3det detector

Those are pictures from the training:

And here the test images:

I printed out the used checkpoint so I know it is loading the trained weight
latest checkpoint: /home/jovyan/work/RotationDetection_v2/output/trained_weights/RetinaNet_OPENPNP_parts_v7_R3Det_2x_20211129/OPENPNP_330767model.ckpt
After reading other discussions I tried to load other checkpoints but the results look pretty much the same.

I also tried to test it against the traing data and still the same issue which is suspicious. Every image contains one and only one object but the final detection 'detects' a lot of them that also makes me wonder.

Do you have an idea what might be the problem? I was first thinking it is overfitting, my dataset is pretty small but it is not working even on the training set so maybe some structural issue? Do you think that data augmentation or even different base model would help?

Many Thanks, Nikola :)