How to optimize EfficientAd trainning？

[watertianyi]

I used EfficientAd for 10 epochs and single category training to get the following results:

Epoch 9/9  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43944/43944 0:23:59 ? 0:00:00 30.44it/s train_st_step: 0.530 train_ae_step: 0.337 train_stae_step: 0.042 train_loss_step: 0.909            
                                                                                            image_AUROC: 0.906 image_F1Score: 0.709 train_st_epoch: 0.553 train_ae_epoch: 0.428                
Epoch 9/9  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43944/43944 0:23:59 ? 0:00:00 30.44it/s train_st_step: 0.530 train_ae_step: 0.337 train_stae_step: 0.042 train_loss_step: 0.909            
                                                                                            image_AUROC: 0.956 image_F1Score: 0.699 train_st_epoch: 0.541 train_ae_epoch: 0.425                
                                                                                            train_stae_epoch: 0.056 train_loss_epoch: 1.022                                                    
F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │    0.6983320713043213     │
│       image_F1Score       │    0.6690346598625183     │
└───────────────────────────┴───────────────────────────┘

predictions = engine.predict(
        datamodule=datamodule,
        model=model,
        ckpt_path="latest/weights/lightning/model.ckpt",
    )

When predicting data, do I have to go through datamodule = Folder()? Is there a way to test the image directly?
The ID of the GPU is specified, but GPU No. 0 is still used in the actual process.
engine = Engine(max_epochs=10,task="classification",accelerator='gpu',devices=[2,3],logger=[logger_tensorboard,logger_comet])

[samet-akcay]

To specify the GPU ID, I have created this PR #2256. Once it is merged, you could do it. Currently Engine always assign the default GPU.

Do you want to test a single image or the entire test set?

[watertianyi]

@samet-akcay test a single image and the entire test set

[samet-akcay]

• Test the Test Set
  engine.train or engine.test already performs predictions on the test set and saves the results to results directory.

• Test a Single Image
  You could use Engine.predict with PredictDataset

For example:

from anomalib.data import PredictDataset

dataset = PredictDataset(path="/path/yo/your/image/or/folder", transform=<your-transforms>)
engine.predict(model=model, dataset=dataset, ckpt_path="path/to/your/ckpt")

[watertianyi]

@samet-akcay
torch.set_float32_matmul_precision('high')
datamodule = Folder(
name="OK_NG_merge",
root="/data/merge",
normal_dir="OK_merge",
abnormal_dir="NG_merge",
task="classification",
train_batch_size=1,
)
model = EfficientAd()
engine = Engine(task="classification",accelerator='gpu',devices=[2,3])
print("validate***********")
engine.validate(model=model,datamodule=datamodule,ckpt_path="EfficientAd/OK_NG_merge/latest/weights/lightning/model.ckpt")
print("test***********")
engine.test(model, datamodule=datamodule,ckpt_path="EfficientAd/OK_NG_merge/latest/weights/lightning/model.ckpt")
When I run the test, I will get the following error. What’s going on? My code is as above.
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for EfficientAd:
While copying the parameter named "model.quantiles.qa_ae", whose dimensions in the model are torch.Size([]) and whose dimensions in the checkpoint are torch.Size([]), an exception occurred : ('Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.See pytorch/rfcs#17 for more details.',).
While copying the parameter named "model.quantiles.qa_st", whose dimensions in the model are torch.Size([]) and whose dimensions in the checkpoint are torch.Size([]), an exception occurred : ('Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.See pytorch/rfcs#17 for more details.',).
While copying the parameter named "model.quantiles.qb_ae", whose dimensions in the model are torch.Size([]) and whose dimensions in the checkpoint are torch.Size([]), an exception occurred : ('Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.See pytorch/rfcs#17 for more details.',).
While copying the parameter named "model.quantiles.qb_st", whose dimensions in the model are torch.Size([]) and whose dimensions in the checkpoint are torch.Size([]), an exception occurred : ('Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update.See pytorch/rfcs#17 for more details.',).

[watertianyi]

@samet-akcay

How are the test set and verification set distributed, and how to check it. The following results show that there is a big gap between the test set and verification set indicators. How to evaluate this effect?
validate***********
Calculate Validation Dataset Quantiles: 100%|██████████| 209/209 [03:03<00:00, 1.14it/s]
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Validate metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ image_AUROC │ 0.9939691424369812 │
│ image_F1Score │ 0.9066702723503113 │
└───────────────────────────┴───────────────────────────┘
Validation ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209/209 0:00:12 • 0:00:00 16.09it/s
F1Score class exists for backwards compatibility. It will be removed in v1.1. Please use BinaryF1Score from torchmetrics instead
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
test***********
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test metric ┃ DataLoader 0 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ image_AUROC │ 0.8142072558403015 │
│ image_F1Score │ 0.7307995557785034 │
└───────────────────────────┴───────────────────────────┘