how to calculate Precision/Recall/Accuracy in v1.1.10?

[stevensagaert]

Discussed in #781

When I do
engine = Engine(max_epochs=epochs, task=TaskType.CLASSIFICATION,image_metrics=["Precision","Recall","Accuracy"])
I get the following output:
Incorrect constructor arguments for Precision metric from TorchMetrics package.
Incorrect constructor arguments for Recall metric from TorchMetrics package.
Incorrect constructor arguments for Accuracy metric from TorchMetrics package

[samet-akcay]

@stevensagaert, what torchmetrics version are you using? Also, can you provide the full error message you are seeing

[stevensagaert]

1.4.0.post0 The error message is was I gave. The code continues.

…

________________________________ From: Samet Akcay ***@***.***> Sent: Tuesday, July 23, 2024 9:03 AM To: openvinotoolkit/anomalib ***@***.***> Cc: Steven Sagaert ***@***.***>; Mention ***@***.***> Subject: Re: [openvinotoolkit/anomalib] how to calculate Precision/Recall/Accuracy in v1.1.10? (Issue #2205) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. @stevensagaert<https://github.com/stevensagaert>, what torchmetrics version are you using? Also, can you provide the full error message you are seeing — Reply to this email directly, view it on GitHub<#2205 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A7A5SMFLBWKRYIZI3YWVELDZNX52RAVCNFSM6AAAAABLI7TCAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBUGQYDIMJTGA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ashwinvaidya17]

@stevensagaert newer version of Torchmetrics require you to also pass the task parameter. One solution would be to use dict to pass the argument

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Padim

if __name__ == "__main__":
    model = Padim()
    data = MVTec()
    engine = Engine(
        image_metrics={
            "Precision": {
                "class_path": "torchmetrics.Precision",
                "init_args": {"task": "binary"},
            }
        },
    )
    engine.train(model, datamodule=data)

[mykappa]

How can I pass the task argument to the metric when using anomalib's CLI and a config.yml?

[stevensagaert]

Got it. Thanks.

…

________________________________ From: Ashwin Vaidya ***@***.***> Sent: Wednesday, July 24, 2024 2:02 PM To: openvinotoolkit/anomalib ***@***.***> Cc: Steven Sagaert ***@***.***>; Mention ***@***.***> Subject: Re: [openvinotoolkit/anomalib] how to calculate Precision/Recall/Accuracy in v1.1.10? (Issue #2205) CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. @stevensagaert<https://github.com/stevensagaert> newer version of Torchmetrics require you to also pass the task parameter. One solution would be to use dict to pass the argument from anomalib.data import MVTec from anomalib.engine import Engine from anomalib.models import Padim if __name__ == "__main__": model = Padim() data = MVTec() engine = Engine( image_metrics={ "Precision": { "class_path": "torchmetrics.Precision", "init_args": {"task": "binary"}, } }, ) engine.train(model, datamodule=data) afbeelding.png (view on web)<https://github.com/user-attachments/assets/aacc4a02-80fc-48d1-9896-2960ee0e54a1> — Reply to this email directly, view it on GitHub<#2205 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A7A5SMCLKR7LAXHQWUB7G2TZN6JVNAVCNFSM6AAAAABLI7TCAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBXG4ZTKMJQGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ashwinvaidya17]

@stevensagaert once this PR #2212 is merged you can directly pass the metric objects to the Engine

[mohandeeprdori]

@ashwinvaidya17 Hi,
How do i get to print these metrics when I'm doing engine.predict(). Below is my code. When i ran this code. It did not print anything.

from torchmetrics.classification import Accuracy, Precision, Recall

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Fastflow

if __name__ == "__main__":
    model = Fastflow()
    datamodule = MVTec()
    ckpt_path = './anomalib/results/Fastflow/MVTec/bottle/v1/weights/lightning/model.ckpt'
    # engine = Engine(image_metrics=[Accuracy(task="binary"), Precision(task="binary"), Recall(task="binary")])
    engine = Engine(
        image_metrics={
            "Precision": {
                "class_path": "torchmetrics.Precision",
                "init_args": {"task": "binary"},
            },
            "Recall": {
                "class_path": "torchmetrics.Recall",
                "init_args": {"task": "binary"},
            },
            "Accuracy": {
                "class_path": "torchmetrics.Accuracy",
                "init_args": {"task": "binary"},
            }
        },
    )
    engine.predict(datamodule=datamodule, model=model, ckpt_path=ckpt_path,)
    # print("Predictions:",predictions)

print(predictions) gave me tensors and masks which i dont require at this point.

[ashwinvaidya17]

Predict is used just to do inference. So, that's why you get tensors and mask. If you want metrics, you need to call Engine.test. Keep in mind, the test_dataloader requires ground truth masks as well.

[mohandeeprdori]

Thanks @ashwinvaidya17 it was helpful. I was able to get the metrics as needed. Is it possible to get metrics for each image or each class?