Being able to pass multiple images as multichannel input into the model in Anomalib

[garima-mahato]

I have multiple images of the same defect in different lighting conditions. How can I merge them as 1 image having multiple channel in anomalib?

[alexriedel1]

Why would you do this? You can pass multiple image one after another to the anomalib models via a datamodule.

One (RGB) image already has 3 channels. If you are working with grayscale image, you could try to concatenate 3 single-channel images to one 3-channel image, but I'm not sure if that helps you with anomaly detection accuracy (would be interesting to know though!)

[garima-mahato]

I have 4 RGB images representing the same defect in different lighting conditions. Each lighting condition highlights a different aspect of the defect. In total 12 channels. They have to be passed together because they represent the same example.

In timm models (which is being used in anomalib), we can pass in_channels. In pytorch dataset module, we can customize the channels. How can we do the same in anomalib?

[alexriedel1]

Ahh ok so the timm model loading function is already designed in a way to support more than three channels even with pre-trained models: https://timm.fast.ai/models#How-is-timm-able-to-use-pretrained-weights-and-handle-images-that-are-not-3-channel-RGB-images?

There is no interface to this functionality in anomalib built-in at the moment but you can do it yourself by modifying how the timm models are created in anomalib

anomalib/src/anomalib/models/components/feature_extractors/timm.py

Line 69 in 4b1e08c

self.feature_extractor = timm.create_model(

give it a try and tell us how this changed the anomaly detection performance!

[bujoi]

If camera and object in fixed position, you can use stereo photometric to generate one image form multiple light.

[watertianyi]

@alexriedel1
Both create_dataset and create_loader are for 3-channel or single-channel normal images. If I input four-channel data, how should I modify the parameters? Because my four-channel data is spliced ​​from four single-channel 2D images.

[watertianyi]

这两个 timms 是什么意思？ 我会这样做： 为 n 维图像编写 da dataloader。如果您使用文件夹datamodule，我认为不需要进行太多更改。然后按照以下指南修改模型以接受 4 维输入：https ://timm.fast.ai/models#How-is-timm-able-to-use-pretrained-weights-and-handle-images-that-are-not-3-channel-RGB-images

These two are the functions used by timm to load data, similar to the dataloader function. If I use the dataloader function, where should I modify the code?

[watertianyi]

@alexriedel1 I want to convert the image into a single channel first, and then merge multiple single channels. Can I modify read_image in anomalib/src/anomalib/data/utils/image.py?

[alexriedel1]

You have to modify the dataloader itself https://github.com/openvinotoolkit/anomalib/blob/main/src/anomalib/data/image/folder.py
The timm model isn't coupled with the dataloader.

[alexriedel1]

@alexriedel1 I want to convert the image into a single channel first, and then merge multiple single channels. Can I modify read_image in anomalib/src/anomalib/data/utils/image.py?

yes you can modify it

[watertianyi]

@alexriedel1
我使用四通道，修改了
anomalib/src/anomalib/data/utils/image.py中read_image中的代码为
`def transform_channel(img):
h, w = img.shape
print("img",img.shape)
mid_width = int((w - 5) / 2)
mid_height = int((h - 5) / 2)
img = Image.fromarray(img)
img_light1 = np.array(img.crop((0, 0, mid_width, mid_height)))
img_light2 = np.array(img.crop((mid_width + 5, 0, img.width, mid_height)))
img_light3 = np.array(img.crop((0, mid_height + 5, mid_width, img.height)))
img_light4 = np.array(img.crop((mid_width + 5, mid_height + 5, img.width, img.height)))

new_img = np.concatenate([np.expand_dims(img_light4, axis=2), np.expand_dims(img_light3, axis=2),
                                np.expand_dims(img_light2, axis=2), np.expand_dims(img_light1, axis=2)], axis=2) 
    
return new_img`

`def read_image(path: str | Path, as_tensor: bool = False) -> torch.Tensor | np.ndarray:
"""Read image from disk in RGB format.

Args:
    path (str, Path): path to the image file
    as_tensor (bool, optional): If True, returns the image as a tensor. Defaults to False.

Example:
    >>> image = read_image("test_image.jpg")
    >>> type(image)
    <class 'numpy.ndarray'>
    >>>
    >>> image = read_image("test_image.jpg", as_tensor=True)
    >>> type(image)
    <class 'torch.Tensor'>

Returns:
    image as numpy array
"""
#TODO
# image = Image.open(path).convert("RGB")
# return to_dtype(to_image(image), torch.float32, scale=True) if as_tensor else np.array(image) / 255.0

 if as_tensor:
    img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    img_transform = transform_channel(img)
    resized_image = cv2.resize(img_transform, (224, 224))
    input = torch.from_numpy(resized_image).permute(2, 0, 1)
    input_image = to_dtype(input, torch.float32, scale=True)

else:
   print("test********************************")
   img = cv2.imread(path)
   input_image=img / 255.0
return input_image`

/home/mnt/anomalib/src/anomalib/models/components/feature_extractors/timm.py
self.feature_extractor = timm.create_model( backbone, pretrained=pre_trained, pretrained_cfg=pretrained_cfg, features_only=True, exportable=True, out_indices=self.idx, in_chans = 4, )
The code can run through, but there is a problem, that is, the image data is four-channel during training and verification, why is it three-channel during testing? How should it be done if inferring an image?