Pre Processing

Zero Padding

Since the frame size of videos differ in each data-set, we add zero padding to smaller size images. The general resolution for training and testing video frames is maintained at 640 x 640 pixel resolution. This resolution comes from finding the maximum size form all frames in samples.

Variance

According to documentation of cilia in ciliary beat frequency: simply, the frequency at which cilia beat back and forth. There is a lot of evidence showing that healthy cilia tend to beat in a range around 10-12Hz (so in 100 frames of a 200fps video, you’d expect to see about 5-6 full “oscillations” of healthy cilia).

Thus we try to remove that area and what we get will be usually cut down on the pixels that are definitely not cilia, but will still give you a lot of artifacts; a loose bound on the regions of cilia.

This method takes 3d image sequence as numpy array argument. The image sequence is then converted to a 2-d matrix with pixels as rows and pixel values for images as columns. Then variance of the pixels are calculated and stored in a variable. The top variant pixels as replaced with a given pixel value.

Below is the reference code form the repository.

def highVariance(imgs,hv=1,pix=125):
    timgs = transform(imgs)`
    varimg = timgs.var(1)
    sortvar = sorted(varimg,reverse=True)
    sortvar = sortvar[:hv]
    a = np.full((1,int(imgs.shape[0])),pix)
    for mv in sortvar:
        timgs[int(np.where(varimg == mv)[0][0])] = a
return timgs

Optical Flow

optical flow algorithms try to find the movement size and angel of objects in a frame sequence. As also mentioned in the documentation, as cilia parts will have the most movement, calculating optical flow should give us better results. to exploit this feature two optical flow based models have been implemented :

• Optical Flow : is an RGB output which is composed of both magnitude and angel of flow
• Optical Flow Magnitude : is a gray scale representation of only magnitudes summed over all frames. this basically means find the areas for which most of movement has been observed.

Final Data-Sets

In order to train the network faster and efficient the pre processor selects one every "skip_count" images in the sample.

For example if the sample has 100 images, out put will have 20 for skip_count =5 The following inputs will be passed to the model for training:

Simple Frames: train_x train_y

Variance Frames: train_vars test_vars