Replication track in NeurIPS 2019 Reproducibility Challenge of Learning where to look: Semantic-Guided Multi-Attention Localization for Zero-Shot Learning

This project is not finished yet, more information will be updated soon!

1. Multi-Attention Subnet:

1. VGG19(?) backbone
   input: Image
   output: Features representation
   
2. K-means
   input: Feature representation
   output: 2 groups of feature representation
   
3. Global averge pooling + 2 Fully connected layers(ReLU) + Sigmoid
   input: 2 groups of feature representation
   (intermediate result: Channel descriptor p1, p2)
   output: Channel-wise attention weight vector a1, a2
   
4. Weighted-sum
   input: Feature representation, channel-wise attention weight
   output: 2 Attention maps
   

2. Region Cropping Subnet:

1. f_CNet(2 fully connected layers)
   input: Attention maps
   output: [t_x, t_y, t_s]
   
2. Boxcar Mask (cropping operation x o V_i)
   input: Attention maps, [t_x, t_y, t_s]
   output: masked images (x_i^part)
   

3. Joint Feature Learning Subnet

1. VGG backbone + Global average pooling
   input: original image/ masked image
   output: visual feature vector $\theta$
   
2. Transformation:
   input: visual feature vector
   output: Semantic feature vector
   

4. Classification

different between seen and unseen classes

Reference:

Pedro Morgado and Nuno Vasconcelos. Semantically consistent regularization for zero-shot recognition. In CVPR, 2017.

Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: l 2 hypersphere embedding for face verification. In ACMMM. ACM, 2017.

Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In CVPR, pages 4438–4446, 2017.

Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. Learning multi-attention convolutional neural network for fine-grained image recognition. In ICCV, 2017.