Question about external memory construction

[Penation]

Hello, could you provide the code for forming external memory? I would like to know how to form the query when using Q-Former to encode external data. Thanks!

[Jiaxuan-Li]

Hi, we use Q-Former to encode the images in external data, which is the same process as encoding input images in EVCap.

            query_outputs_img = self.Qformer.bert(
                query_embeds=query_tokens,
                encoder_hidden_states=image_embeds,
                encoder_attention_mask=image_atts,
                return_dict=True,
            )
            query_output_img = query_outputs_img.last_hidden_state

[Penation]

Thanks for providing the code! I have an additional request. Could you provide the original images of EVCap's External memory( 14596 (8581 + 5 × 1203) images) instead of tensors? I want to try different operations on RAG images and evaluate the quality of the final generated captioning. Sincerely look forward to your reply!

[emptykk]

哥们能交流下吗