The official codes for PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
We propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model, and establish a scalable pipeline to construct a large-scale medical visual question-answering dataset, named PMC-VQA, which contains 227k VQA pairs of 149k images that cover various modalities or diseases.
The dataset is available at Huggingface
The model checkpoints are available at MedVInT-TE and MedVInT-TD. The previous checkpoint of MedVInT-TD was mistakenly uploaded. We have rectified the issue and updated the model's checkpoint on July 31. Now, you can access the correct and improved version of the model.
Please refer to https://github.com/chaoyi-wu/PMC-LLaMA
Download from Huggingface and save into ./PMC-VQA
Download the pre-trained MedVInT-TE, and save into ./src/MedVInT_TE/Results directly.
Download the pre-trained MedVInT-TD, and save into ./src/MedVInT_TD/Results directly.
See MedVInT_TE and MedVInT_TD for the details of training MedVInT_TE and MedVInT_TD.
CLIP -- https://github.com/openai/CLIP
PMC-CLIP -- https://github.com/WeixiongLin/PMC-CLIP
PMC-LLaMA -- https://github.com/zphang/minimal-llama
LLaMA: Open and Efficient Foundation Language Models -- https://arxiv.org/abs/2302.13971
We thank the authors for their open-sourced code and encourage users to cite their works when applicable.
Please raise an issue if you need help, any contributions are welcomed.
If you use this code or use our pre-trained weights for your research, please cite our paper
@article{zhang2023pmcvqa,
title={PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering},
author={Xiaoman Zhang and Chaoyi Wu and Ziheng Zhao and Weixiong Lin and Ya Zhang and Yanfeng Wang and Weidi Xie},
year={2023},
journal={arXiv preprint arXiv:2305.10415},
}