SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference

Yuan Zhang^1,3* , Chun-Kai Fan^1*, Junpeng Ma^2*, Wenzhao Zheng^3✉️, Tao Huang⁴, Kuan Cheng¹,

Denis Gudovskiy⁵, Tomoyuki Okuno⁵, Yohei Nakata⁵, Kurt Keutzer³, Shanghang Zhang^1✉️

¹School of Computer Science, Peking University, ²Fudan University,

³UC Berkeley, ⁴The University of Sydney, ⁵Panasonic Holdings Corporation

📜 News

🔥 [2024/10/15] We released SparseVLM and its Project Page! The Code is now open-source!

✒️ Contents

• News
• Contents
• Overview
• Preparation
• Usage
• Citation
• Acknowledgment

👀 Overview

In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, existing methods extract more compact image representations by modifying the image encoder or projector. While some recent works further sparsify vision tokens during the decoding, they still ignore the guidance from the language tokens, which contradicts the multimodality paradigm. We argue that visual tokens should be sparsified adaptively based on the question prompt, as the model might focus on different parts (e.g., foreground or background) when dealing with various questions, as shown in Figure below. Unlike previous methods with text-agnostic visual sparsification (c) e.g., recent FastV, our SparseVLM (b) is guided by question prompts to select relevant visual patches.

👨‍💻 Preparation

1. Clone this repository and navigate to SparseVLMs folder

git clone https://github.com/Gumpest/SparseVLMs.git
cd SparseVLMs

2. Install necessary package

conda create -n SparseVLMs python=3.10 -y
conda activate SparseVLMs
pip install -e .

3. Download Multimodal Benchmark

Please follow the detailed instruction in LLaVA-Evaluation.

🎯 Usage

Specifically, --sparse in script indicates whether to perform sparseness, while --scale and --bias control the degree of token sparsity.

1. Example for evaluating MME results (192 tokens, scale = 13.5, bias = 0.0):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/mme.sh

2. Example for evaluating POPE results (128 tokens, scale = 9, bias = 6):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/pope.sh

3. Example for evaluating TextVQA results (64 tokens, scale = 0.8, bias = 0.0):

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/textvqa.sh

License

This project is released under the Apache 2.0 license.

Citation

If you use SparseVLM in your research, please cite our work by using the following BibTeX entry:

@article{zhang2024sparsevlm,
  title={SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference},
  author={Zhang, Yuan and Fan, Chun-Kai and Ma, Junpeng and Zheng, Wenzhao and Huang, Tao and Cheng, Kuan and Gudovskiy, Denis and Okuno, Tomoyuki and Nakata, Yohei and Keutzer, Kurt and others},
  journal={arXiv preprint arXiv:2410.04417},
  year={2024}
}

Acknowledgment

We extend our gratitude to the open-source efforts of TCFormer, LLaVA, MiniGemini and VideoLLaVA.