Understanding Multi-Granularity
for Open-Vocabulary Part Segmentation

Jiho Choi^{1 *}, Seonho Lee^{1 *}, Seungho Lee², Minhyun Lee², Hyunjung Shim¹
(* indicates equal contributions)
¹Graduate School of Artificial Intelligence, KAIST, Republic of Korea
²School of Integrated Technology, Yonsei University, Republic of Korea
{jihochoi, glanceyes, kateshim}@kaist.ac.kr, {seungholee, lmh315}@yonsei.ac.kr

---

PartCLIPSeg

PartCLIPSeg is a framework designed to enhance open-vocabulary part segmentation by integrating generalized parts with object-level contexts and attention controls, improving the model's ability to generalize and segment fine-grained parts accurately.

Updates

• [2024.10.10] 👨‍💻 The official codes have been released!
• [2024.09.26] 🎉 Our paper has been accepted to NeurIPS 2024!
• [2024.07.17] 📄 Our paper is now available! You can find the paper here.

Installation

# ------------------
#     Init conda
# ------------------
conda create --name partclipseg python=3.8 -y
conda activate partclipseg
pip install --upgrade pip
conda install cuda=12.4.1 -c nvidia -y
pip install torch==2.2.2 torchvision==0.17.2 --index-url https://download.pytorch.org/whl/cu121
pip install timm==0.9.1
pip install scikit-image==0.21.0
pip install scikit-learn==0.24.2
pip install opencv-python==4.5.5.64
pip install hydra-core==1.3.2
pip install openmim==0.3.6
pip install mmsegmentation==0.29.1
pip install tokenizers==0.11.1
pip install Pillow~=9.5
pip install numpy==1.23.0
pip install einops ftfy regex fire ninja psutil gdown

# --------------------------
#     Install Detectron2
# --------------------------
pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -c "import detectron2; print(detectron2.__version__)"  # 0.6

# --------------------------
#     Install mmcv
# --------------------------
# pip install mmcv-full==1.7.1
# => if an error occurs
pip install mmcv-full==1.7.1 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.html
python -c "import mmcv; print(mmcv.__version__)"  # 1.7.1

Prepare Datasets

cd datasets

PascalPart116

• You can find further information in the OV-PARTS GitHub repository.

gdown https://drive.google.com/uc?id=1QF0BglrcC0teKqx15vP8qJNakGgCWaEH
tar -xzf PascalPart116.tar.gz
find datasets/PascalPart116/images/val/ -name '._*' -delete
find datasets/PascalPart116/ -name '._*' -delete

ADE20KPart234

gdown https://drive.google.com/uc?id=1EBVPW_tqzBOQ_DC6yLcouyxR7WrctRKi
tar -xzf ADE20KPart234.tar.gz

PartImageNet

• Download the LOC_synset_mapping.txt file from this link and place it in the datasets folder.
• Download PartImageNet_Seg from PartImageNet and extract it into the datasets folder.

Preprocess Datasets

• PascalPart116
• ADE20KPart234
• PartImageNet

# PascalPart116
python baselines/data/datasets/mask_cls_collect.py \
    datasets/PascalPart116/annotations_detectron2_part/val \
    datasets/PascalPart116/annotations_detectron2_part/val_part_label_count.json

python baselines/data/datasets/mask_cls_collect.py \
    datasets/PascalPart116/annotations_detectron2_obj/val \
    datasets/PascalPart116/annotations_detectron2_part/val_obj_label_count.json

# ADE20KPart234
# (no preprocessing required)

# PartImageNet
cd datasets
python partimagenet_preprocess.py --data_dir PartImageNet
# Make sure to have LOC_synset_mapping.txt in the datasets folder mentioned above.

Pre-trained Weights

Model	Setting	Dataset	Checkpoint
PartCLIPSeg	zero-shot	Pascal-Part-116	model
PartCLIPSeg	zero-shot	ADE20K-Part-234	model

mkdir weights && cd weights

# Pascal-Part-116
# PartCLIPSeg + Zero-shot (partclipseg_voc_oracle.pth (829M))
gdown https://drive.google.com/uc?id=1ytZpfutk__67GLwB-4jMClehJlnAzYFS

# ADE20K-Part-234
# PartCLIPSeg + Zero-shot (partclipseg_voc_oracle.pth (829M))
gdown https://drive.google.com/uc?id=10UTxrlamYq2rq3Uuv1GdOxpLy8f-QXoz

Usage (Run)

Zero-Shot Prediction

# -------------
#     Train
# -------------
python train_net.py \
    --num-gpus 8 \
    --config-file configs/zero_shot/partclipseg_voc_oracle.yaml

# -----------------
#     Inference
# -----------------
python train_net.py \
    --num-gpus 8 \
    --config-file configs/zero_shot/partclipseg_voc_oracle.yaml \
    --eval-only MODEL.WEIGHTS ./weights/partclipseg_voc_oracle.pth

Acknowledgement

We would like to express our gratitude to the open-source projects and their contributors, including OV-PARTS, CLIPSeg, Mask2Former, CLIP, and OV-DETR.