Accepted by ISBI 2025

CLISC: Bridging CLIP and SAM by Enhanced CAM for Unsupervised Brain Tumor Segmentation

Structure

In this paper, we propose a novel unsupervised brain tumor segmentation method by adapting foundation models (CLIP and SAM) rather than directly using them for inference. The contribution is three-fold. First, we propose a framework named CLISC that bridges CLIP and SAM by enhanced Class Activation Maps (CAM) for unsupervised brain tumor segmentation, where image-level labels obtained by CLIP is used to train a classification model that obtains CAM, and the CAM is used to generate prompts for SAM to obtain pseudo segmentation labels. Second, to obtain high-quality prompts, we propose an Adaptive Masking-based Data Augmentation (AMDA) strategy for improved CAM quality. Thirdly, to reject low-quality segmentation pseudo-labels, we propose a SAM-Seg Similarity-based Filtering (S3F) strategy in a self-learning method for training a segmentation model. Evaluation on the BraTS2020 dataset shows that our method outperforms five state-of-the-art unsupervised segmentation methods by more than 10 percentage points with an average DSC of 85.60%. Besides, our approach outperforms zero-shot SAM inference, achieving performance on par with fully supervised learning.

How to use

Click here download sam vit-b checkpoint and place it at "./checkpoint"

1. Environment

conda create -n CLISC python=3.10 
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
cd CLISC
pip install -e . ./CLIP ./segment-anything/ ./pytorch-grad-cam/

2. Dataset and Preprocessing

We use the BraTS2020_TrainingData to evaluate the effectiveness of our framework. A total of 369 cases are split into training, validation, and test sets with a ratio of 7:1:2, which are shown in "data/BraTS2020/splits"

Data Download choice 1: preprocessed data (Recommeded)

click here to download the processed data and place it as follows: "./data/BraTS2020/image" and "./data/BraTS2020/label"

Data Download choice 2: raw data and run preprocessing

click here to download the raw data

and place it as:

"./data/raw_data/BraTS2020_TrainingData"

Note that you need to rename the seg file in BraTS20_Training_355 "W39_1998.09.19_Segm.nii" to "BraTS20_Training_355_seg.nii" like others to ensure that there are no problems with preprocessing. Then run data preprocess.

python ./data/BraTS2020/preprocessing.py

3. Training

cd train
bash run.sh

4. Test

python test_3D.py --stage "test"

Citation

@inproceedings{ma2025clisc,
  title={CLISC: Bridging clip and sam by enhanced cam for unsupervised brain tumor segmentation},
  author={Ma, Xiaochuan and Fu, Jia and Liao, Wenjun and Zhang, Shichuan and Wang, Guotai},
  booktitle={2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI)},
  pages={1--5},
  year={2025},
  organization={IEEE}
}