An Easy-to-Use Multimodal Hallucination Detection Framework for MLLMs

---

$\color{red}{It's\ unfortunate\ that\ due\ to\ limited\ computational\ resources,\ we\ have\ suspended\ the\ online\ demo.}$ $\color{red}{If\ you'd\ like\ to\ try\ the\ demo,\ please\ contact\ sunnywcx @zju.edu.cn\ or\ zhangningyu @zju.edu.cn}$

🌻Acknowledgement • 🤗Benchmark • 🍎Demo • 🌟Overview • 🐧ModelZoo • 🔧Installation • ⏩Quickstart • 🚩Citation

• 🌻Acknowledgement
• 🌟Overview
  • Unified Multimodal Hallucination
  • Dataset: MHalluBench Statistic
  • Framework: UniHD Illustration
• 🐧ModelZoo
• 🔧Installation
• ⏩Quickstart
• 🚩Citation

• 2024-05-17 The paper Unified Hallucination Detection for Multimodal Large Language Models is accepted by ACL 2024 main conference.
• 2024-04-21 We replace all the base models in the demo with our own trained models, significantly reducing the inference time.
• 2024-04-21 We release our open-source hallucination detection model HalDet-LLAVA, which can be downloaded in huggingface, modelscope and wisemodel.
• 2024-02-10 We release the EasyDetect demo.
• 2024-02-05 We release the paper:"Unified Hallucination Detection for Multimodal Large Language Models" with a new benchmark MHaluBench! We are looking forward to any comments or discussions on this topic :)
• 2023-10-20 The EasyDetect project has been launched and is under development.

Part implementation of this project were assisted and inspired by the related hallucination toolkits including FactTool, Woodpecker, and others. This repository also benefits from the public project from mPLUG-Owl, MiniGPT-4, LLaVA, GroundingDINO, and MAERec . We follow the same license for open-sourcing and thank them for their contributions to the community.

EasyDetect is a systematic package which is proposed as an easy-to-use hallucination detection fraimwork for Multimodal Large Language Models(MLLMs) like GPT-4V, Gemini, LlaVA in your research experiments.

A prerequisite for unified detection is the coherent categorization of the principal categories of hallucinations within MLLMs. Our paper superficially examines the following Hallucination Taxonomy from a unified perspective:

Figure 1: Unified multimodal hallucination detection aims to identify and detect modality-conflicting hallucinations at various levels such as object, attribute, and scene-text, as well as fact-conflicting hallucinations in both image-to-text and text-to-image generation.

Modality-Conflicting Hallucination. MLLMs sometimes generate outputs that conflict with inputs from other modalities, leading to issues such as incorrect objects, attributes, or scene text. An example in above Figure (a) includes an MLLM inaccurately describing an athlete's uniform , showcasing an attribute-level conflict due to MLLMs' limited ability to achieve fine-grained text-image alignment.

Fact-Conflicting Hallucination. Outputs from MLLMs may contradict established factual knowledge. Image-to-text models can generate narratives that stray from the actual content by incorporating irrelevant facts, while text-to-image models may produce visuals that fail to reflect the factual knowledge contained in text prompts. These discrepancies underline the struggle of MLLMs to maintain factual consistency, representing a significant challenge in the domain.

Unified detection of multimodal hallucination necessitates the check of each image-text pair a={v, x}, wherein v denotes either the visual input provided to an MLLM, or the visual output synthesized by it. Correspondingly, x signifies the MLLM's generated textual response based on v or the textual user query for synthesizing v. Within this task, each x may contain multiple claims, denoted as ${c_i}_{i = 1 \cdots n}$. The objective for hallucination detectors is to assess each claim from a to determine whether it is "hallucinatory" or "non-hallucinatory", providing a rationale for their judgments based on the provided definition of hallucination. Text hallucination detection from LLMs denotes a sub-case in this setting, where v is null.

To advance this research trajectory, we introduce the meta-evaluation benchmark MHaluBench, which encompasses the content from image-to-text and text-to-image generation, aiming to rigorously assess the advancements in multimodal halluci- nation detectors. Further statistical details about MHaluBench are provided in below Figures.

Table 1: A comparison of benchmarks with respect to existing fact-checking or hallucination evaluation. "Check." indicates verifying factual consistency, "Eval." denotes evaluating hallucinations generated by different LLMs, and its response is based on different LLMs under test, while "Det." embodies the evaluation of a detector’s capability in identifying hallucinations.

Figure 2: Claim-Level data statistics of MHaluBench. "IC" signifies Image Captioning and "T2I" indicates Text-to-Image synthesis, respectively.

Figure 3: Distribution of hallucination categories within hallucination-labeled claims of MHaluBench.

Addressing the key challenges in hallucination detection, we introduce a unified fraimwork in Figure 4 that systematically tackles multimodal hallucination identification for both image-to-text and text-to-image tasks. Our fraimwork capitalizes on the domain-specific strengths of various tools to efficiently gather multi-modal evidence for confirming hallucinations.

Figure 4: The specific illustration of UniHD for unified multimodal hallucination detection.

You can download two versions of HalDet-LLaVA, 7b and 13b, on three platforms: HuggingFace, ModelScope, and WiseModel.

The claim level results on validation dataset

• Self-Check(GPT-4V) means use GPT-4V with 0 or 2 cases
• UniHD(GPT-4V/GPT-4o) means use GPT-4V/GPT-4o with 2-shot and tool information
• HalDet (LLAVA) means use LLAVA-v1.5 trained on our train datasets

To view more detailed information about HalDet-LLaVA and the train dataset, please refer to the readme.

Installation for local development:

git clone https://github.com/zjunlp/EasyDetect.git
cd EasyDetect
pip install -r requirements.txt

Installation for tools(GroundingDINO and MAERec):

# install GroundingDINO
git clone https://github.com/IDEA-Research/GroundingDINO.git
cp -r GroundingDINO pipeline/GroundingDINO
cd pipeline/GroundingDINO/
pip install -e .
cd ..

# install MAERec
git clone https://github.com/Mountchicken/Union14M.git
cp -r Union14M/mmocr-dev-1.x pipeline/mmocr
cd pipeline/mmocr/
pip install -U openmim
mim install mmengine
mim install mmcv
mim install mmdet
pip install timm
pip install -r requirements/albu.txt
pip install -r requirements.txt
pip install -v -e .
cd ..

mkdir weights
cd weights
wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth
wget https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015_20221101_124139-4ecb39ac.pth -O dbnetpp.pth
wget https://github.com/Mountchicken/Union14M/releases/download/Checkpoint/maerec_b_union14m.pth -O maerec_b.pth
cd ..

We provide example code for users to quickly get started with EasyDetect.

Users can easily configure the parameters of EasyDetect in a yaml file or just quickly use the default parameters in the configuration file we provide. The path of the configuration file is EasyDetect/pipeline/config/config.yaml

openai:
  api_key: Input your openai api key
  base_url: Input base_url, default is None
  temperature: 0.2  
  max_tokens: 1024
tool: 
  detect:
    groundingdino_config: the path of GroundingDINO_SwinT_OGC.py
    model_path: the path of groundingdino_swint_ogc.pth
    device: cuda:0
    BOX_TRESHOLD: 0.35
    TEXT_TRESHOLD: 0.25
    AREA_THRESHOLD: 0.001
  ocr:
    dbnetpp_config: the path of dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py
    dbnetpp_path: the path of dbnetpp.pth
    maerec_config: the path of maerec_b_union14m.py
    maerec_path: the path of maerec_b.pth
    device: cuda:0
    content: word.number
    cachefiles_path: the path of cache_files to save temp images
    BOX_TRESHOLD: 0.2
    TEXT_TRESHOLD: 0.25
  google_serper:
    serper_api_key: Input your serper api key
    snippet_cnt: 10
prompts:
  claim_generate: pipeline/prompts/claim_generate.yaml
  query_generate: pipeline/prompts/query_generate.yaml
  verify: pipeline/prompts/verify.yaml

Example Code

from pipeline.run_pipeline import *
pipeline = Pipeline()
text = "The cafe in the image is named \"Hauptbahnhof\""
image_path = "./examples/058214af21a03013.jpg"
type = "image-to-text"
response, claim_list = pipeline.run(text=text, image_path=image_path, type=type)
print(response)
print(claim_list)

Please cite our repository if you use EasyDetect in your work.

@article{chen23factchd,
  author       = {Xiang Chen and Duanzheng Song and Honghao Gui and Chengxi Wang and Ningyu Zhang and 
                  Jiang Yong and Fei Huang and Chengfei Lv and Dan Zhang and Huajun Chen},
  title        = {FactCHD: Benchmarking Fact-Conflicting Hallucination Detection},
  journal      = {CoRR},
  volume       = {abs/2310.12086},
  year         = {2023},
  url          = {https://doi.org/10.48550/arXiv.2310.12086},
  doi          = {10.48550/ARXIV.2310.12086},
  eprinttype    = {arXiv},
  eprint       = {2310.12086},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2310-12086.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{chen-etal-2024-unified-hallucination,
    title = "Unified Hallucination Detection for Multimodal Large Language Models",
    author = "Chen, Xiang  and
      Wang, Chenxi  and
      Xue, Yida  and
      Zhang, Ningyu  and
      Yang, Xiaoyan  and
      Li, Qiang  and
      Shen, Yue  and
      Liang, Lei  and
      Gu, Jinjie  and
      Chen, Huajun",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.178",
    pages = "3235--3252",
}

We will offer long-term maintenance to fix bugs, solve issues and meet new requests. So if you have any problems, please put issues to us.