|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
library_name: transformers |
|
tags: |
|
- misinformation |
|
- fake news |
|
- vlm |
|
- mllm |
|
- llm |
|
--- |
|
# Model Card |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
SNIFFER is a multimodal large language model specifically engineered for Out-Of-Context misinformation detection and explanation. |
|
It employs two-stage instruction tuning on [InstructBLIP](https://huggingface.co/Salesforce/instructblip-vicuna-13b), including news-domain alignment and task-specific tuning. |
|
|
|
The whole model is composed of three parts: 1) _internal checking_ that analyzes the consistency of the image and text content; 2) _external checking_ that analyzes the relevance between the context of the retrieved image and the provided text, and 3) _composed reasoning_ that combines the two-pronged analysis to arrive at a final judgment and explanation. |
|
|
|
Here the checkpoint is used for the _internal checking_ part. |
|
|
|
|
|
## Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
- **Paper:** https://arxiv.org/abs/2403.03170 (to be appear in CVPR 2024) |
|
- **Project:** https://pengqi.site/Sniffer/ |
|
- **Repository:** https://github.com/MischaQI/Sniffer |
|
|
|
|
|
## Results |
|
|
|
Dataset: [NewsCLIPpings](https://github.com/g-luo/news_clippings) |
|
|
|
<div align="center"> |
|
</div> |
|
|
|
| Model | All | Fake | Real | |
|
| :-------------------- | :----| :----| :----| |
|
| SAFE | 52.8 | 54.8 | 52.0 | |
|
| EANN | 58.1 | 61.8 | 56.2 | |
|
| VisualBERT | 58.6 | 38.9 | 78.4 | |
|
| CLIP | 66.0 | 64.3 | 67.7| |
|
| DT-Transformer | 77.1 | 78.6 | 75.6 | |
|
| CCN | 84.7 | 84.8 | 84.5 | |
|
| Neu-Sym detector | 68.2 | - | - | |
|
| **SNIFFER (ours)** | **88.4** | **86.9** | **91.8** | |
|
|
|
|
|
## Citation |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
``` |
|
@inproceedings{qi2023sniffer, |
|
author = {Qi, Peng and Yan, Zehong and Hsu, Wynne and Lee, Mong Li}, |
|
title = {SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection}, |
|
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, |
|
year = {2024} |
|
} |
|
``` |
|
|
