README.md

---
base_model: mistral-community/pixtral-12b
library_name: peft
license: cc-by-4.0
datasets:
- daniel3303/GroundCap
language:
- en
metrics:
- bleu
- meteor
- cider
- spice
- f1
- recall
- precision
- gmeteor
- rouge

model-index:
  - name: PixtralGroundCap
    results:
      - task:
          type: image-captioning
          subtype: grounded-image-captioning
        dataset:
          name: daniel3303/GroundCap
          type: grounded-image-captioning
          split: test
        metrics:
          - name: Precision
            type: grounding-precision
            value: 0.58
          - name: Recall
            type: grounding-recall
            value: 0.96
          - name: F1
            type: grounding-f1
            value: 0.69
          - name: BLEU-4
            type: bleu-4
            value: 0.19
          - name: METEOR
            type: meteor
            value: 0.23
          - name: CIDEr
            type: cider
            value: 0.51
          - name: SPICE
            type: spice
            value: 0.30
          - name: gMETEOR
            type: gmeteor
            value: 0.35
       

---

# Model Card for PixtralGroundCap

This model is a fine-tuned version of Pixtral-12B on the GroundCap dataset for grounded image captioning. It generates detailed image descriptions with explicit grounding tags that link textual descriptions to specific visual elements in the image. The model was trained on the GroundCap dataset and uses a novel tag system to ground objects (`<gdo>`), actions (`<gda>`), and locations (`<gdl>`) to specific regions in images.

## Model Details

### Model Description

- **Developed by:** Daniel A. P. Oliveira, Lourenço Teodoro, and David Martins de Matos (INESC-ID Lisboa and Instituto Superior Técnico, Universidade de Lisboa)
- **Model type:** Fine-tuned Pixtral-12B model for grounded image captioning
- **Language(s):** English
- **License:** Creative Commons Attribution 4.0
- **Finetuned from model:** mistral-community/pixtral-12b




### Model Sources

- **Paper:** https://arxiv.org/abs/2502.13898
- **Dataset:** https://huggingface.co/datasets/daniel3303/GroundCap



## Uses

### Direct Use

The model is designed for generating grounded image captions that explicitly link textual descriptions to visual elements using three types of grounding tags:
- `<gdo>` for objects
- `<gda>` for actions
- `<gdl>` for locations

Each tag maintains object identity through unique IDs, enabling consistent reference tracking throughout the caption.

### Downstream Use

The model can be integrated into:
- Accessibility applications requiring detailed image descriptions
- Content management systems needing verifiable image captions
- Visual question answering systems
- Image retrieval systems

### Out-of-Scope Use

The model is not designed for:
- General image classification
- Object detection (requires separate object detection pipeline)
- Video captioning
- Non-English language captioning

## How to Get Started with the Model

### Input Format

The model expects input in the following format:

```
You are an AI assistant that can see and understand images. I will provide you with an image and the detected objects in it along with their positions and dimensions in the format [id, x,y,width,height].
[DETECTIONS]
[sky-0: 0.41,0.00,0.20,0.15]
[sky-1: 0.62,0.00,0.26,0.10]
[wall-0: 0.01,0.02,0.35,0.86]
[person-0: 0.38,0.35,0.12,0.40]
[person-1: 0.45,0.35,0.08,0.39]
[wall-1: 0.39,0.10,0.35,0.48]
[person-2: 0.71,0.29,0.20,0.51]
[wall-2: 0.75,0.03,0.24,0.88]
[person-3: 0.00,0.57,0.22,0.42]
[handbag-0: 0.21,0.75,0.11,0.23]
[person-4: 0.26,0.48,0.20,0.52]
[floor-wood-0: 0.40,0.59,0.60,0.41]
[/DETECTIONS]
[IMG]
```

### Example Output

The model will generate a grounded caption using three types of tags:
- ```<gdo>``` for objects
- ```<gda>``` for actions
- ```<gdl>``` for locations

Example output:

```
In this scene, a group of individuals is gathered in what appears to be a <gdl class="wall" wall-0 wall-1 wall-2>rugged, makeshift shelter</gdl>. The <gdl class="wall" wall-0 wall-1 wall-2>walls</gdl> are constructed from <gdl class="wall" wall-0 wall-1 wall-2>rough materials</gdl>, giving the space a temporary and utilitarian feel. The <gdl class="sky" sky-0 sky-1>sky</gdl> is visible in the background, suggesting that the shelter is partially open or lacks a complete roof.
Several <gdo class="person" person-0 person-1 person-2 person-3 person-4>people</gdo> are present in the scene, each engaged in their own activities. <gdo class="person" person-0>One individual</gdo> <gda class="sit" person-0>sits</gda> on the <gdl class="floor-wood" floor-wood-0>ground</gdl>, while <gdo class="person" person-1>another person</gdo> <gda class="sit" person-1>is seated</gda> nearby. <gdo class="person" person-2>Another person</gdo> <gda class="sit" person-2>is also sitting</gda> on the <gdl class="floor-wood" floor-wood-0>ground</gdl>, and <gdo class="person" person-3>a fourth individual</gdo> <gda class="sit" person-3>is seated</gda> as well. <gdo class="person" person-4>An additional person</gdo> <gda class="sit" person-4>is sitting</gda> close by.
The <gdo class="handbag" handbag-0>handbag</gdo> is placed on the <gdl class="floor-wood" floor-wood-0>ground</gdl> near one of the individuals, suggesting they might have brought some personal belongings with them. The overall atmosphere of the scene is one of simplicity and resilience, with the individuals making the best of their surroundings in this temporary shelter.
```


## Bias, Risks, and Limitations

- The model was trained on movie scenes from MovieNet, which may introduce biases in terms of scene composition, lighting, and camera angles
- Performance may vary for real-world images that differ significantly from movie scenes
- The model relies on pre-detected objects and their bounding boxes, Mask2Former was used for object detection in the original paper


### Recommendations

- Use in conjunction with a robust object detection system
- Verify grounding accuracy for critical applications
- Consider the movie-centric nature of the training data when applying to other domains

## Training Details

### Training Data

The model was trained on the GroundCap dataset, which contains:
- 52,016 images from 77 movies
- 344 human-annotated captions
- 52,016 automatically generated captions
- 
### Training Procedure

The training followed a two-stage approach:

#### Stage 1:
- Training on 52,016 automatically generated captions
- Learning rate: 2×10^-4
- Epochs: 2
- Batch size: 64 (with gradient accumulation)

#### Stage 2:
- Fine-tuning on 344 human-refined captions
- Learning rate: 2×10^-6
- Epochs: 2
- Batch size: 32 (with gradient accumulation)

#### Training Hyperparameters

- **LoRA Configuration:**
  - Rank: 16
  - Alpha: 32
  - Targeted layers: Self-attention (query, key, value, output) and MLP (gate, up, down)
- **Optimizer:** AdamW
- **Weight decay:** 0.01
- **Precision:** bfloat16
- **Hardware:** 2x NVIDIA A100 (80GB)
- **Training time:** 1 day

## Evaluation

### Testing Data, Factors & Metrics

The model was evaluated on:
- 10,000 test images from GroundCap from which 70 are human-annotated test cases

### Metrics

- **Grounding metrics:** 
  - Precision (P): Correctly grounded objects / Total objects mentioned in caption
  - Recall (R): Correctly grounded objects / Total detected objects
  - F1 score: Harmonic mean of precision and recall

- **Caption quality metrics:** 
  - BLEU-4: N-gram overlap with reference captions
  - METEOR: Semantic similarity with reference captions
  - CIDEr: Consensus-based image description evaluation
  - SPICE: Semantic propositional image caption evaluation
  - ROUGE-L: Longest common subsequence based evaluation

- **Combined metric:** 
  - gMETEOR: Harmonic mean of METEOR and grounding F1 score, combining language quality with grounding accuracy

- **Human evaluation:** (5-point Likert scale)
  - Object precision: Accuracy of object grounding and tag classification
  - Grounding recall: Coverage of detected objects in captions
  - Description accuracy: Correctness of described actions and relationships
  - Language quality: Grammar, readability, and coherence
  - Overall quality: Assessment of caption effectiveness

- **ChatGPT-4o evaluation:** (5-point Likert scale)
  - Uses same criteria as human evaluation
  - Correlations with human judgments:
    - Object Precision: 0.81 (Pearson), 0.73 (Spearman)
    - Grounding Recall: 0.76 (Pearson), 0.67 (Spearman)
    - Description Accuracy: 0.79 (Pearson), 0.77 (Spearman)
    - Language Quality: 0.59 (Pearson), 0.44 (Spearman)
    - Overall Quality: 0.78 (Pearson), 0.68 (Spearman)

### Results

Automatic metrics on test set for PixtralGroundCap:
- Precision: 0.58
- Recall: 0.96
- F1 Score: 0.69
- BLEU-4: 0.19
- METEOR: 0.23
- CIDEr: 0.51
- SPICE: 0.30
- ROUGE-L: 0.37
- gMETEOR: 0.35

Human evaluation results (scale 1-5):
- Object Precision: 4.22
- Grounding Recall: 4.19
- Description Accuracy: 4.08
- Language Quality: 4.91
- Overall Quality: 4.22

ChatGPT-4o evaluation results (scale 1-5):
- Object Precision: 4.21
- Grounding Recall: 4.13
- Description Accuracy: 4.01
- Language Quality: 4.90
- Overall Quality: 4.19


## Environmental Impact

- **Hardware Type:** 2x NVIDIA A100 GPUs
- **Hours used:** 24 hours
- **Cloud Provider:** INESC-ID
- **Compute Region:** Lisbon, Portugal

## Paper

[ArXiv link](https://arxiv.org/abs/2502.13898).

## Citation

**BibTeX:**
```bash
@article{Oliveira2025GroundCapAV,
  title={GroundCap: A Visually Grounded Image Captioning Dataset},
  author={Daniel A. P. Oliveira and Louren{ç}o Teodoro and David Martins de Matos},
  year={2025},
  url={https://api.semanticscholar.org/CorpusID:276450057}
}
```

## Model Card Authors

Daniel A. P. Oliveira, Lourenço Teodoro, and David Martins de Matos

## Model Card Contact

[email protected]


### Framework versions

- PEFT 0.13.2