|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# PVIT model |
|
|
|
This is the model weights of paper: [Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models](https://arxiv.org/abs/2308.13437). |
|
|
|
## Model description |
|
|
|
Position-enhanced Visual Instruction Tuning (PVIT) extends the MLLM by incorporating an additional region-level vision encoder to facilitate support for region-based inputs. Specifically, we adopt the vision encoder from RegionCLIP and utilize it to extract region-level features by taking images and regions as inputs. As an additional source of information, the incorporation of region-level features in this way has a minimal impact on the original MLLM. Furthermore, since the features provided by RegionCLIP are themselves already aligned to the language at a fine-grained level, the overhead of aligning it to the MLLM will be relatively small. Following [LLaVA](https://github.com/haotian-liu/LLaVA), we design a two-stage training strategy for PVIT that first pre-training a linear projection to align the region features to the LLM word embedding, followed by end-to-end fine-tuning to follow complex fine-grained instructions. |
|
|
|
For more details, please refer to our [paper](https://arxiv.org/abs/2308.13437) and [github repo](https://github.com/THUNLP-MT/PVIT). |
|
|
|
## How to use |
|
|
|
Users have to apply it on top of the original LLaMA weights to get actual PVIT weights. See [here](https://github.com/THUNLP-MT/PVIT#pvit-weights) for instructions. |
|
|
|
## Intended use |
|
|
|
Primary intended uses: The primary use of PVIT is research on large multimodal models and chatbots. |
|
|
|
Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence. |
|
|
|
## BibTeX entry and citation info |
|
|
|
```bibtex |
|
@misc{chen2023positionenhanced, |
|
title={Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models}, |
|
author={Chi Chen and Ruoyu Qin and Fuwen Luo and Xiaoyue Mi and Peng Li and Maosong Sun and Yang Liu}, |
|
year={2023}, |
|
eprint={2308.13437}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CV} |
|
} |
|
``` |
|
|
