Update README.md

README.md

@@ -5,4 +5,84 @@ language:
 base_model:
 - meta-llama/Llama-3.2-11B-Vision-Instruct
 pipeline_tag: visual-question-answering
----
+---
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
+
+## Model Details
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **License:** apache-2.0
+- **Finetuned from model:** meta-llama/Llama-3.2-11B-Vision-Instruct
+
+## Reproduction
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+To reproduce our results, you should use [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) and the following settings.
+
+| Parameter         | Value   |
+|-------------------|---------|
+| do_sample         | True    |
+| temperature       | 0.6     |
+| top_p             | 0.9     |
+| max_new_tokens    | 2048    |
+
+You may change them in [this file](https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/vlm/llama_vision.py), line 80-83, and modify the max_new_tokens throughout the file.
+
+Note: We follow the same settings as Llama-3.2-11B-Vision-Instruct, except that we extend the max_new_tokens to 2048.
+
+After you get the results, you should filter the model output and only **keep the outputs between \<CONCLUSION\> and \</CONCLUSION\>**.
+
+This shouldn't have any difference in theory, but empirically we observe some performance difference because the jugder GPT-4o can be inaccurate sometimes.
+
+By keeping the outputs between \<CONCLUSION\> and \</CONCLUSION\>, most answers can be direclty extracted using VLMEvalKit system, which can be much less biased.
+
+## How to Get Started with the Model
+
+You can use the inference code for Llama-3.2-11B-Vision-Instruct.
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+The model is trained on the LLaVA-o1-100k dataset (to be released).
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+The model is finetuned on [llama-recipes](https://github.com/Meta-Llama/llama-recipes) with the following settings.
+Using the same setting should accurately reproduce our results.
+
+| Parameter                     | Value                                             |
+|-------------------------------|---------------------------------------------------|
+| FSDP                          | enabled                                           |
+| lr                            | 1e-5                                              |
+| num_epochs                    | 3                                                 |
+| batch_size_training           | 4                                                 |
+| use_fast_kernels              | True                                              |
+| run_validation                | False                                             |
+| batching_strategy             | padding                                           |
+| context_length                | 4096                                              |
+| gradient_accumulation_steps   | 1                                                 |
+| gradient_clipping             | False                                             |
+| gradient_clipping_threshold   | 1.0                                               |
+| weight_decay                  | 0.0                                               |
+| gamma                         | 0.85                                              |
+| seed                          | 42                                                |
+| use_fp16                      | False                                             |
+| mixed_precision               | True                                              |
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+The model may generate biased or offensive content, similar to other VLMs, due to limitations in the training data. 
+Technically, the model's performance in aspects like instruction following still falls short of leading industry models.