Add pipeline tag and library name

README.md

@@ -1,20 +1,21 @@
 ---
 license: apache-2.0
+library_name: transformers
+pipeline_tag: video-text-to-text
 ---
+
 # R1-Omni: Explainable Omni-Multimodal Emotion Recognition with Reinforcement Learning
 
 [![GitHub](https://img.shields.io/badge/GitHub-R1Omni-green)](https://github.com/Jiaxing-star/R1-Omni)
 [![ModelScope](https://img.shields.io/badge/ModelScope-R1Omni-blue)](https://modelscope.cn/models/iic/R1-Omni-0.5B)
 [![arXiv](https://img.shields.io/badge/arXiv-2503.05379-red)](https://arxiv.org/abs/2503.05379)
 
-
 ## 📖 Introduction
 **R1-Omni** is the industry’s first application of Reinforcement Learning with Verifiable Reward (RLVR) to an Omni-multimodal large language model. We focus on emotion recognition, a task where both visual and audio modalities play crucial roles, to validate the potential of combining RLVR with Omni model. Our findings reveal several key insights:
 1) **Enhanced Reasoning Capability**: R1-Omni demonstrate superior reasoning abilities, enabling a clearer understanding of how visual and audio information contribute to emotion recognition.
 2) **Improved Understanding Capability**: Compared to SFT, RLVR significantly boosts performance on emotion recognition tasks.
 3) **Stronger Generalization Capability**: RLVR models exhibit markedly better generalization capabilities, particularly excelling in out-of-distribution scenarios.
 
-
 ## 🏆 Performance
 
 Below are the performance on emotion recognition datasets. We use symbols to indicate whether the data is **in-distribution (⬤)** or **out-of-distribution (△)**.
@@ -26,12 +27,10 @@ Below are the performance on emotion recognition datasets. We use symbols to ind
 | MAFW-DFEW-SFT                    | 60.23         | 44.39         | 50.44         | 30.39         | 29.33            | 30.75            |
 | R1-Omni                          | 65.83         | 56.27         | 57.68         | 40.04         | 43.00            | 44.69            |
 
-
 ### Legend
 - **⬤**: Indicates **in-distribution data** (DFEW and MAFW).
 - **△**: Indicates **out-of-distribution data** (RAVDESS).
 
-
 ## 🛠️ Environment Setup
 Our code is built on the R1-V framework. To set up the environment, please follow the installation instructions in the [R1-V repository](https://github.com/Deep-Agent/R1-V/)
 
@@ -46,7 +45,6 @@ Our inference code is based on the implementation from **HumanOmni**. To ensure
    - In the directory where you downloaded the R1-Omni model, locate the config.json file.
    - Update the paths on line 23 and line 31 to point to the local folders where you saved the models.
 
-
 #### Example: Updating config.json
 If you saved the models to the following local paths:：
 - `/path/to/local/models/siglip-base-patch16-224`
@@ -66,8 +64,6 @@ python inference.py --modal video_audio \
   --instruct "As an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you?  Output the thinking process in <think> </think> and final emotion in <answer> </answer> tags."
 ```
 
-
-
 ## 🧠 Training
 ### Cold Start
 we initialize the HumanOmni-0.5B by fine-tuning it on a combined dataset consisting of 232 samples from the [Explainable Multimodal Emotion Reasoning](https://github.com/zeroQiaoba/AffectGPT) dataset and 348 samples from HumanOmni dataset.
@@ -79,11 +75,14 @@ An example json file of the training data:
         "conversations": [
             {
                 "from": "human",
-                "value": "<video>\n<audio>\nAs an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you? Output the thinking process in <think> </think> and final emotion in <answer> </answer> tags."
+                "value": "<video>
+<audio>
+As an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you? Output the thinking process in <think> </think> and final emotion in <answer> </answer> tags."
             },
             {
                 "from": "gpt",
-                "value": "<think>The video depicts a bright and tranquil indoor setting, where a man in a white Polo shirt stands by the window, engaged in a phone call. His furrowed brow and open mouth suggest he is experiencing tension and anxiety. According to the audio content of the video, his speech is fast-paced, and his tone is filled with confusion and stress. A comprehensive analysis reveals that the man is facing a moderate level of anxiety, closely linked to the challenging phone conversation he is having. Consequently, the entire emotional analysis report emphasizes his anxiety and nervousness in handling challenging situations.</think>\n<answer>anxious</answer>"
+                "value": "<think>The video depicts a bright and tranquil indoor setting, where a man in a white Polo shirt stands by the window, engaged in a phone call. His furrowed brow and open mouth suggest he is experiencing tension and anxiety. According to the audio content of the video, his speech is fast-paced, and his tone is filled with confusion and stress. A comprehensive analysis reveals that the man is facing a moderate level of anxiety, closely linked to the challenging phone conversation he is having. Consequently, the entire emotional analysis report emphasizes his anxiety and nervousness in handling challenging situations.</think>
+<answer>anxious</answer>"
             }
         ]
     },
@@ -102,7 +101,9 @@ An example json file of the training data:
         "conversations": [
             {
                 "from": "human",
-                "value": "<video>\n<audio>\nAs an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you?"
+                "value": "<video>
+<audio>
+As an emotional recognition expert; throughout the video, which emotion conveyed by the characters is the most obvious to you?"
             },
             {
                 "from": "gpt",
@@ -114,6 +115,8 @@ An example json file of the training data:
 ]
 ```
 
+### wandb
+![image](https://github.com/user-attachments/assets/3395bafa-aaba-4212-902d-91067a1cd19a)
 
 ## 🤝 Related Work
 - [R1-V](https://github.com/Deep-Agent/R1-V)