Improve model card for WebDancer

README.md

@@ -1,12 +1,171 @@
 ---
 license: apache-2.0
+pipeline_tag: image-text-to-text
+library_name: transformers
 ---
-You can download the model then run the inference scipts in https://github.com/Alibaba-NLP/WebAgent.
 
-- **WebSailor** is a complete post-training methodology designed to teach LLM agents sophisticated reasoning for complex web navigation and information-seeking tasks. It addresses the challenge of extreme uncertainty in vast information landscapes, a capability where previous open-source models lagged behind proprietary systems.
+# WebDancer: Towards Autonomous Information Seeking Agency
 
-- We classify information-seeking tasks into three difficulty levels, where **Level 3** represents problems with both high uncertainty and a complex, non-linear path to a solution. To generate these challenging tasks, we introduce **SailorFog-QA**, a novel data synthesis pipeline that constructs intricate knowledge graphs and then applies information obfuscation. This process creates questions with high initial uncertainty that demand creative exploration and transcend simple, structured reasoning patterns.
+<div align="center">
 
-- Our training process begins by generating expert trajectories and then reconstructing the reasoning to create concise, action-oriented supervision signals, avoiding the stylistic and verbosity issues of teacher models. The agent is first given a "cold start" using rejection sampling fine-tuning (RFT) on a small set of high-quality examples to establish a baseline capability. This is followed by an efficient agentic reinforcement learning stage using our **Duplicating Sampling Policy Optimization (DUPO)** algorithm, which refines the agent's exploratory strategies.
+[📄Paper](https://huggingface.co/papers/2505.22648) | [🏠Homepage](https://osatlas.github.io/) | [💻Code](https://github.com/Alibaba-NLP/WebAgent)
 
-- WebSailor establishes a **new state-of-the-art for open-source agents**, achieving outstanding results on difficult benchmarks like BrowseComp-en and BrowseComp-zh. Notably, our smaller models like WebSailor-7B outperform agents built on much larger backbones, highlighting the efficacy of our training paradigm. Ultimately, WebSailor closes the performance gap to proprietary systems, achieving results on par with agents like Doubao-Search.
+</div>
+
+WebDancer is a native agentic search reasoning model using the ReAct framework, aiming towards autonomous information seeking agency and _Deep Research_-like capabilities. This model was presented in the paper [WebDancer: Towards Autonomous Information Seeking Agency](https://huggingface.co/papers/2505.22648).
+
+Addressing intricate real-world problems necessitates in-depth information seeking and multi-step reasoning. WebDancer presents a cohesive paradigm for building end-to-end agentic information seeking agents from a data-centric and training-stage perspective. The approach consists of four key stages: (1) browsing data construction, (2) trajectories sampling, (3) supervised fine-tuning for effective cold start, and (4) reinforcement learning for enhanced generalisation. Empirical evaluations on the challenging information seeking benchmarks, GAIA and WebWalkerQA, demonstrate the strong performance of WebDancer, achieving considerable results and highlighting the efficacy of this training paradigm.
+
+<div align="center">
+<p align="center">
+  <img src="https://github.com/user-attachments/assets/cf2ee020-5e15-4087-9a7e-75cc43662494" width="800px" />
+</p>
+</div>
+
+## Key Features
+
+*   **Native agentic search reasoning model**: Utilizes the ReAct framework for autonomous information seeking, inspired by "Deep Research"-like models.
+*   **Four-stage training paradigm**: Comprises browsing data construction, trajectory sampling, supervised fine-tuning for effective cold start, and reinforcement learning for improved generalization, enabling the agent to autonomously acquire search and reasoning skills.
+*   **Data-centric approach**: Integrates trajectory-level supervision fine-tuning and reinforcement learning (DAPO) to develop a scalable pipeline for training agentic systems via SFT or RL.
+*   **Strong Performance**: WebDancer achieves a Pass@3 score of 64.1% on GAIA and 62.0% on WebWalkerQA.
+
+## Quick Start
+
+You need to enter the `WebDancer` folder in the [WebAgent repository](https://github.com/Alibaba-NLP/WebAgent) for the following commands.
+
+### Step 0: Set Up the Environment
+
+```bash
+conda create -n webdancer python=3.12
+pip install -r requirements.txt
+```
+
+For `WebDancer-QwQ-32B` (which is based on Qwen2-VL), ensure that the necessary dependencies are installed:
+```bash
+pip install transformers
+pip install qwen-vl-utils
+```
+
+### Step 1: Deploy the Model
+
+Download the WebDancer model from [🤗 HuggingFace](https://huggingface.co/Alibaba-NLP/WebDancer-QwQ-32B) and deploy it using the provided scripts with [sglang](https://github.com/sgl-project/sglang).
+
+```bash
+cd scripts
+bash deploy_model.sh WebDancer_PATH
+```
+
+> **Note:** Replace `WebDancer_PATH` with the actual path to the downloaded model.
+
+### Step 2: Run the Demo
+
+Edit the following keys in [`WebDancer/scripts/run_demo.sh`](https://github.com/Alibaba-NLP/WebAgent/blob/main/WebDancer/scripts/run_demo.sh):
+
+-   `GOOGLE_SEARCH_KEY`, you can get it from [serper](https://serper.dev/).
+-   `JINA_API_KEY`, you can get it from [jina](https://jina.ai/api-dashboard/).
+-   `DASHSCOPE_API_KEY`, you can get it from [dashscope](https://dashscope.aliyun.com/).
+
+Then, launch the demo with Gradio to interact with the WebDancer model:
+
+```bash
+cd scripts
+bash run_demo.sh
+```
+
+### Inference Example
+
+WebDancer models accept images of any size as input. The model outputs are normalized to relative coordinates within a 0-1000 range (either a center point or a bounding box defined by top-left and bottom-right coordinates). For visualization, remember to convert these relative coordinates back to the original image dimensions.
+
+Here's a minimal Python inference example for `WebDancer-QwQ-32B` (based on Qwen2-VL):
+
+```python
+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
+from qwen_vl_utils import process_vision_info
+from PIL import Image
+import torch
+
+# Load the model and processor
+# Default: Load the model on the available device(s)
+model_name = "Alibaba-NLP/WebDancer-QwQ-32B"
+model = Qwen2VLForConditionalGeneration.from_pretrained(
+    model_name, torch_dtype="auto", device_map="auto"
+)
+processor = AutoProcessor.from_pretrained(model_name)
+
+# Define the task with an image and text instruction
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                # Replace with a local image path or a URL
+                "image": "./examples/images/web_6f93090a-81f6-489e-bb35-1a2838b18c01.png",
+            },
+            {"type": "text", "text": "In this UI screenshot, what is the position of the element corresponding to the command \"switch language of current page\" (with bbox)?"},
+        ],
+    }
+]
+
+# Preparation for inference
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+image_inputs, video_inputs = process_vision_info(messages) # video_inputs will be empty for this task
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda") # Ensure inputs are on the correct device
+
+# Inference: Generation of the output
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+
+generated_ids_trimmed = [
+    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=False, clean_up_tokenization_spaces=False
+)
+print(output_text[0])
+# Expected output: <|object_ref_start|>language switch<|object_ref_end|><|box_start|>(576,12),(592,42)<|box_end|><|im_end|>
+```
+
+## Demos
+
+WebDancer can execute long-horizon tasks with multiple steps and complex reasoning, such as web traversal, information seeking, and question answering.
+
+<div align="center">
+    <h3>WebWalkerQA</h3>
+    <video src="https://github.com/user-attachments/assets/0bbaf55b-897e-4c57-967d-a6e8bbd2167e" />
+</div>
+
+<div align="center">
+    <h3>GAIA</h3>
+    <video src="https://github.com/user-attachments/assets/935c668e-6169-4712-9c04-ac80f0531872" />
+</div>
+
+<div align="center">
+    <h3>Daily Use</h3>
+    <video src="https://github.com/user-attachments/assets/d1d5b533-4009-478b-bd87-96b86389327d" />
+</div>
+
+## Citation
+
+If this work is helpful, please kindly cite as:
+
+```bibtex
+@misc{wu2025webdancer,
+      title={WebDancer: Towards Autonomous Information Seeking Agency},
+      author={Jialong Wu and Baixuan Li and Runnan Fang and Wenbiao Yin and Liwen Zhang and Zhengwei Tao and Dingchu Zhang and Zekun Xi and Yong Jiang and Pengjun Xie and Fei Huang and Jingren Zhou},
+      year={2025},
+      eprint={2505.22648},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2505.22648},
+}
+```