license: apache-2.0
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- web-agent
- gui-agent
- information-seeking
WebDancer: Towards Autonomous Information Seeking Agency
This repository contains the WebDancer model, a native agentic search reasoning model designed for autonomous information seeking, as introduced in the paper WebDancer: Towards Autonomous Information Seeking Agency.
WebDancer aims to address intricate real-world problems necessitating in-depth information seeking and multi-step reasoning by understanding GUI screenshots and generating autonomous actions.
- 🏠 Project Homepage: https://osatlas.github.io/
- 💻 Code: https://github.com/OS-Copilot/OS-Atlas
Abstract
Addressing intricate real-world problems necessitates in-depth information seeking and multi-step reasoning. Recent progress in agentic systems, exemplified by Deep Research, underscores the potential for autonomous multi-step research. In this work, we present a cohesive paradigm for building end-to-end agentic information seeking agents from a data-centric and training-stage perspective. Our approach consists of four key stages: (1) browsing data construction, (2) trajectories sampling, (3) supervised fine-tuning for effective cold start, and (4) reinforcement learning for enhanced generalisation. We instantiate this framework in a web agent based on the ReAct, WebDancer. Empirical evaluations on the challenging information seeking benchmarks, GAIA and WebWalkerQA, demonstrate the strong performance of WebDancer, achieving considerable results and highlighting the efficacy of our training paradigm. Further analysis of agent training provides valuable insights and actionable, systematic pathways for developing more capable agentic models. The codes and demo will be released in this https URL .
🌐 Features for WebDancer
- Native agentic search reasoning model using ReAct framework towards autonomous information seeking agency and Deep Research-like model.
- We introduce a four-stage training paradigm comprising browsing data construction, trajectory sampling, supervised fine-tuning for effective cold start, and reinforcement learning for improved generalization, enabling the agent to autonomously acquire autonomous search and reasoning skills.
- Our data-centric approach integrates trajectory-level supervision fine-tuning and reinforcement learning (DAPO) to develop a scalable pipeline for training agentic systems via SFT or RL.
- WebDancer achieves a Pass@3 score of 64.1% on GAIA and 62.0% on WebWalkerQA.
💎 Results Showcase
🚀 Quick Start
This section provides instructions on how to inference the WebDancer model using the Hugging Face transformers library.
Notes: This model can accept images of various sizes as input. The model outputs are typically normalized to relative coordinates within a 0-1000 range (either a center point or a bounding box). For visualization or interaction, remember to convert these relative coordinates back to the original image dimensions.
First, ensure that the necessary dependencies are installed:
pip install transformers qwen-vl-utils
Inference code example:
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch
from PIL import Image
# Default: Load the model on the available device(s)
# Replace "Alibaba-NLP/WebDancer-32B" with the actual model ID if different
model_id = "Alibaba-NLP/WebDancer-32B"
model = Qwen2VLForConditionalGeneration.from_pretrained(
model_id, torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)
# Example: Prepare an image and a text query
# You would load your UI screenshot here. For demonstration, we use a placeholder.
# In a real scenario, './examples/images/web_6f93090a-81f6-489e-bb35-1a2838b18c01.png'
# would be replaced with your actual image file path.
# For simplicity, let's create a dummy image for this example to make it runnable without a file.
# In a real scenario, you would have an actual image file.
try:
image_path = "./examples/images/web_6f93090a-81f6-489e-bb35-1a2838b18c01.png"
# Attempt to open the image. If it doesn't exist, create a dummy.
_ = Image.open(image_path)
except FileNotFoundError:
print(f"Image not found at {image_path}. Creating a dummy image for demonstration.")
dummy_image = Image.new('RGB', (800, 600), color = 'red')
dummy_image.save("dummy_web_screenshot.png")
image_path = "dummy_web_screenshot.png"
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": image_path, # Replace with your actual image path
},
{"type": "text", "text": "In this UI screenshot, what is the position of the element corresponding to the command \"switch language of current page\" (with bbox)?"},
],
}
]
# Preparation for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages) # Handles image loading
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors="pt",
)
inputs = inputs.to(model.device) # Move inputs to the same device as the model
# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=False, clean_up_tokenization_spaces=False
)
print(output_text)
# Example output might look like: <|object_ref_start|>language switch<|object_ref_end|><|box_start|>(576,12),(592,42)<|box_end|><|im_end|>
🎥 WebDancer Demos
The model can execute long-horizon tasks with multiple steps and complex reasoning, such as web traversal, information seeking, and question answering.
WebWalkerQA
GAIA
Daily Use
📃 License
This project is licensed under the Apache-2.0 License.
🚩 Citation
If this work is helpful, please kindly cite as:
@misc{wu2025webdancer,
title={WebDancer: Towards Autonomous Information Seeking Agency},
author={Jialong Wu and Baixuan Li and Runnan Fang and Wenbiao Yin and Liwen Zhang and Zhengwei Tao and Dingchu Zhang and Zekun Xi and Yong Jiang and Pengjun Xie and Fei Huang and Jingren Zhou},
year={2025},
eprint={2505.22648},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2505.22648},
}