RUC-NLPIR/Agent-Reflex

 Agent-Reflex: Teaching Language Agents to Act Reflectively

If you like our project, please give us a star ⭐ on GitHub for the latest update.

A training framework that elicits proactive reflection in tool-using LLM agents: noticing noisy tool feedback and user corrections, and recovering from them.

Demo · Overview · Framework · Results · Installation · Data · Quick Start · Citation

• [2026-10] 🤗 The ReSFT data (54,663 trajectories) and ORPO tasks (2,550 with skills) are now available on Hugging Face: SFT-54K · RL-2K.
• [2026-10] 🤗 The Agent-Reflex-8B checkpoint (Qwen3-8B, ReSFT + ORPO) is released.
• [2026-10] 🚀 Full codebase released: the RSD skill library (1,170 meta-skills), ReSFT and ORPO training recipes, and evaluation on six benchmarks.
• [2026-10] 📄 Our paper Agent-Reflex: Teaching Language Agents to Act Reflectively is now available on arXiv and Hugging Face Papers.

_{🎥 Download the 1080p video (60 seconds) · 💻 original task interactive replay (open locally in a browser)}

Agents are trained in idealized environments but deployed into noisy ones. Two gaps follow:

• (1) No noisy feedback. Real MCP tools time out, reject permissions or return malformed results, yet training environments return clean feedback, so agents never learn to detect and recover from failures.
• (2) No user correction. Real users interrupt and correct the agent mid-task, yet user simulators rarely issue such corrections.

Figure 1 (a, b) · The two gaps. Idealized interaction (a) vs. real-world interaction with user corrections and noisy feedback (b).

Figure 1 (c, d) · The cost of noise. Strong agents lose success rate under user or tool noise (c); tool errors fall into eight recurring families (d).

Agent-Reflex treats reflection as a learnable meta-skill: it mines skills from failures in noisy environments, distills them into supervised data and reinforces them with noisy agentic RL. On Qwen3-8B it improves six agent benchmarks and stays robust under noise.

Build failure-aware executable environments by perturbing tool returns with real-world failure families, and mine verifier-matched success / failure pairs into an offline library of 1,170 reflection meta-skills over 557 environments. Environment synthesis and the environment infrastructure follow Agent-World.

Convert each skill into verifier-validated self-reflection (SR), skill recovery (KR) and user-correction (UR) data, with skill prompts stripped, and internalize them by supervised fine-tuning.

Combine Noisy Agent RL with same-token, skill-guided on-policy distillation (OPD) to provide dense reflection supervision. Skills are used only in training; the deployed agent is skill-free.

We evaluate on six agent benchmarks under each benchmark's official metric:

• General Function Calling:
  • BFCL V4 (paper): the Berkeley Function Calling Leaderboard extended to agentic evaluation (web search, memory, multi-turn, live / non-live, irrelevance); 5,217 tasks.
  • ACEBench: normal, special and agentic tool use, emphasizing when to call a tool and how to handle underspecified requests.
  • ToolSandbox: stateful, conversational tool use with implicit state dependencies, an LLM user simulator and deliberately unsolvable tasks; 200 offline scenarios.
• Long-Horizon Interactive Workflows:
  • τ²-Bench: dual-control airline, retail and telecom domains where agent and user both act on shared state under a policy document.
  • VitaBench: delivery, in-store and OTA domains plus cross-scenario tasks spanning domains within one episode.
• Noisy Interaction:
  • AgentNoiseBench: tool-side and user-side perturbations injected into τ²-Bench and VitaBench environments.

On Qwen3-8B, Agent-Reflex reaches 58.6 Overall on the five long-horizon tool-use benchmarks (+18.4 over the backbone, +7.7 over EnvScaler-8B) and 29.1 on AgentNoiseBench, the best among all 8B methods:

Each part has its own environment, built by one script with uv (pip install uv) from a pinned requirements.txt:

• Hardware: NVIDIA 80 GB GPUs with a driver supporting CUDA 12.8. ReSFT uses 2 × 8 GPUs; ORPO and evaluation use 1 × 8.
• Network: the setup scripts need GitHub, PyPI and Hugging Face. Behind a proxy, export HTTP_PROXY / HTTPS_PROXY first.

Training needs no API keys. Evaluation reads them from evaluation/.env (git-ignored):

cp evaluation/.env.example evaluation/.env
# then edit evaluation/.env:
BOYUE_API_KEY=sk-xxxxxxxx
BOYUE_API_BASE_URL=http://35.220.164.252:3888/v1   # outside CN/HK: https://api.boyuerichdata.opensphereai.com/v1
SERPER_API_KEY=xxxxxxxx

• BOYUE_API_KEY: the user simulator (gpt-4.1) and judge (claude-sonnet-4-6) for every benchmark except BFCL. Register at Boyue API (docs), open the API令牌 (API tokens) page and create a token in the auto group; the key starts with sk-.

• SERPER_API_KEY: BFCL V4 web search only, from serper.dev.

Agent-Reflex/
├── RSD/             🪞 Stage 1 · the reflection meta-skill library (1,170 skills)
├── LLaMA-Factory/   📘 Stage 2 · ReSFT recipe and launch scripts
├── ORPO/            🎯 Stage 3 · verl-based ORPO: noisy rollouts + skill-guided OPD
├── evaluation/      📊 τ²-Bench, BFCL V4, ACEBench, VitaBench, AgentNoiseBench, ToolSandbox
└── assets/          🖼️ figures used in this README; demo/ holds the demo video and replay page

pip install -U huggingface_hub
hf download dongguanting/Agent-Reflex-SFT-54K --repo-type dataset --include "train/*" \
  --local-dir LLaMA-Factory/data/agent_reflex_resft
hf download dongguanting/Agent-Reflex-RL-2K train_envscaler_2550_with_skills.jsonl \
  --repo-type dataset --local-dir ORPO/data
hf download dongguanting/Agent-Reflex-8B --local-dir /path/to/Agent-Reflex-8B

cd LLaMA-Factory
NNODES=2 NODE_RANK=<0|1> MASTER_ADDR=<node0> MODEL_PATH=/path/to/Qwen3-8B \
  bash scripts/run_resft_qwen3_8b.sh

Start from the ReSFT model checkpoint:

cd ORPO
python scripts/prepare_init_checkpoint.py /path/to/resft_checkpoint --apply
MODEL_PATH=/path/to/resft_checkpoint bash scripts/run_orpo_qwen3_8b.sh

Full task sets under the official settings (fill in evaluation/.env first):

cd evaluation
MODEL_PATH=/path/to/Agent-Reflex-8B RUN_NAME=agent-reflex bash run_all.sh

run_all.sh serves the model with vLLM and runs all six benchmarks; set BENCHES="Tau2 BFCL-v4" to run a subset. Per-benchmark details are in evaluation/README.md.

Agent-Reflex builds on verl, LLaMA-Factory, EnvScaler, Agent-World, tau2-bench, BFCL, VitaBench, ToolSandbox and AgentNoiseBench. Thanks to their authors.

Agent-Reflex code and the RSD skill library are released under the MIT License. Vendored components keep their own licenses:

ToolSandbox is cloned from apple/ToolSandbox at setup time and is not redistributed here. The AgentNoiseBench noise injection and data follow AgentNoiseBench; please refer to its authors for terms of use.

@misc{dong2026agentreflex,
  title={Agent-Reflex: Teaching Language Agents to Act Reflectively},
  author={Guanting Dong and Xinyu Yang and Jiajie Jin and Yuyang Hu and Xiaoshuai Song and Yifei Chen and Xiaoxi Li and Jiejun Tan and Yutao Zhu and Tat-Seng Chua and Ji-Rong Wen and Zhicheng Dou},
  year={2026},
  eprint={2610.17953},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2610.17953},
}