fudan-generative-vision/Remory

Remory supplements an agent's compaction summary with learned soft memory from its history. The memory is conditioned on the summary and appended after it, helping the frozen model continue its work across compactions.

📄 Paper | 💻 Code | 🤗 Models

_{Music: Electric Dreams by Scott Buckley · CC BY 4.0 · Credits & edit details}

The GLM release is the 1.24B checkpoint used in local evaluations, distinct from the 1.502B encoder described in the paper. Both models use the same pinned SGLang version and launcher. The quick start below uses Qwen.

Plan for at least 2 × 80 GB GPUs to deploy Qwen3.8-27B + Remory with SGLang. The command below loads both models and serves generation and compaction through one endpoint. See deployment for more options.

git clone https://github.com/1ring2rta/Remory.git
cd Remory
python deploy/install.py
source .remory/venv/bin/activate

The installer prepares a local environment with the pinned SGLang version, Remory hook, and CUDA compiler components.

CUDA_VISIBLE_DEVICES=0,1 python -m remory.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --remory-checkpoint mocoV3/Qwen3.8-27B-REMORY-1.9B \
  --tp 2 \
  --host 127.0.0.1 \
  --port 8421

--tp 2 splits the LLM across the two GPUs; each worker also loads Remory. The context window defaults to the model maximum (256K for Qwen3.8-27B). The first run downloads weights and compiles GPU kernels.

For an existing clone, follow the upgrade instructions. The launcher downloads the pinned checkpoint automatically.

In another terminal:

source .remory/venv/bin/activate
r-codex -C /path/to/project

This opens the native Codex UI with Remory compaction. Use /compact during a session and r-codex resume --last to return later. Codex is installed on first use. See Codex for sessions, sub-agents, and MCP tools.

curl http://127.0.0.1:8421/generate \
  -H 'Content-Type: application/json' \
  -d '{"text":"The capital of France is", "sampling_params":{"temperature":0,"max_new_tokens":16}}'

In another terminal, run the full compaction example from the repository root:

source .remory/venv/bin/activate
python examples/quickstart.py

The example generates an answer, creates a summary and residual memory, then continues from that checkpoint. All model calls use the same SGLang /generate endpoint. During residual encoding, the Remory hook reads the actor's hidden states and runs the memory network. During continuation, it inserts the saved soft tokens after the summary.

Harness.compact is the compaction method to connect to your agent: generate a summary, encode residual memory, then return the new checkpoint. The agent retains its own tools and compaction trigger.

from remory.adapters import Harness
from remory.client import Client

with Client("http://127.0.0.1:8421", session_id="my-session") as client:
    agent = Harness(client, codec)  # The actor's native chat template
    checkpoint = agent.compact(
        prefix_messages=prefix,    # System prompt, tools, and original task
        removed_messages=history,
        summary_prompt=summary_prompt,
        summary_schema=summary_schema,
    )
    result = agent.generate(
        checkpoint=checkpoint, prefix_messages=prefix, messages=new_messages,
    )

The complete example loads the tokenizer and summary prompt from the released checkpoint. If your harness already generates a summary, use on_compact(..., summary=summary) to add residual memory.

Save the checkpoint with the conversation. For the next compaction, pass it as previous and supply the newly removed history. Memory is stored in .remory/memory.sqlite. The Codex integration provides a complete Responses provider and an interactive launcher. See the API reference for native SGLang requests with memory.

The Qwen3.8-27B training guide includes the Stage I reconstruction trainer, Stage II summary-conditioned residual-memory trainer, four-GPU launch commands, and checkpoint/resume instructions. Both stages freeze Qwen and train only the memory network. Synthetic JSONL examples show the document and history/summary/continuation formats, with CPU preparation commands.

Qwen3.8-27B and GLM-5.3-Flash improve across long-horizon agent benchmarks. Residual memory also reduces repeated tool outputs and tool errors on BrowseComp and Terminal-Bench 2.1. On SummHay, it approaches the full-context joint score using 5.2% of the input positions.

Gray bars show published frontier results under their respective evaluation protocols. Full results and settings are in the paper.

The Qwen memory network builds on DFlash and SpecForge. Inference uses SGLang and Transformers.

@misc{xia2026remory,
  title  = {Remory: Learning Residual Memory for Context Compaction},
  author = {Xia, Hanchen and Chen, Baoyou and Ge, Yutang and Deng, Naihao and
            Yang, Senqiao and Dong, Zilong and Yuan, Weihao and Zhu, Siyu},
  year   = {2026},
  eprint = {2610.11287},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url    = {https://arxiv.org/abs/2610.11287}
}

Development · MIT License · Third-party notices